A/B Testing Models Quiz

Quiz covering Model Analysis

Questions in this quiz

A preview of the 5 questions covered. Start the quiz above to answer them, check your score, and read the explanations.

1. 1

   What is the primary purpose of conducting an A/B test when deploying a new machine learning model?

2. 2

   In an A/B test setup, why is it critical to ensure that users are randomly assigned to either the control group or the treatment group?

3. 3

   You are running an A/B test for a recommendation model. Which of the following scenarios best describes a 'novelty effect' that might skew your results?

4. 4

   When evaluating the results of an A/B test, what does a p-value of 0.03 indicate, assuming a significance level (alpha) of 0.05?

5. 5

   You observe a statistically significant improvement in click-through rate (CTR) for the treatment group, but notice that the model's latency increased by 200ms. What is the most appropriate next step?