Module 34

Experiments and uncertainty: A/B testing

Advanced module (after graduation): chance variation, bootstrap interval, permutation test, p-value, A/B test, power

When you finish this module you can say:

I can test whether the difference between two versions is chance or real with a bootstrap interval, a permutation test and a p-value, find by simulation how many examples I need, and write an honest decision report without falling into traps like peeking and multiple comparisons.

By the end of this module you will be able to test by simulation whether the score difference between two versions or two models is chance or real, work out how many examples you need, and report your decision with an interval, a p-value and a sample size.

Lessons

6 lessons · ≈ 64 min

1. How much does chance wobble10 min
2. A confidence interval with the bootstrap10 min
3. Permutation test and the p-value11 min
4. The A/B test11 min
5. Traps: peeking and multiple comparisons11 min
6. Project: compare two prompts honestly11 min · project

Practice exercises

Short writing exercises, independent of the lessons: Go to the exercises

1. An interval for the median
2. A p-value from counts
3. The A/B decision