Lab: Simulation on Sequential AB Test

In this assignment, you will run simulation studies to compare three different A/B testing strategies under the hypothesis \[ H_0: p_A = p_B, H_a: p_A > p_B \]

Your goal is to evaluate the empirical behavior of p-values, Type I error inflation, and the behavior of an always-valid Bayesian sequential test (mSPRT).

All simulations should assume independent Bernoulli outcomes, and all results must be presented clearly with tables and visualizations.

Simulation Setup (Used in all parts)

• True conversion rates:

\[ p_A = p_B = 0.3 \]

• Sample Size: \(n_A = n_B=100\) per test

• Number of simulation iteration: 100,000.

Simulation 1: Fixed Sample AB Test

Goal: Show that under a true null, classical p-values follow a Uniform(0,1) distribution.

1. For each iteration, compute the p-values.

2. Report:

• A histogram of the empirical p-value distribution

• A table reporting proportion of p-values <0.10, <0.05, <0.01.

3. Conclusion: Verify the empirical distribution is approximately Uniform(0,1).

Simulation 2: Sequential AB Test with Peeking

Goal: Show that repeated interim looks inflate the Type I error rate.

1. For each arm at each iteration, have an interim look every 10 visitors.

2. For each iteration, compute the p-value. Estimate the empirical type I error for \(\alpha = 0.05\).

3. Report

• The inflated false-positive rate.

• A short explanation of why sequential peeking increases Type I error.

Simulation 3: Sequential AB Test with mSPRT

Goal: Compare the Bayesian mSPRT method to the inflated classical sequential method and show it controls false positives properly.

1. Use Beta(1,1) as the prior.

2. Reveal outcomes in batches of 10. After each update:

• Update the posterior parameter and mixture likelihood ratio.

• Compute the p-value for each iteration.

3. Report:

• Estimate the empirical Type I error.

• A short explanation of why mSPRT didn’t increase Type I error.