A/B Testing

A/B Testing in ML compares two model variants — a control (current production model) and a treatment (candidate model) — by randomly splitting traffic and measuring a predefined metric (conversion rate, click-through, revenue per user) over a statistically significant period. The required sample size n per variant for detecting a minimum detectable effect δ with 80% power at α = 0.05 is n ≈ 16σ²/δ², meaning a 1% click-through improvement (from 5% baseline, σ ≈ 0.22) requires ~77K observations per variant. Online A/B tests complement offline evaluation: a model with +0.5% AUC on holdout data may produce -2% revenue when deployed due to distribution shift or interaction effects

A/B testing compares two model versions by splitting traffic — 50% to model A, 50% to model B — and measuring which performs better on business metrics (revenue, engagement, user satisfaction). It's the gold standard for model deployment decisions. Never deploy based on offline metrics alone.

A/B testing deployment: (1) deploy model A (current) and model B (candidate) simultaneously, (2) randomly assign each user/request to A or B, (3) track business metrics (not just ML metrics), (4) run test until statistical significance (p < 0.05, but beware p-hacking), (5) minimum sample size: compute before starting — don't peek and stop early, (6) for LLMs: A/B test different prompts, models, temperatures, or generation configs — small changes can have large business impacts.