20. A/B Testing Prompts

Chapter 20 of 25 · 15 min

KEY INSIGHT

Meaningful prompt comparisons require traffic allocation independent of model selection and sufficient sample size per variant before drawing conclusions. ```python import hashlib import json class PromptABTest: def __init__(self variants, traffic_split=None): """ Args: variants: dict of {variant_name: prompt_template} traffic_split: dict of {variant_name: proportion}, defaults to equal split """ self.variants = variants self.traffic_split = traffic_split or { name: 1/len(variants) for name in variants } self.metrics = {name: [] for name in variants} def assign_variant(self, user_id, prompt_name=None): """Assign user to variant deterministically based on user_id. Deterministic assignment ensures: 1. Same user always sees same variant (consistency) 2. Assignment is independent of request timing (no temporal bias) """ if prompt_name: return prompt_name # Override for debugging hash_input = f"{user_id}:{':'.join(self.variants.keys())}" hash_value = int(hashlib.md5(hash_input.encode()).hexdigest(), 16) normalized = (hash_value % 1000) / 1000 cumulative = 0 for variant_name, proportion in self.traffic_split.items(): cumulative += proportion if normalized < cumulative: return variant_name return list(self.variants.keys())[-1] def record_result(self, variant_name, input_data, latency_ms, success): """Record metrics for statistical analysis.""" self.metrics[variant_name].append({ 'input_hash': hashlib.md5(json.dumps(input_data).encode()).hexdigest()[:8], 'latency_ms': latency_ms, 'success': success, 'timestamp': time.time() }) def analyze(self): """Calculate significance and effect sizes per variant.""" from scipy import stats results = {} for variant, metrics in self.metrics.items(): successes = [m['success'] for m in metrics] latencies = [m['latency_ms'] for m in metrics] results[variant] = { 'n': len(metrics), 'success_rate': np.mean(successes), 'avg_latency_ms': np.mean(latencies), 'p95_latency_ms': np.percentile(latencies, 95) } return results ``` **Failure mode:** A/B tests run without minimum sample size produce noise. A 3% difference between variants with 20 samples is not statistically significant (p > 0.3). Required sample size depends on expected effect size: detecting 5% improvement requires approximately 600 samples per variant. ```python def required_sample_size(baseline_rate, min_detectable_effect, alpha=0.05, power=0.8): """Calculate minimum samples needed per variant.""" from scipy.stats import norm p1 = baseline_rate p2 = baseline_rate * (1 + min_detectable_effect) z_alpha = norm.ppf(1 - alpha/2) z_beta = norm.ppf(power) pooled_p = (p1 + p2) / 2 effect = abs(p2 - p1) n = ((z_alpha + z_beta)**2 * (2*pooled_p*(1-pooled_p))) / effect**2 return int(np.ceil(n)) # Example: detecting 10% relative improvement from 80% baseline required = required_sample_size(0.80, 0.10) # Result: ~520 samples per variant needed ```

A/B testing compares prompt variants under production conditions. Unlike ad-hoc comparison, production A/B testing handles statistical significance, traffic distribution, and guardrails against regression.

EXERCISE

Design an A/B test comparing two prompt variants for a customer support task. Calculate required sample size for detecting 5% improvement in resolution rate. Simulate running the test with Python and analyze when significance is reached.