19. Prompt Evaluation

Chapter 19 of 25 · 15 min

KEY INSIGHT

Prompt quality splits into capability (does it work?) and reliability (does it work consistently?). Both dimensions require different measurement approaches. ```python def evaluate_prompt(prompt, test_cases, model): """ Multi-dimensional prompt evaluation. Args: prompt: PromptTemplate instance test_cases: list of {'input': dict, 'expected': str} model: callable that takes prompt string, returns output Returns: dict with evaluation metrics """ results = [] latencies = [] for case in test_cases: input_dict = case['input'] expected = case['expected'] start = time.time() output = model(prompt.render(**input_dict)) latency_ms = (time.time() - start) * 1000 latencies.append(latency_ms) # Multi-label scoring for partial correctness correctness = calculate_edit_similarity(output, expected) results.append({ 'input': input_dict, 'output': output, 'expected': expected, 'correctness': correctness, 'latency_ms': latency_ms }) return { 'avg_correctness': np.mean([r['correctness'] for r in results]), 'p95_correctness': np.percentile([r['correctness'] for r in results], 95), 'avg_latency_ms': np.mean(latencies), 'min_correctness': min([r['correctness'] for r in results]), 'failure_cases': [r for r in results if r['correctness'] < 0.5] } def calculate_edit_similarity(output, expected): """Levenshtein distance normalized to 0-1 score.""" from difflib import SequenceMatcher return SequenceMatcher(None, output, expected).ratio() ``` **Failure mode:** Single-metric evaluation (accuracy only) misses latent instabilities. A prompt scoring 95% accuracy may fail entirely on 5% of inputs that are common in production traffic. Tracking p5 correctness (5th percentile) surfaces these failure cases. ```python # Counterintuitive case: p5 matters more than average test_results = { 'avg_correctness': 0.94, 'p5_correctness': 0.12, # Bottom 5% are catastrophic failures 'min_correctness': 0.0, 'failure_cases': 23 # Out of 100 test cases } # This prompt is not production-ready despite high average ``` Recommended evaluation dimensions: correctness (p5, p50, p95), latency (p50, p99), format compliance rate, and input-length sensitivity. Track each dimension separately and set thresholds per dimension for production readiness.

Evaluating prompts requires metrics outside accuracy. A prompt may produce correct answers occasionally while being unreliable, slow, or brittle under input variation. Production evaluation tracks multiple dimensions.

EXERCISE

Build an evaluation harness for your most-used prompt. Create 50 test cases covering edge cases, run evaluation, and document which cases fail and why. Report p5 and p95 correctness alongside average.