06. Baseline Selection

Chapter 6 of 18 · 10 min

KEY INSIGHT

Baseline selection is a strategic decision that shapes how reviewers perceive your contribution. Select baselines that are (a) strong, (b) relevant, and (c) reproducible. Choosing baselines too weak makes your improvements appear artificially large. Choosing too strong baselines may require proprietary resources. The art lies in finding the right tier. **Baseline Tiers:** | Tier | Description | Selection Criteria | |------|-------------|-------------------| | Published State-of-Art | Highest-performing published method | Use when your contribution builds on this directly | | Standard Baseline | Well-established method in the field | Use when your contribution is domain-adjacent | | Ablation Anchor | Minimal viable version of your approach | Use when isolating contribution components | **Selection Process:** 1. **Survey Literature:** Identify the top-5 performing methods on your benchmark. 2. **Assess Availability:** Check if code is released, if training is feasible within your compute budget. 3. **Verify Reproducibility:** Run reported numbers to confirm baseline implementation matches paper. 4. **Select 2-3 Baselines:** One published, one standard, one ablation anchor. **Example Baseline Selection:** Research Question: "Can linear attention replace softmax attention in NMT with <1% BLEU degradation?" Selected Baselines: - **Transformer (Vaswani et al., 2017):** Standard baseline, widely implemented - **Linear Transformer (Katharopoulos et al., 2020):** Direct comparison to prior linear attention work - **Simplified Transformer (no positional encoding):** Ablation anchor isolating attention mechanism **Failure Mode:** "Paper baseline overfitting"—tuning your method extensively while using reported numbers for baselines without verification. Always run baselines yourself.

Local verification checkpoint

Run the smallest example from this chapter in a local workspace and record the package version, runtime, data path, and observed output. If the result depends on model size, vector count, CPU/GPU backend, or available memory, note that constraint beside the exercise so the lesson remains reproducible.

Local verification checkpoint

EXERCISE

List your selected baselines with: (1) paper citation, (2) implementation source, (3) expected performance range, and (4) compute requirements.