How to benchmark models with varying context lengths

What this does

Benchmarks a model inference speed and throughput at different context window sizes (512, 2048, 8192 tokens) to reveal how context length affects latency and memory consumption. After this guide an optimal context configuration for specific workloads will be identifiable.

Steps

1. Create a benchmark script with variable num_ctx. Loops across multiple context sizes running the same prompt.

   import requests, time
   for ctx in [512, 2048, 8192]:
       payload = {"model": "llama3.2:3b", "prompt": "Explain transformers. " * 10, "options": {"num_ctx": ctx}, "stream": False}
       start = time.time()
       resp = requests.post("http://localhost:11434/api/generate", json=payload)
       elapsed = time.time() - start
       data = resp.json()
       tokens = len(data.get("response", "").split())
       print(f"ctx={ctx} elapsed={elapsed:.2f}s tps={tokens/elapsed:.1f}")
   

   Expected: Throughput decreases as context length increases.

2. Run the script and observe the trend. Higher context lengths consume more KV cache memory, reducing tokens-per-second.

   python3 benchmark_context.py
   

   Expected output: ctx=512 elapsed=2.34s tps=17.9, ctx=2048 elapsed=4.10s tps=10.2, ctx=8192 elapsed=8.75s tps=4.8.

3. Measure memory impact alongside speed. Capture VRAM consumption during each run.

   nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits
   

   Expected output: VRAM usage increases as num_ctx grows.

• Record the local run evidence. Save the exact command, runtime or package version, model name if applicable, and observed output so the result can be reproduced later.

Verification

python3 -c "import requests; r=requests.post('http://localhost:11434/api/generate', json={'model':'llama3.2:3b','prompt':'test','options':{'num_ctx':8192},'stream':False}); print(f\"tps={len(r.json()['response'].split())/(r.elapsed.total_seconds()):.1f}\")"
# Expected: tps lower than with num_ctx=512 (typically 50-70% of the short-context throughput)

Common failures

• num_ctx exceeds model maximum - Setting larger than trained context length causes errors; check model documentation for supported window.
• throughput increases with context - Measurement noise or warm cache; clear cache between runs or increase prompt length.
• OOM at high context lengths - Model plus KV cache exceeds VRAM; reduce context size or use a smaller model.

Related guides

• How to measure memory usage during model inference
• How to run a standardized benchmark suite against multiple models