HumanEval

HumanEval is a benchmark dataset of 164 hand-written programming problems, each with a function signature, docstring, and unit tests. It measures a model's ability to generate correct Python code from natural language descriptions. Operators encounter HumanEval when evaluating code-generation models (e.g., CodeLlama, DeepSeek-Coder) to compare pass@k scores, which indicate how often the model produces functionally correct code. Because HumanEval tests logical correctness rather than surface-level fluency, it is a standard filter for deciding whether a model is suitable for coding assistants.

HumanEval was introduced by OpenAI in 2021 to address the lack of rigorous functional correctness benchmarks for code generation. Each problem includes a prompt (docstring + signature) and multiple test cases. The metric pass@k estimates the probability that at least one of k generated samples passes all tests. For example, pass@1 uses a single sample per problem; pass@100 uses 100 samples and corrects for sampling variance. HumanEval is relatively small (164 problems) but hand-crafted to avoid data contamination. Variants like HumanEval+ add more tests to reduce false positives. Operators running local models often use HumanEval to validate fine-tuned code models before deploying them in tools like Continue.dev or local IDEs.

An operator downloads CodeLlama-7B-Python and runs evaluation with lm-evaluation-harness using the HumanEval task. The model achieves pass@1 of 28.5% — meaning about 47 of 164 problems are solved in one attempt. By contrast, a larger model like DeepSeek-Coder-33B might score 45% pass@1. The operator uses these numbers to decide whether the smaller model is sufficient for autocomplete tasks or if the larger model's VRAM cost (~20 GB at Q4) is justified.

To evaluate a local model on HumanEval, an operator runs: lm_eval --model hf --model_args pretrained=code-llama-7b --tasks humaneval --num_fewshot 0. The harness generates code, runs it in a sandboxed Python environment, and reports pass@1 and pass@100. In LM Studio, the operator can load the model and manually test a few HumanEval prompts via the chat interface, but automated evaluation requires the harness. Results guide decisions on which quantized model to serve via Ollama or vLLM.