Evaluation
Reading benchmark results carefully
A benchmark score is the output of a complete evaluation setup, not a permanent property of a model. Review the setup before comparing the number.
Model comparisons often compress thousands of prompts into one number. That number can be useful, but it hides the decisions that produced it: which revision was loaded, how prompts were formatted, whether examples were sampled, how answers were extracted, and which failures were excluded. A careful review reconstructs those decisions before it asks which model is ahead.
Start with the evaluation contract
Write the comparison as a small contract. Name the task and dataset revision, the exact model and runtime revision, the prompt template, generation settings, scoring rule, and execution environment. If one of those fields is missing, the result may still be interesting, but it is not yet reproducible.
The contract should also state what the test is intended to measure. Multiple-choice accuracy can test selection among supplied options; it does not automatically measure whether a model can produce a correct answer unaided. A judge-model score may capture preferences expressed in its rubric; it does not become an objective measure merely because it is numeric.
| Field | Why it changes the result |
|---|---|
| Model revision | Weights and chat templates can change behind a stable product name. |
| Prompt format | System text, separators, and answer instructions affect task interpretation. |
| Decoding | Temperature, sampling, and token limits alter both accuracy and variance. |
| Scoring | Normalization and answer extraction decide which outputs count as correct. |
| Runtime | Quantization and backend versions can change numerics and supported context. |
Read the score as a distribution
An aggregate score is an estimate from a finite collection of examples. Small differences can reverse when the sample, prompt wording, or random seed changes. Prefer reports that include per-task results, the number of evaluated items, and uncertainty such as a confidence interval or bootstrap distribution.
For a quick review, compare the reported gap with ordinary run-to-run variation. If two models differ by less than that variation, describe them as unresolved under this setup. Do not turn a narrow numeric lead into a general ranking. When practical, repeat stochastic generations and record the median together with a spread measure rather than selecting the best run.
Inspect the aggregate before trusting it
A mean can hide opposing changes. One model may improve on short factual questions while regressing on multi-step instructions. Break the result down by task family, language, answer length, and any slice that matters to the intended use. Also inspect failure examples: a score cannot reveal whether errors are harmless formatting misses, systematic misconceptions, or unsafe behavior.
Weighting deserves attention. A macro average gives each task equal influence; a micro average gives larger tasks more influence. Neither is universally correct. The report should say which was used and retain the component scores so another reader can recompute an alternative summary.
Check contamination and prompt sensitivity
Public benchmarks may appear in training data, documentation, or derived datasets. Exact overlap is not the only concern: memorized solutions and benchmark-specific prompting can also inflate results. Look for overlap checks, held-out or newly written items, and a clear statement of what could not be ruled out.
Prompt sensitivity is easier to test directly. Run a small set of defensible prompt variants: for example, change answer-label order, remove incidental prose, or compare zero-shot and few-shot formats. The goal is not to search for the highest score. It is to learn whether the conclusion survives reasonable presentation changes.
Compare like with like
Keep tool access, retrieval, reasoning traces, context limits, and sampling budgets aligned. If one system uses retrieval or several sampled answers, report that as a system comparison rather than attributing the outcome only to the underlying model. Cost and latency belong beside quality when the systems consume different resources.
Version every input that can drift. Dataset aliases, hosted model names, and API defaults are convenient but insufficient records. Save immutable revisions where possible and record the date for services that cannot be pinned.
A compact review checklist
- Can another person identify the exact model, dataset, prompt, runtime, and scoring rule?
- Does the task actually represent the capability claimed in the conclusion?
- Are sample size, per-task results, and uncertainty visible?
- Were contamination and reasonable prompt variants considered?
- Were tools, retrieval, token budgets, and generation settings comparable?
- Do inspected failures support the interpretation of the aggregate score?
Primary references
- Holistic Evaluation of Language Models (HELM) — scenarios, metrics, and transparent reporting.
- Beyond the Imitation Game (BIG-bench) — diverse tasks and limits of aggregate capability claims.
- Language Model Evaluation Harness — versioned task configurations and reproducible execution tooling.