LLM Model Notes

Methods

Repeatable model comparisons

A useful comparison starts with a decision, freezes its moving parts, and leaves behind enough evidence to rerun or challenge the conclusion.

By · · 10 minute read

Small evaluations are valuable when they are designed as decision tools rather than miniature leaderboards. A few dozen representative inputs can reveal whether a model follows the required format, handles a difficult language mix, or meets a latency budget. The same experiment becomes misleading when its question, inputs, and acceptance criteria change after results arrive.

Write the decision before the test

Begin with a sentence that names the decision and the evidence it needs. For example: “Choose a model for extracting five fields from short invoices while keeping field-level error below two percent and p95 latency below three seconds.” This is narrower and more testable than “find the best model.”

Define acceptance criteria before running candidates. Include quality, latency, cost, failure handling, and any operational constraint such as local execution or a fixed context limit. Decide which criterion is a hard gate and which can be traded. Precommitting does not remove judgment; it makes later judgment visible.

Build representative cases

Sample the situations the system must handle, including ordinary cases, rare but costly cases, malformed inputs, and explicit non-goals. Keep a development subset for prompt iteration and a held-out subset for the final comparison. If the same cases guide prompt changes and certify the result, the estimate becomes optimistic.

Assign stable case identifiers and record why each case exists. Store source data only when permission allows; otherwise publish a generator or a redacted equivalent. A useful case set is not necessarily large, but it should make omissions obvious.

Freeze the comparison manifest

Put every input that can alter behavior into one versioned manifest. At minimum, record the immutable model revision where available, runtime and adapter versions, system and user prompt templates, few-shot examples, tool definitions, generation settings, maximum input and output lengths, stop sequences, seeds, and scoring code revision.

Hosted services may not expose immutable weights. In that case, record the provider model identifier, request date, API version, region when behavior can vary, and the complete request parameters. The result is then reproducible as an observation of a dated service, not a permanent checkpoint.

experiment: invoice-fields-v1
cases: cases-v3.jsonl
model: provider/model-revision
prompt: prompts/extract-v2.txt
temperature: 0
max_output_tokens: 300
scorer: exact-fields-v2
runs_per_case: 3

Use paired observations

Run every candidate on the same cases and score each case separately. Pairing makes the comparison depend on differences for the same input rather than on two unrelated averages. It also exposes where candidates disagree, which is usually more useful than the headline mean.

Randomize candidate order when shared services, caches, or time-of-day effects might matter. For stochastic generation, repeat each case using a declared seed policy. Never discard a run because its output looks unrepresentative unless the rejection rule was written in advance and applied to every candidate.

Save raw responses before parsing. A parser may convert a formatting failure into an empty answer or silently accept extra text. Keeping the original response makes scoring errors auditable and allows a corrected scorer to run without paying for inference again.

Score behavior, not presentation accidents

Separate semantic correctness from format validity, refusals, truncation, and transport errors. A strict production parser may correctly reject malformed JSON, but the report should still show whether the underlying values were present. This distinction tells you whether to change the model, prompt, parser, or retry policy.

If humans or another model judge open-ended answers, publish the rubric and blind the judge to candidate identity. Calibrate with a small set of examples that have agreed ratings. Report disagreement instead of forcing uncertain cases into a precise-looking score.

Estimate uncertainty at the case level

Report counts and per-case outcomes alongside aggregates. For paired candidates, bootstrap over cases or inspect the distribution of paired differences. A narrow lead concentrated in two unusual examples deserves a different conclusion from a broad improvement across the set.

Choose wording that matches the evidence. “Candidate B passed 47 of 50 held-out cases versus 44 for A” is observable. “B is generally more reliable” requires broader cases, repeated runs, and evidence that the tested distribution represents future use.

Publish a result bundle

A compact result bundle should contain the decision statement, manifest, case identifiers, raw or redacted responses, parsed outputs, per-case scores, environment metadata, summary tables, and a short limitations note. Generate the report from those artifacts instead of copying numbers by hand.

Give the bundle a content hash or release tag. A later rerun should create a new bundle rather than mutate the old one. When a service or dataset changes, readers can then distinguish a changed system from a changed interpretation.

A minimal comparison sequence

  1. State the decision, hard gates, and trade-offs before evaluating.
  2. Split representative cases into development and held-out sets.
  3. Freeze model, prompt, runtime, generation, and scoring revisions.
  4. Run candidates on paired cases and retain every raw response.
  5. Report per-case outcomes, uncertainty, cost, latency, and failures.
  6. Publish an immutable bundle with limitations and a rerun command.

Primary references