LLM Model Notes

Methodology

Prefer repeatable observations

Every note begins with a defined question, fixed inputs, recorded model and runtime settings, and evidence another person can inspect. Results distinguish measured behavior from interpretation and state the conditions under which the conclusion may stop applying.

Define the question

A measurement should support a specific decision. Before choosing tasks or metrics, write down the intended use, the important failure modes, and the acceptance criteria. This prevents a convenient benchmark from replacing the capability that actually matters.

Task coverage is selected from the target workload rather than from a desire to produce one broad rank. A small representative set with documented gaps is preferred to a larger collection whose relationship to the decision is unclear.

Record every moving part

The run record includes model and runtime revisions, prompt messages, chat template, few-shot examples, generation parameters, context and output limits, tool definitions, dataset revision, scorer revision, hardware, and numerical precision. Hosted services are dated when an immutable revision is unavailable.

Inputs receive stable identifiers. Randomness is controlled with a declared seed policy, and repeated generations are retained rather than selecting a convenient run. Cache state, warm-up, concurrency, and request-length distributions are part of inference measurements.

Use direct measurements

Quality is recorded per case before it is summarized. Latency is separated into first-token, inter-token, and end-to-end observations. Server capacity is measured under a stated service objective, with failed and cancelled requests included in the accounting.

When a human or model judge is needed, the rubric is published and candidate identity is hidden where practical. Disagreement and unscorable cases remain visible. A precise-looking number is not substituted for uncertain evidence.

Retain raw observations

Raw responses, parsed outputs, scores, timestamps, and environment metadata are saved before aggregation. Reports are generated from those artifacts so a corrected parser or summary can be applied without rerunning inference.

Each result bundle is immutable and identifies its inputs. A later run creates a new bundle and states what changed. Public notes link to primary papers or versioned project documentation so claims can be checked against their source.

Report uncertainty and trade-offs

Counts, sample sizes, per-task outcomes, and an uncertainty estimate accompany aggregate scores. Comparisons use paired cases where possible. Quality, latency, cost, memory, and operational constraints are reported together when a decision trades one for another.

Conclusions use language no broader than the experiment. “Higher on this task under this setup” is preferred to a universal claim about model quality.

State limits

Every note identifies known omissions: unavailable revisions, possible training overlap, synthetic workloads, untested languages, judge limitations, or hardware-specific behavior. Missing evidence is not converted into a positive result.

These rules do not guarantee that an experiment is correct. They make its assumptions visible enough for another reader to reproduce the run, recompute the summary, or explain why the evidence does not transfer to a different use.

Minimum run record