Inference

Latency, throughput, and context

A serving benchmark is a workload description plus a distribution of observations. One requests-per-second number is not enough.

By LLM Model Notes · 16 July 2026 · 9 minute read

Interactive language-model serving has at least two visibly different phases. The server first processes the prompt, then generates new tokens one step at a time. Hardware, batching, cache pressure, and request shape affect those phases differently. Useful measurements keep them separate and record the workload that produced them.

Separate latency from throughput

Time to first token measures the delay from sending a complete request until the first generated token arrives. It includes queueing, prompt processing, and the first decoding step. Inter-token latency measures the spacing between later tokens. Its inverse is often reported as output tokens per second for one request.

End-to-end latency covers the complete response. It depends strongly on output length, so it should never be compared without a length distribution. Throughput measures completed requests or processed tokens across the whole server. A system can increase aggregate throughput through batching while making individual requests wait longer. Neither metric replaces the other.

Record the workload envelope

A result is meaningful only inside its workload envelope. Record the prompt-length and output-length distributions, arrival pattern, concurrency, model revision, numerical precision, parallelism strategy, runtime revision, accelerator type, memory capacity, and power limits. State whether prefixes repeat and whether prefix caching is enabled.

Use request traces that resemble the intended application. A fixed 128-token prompt says little about document analysis with 20,000-token inputs. Uniform arrivals can hide queue behavior that appears under bursts. If production data cannot be shared, publish a synthetic generator and its seed so the shape remains reproducible.

Control warm-up and cache state

Cold model loading, kernel compilation, memory allocation, and cache population can dominate the first requests. Decide whether the benchmark represents cold start or steady service and label it accordingly. For a steady-state run, warm the server using the same request shape, then begin measurement only after resource use and latency stabilize.

Cache policy can create large differences. Reused prefixes reduce prompt work; response caches can avoid generation entirely. Both may be legitimate product features, but a comparison must give every system the same opportunity and disclose hit rates. Flush caches only when the workload contract calls for cold behavior.

Measure a steady run

Begin below saturation and increase offered load in steps. At each step, hold the workload long enough to observe queue growth, cache turnover, and thermal behavior. Capture completed requests, rejected requests, queue delay, input tokens, output tokens, latency percentiles, memory use, and accelerator utilization.

The useful capacity boundary is usually the highest load that still meets an explicit service objective, such as a p95 first-token limit and a minimum generation rate. Reporting only the highest throughput can reward a configuration whose queue has already made interaction unusable.

Keep failed and cancelled requests in the accounting. Removing them makes an overloaded server appear faster. If clients abandon slow responses, report both server completion latency and user-observed cancellation.

Treat long context as a separate experiment

Longer prompts increase prompt-processing work and key-value cache memory. They can reduce available batch size and move the server into a different capacity regime. Test several fixed prompt-length bands rather than mixing them into one average, and report whether requests were truncated or shifted to another backend.

Performance is not the only concern. A model may accept a long sequence while failing to use information located in the middle. Pair serving measurements with a task that checks retrieval or reasoning across the same context lengths. Otherwise a fast number may describe tokens the model does not use reliably.

Report quality and cost together

Quantization, speculative decoding, reduced precision, and smaller models can change both speed and output behavior. Validate quality under the exact serving configuration, not only with an unoptimized checkpoint. State acceptance criteria before tuning so speed work does not silently trade away the capability being served.

For cost, report the denominator: per completed request, per one million input tokens, per one million output tokens, or per hour at the declared service objective. Include idle capacity if the deployment must reserve it. This turns a benchmark into a decision record instead of a hardware demonstration.

A compact run record

1. Pin model, runtime, numerical precision, hardware, and parallelism revisions.
2. Publish prompt lengths, output lengths, arrival pattern, and concurrency.
3. Separate first-token, inter-token, end-to-end, and aggregate measurements.
4. Label warm-up, cache policy, prefix reuse, failures, and cancellations.
5. Find capacity under a declared latency objective rather than at any cost.
6. Recheck task quality using the exact optimized serving configuration.

Primary references

• Efficient Memory Management for Large Language Model Serving with PagedAttention — memory layout and serving throughput trade-offs.
• MLPerf Inference — load-generation scenarios, latency constraints, and reproducible result rules.
• vLLM — an open serving runtime whose versioned configuration can be captured with a benchmark.