A single speed measurement can look impressive. Ten repeated runs under the same conditions usually tell a more useful story. Speed Index therefore runs every core latency and throughput test ten times and reports the average, median, range, and spread instead of highlighting only the best result. The goal is to show how stable the model actually is when the same work is requested repeatedly.
I treat repeated runs the same way I treated performance instrumentation in product teams: one data point is an anecdote, a small series is a pattern. Cherry-picking the fastest call creates a number that marketing can use and that operators cannot trust. Publishing the full spread gives readers the information they need to decide whether the model belongs in an automated or high-volume workflow.
Why Variance Matters More Than Peak Speed
In real use, a model is rarely called once. Support triage scripts, content-generation loops, and internal tools fire the same type of request many times a day. When latency jumps from 1.8 seconds to 6 seconds on the eighth call, the average user experience and the downstream automation both suffer.
Variance affects three practical areas:
User-facing interfaces: inconsistent first-token times make the product feel unreliable even if the average looks acceptable.
Automated pipelines: wide spreads force longer timeouts, more complex retry logic, and higher operational overhead.
Cost forecasting: if some runs produce longer outputs or trigger retries, the token bill becomes harder to predict.
A model whose ten runs cluster tightly is easier to plan around than a model whose average is attractive but whose slowest run is several times longer than its fastest. The ten-run method exists to make that difference visible.
How the Ten-Run Series Is Executed
The procedure stays consistent across every Speed Tests article:
Fix the model version, endpoint, prompt, and network conditions.
Send the request and record first-token latency, total completion time, and output token count.
Observe a short cool-down interval so that temporary warm-up or rate-limit effects are less likely to cascade.
Repeat until ten successful or failed attempts have been logged.
Calculate the arithmetic mean, the median, the minimum, the maximum, and a simple view of the spread.
Record any failures, truncations, or timeouts as part of the series rather than discarding them.
Ten is a deliberate compromise. It is large enough to reveal a usable distribution yet small enough to remain practical for regular testing. When the observed spread is unusually wide, the article notes that fact and may add further runs or a concurrent-load follow-up.
The same prompt is used for every run in the series. Changing the prompt between calls would mix content variance with model variance and make the results harder to interpret.
What Is Reported and What Is Not
Every ten-run test publishes:
Mean and median for first-token latency and total completion time
Observed minimum and maximum
A short description or simple distribution of the ten values when the spread is noteworthy
Failure or truncation count inside the series
Tokens-per-second figures calculated on the same consistent basis
What is deliberately not done:
Highlighting only the single fastest run as the headline number
Dropping outliers without explanation
Averaging across different prompts or different times of day as if they were identical conditions
Claiming that ten runs represent every possible network or load condition
The method is observational under stated conditions. It is designed to support comparison between models tested the same way and to give readers a realistic sense of consistency, not to produce a laboratory certificate.
Cherry-Picking and Why It Misleads
Cherry-picking the best result is common in public demos and selective benchmarks. It produces clean charts and strong headlines. It also creates a systematic optimistic bias. Operators who size timeouts, user expectations, or cost forecasts against the best-case number quickly discover the gap.
By contrast, reporting the full series forces the conversation onto the numbers that actually appear in repeated use. A model that looks only moderately fast but stays inside a narrow band can be more valuable for production than a model that occasionally posts an impressive low number and then drifts.
Speed Index therefore treats the spread itself as a first-class result. When the range is tight, that fact is stated. When the range is wide, that fact is stated with equal weight.
How Results Appear in Articles
When a ten-run series is published it follows the standard Speed Index closing structure:
Use it when…
Skip it when…
What we measured
What we still do not know
The measured section lists the mean, median, and range for the key timing metrics and notes the failure count. The open-questions section records variables still untested—different times of day, longer prompts, higher concurrency, or newer model versions.
Because model behavior and provider-side capacity can change, each series is dated and version-stamped. A later re-test is published as new data rather than an invisible overwrite of the earlier spread.
Why This Discipline Belongs in Every Speed Test
Single-run numbers are easy to collect and easy to misuse. Ten repeated runs under controlled conditions take more time and produce less glamorous headlines. They also produce information that product managers, operations leads, and small-team developers can actually plan around.
Fast is a feature. Reliable is the product. Reliability includes the ability to deliver similar performance when the same request is made ten times in a row. Measuring that consistency, instead of cherry-picking the best single result, is one of the core disciplines of Speed Index. The stopwatch stays running for the full series, and the full series is what gets published.
Letters
No letters yet — be the first to write.