Most model comparisons end with a single ranking. Real work rarely works that way. A model that leads on first-token latency may trail on cost; a model that stays cheap may require more editing; a model that produces cleaner output may not be the quickest. Speed Index therefore runs a three-model comparison that deliberately allows different leaders on different axes—speed, cost, and practical accuracy—using the same tasks and the same scoring rules.
I design these multi-axis tests the same way I used to design product trade-off discussions: make the dimensions visible, keep the inputs identical, and refuse to collapse everything into one number that hides the compromises. Readers who weight speed highest should be able to see the speed leader. Readers who weight cost or output reliability highest should be able to see those leaders too.
Why Three Axes Instead of One Ranking
A single composite score forces a weighting decision that may not match the reader’s situation. Product teams under tight interactive deadlines often care most about first-token latency and total completion time. Operations teams running high volumes often care most about cost per completed task. Teams that pass model output to customers or executives often care most about how much editing is still required before the text is usable.
By keeping the three axes separate, the comparison surfaces those different priorities instead of burying them. The same matched tasks are scored for:
Speed: first-token latency, sustained tokens per second, total completion time, and consistency
Cost: observed input and output tokens multiplied by the public rates in effect on the test date, expressed as cost per task
Practical accuracy: instruction following, factual fidelity to the source material, structural cleanliness, and the amount of human editing still required
No attempt is made to average the three into a final “winner.” The article simply places the results side by side so readers can apply their own weights.
Model Selection and Matched Conditions
Three models are selected for the comparison. One is GLM 5.3 Fast, the primary model of the site. The other two are chosen from widely available fast alternatives that small U.S. teams are likely to evaluate (commonly a GPT-family model and a Claude-family model, or another strong open or commercial option available at the time of testing). Exact version strings are recorded for all three.
Every model receives the identical set of tasks and the identical prompts. Tasks are drawn from the everyday product and operations work already used in earlier Versus articles—short summaries, structured extraction, plain-language rewriting, and triage-style categorization—so that the multi-axis results remain comparable with previous single-pair tests.
Test conditions stay fixed: the same Austin home-office machine, the same residential fiber connection, the same number of runs per task, and the same cool-down practice between calls. Order of models is rotated across the task set to reduce simple sequence effects.
How Each Axis Is Measured
Speed follows the standard Speed Index baseline method. First-token latency, tokens per second, and total completion time are captured for every run. Consistency (average, median, range) is reported so that a model with attractive averages but wide scatter is visible.
Cost is calculated from the observed token counts rather than from theoretical maxima. Public input and output rates on the test date are applied, and the result is shown as cost per completed task. Any retries required by truncated or malformed outputs are included in the token totals.
Practical accuracy is scored with short observational notes on four points: whether the output followed the explicit instructions, whether it stayed faithful to the supplied source material, whether the structure matched what was requested, and roughly how much editing a human still needed before the text could be used. These notes remain qualitative and separate; they are not collapsed into a numeric quality score that would re-introduce hidden weighting.
What the Comparison Does Not Claim
A three-axis comparison still has clear limits. It does not claim:
That the selected tasks represent every possible use case
That the three models exhaust the current market
That laboratory conditions free of ordinary network variation were achieved
That future model versions will preserve the same relative positions
It claims only that the three models were given the same tasks under the same documented conditions and scored on the same three practical axes. That is enough to show where the leaders differ and to give readers a concrete reference for their own priorities.
Failures and incomplete outputs are recorded on every axis. A model that looks strong on speed but produces frequent truncations will show that cost and accuracy impact alongside the latency numbers.
How Results Will Be Presented
When the full three-model results appear they will follow the standard Speed Index closing structure, adapted for multiple leaders:
Use it when… (per axis where helpful)
Skip it when…
What we measured
What we still do not know
The measured section will present three clear side-by-side views: the speed ordering, the cost ordering, and the practical-accuracy notes. Readers will be able to see at a glance whether one model led across the board or whether the leaders split. The open-questions section will note task types, concurrency levels, or newer model versions still untested.
Because models and pricing change, each multi-axis comparison is dated and version-stamped. A later re-test is published as a new data point rather than an invisible replacement of the earlier one.
Why Allowing Different Winners Is More Useful
Forcing a single ranking creates a false sense of certainty and often mismatches the reader’s actual constraints. A team that lives on interactive latency needs a different primary signal from a team that lives on monthly invoice size or on the volume of post-editing required. Keeping the axes separate respects those differences.
The format also makes the trade-offs honest. If the fastest model is also the most expensive, or if the cheapest model requires the most cleanup, that tension remains visible instead of being averaged away. Readers can then decide which compromise fits their situation.
Fast is a feature. Reliable is the product. Cost predictability is part of reliability. A comparison that lets different models lead on different dimensions keeps all three statements measurable. Speed Index runs the tasks, publishes the three axes, and leaves the final weighting to the people who have to live with the choice.
Letters
No letters yet — be the first to write.