Overview

LLMRouterBench treats routing as a choice among operating points with different accuracy, price, and response time. This chapter distinguishes performance gain from cost saving, defines why Best Single and the empirical Pareto frontier matter, and shows how latency can turn a nominally good route into a timeout or fallback trigger. Its figures are conditional on the evaluated pool, prices, workload, provider configuration, and accuracy target.

Learning outcomes

Under the reported setup, some routing configurations achieved roughly 4% relative performance gain and up to 31.7% cost reduction while matching Best Single.
A cheaper route counts as a saving only when it meets the workflow’s accuracy target; otherwise it is an explicit quality trade-off.
Latency adds a third constraint, and provider-specific tail behavior can turn an accurate route into a timeout or fallback trigger.

Keywords

LLMRouterBenchCostAccuracyLatencyParetoCostSave

Cost Is Only One Coordinate of the Route

A routed LLM system does not choose between “good” and “bad” models; it chooses among operating points with different capability, price, and response time. This chapter examines how LLMRouterBench turns that choice into a conditional comparison, and why the benchmark’s cost-saving results cannot be separated from accuracy targets or latency. The practical judgment is straightforward: a route is an improvement only when it meets the workflow’s required quality and service conditions, not merely when its invoice is smaller.

Cheaper only counts when the target survives

Cost is attractive as a routing objective because model calls have visibly different prices. A system can send routine requests to a less expensive model and reserve a more capable one for work that appears difficult. But “send difficult work to the strong model” is not yet a decision rule. The system needs a performance target, a way to compare the cost of meeting that target, and a way to identify when a particular operating point falls short.

LLMRouterBench defines those comparisons relative to Best Single, the strongest fixed model under the evaluation. For each routing method, the benchmark considers a set of configurations. One configuration, written conceptually as the best-accuracy point, maximizes average accuracy without regard to cost. Another is the least-cost configuration whose average accuracy is at least as high as Best Single. Those two points answer different questions. The first asks how much performance the router can reach at its best. The second asks how much money it can save without giving up the fixed-model reference level.

This distinction produces two useful measures. PerfGain is the relative improvement of the router’s best average accuracy over Best Single. CostSave is the fraction of Best Single’s cost removed by the least-cost configuration that still matches its average accuracy. A method can have a positive performance gain but no cost-saving configuration that reaches the target. It can also find a cheaper configuration only by accepting lower accuracy. Those outcomes should not be described with the same word—“efficient”—because they represent different product choices.

The word “relative” matters here. A gain expressed relative to Best Single is not the same as a percentage-point increase in accuracy, and a saving is measured against the cost of the fixed reference rather than against an arbitrary absolute price. Keeping the denominator visible prevents a small baseline change from making two otherwise similar routes look incomparable. It also clarifies why a method marked not applicable for CostSave has failed a performance condition before cost can be credited; the missing value is not evidence of zero expense.

A small example makes the denominator matter. Suppose a fixed model answers 80% of a workload correctly at a cost of 100 units. A routed configuration that answers 84% correctly at 110 units has a performance gain but no cost saving at that operating point. If another configuration answers 80% at 70 units, it has a 30% cost saving under the matching-accuracy rule.

Published by DeepMox

FAQ

When does a cheaper model count as a routing cost saving?

A cheaper route counts as a cost saving only when it reaches the required accuracy target, such as Best Single in LLMRouterBench’s rule. A lower-cost route that misses the target is a quality trade-off, not a saving under that comparison.

What does the performance-cost Pareto frontier tell you?

The performance-cost Pareto frontier shows which evaluated operating points are not dominated by alternatives that are at least as accurate and no more expensive. It describes observed trade-offs, not safety, reliability, or operational ease.

Why should latency be evaluated with cost and accuracy?

Latency belongs in the routing decision because a slow route can miss the application’s time budget and trigger retries or fallback even when its price and accuracy are acceptable. LLMRouterBench’s latency estimates are provider-specific rather than a universal ranking.

Previous10 minModel Recall in LLM Routing

Next9 minIntended vs Effective LLM Routes

LLMRouterBench: Cost, Quality, and Latency

Cost Is Only One Coordinate of the Route

Cheaper only counts when the target survives

FAQ