When Average Accuracy Hides a Missed Expert
The difficult problem in model routing is not proving that a model pool contains useful specialists. It is getting the right specialist to answer the particular request in front of the system. This chapter treats the gap between a router’s average accuracy and the per-query Oracle as a selection diagnosis: the reader will learn how to recognize model-recall failure, why it matters more than a small average-point loss, and why enlarging a pool cannot replace curating it.
The Oracle gap is a selection diagnosis
The benchmark’s leading routing methods sit close together on average, but they remain far below the per-query Oracle. That pattern is easy to misread. One interpretation is that the candidate models are not capable enough. Another is that the routing methods are not choosing well. LLMRouterBench’s analysis supports the second explanation as a major part of the gap: many requests can be answered correctly by only one or a few candidates, and current routers often fail to select those candidates.
Here, model recall has a specific meaning. It is not the recall of a search engine or a retrieval-augmented generation system. It describes whether the router selects a model that has the capability to answer the request correctly. The capability must already exist somewhere in the pool; the routing decision must find it. A query can therefore fail in two different ways. No candidate may know how to solve it, which is a capacity limitation. Or a candidate can solve it, but the router sends the request elsewhere, which is a selection limitation. The final response may look identical, but the remedies are different.
The distinction becomes especially important for rare cases. Imagine a pool in which eight models can answer a common factual question. Almost any reasonable selector will often choose one of them. Now consider a technical problem for which only one model has the relevant reasoning or coding behavior. A router that chooses any of the other seven has a high-confidence decision only in the sense that it has made a decision; it has not delivered reliable model selection. The average score can remain healthy if common questions dominate the workload, while the rare specialist cases carry most of the practical risk.
The benchmark reports a concrete version of this pattern. In its query-hardness analysis, requests for which at most three candidate models answered correctly made up 410 queries, or 11.9% of the test set. On that subset, the reported accuracy of Avengers was 24.6% and EmbedLLM’s was 23.2%. These figures do not mean that every difficult request requires a different model, nor do they prove that either router fails for the same reason on every item. They show that the cases with a small set of capable experts are a visible source of error, and that aggregate performance can conceal how poorly a selector handles them.
This is why the per-query Oracle is a useful diagnostic even though it cannot be deployed. The Oracle has access to the completed answers and can choose a correct model after the fact. If it performs much better than a live router, the pool contains unrealized opportunity. The difference does not tell us that an Oracle-like system is easy to build. It tells us where to look: at the information available before selection, the router’s uncertainty, the representation of the query, and the way rare capabilities are prioritized.
The benchmark’s reported performance-oriented values make the scale of the gap visible. EmbedLLM, GraphRouter, MODEL-SAT, and Avengers are all around the low seventies in average accuracy under the shared protocol, while the per-instance Oracle is 91.64. The methods are not equally good on every dataset, and the values are not production guarantees. Still, the distance between the cluster of live methods and the hindsight ceiling is too large to explain only as random noise or a shortage of models. A selector can be broadly competent and still systematically miss the model that matters on a narrow class of requests.
Average metrics make this failure easy to overlook for structural reasons. An average compresses different errors into one number. A route that improves many routine tasks can offset a serious regression on a small specialist domain. Dataset-level averaging adds another layer: a method can appear stable across the suite while its failures concentrate in the hardest items of one dataset. The reader should therefore ask not only “What is the average accuracy?” but also “How many candidates were capable of this query, and did the router select one of them?”
That question changes what a routing system must observe during evaluation. A useful analysis should record the set of candidate outcomes for each query, the number of models that were correct, and whether the selected model belonged to that set. It can then distinguish at least three cases: broad agreement among models, genuine lack of capability in the pool, and failure to recall a rare capable model. The sources do not present this as a complete operational recipe, but their query-hardness analysis points in this direction. Without the per-query view, a router’s most consequential misses disappear into the average.
The diagram separates the location of capability from the act of selecting it. On common requests, those two often align by chance. On rare requests, the selection step becomes the dominant risk. A router can only recall a specialist that is present in its candidate pool, but presence alone does not make the route dependable.
What could improve recall? The benchmark points to better uncertainty or difficulty estimation and to mechanisms that explicitly prioritize rare-but-critical experts. Those are directions, not demonstrated solutions. A router may estimate that a query is hard and still choose the wrong model; it may identify a domain correctly but miss a capability that does not follow ordinary topic boundaries. The evidence supports a change in diagnostic emphasis, not a promise that one new feature closes the Oracle gap.
There is also a temptation to treat a larger model as the answer to every recall problem. A stronger general model may reduce some misses, but the benchmark’s complementarity result means that strength is not a single axis. A generally strong model can still be weaker on a particular coding, mathematical, tool-use, or conversational behavior. Replacing a portfolio with one expensive default may reduce selection errors in some workloads while discarding the cost or specialist advantages that made routing worthwhile.
The safer conclusion is narrower: route evaluation should include tail selection behavior. The relevant question is not whether the router usually chooses a good model in the abstract. It is whether the selector recognizes when the request falls outside the region where many candidates are interchangeable, and whether it responds with enough caution to preserve access to the few models that can succeed.
A larger pool is not the same as a better pool
Once model recall is separated from model capacity, adding candidates looks less obviously beneficial. More models can increase the number of capabilities available in principle. They can also create more choices for the router, more provider interfaces to maintain, more cost and latency differences to characterize, and more behaviors to test after a model or endpoint changes. The relevant design question is not “How many models can we connect?” but “Which set gives the route useful coverage that the selector can reliably exploit?”
LLMRouterBench uses the Oracle as a pool-capacity probe. When the study expands a very small pool to a moderate one, the potential gains are substantial. After that, adding more models yields diminishing returns. A carefully chosen small subset can outperform a larger random subset. This result concerns the benchmark’s pools and Oracle measure, but it supports an important systems interpretation: candidate-pool composition is part of routing quality, not merely a procurement or infrastructure decision.
The reason is coverage. Suppose the first few models collectively cover mathematics, code, and general knowledge, while later additions are near-duplicates of capabilities already represented. The pool grows, but the set of requests for which a new model is uniquely useful grows slowly. By contrast, one carefully selected specialist can add more useful coverage than several broadly similar models. The value of a candidate is therefore its marginal contribution to the pool, not its standalone reputation or parameter count.
The benchmark’s comparison between random selection and a top-performing subset illustrates the point. A random pool samples whatever strengths happen to be available. A curated subset deliberately keeps models that perform well across the evaluation’s task structure. The study reports that the carefully selected subsets produce stronger Oracle outcomes than random subsets, especially once the pool is no longer tiny. This does not mean that ranking models by average accuracy is the final curation method. Average strength can favor generalists and overlook a rare specialist. It does mean that pool composition deserves an explicit method rather than an assumption that more candidates are automatically better.
Curation also reduces hidden coupling. Every additional model can introduce a different context limit, output convention, tool capability, price, latency profile, or provider failure mode. The source does not measure all of these operational costs in the pool-size experiment, so they should not be presented as measured reasons for the observed diminishing returns. They are the engineering consequences a production team should examine before treating a benchmark gain in potential capacity as a practical gain in reliability. If the router cannot distinguish the new model’s conditions or cannot test its behavior, the added candidate may enlarge the failure surface faster than it enlarges useful coverage.
This is why pool selection and router selection should be evaluated jointly. A model pool can be too small to contain the needed expert. It can also be large enough in hindsight but poorly organized for the router’s decision process. The first problem calls for adding or replacing candidates. The second calls for better selection signals, uncertainty handling, or evaluation. Treating both as “we need more models” hides the actual cause.
A curated pool should therefore answer three practical questions. What distinct work does each candidate cover? The answer should be based on observed task behavior, not only a model’s general label. Where is the overlap useful? Redundancy can provide fallback options, but it also consumes selection and testing capacity. Which gaps remain dangerous? A model that is rarely selected may still be essential if it is the only reliable candidate for an important class of requests. These questions are a decision framework, not findings that the benchmark has already answered for every application.
The trade-off can be expressed as a simple sequence. First, use the intended workload to identify the capabilities the route must cover. Next, test candidate models on that workload and identify where their strengths are genuinely complementary. Then evaluate the router on the curated pool, including per-query cases in which only one or a few candidates are correct. Finally, recheck the pool when models, prices, or request patterns change. A pool that was well curated for one distribution may no longer be well curated after deployment conditions move.
This last point is important because the benchmark is a fixed evaluation. Its five-seed repetitions test variation within the selected protocol, not all of the changes a live system will experience. A new model release can alter the relative ordering. A new customer segment can make a previously rare task common. A provider change can make a candidate slower or less available. Curation is therefore not a one-time list of approved models; it is an ongoing claim that the current pool still covers the work the application promises to handle.
The combined diagnosis is now clearer. The Oracle gap says that the available capability is not being recalled reliably. The pool-size result says that adding candidates eventually contributes less than selecting the right ones. Together they argue for a routing design that treats specialist coverage, uncertainty, and pool composition as linked controls. They do not prescribe one universal router, and they do not show that a small pool is always superior. A regulated workflow may need extra candidates for policy or availability reasons; a low-stakes drafting tool may prefer a smaller, simpler set. The correct pool depends on the promises the route must keep.
Once the router is judged this way, cost and speed can no longer be added as afterthoughts. A specialist may be accurate but expensive. A cheaper candidate may be adequate for common work but unsafe for the rare cases just identified. A route that takes too long may trigger the very failover behavior that changes its state and outcome. The next question is how to compare these competing objectives without calling every cheaper choice an improvement.