How OmniRoute Chooses and Recovers
OmniRoute's auto routing and resilience system answers a practical question: when several provider paths could serve a request, which one should go first, and what should happen when that path changes state? This chapter treats selection and recovery as one control loop. The useful result is not a universal “best model,” but a decision that reflects the request's objective, the current evidence, and the narrowest failure domain that needs to be avoided.
auto is a live policy, not a magic model
A manually configured combo is a stored list of targets. auto starts from a different premise: the candidate list should reflect the connections that are active now. When a client sends a model such as auto, auto/coding, auto/fast, or auto/cheap, OmniRoute builds a virtual combo in memory for that request. It reads active provider connections, keeps those with usable credentials, resolves a model for each connection, checks the provider registry for capability and pricing information, and creates a candidate for each provider/model/connection combination.
That virtual combo is not saved as a new database object. This matters operationally. Adding a provider, reconnecting an OAuth account, or bringing a local model back online can change the next candidate pool without requiring an operator to edit a combo. The same design also avoids accumulating one stored combo for every temporary combination of accounts. The trade-off is that the decision must be rebuilt and interpreted at request time, using whatever catalog and telemetry are available then.
The candidate pool is not simply a list of names. Each candidate carries conditions that affect whether it is usable and how attractive it is. A connection may be active in configuration but lack a valid API key or non-expired OAuth token. A model may be present in a static registry but unavailable to that account. A provider may be healthy in general while one of its connections is cooling down. A request may require a large context window, vision capability, or a particular kind of reasoning. Before ranking begins, these distinctions keep the pool connected to actual operating state.
The next step is scoring. OmniRoute documents a 16-factor scorer rather than a single priority number supplied by a provider. The factors cover several families of evidence:
- Availability: remaining quota or rate-limit headroom, provider health, and the availability of an OAuth session.
- Performance and cost: inverse blended token cost, inverse p95 latency, and observed stability rather than an isolated fast response.
- Task fit: how well the candidate suits coding, review, planning, analysis, debugging, or documentation.
- Context and account shape: context-window fit, account tier, affinity between the task and tier, connection density, and whether a session can remain on a useful connection.
- Observed outcomes: feedback-driven quality and reliability based on recent routing history when enough observations exist.
- Continuity and timing: optional prompt-cache affinity and quota-reset affinity, which can be enabled by relevant profiles even when they do not vote in the default weighting.
These categories explain why “best” is conditional. A provider with the lowest listed price can lose to one with more quota headroom or better task fit. A fast connection can lose to a stable one when p95 latency and error history are considered. A high-tier model can lose when the request is a background task that does not need it. A provider that answered the previous turn can be favored for session continuity, but only while it remains eligible.
The score is a policy-weighted ranking, not a measurement of intelligence. Some values come from catalogs, some from local observations, and some are defaults for a cold candidate with little history. The system can estimate that a model is suitable for coding; it cannot prove that the model will produce a correct patch for a particular repository. Nor does the score override a hard incompatibility. If the target cannot accept the request's modality or context size, a high score does not turn it into a suitable target.
The variants make the objective visible to the caller. auto/fast emphasizes low latency. auto/cheap emphasizes cost. auto/coding gives more weight to task fitness and quality for code work. auto/offline favors quota or local availability, while reliability-oriented profiles give more weight to health, stability, and observed success. Category and tier suffixes can further constrain the candidate pool, such as a coding pool or free-tier pool. The important design choice is that these modes alter the decision policy over a shared pool; they do not claim that one provider is intrinsically optimal for every use.
A budget adds a different kind of control. The scoring profile expresses preference, but a per-request budget can act as a ceiling: candidates whose estimated cost exceeds the limit can be removed before selection. If every candidate exceeds the limit, the configured fallback determines whether OmniRoute uses the cheapest viable option anyway or refuses to select one. In strict mode, the request fails with HTTP 402 rather than silently overspending. This is a valuable distinction between “optimize for cheap” and “do not cross this boundary.”
The system also supports strategies that are easier to reason about than a multi-factor score. A priority combo can try targets in a fixed order. Round-robin can rotate through them. Cost-oriented or latency-oriented strategies can make one criterion dominant. Last-Known-Good-Provider, or LKGP, favors the provider that most recently succeeded and falls back to broader rules if that provider is no longer eligible. Fusion and pipeline have different semantics again: one gathers several model answers for synthesis, while the other threads outputs through sequential steps. The presence of many strategies does not mean they are interchangeable. They encode different answers to the question of what the operator is optimizing.
One concrete example shows the difference. Imagine two eligible connections for a coding request. Connection A is cheaper and has ample quota, but its recent p95 latency is unstable. Connection B costs more but has steadier latency and a stronger task-fit signal. A balanced policy may select B; a cost-focused policy may select A; a strict budget may exclude B; a reliability-focused policy may continue preferring B. None of these outcomes is an algorithmic contradiction. They are different policy choices over the same evidence.
The decision is also observable. OmniRoute exposes routing-decision metadata and candidate information, while its dashboard can show quota, health, usage, latency, and fallback behavior. That feedback is essential because a score without an explanation is difficult to tune. If auto repeatedly selects a path that looks good in the catalog but fails in practice, the operator needs to see whether the cause is stale model data, an exhausted quota, a connection state, sparse telemetry, or a weighting choice.
Recovery works by shrinking the failure domain
Selection becomes meaningful only when the system knows how to react to failure. OmniRoute separates three primary mechanisms because different symptoms identify different affected scopes. The provider circuit breaker covers the whole provider. Connection cooldown covers one account, key, or provider connection. Model lockout covers one provider-plus-connection-plus-model combination. Treating these as one generic “unavailable” flag would make recovery either too aggressive or too repetitive.
A provider circuit breaker responds to repeated provider-level failures. Its states are CLOSED, DEGRADED, OPEN, and HALF_OPEN. In the normal CLOSED state, traffic is allowed. DEGRADED is an early warning: failures have crossed a warning threshold, but traffic still flows. OPEN temporarily excludes the provider from routing. After the reset period, HALF_OPEN permits a probe. A successful probe can close the breaker; another qualifying failure opens it again. By default, only provider-level statuses such as 408 and 5xx trip this breaker. Ordinary account or model errors, including most 401, 403, and 429 cases, belong to narrower mechanisms instead.
Connection cooldown answers a more local problem. Suppose one API key receives a rate-limit response while another connection for the same provider is still available. Disabling the entire provider would throw away a usable route. OmniRoute records fields such as a future rateLimitedUntil, an unavailable status, the last error, and a backoff level for that connection. Selection skips it while the cooldown is active. OAuth and API-key connections have different base cooldowns, repeated recoverable failures can increase the delay exponentially, and an upstream Retry-After or reset hint can inform the wait. A successful use clears the transient error state.
The anti-thundering-herd guard matters when many requests fail at once. Without coordination, each concurrent failure could extend the same cooldown independently, increase the backoff counter several times, and cause a storm when all callers retry. The guard limits that amplification. It does not make the upstream healthy; it keeps local recovery behavior from multiplying the damage.
Model lockout is narrower still. Some providers expose quota or availability at model level. A 429 for one model should not necessarily disable the other models on the same connection. A local provider may return 404 because one model is not installed, while its other models remain usable. A provider may reject one mode/model combination without rejecting the account. Model lockout records that specific provider, connection, and model tuple as unavailable, allowing routing to preserve the rest of the connection.
The following state map shows the relationship. The states are separate even when one request can move through more than one layer: a provider may be healthy while one connection cools, or a connection may be usable for one model while another is locked out.
The model's most important consequence is diagnostic. If every key for a provider is skipped, the operator should inspect both the provider breaker and each connection's cooldown. If only one key fails, connection cooldown is the right mental model. If only one model fails, model lockout is more precise. If a provider remains excluded after its reset window, the problem may be a stale read of raw state rather than the lazy recovery path that refreshes expired status.
Recovery can also preserve conversation continuity. Session affinity lets a client supply a session key so follow-up requests prefer the same connection across providers, reducing context loss and repeated cold starts on systems with account-level session state. That preference is not an absolute lease: normal health, quota, cooldown, model, and allowlist rules can still make another connection the only eligible choice. In other words, continuity is a routing signal, not permission to ignore failure.
Concurrency controls address a different failure trigger. Subscription accounts may accept only a small number of simultaneous requests. In quota-share routing, several local keys can point at one upstream account, so routing that looks fair at the key level can still flood the account. OmniRoute can deprioritize saturated connections and serialize quota-share dispatch through a per-connection semaphore when a maximum concurrency is configured. The queue is designed to contain pressure without turning every saturated condition into a hard rejection. Request-queue admission can separately bound how many callers wait for a provider/connection before a typed queue-full error is returned.
Waiting is another trade-off. When all candidates are in a short transient cooldown, a combo can wait for the earliest eligible time and retry within a bounded budget. That may improve continuity for a temporary rate window, but it spends client latency. The system does not treat every error as worth waiting for: terminal quota exhaustion, authentication problems, or a missing model need a different response. A bounded wait is a policy choice, not proof that capacity will return.
The distinction between temporary and terminal state is especially important. A connection cooling for a few seconds should become eligible again. A credential marked banned, permanently expired after bounded refresh attempts, or credits-exhausted is intended to remain unavailable until credentials or settings change or an operator resets it. Overwriting a terminal state with a transient cooldown would make the dashboard look healthy while the same dead credential keeps returning to the pool.
All of these mechanisms depend on observability. Quota snapshots, response status, retry hints, connection timestamps, breaker state, model lockouts, p50/p95/p99 latency, fallback attempts, and routing decisions are different signals. A single success rate cannot tell an operator whether the provider is healthy, one account is exhausted, or one model is unavailable. The system's ability to expose those distinctions is what lets a human correct the policy rather than merely raise or lower a global retry count.
There are limits to the control loop. The gateway may receive incomplete or misleading upstream information. Local catalog data can drift from a provider's live model list. A score can be based on sparse observations. Model lockout state is in memory and is lost on restart, while other domain state is persisted. Quota and terms remain external. Auto-routing can reduce the chance that one bad path interrupts work, but it cannot manufacture quota, repair an upstream outage, or certify that the selected model is correct for the task.
The central lesson is therefore narrower than “OmniRoute automatically picks the best AI.” It builds a live candidate pool, applies an explicit policy, records the evidence, and responds to failure at a chosen scope. The quality of that behavior depends on the policy's objective and the accuracy of the state it sees. Once the reader understands that, the next question is how to interpret the gateway's efficiency promises: token compression and free capacity can improve the same control loop, but only when their eligibility and limits are made visible.