From One Endpoint to Many Providers
This chapter follows a single chat request through OmniRoute, from the moment a client sends it to the moment a normalized response returns. The focus is the gateway's operating path: admission, policy, state lookup, translation, provider execution, and feedback. Understanding that path lets you judge what OmniRoute's compatibility promise really covers—and where the behavior of the selected upstream still matters.
A request is admitted before it is routed
A request does not begin with “pick a model.” It begins at a boundary where OmniRoute must decide whether the request is shaped correctly, whether the caller may use the route, and whether the payload is safe to pass into the rest of the system.
Consider a coding assistant sending a streaming chat completion. The client may use an OpenAI-shaped request even if the eventual model is reached through an Anthropic-compatible or provider-specific service. The request first reaches a compatibility route such as /v1/chat/completions. The route performs the ordinary boundary work: it handles cross-origin rules where relevant, validates the body against the API schema, and passes the request through the authorization pipeline. The same gateway also accepts other client-facing shapes, including /v1/messages, /v1/responses, and Gemini-compatible /v1beta routes. These are different entry contracts into a shared routing core, not separate products with unrelated behavior.
Authorization is part of the request architecture because the gateway has more than one kind of caller. A client API request is allowed to ask for model inference with a Bearer key when the configured policy requires one. A management request can change providers, keys, combos, settings, or diagnostics and therefore requires a dashboard session or a key with management scope. The authorization pipeline classifies routes as PUBLIC, CLIENT_API, or MANAGEMENT; an unknown route falls toward the more protected class rather than being treated as harmless. Trusted internal headers are stripped before classification so a caller cannot simply declare itself authorized.
That distinction prevents a common design mistake: treating “the same server” as “the same permission.” A key created for a coding client should not automatically be able to rewrite the routing table. Conversely, the dashboard needs to manage connections without pretending to be an upstream model client. The gateway's first decision is therefore not which provider is cheapest. It is which policy applies to this kind of request.
After route policy permits the request, pre-call guardrails may inspect it. OmniRoute's registry can run transformations or blocks before the upstream call and can inspect the response afterward. A prompt-injection guard may warn or block according to its configured mode; a modality bridge may convert an image or audio part when the target cannot accept it; PII or credential masking can rewrite content when those opt-in features are enabled. These are not interchangeable with authorization. Authorization asks who may use a surface. A guardrail asks what the payload contains or whether it needs a controlled transformation.
The request then reaches the shared chat core. Before an upstream call, that core can consult cache and idempotency state, resolve the requested model or combo, choose a credentialed connection, and prepare the target-specific request. Each step changes what the next step is allowed to assume. A cache hit may end the path without any provider call. A combo name may expand into several possible targets. A model identifier may resolve to an alias or a provider-qualified target. A connection may be skipped because its credential is unavailable or its temporary failure state has not expired.
The important idea is that routing is a sequence of constrained decisions, not a single switch statement. Every boundary carries a different kind of state: route class and scopes for authorization, transformed payload and guardrail results for content handling, model and combo configuration for selection, connection status for credential choice, and provider format for dispatch.
The sequence also explains why the dashboard can show more than “success” or “failure.” The gateway knows which route class was used, which provider and model were selected, whether a fallback occurred, how many tokens were observed, how long the upstream took, and whether a cache served the answer. That evidence is produced by the same path that made the decision, so it can be used to diagnose the decision later.
Translation is a contract, not a universalizer
Once OmniRoute has a target, it must reconcile two kinds of difference: the shape expected by the client and the shape expected by the provider. The architecture uses OpenAI as a hub format for many conversions. In simplified form, a request moves from its source format into an OpenAI representation and then from that representation into the provider's target format. This reduces the number of direct client-to-provider translators that must be maintained.
The hub works because many concepts have a common core. A message has a role and content. A request has a model, generation controls, and possibly tools. A response has text, usage, and a completion state. An intermediate representation can give the routing core a stable place to apply policies, compression, caching, and account selection before the final wire shape is built.
But the common core is not the whole protocol. OmniRoute's translators also perform targeted normalization. A developer role may become system for a target that does not recognize the former. A provider that rejects a system role may require system material to be merged into a user message. Structured output expressed as an OpenAI JSON schema may need to become Gemini's response MIME type and schema fields. Reasoning blocks such as <think>...</think> can be extracted into a distinct reasoning field when the target and client formats support that interpretation. Streaming responses need their own conversion rules because an event stream carries incremental state, termination markers, and usage information rather than one finished JSON object.
These adaptations are useful precisely because they are selective. A translator is not declaring that every provider supports the same semantics; it is deciding how far a particular request can be carried across a known boundary. If the target has no native tool-calling capability, a tool array may be emulated or dropped according to that provider's integration. If a model does not accept a reasoning control, the request may need a compatible alternative or may have to reject the setting. If a provider uses a different authentication flow, the executor—not the generic client contract—handles refresh or session details.
This is why provider executors remain separate from the hub. An executor knows how to build the upstream URL, headers, body, retry behavior, credential refresh, and response handling for a provider family. Some providers use ordinary API keys and OpenAI-shaped endpoints. Others use OAuth, browser sessions, native event streams, signed requests, or a special Responses path. The gateway can hide those differences from a basic client, but it cannot remove the operational facts that create them.
The same principle applies to models. A provider prefix such as provider/model gives OmniRoute a routing identity, but it does not promise that two models with similar names have equal context windows, tool limits, reasoning behavior, or safety policies. The generated catalog records capabilities and model availability, yet some catalogs are synchronized dynamically and some provider lists can drift. A model picker that shows an entry is not the same thing as a live guarantee that the upstream will accept every request for it.
The result is a layered compatibility promise:
- The client can use a documented surface and a stable request shape.
- OmniRoute can apply common policy, state, routing, and telemetry around the request.
- The translator and executor can adapt the request to a supported target path.
- The response can be normalized back into the client's expected format.
- Provider-specific semantics, limits, and availability remain attached to the selected target.
The last point is the one most likely to be lost in a product overview. A common wire format improves reach, but it does not create equal model quality. A client that only asks for text may move cleanly between providers. A client that depends on a particular tool protocol, a long reasoning trace, an image modality, a cached prompt prefix, or a session-bound account is asking for more than syntax. It is asking for a capability and state contract that may not survive a provider change.
State turns a relay into a control plane
The request path becomes more valuable when its local state changes future choices. OmniRoute stores provider connections, model aliases, combos, settings, pricing, usage history, and domain state in SQLite-backed modules. Runtime decisions can also use in-memory state for fast health and availability checks. The distinction matters: a persisted configuration can survive a restart, while a transient lockout or telemetry sample may not. The provider's authoritative quota is still external and may disagree with a locally cached view.
Usage and decision records are not merely for billing reports. A response can carry metadata identifying the provider, model, latency, cost, cache result, fallback attempts, request ID, and version. Health views aggregate provider breaker state, connection cooldowns, model lockouts, cache statistics, and latency percentiles. An operator can therefore ask a more useful question than “did the request return 200?” They can ask which path returned it, under which conditions, and whether that path remains appropriate.
The feedback loop also has a failure side. If a credential expires, an executor may refresh it and retry. If a provider returns a failure eligible for fallback, the routing layer can update local availability state and try another target. If the response has already committed bytes to the client, the system cannot always replay the whole request without changing what the client saw; streaming safety therefore constrains which recovery actions are possible. A gateway that exposes these boundaries is easier to operate than one that silently retries until a timeout.
There is a cost to centralization. One endpoint becomes a valuable control surface, so a configuration error can affect many clients at once. A bad alias, an overly broad fallback, an incorrect budget, or an exposed management key can have a wider blast radius than a mistake in one client file. Centralization is beneficial only when the policy and state behind it are visible, scoped, and recoverable. The route pipeline addresses this with separate auth classes, management scopes, local persistence, health signals, and explicit response metadata, but the operator still owns the configuration.
This also explains why OmniRoute supports both direct provider paths and higher-level combos. A direct model request is a precise choice when the caller knows the target it needs. A combo is a policy choice: it lets the gateway decide how to proceed across a set of targets when availability, cost, or continuity changes. The same translation and execution machinery can serve both, but the decision responsibility differs. Direct routing exposes more provider-specific behavior to the caller; combo routing delegates more choice to local policy and telemetry.
By the end of the request, the client sees a response in its expected shape, but OmniRoute has done more than forward it. It has admitted the request under a policy, possibly transformed its payload, resolved a target, selected stateful credentials, crossed a format boundary, observed the result, and recorded enough context to inform the next decision. That is the concrete meaning of a local control plane.
The remaining uncertainty is not how the gateway can reach many providers. It is how it should choose among them when several paths are eligible—and how narrowly it should react when quota, health, latency, or account state changes. The next problem is therefore selection under changing conditions, not translation alone.