When Failover Keeps the API Alive but Loses the Conversation

A multi-provider gateway can keep returning responses after a primary model fails, yet still break the application that depends on the conversation. This chapter examines that continuity gap and the mechanism used to measure it: compare a fallback that receives only the latest request with one that reconstructs the preceding dialogue. The practical judgment is not whether the fallback answered, but whether it answered with the state that made the request meaningful.

Availability is not continuity

Availability is the property most failover systems expose clearly. A provider times out or returns an error; the gateway selects another endpoint; the user receives a response. From the gateway’s perspective, the failure has been contained. The request completed.

Continuity asks a different question: did the response preserve the relevant context established before the provider changed? A user might ask, “Which cuisine did I say I preferred?” after several unrelated turns. If the fallback receives only that final question, it can return a fluent answer, but it has no evidence for the preference. The system is available in the narrow sense and disconnected in the sense that matters to the conversation.

That distinction is easy to miss because ordinary operational signals favor availability. Health checks can be green. The fallback can return a valid status code. The response can contain no obvious error marker. None of those signals shows whether the new provider saw the earlier dialogue. A user experiences the difference immediately: they have to repeat information, or they receive an answer that sounds plausible but is not grounded in what they already said.

The consequences become more serious when an LLM is part of a multi-step process. An agent may carry a decision from one turn to the next, use a retrieved document, or wait for a tool result before continuing. If a failover silently removes an earlier assumption, the agent can keep working with a truncated state. Later steps may look reasonable even though the root error occurred at the provider boundary. A request-level success metric can miss that failure because it measures the response, not the continuity of the work.

ContinuityBench formalizes this as a gap between request-level failover and dialogue-level state preservation. Existing multi-provider gateways are commonly organized around independent API calls. When a call fails, the gateway forwards a payload to another provider. Unless the gateway has deliberately retained and reconstructed the conversation, the fallback receives the current request without the history that preceded it. This is not necessarily an accidental omission. A request-level abstraction is designed to provide a response, not to make conversation history a first-class object.

The difference can be stated as a boundary. Availability crosses a provider boundary with a response. Continuity crosses that boundary with the information required to interpret the response. In a one-shot request, the two may be nearly identical. In a stateful assistant or agent, they are separate reliability properties and need separate tests.

What changes at failover

The ContinuityBench comparison is intentionally narrow. The baseline and treatment share the same request interception, fault injection, provider abstraction, logging, and evaluation harness. They differ in the payload forwarded after a primary-provider failure.

In the stateless baseline, the gateway extracts only the most recent user message and sends that as a new request to the fallback.