A Stateful Fallback Can Fail in Two New Ways

Stateful failover solves one problem by creating another class of risks. This chapter examines the two systems failures that defeated an otherwise effective continuity design in the ContinuityBench study: shared state was corrupted under concurrency, and retries turned a rate-limited fallback into an escalating outage. The practical lesson is that a reliable route must protect both the correctness of the state it forwards and the capacity of the provider that receives it.

When state ownership breaks

The first failure appeared as a quality problem. At low concurrency, the History-Forwarding proxy preserved context well. When the test was raised to 100 simultaneous conversations, Continuity Preservation Rate fell to about 28%. The fallback responses were not merely forgetful. They contained fragments of other conversations, so a request could receive a factual anchor that belonged to a different session.

The symptom could easily be blamed on the model. A fallback provider that answers a question incorrectly looks like a model-quality problem in an ordinary dashboard. The system’s logs, however, showed a different cause: an early version of the harness maintained a shared, global cache of conversation histories. Concurrent tasks updated and transmitted that shared history while asynchronous work was interleaved. A failover payload for Conversation A could capture information that Conversation B had just written.

This is a state-ownership failure. The forwarding idea was correct in the abstract—the fallback needed the preceding conversation—but the system did not establish which conversation owned each mutable record at the moment it was read. The more faithfully the proxy tried to forward history, the more damaging the race became. A stateless gateway might lose context; a stateful gateway can send the wrong context unless the state boundary is explicit.

The causal sequence is short:

  1. Several conversations are active at once.
  2. A shared history object is read or changed by more than one task.
  3. An asynchronous yield lets another task modify the object before the failover payload is fixed.
  4. The gateway forwards a mixture of messages to the fallback.
  5. The fallback produces a response that is fluent but grounded in another session’s state.

The important point is not the programming language or the particular cache. It is the timing of ownership. A conversation history must be isolated before concurrent work can mutate it, and the snapshot used for failover must remain stable while the provider switch is in progress. The paper reports a fix that deep-copied each conversation’s messages at the local thread boundary before asynchronous yielding. It also draws the broader production lesson that internally cached histories require robust per-session read and write coordination.

This failure is more serious than an ordinary wrong answer because state can cross a tenant boundary. The issue is not only that a user receives an inaccurate response. They may receive details from another user’s conversation. Even when the synthetic harness contains no personal information, the mechanism identifies a real class of isolation defect in any gateway that stores conversation state for multiple sessions.