Inside the Serving Path: Scheduling, Streaming, and Acceleration

Once Magnitude has selected and admitted a local model, the user's problem changes. The question is no longer simply whether the configuration fits, but why an agent request takes the time it does and how the service keeps that time understandable. This chapter follows one live request through queueing, prompt processing, token generation, progress reporting, and speculative acceleration so the reader can separate responsiveness from raw model speed.

A request is more than token generation

When an agent sends a request to Magnitude, several kinds of work can occur before the first useful token appears. The service must admit the request, possibly wait behind other work, prepare the prompt using the model's chat format and tools, evaluate those input tokens, and then begin autoregressive decoding. If the selected model is not resident, loading comes earlier still. These stages produce different user experiences and have different causes.

Loading is the cost of making model resources resident. It can dominate the first request but does not describe the model's steady-state generation rate. Queueing is time spent waiting for an available sequence or a scheduling opportunity. Prefill is the evaluation of the input prompt and its existing context. It can be substantial for an agent that sends a large codebase excerpt or a long tool history. Decode is the repeated generation of output tokens after the prompt has been processed. Time to first token includes the work before the first visible output; sustained decode rate describes what happens after generation is underway.

Those quantities should not be folded into one “tokens per second” number. A model can have a strong decode rate but a slow first response because it loads a large artifact or receives a long prompt. A shorter prompt can make the same model feel faster without changing its kernel performance. Under concurrency, another request's prefill can delay an active response even when the model's isolated decode rate is unchanged. Magnitude's progress vocabulary reflects this: the stream can identify model loading, queueing, preparation, prefill, and generating rather than leaving the user to infer everything from silence.

The engine's scheduler makes a deliberate trade-off between these stages. It places runnable decode work before prompt work in each batch assembly. For every resident sequence, at most one next-token decode is selected per iteration, preserving a stable order. Any remaining batch capacity is then assigned to prefill work in rotating round-robin quanta. A long prompt is split into bounded pieces and shared among waiting sequences instead of occupying the entire batch until it finishes.

The unit being scheduled is a piece of work for a resident sequence, not an entire user conversation treated as an indivisible job. When several requests are waiting, the engine can admit a group during an idle coalescing interval and then assemble a batch from the active set. Decode work is selected first because an already visible generation has a continuity expectation. Prefill is selected in bounded pieces because a new request also needs a path to its first token. The planner's policy is intentionally narrow: it decides how to fill the current native batch, while sequence state, cancellation, and cleanup determine whether that work remains safe to reuse.

This view also explains why a service-level benchmark has to look beyond an isolated decode loop.