Controlling the Next Token: Structure, Speculation, and Trade-offs

SGLang can optimize serving at the moment a token is chosen, not only before generation begins. This chapter examines two different interventions: structured generation narrows the set of legal next tokens, while speculative decoding proposes several future tokens for the target model to verify. Understanding the two paths lets a reader distinguish a format guarantee from a quality guarantee, and evaluate a speed claim by its acceptance, memory, and compatibility conditions.

The difference is easiest to see in an application that needs reliable machine-readable output quickly. A JSON response must not merely look plausible; it must satisfy a schema that downstream code can parse. A long explanation, meanwhile, may be slow because the target model must perform a separate decode step for each token. SGLang addresses these problems in different ways. One changes the admissible output space. The other tries to make each target-model step advance farther.

Correctness starts before the next token

A structured-output request gives SGLang a formal description of what may be generated. The documented interface accepts one of a JSON schema, a regular expression, or an EBNF grammar per request. The default XGrammar backend supports all three; Outlines supports JSON schema and regular expressions; Llguidance supports all three. The backend choice is therefore part of the request’s execution contract. A server launched without a grammar backend cannot silently provide the same behavior.

The runtime does not wait until the full answer has been generated to inspect it. A grammar object represents the current state of the constraint. It can accept a token, determine whether the grammar has terminated, and fill a vocabulary mask for the current decoding step. That mask is applied to the model’s logits so tokens that would violate the grammar are excluded before sampling. After a token is accepted, the grammar advances and produces the mask for the next state.

This is a stronger mechanism than post-processing, but its guarantee has a precise scope. If the schema says that an object must contain a string and an integer, constrained decoding can enforce that structural form. It cannot make the model name the correct city, select a useful field definition, or satisfy a business rule that the schema does not express. SGLang’s documentation accordingly recommends putting explicit format instructions in the prompt as well. Syntax and semantics are separate responsibilities.

Constraint compilation also participates in scheduling. A new schema or grammar may not yet have a ready grammar object. SGLang’s grammar manager looks for a cached object by constraint key; on a miss, it starts compilation asynchronously and places the request in a grammar queue. The scheduler admits requests only after their grammar is ready. This avoids running a request with an unavailable mask, but it means a feature that looks like a per-token decoding option can add latency before the first forward pass.

Caching changes the cost for repeated schemas. Once a grammar has compiled, later requests with the same constraint can reuse the cached grammar object rather than paying the compilation work again. That reuse is conceptually parallel to prefix caching, but the state is different.