When a Prompt Becomes an Execution Plan
SGLang’s frontend addresses a deceptively important problem: how can an application express the structure around generation without reducing everything to one finished string? This chapter focuses on the boundary between prompt logic and serving machinery—roles, generated values, branches, streams, and common prefixes—and shows how SGLang turns those relationships into runtime work. The practical payoff is a way to choose between a familiar API and a more expressive interface while understanding what each choice makes visible to the system.
Imagine an application that asks a model for two possible answers, compares them, and then asks for a final response using the better candidate. A conventional client might issue three unrelated requests and manually concatenate their text. SGLang can represent the conversation and its dependencies as a program: establish a system instruction, generate named values, fork from a shared state, and join the results. The difference is not cosmetic. It tells the runtime which pieces are fixed context, which pieces are model-produced values, and which pieces may proceed independently.
A prompt is structured state, not just text
Chat applications already have structure even when their API hides it. A system instruction sets behavior; a user message supplies a task; an assistant turn contains a response. The model ultimately receives tokens, but the route from roles to tokens matters. Different models expect different separators, control tokens, and chat templates. If an application assembles those details itself, it must keep its formatting synchronized with the model it serves.
SGLang’s frontend represents roles explicitly. A program can add system, user, and assistant content, while the runtime’s chat-template logic supplies the appropriate prefixes and suffixes. When a role begins, the executor can insert a default system message if the conversation requires one, then append the template’s role prefix. When the role ends, it records the resulting message in an OpenAI-style representation and applies the corresponding suffix. This preserves two views at once: a textual prompt that can be sent to the model and a structured conversation that the application can inspect or continue.
That separation becomes valuable when a prompt contains model-produced values. A generation operation is not merely “some text that happens to appear here.” In SGLang’s intermediate representation, a generation node can carry a name and sampling parameters. The returned value is stored under that name, and later program logic can read it. A selection operation is different: it asks the backend to score or choose among explicit alternatives, then appends the selected decision to the running state. Both produce text, but they express different dependencies.
The distinction is easiest to see in a tool-selection flow. The program can ask the model to choose between “calculator” and “search,” store that choice, and use ordinary Python control flow to decide what generation should follow. The branch is not a post-hoc interpretation of a completed transcript; it is part of the program that constructs the next request. The same mechanism supports named answers, intermediate summaries, and values that must be reused in later turns.
SGLang also allows generation parameters to travel with the operation. Maximum output length, stopping conditions, temperature, logprob requests, and some output constraints are associated with the generation node rather than scattered through global client state. That association matters because a program may contain several generations with different roles. One step may be a short classifier-like choice; another may be a long explanation; a later one may require a regular expression or JSON-shaped result. The frontend can describe those differences at the point where they affect execution.
This does not make the frontend a second model. The model still determines the generated content, and the backend still determines which capabilities are available. The frontend’s contribution is to preserve the relationships among operations so that the runtime does not have to infer them from an opaque transcript.
The executor carries the relationships forward
Behind the surface syntax, SGLang’s interpreter uses a stream executor that maintains more than a prompt buffer. It tracks the current text, chat messages, images, variables, metadata, the active role, and events used for streaming. Expressions are submitted to a worker queue. A constant string extends the current state; a generation expression calls the backend and stores the result; a variable expression waits for another value and appends it; role expressions update both text and message structure.
The queue gives the program a useful execution discipline. A frontend function can be written in ordinary Python, but operations that touch the model are submitted as expressions and executed in order by the executor. A caller can ask for a value and the executor will wait for the corresponding event rather than racing ahead with an incomplete dependency. Streaming uses the same state: as chunks arrive, the executor extends both the full text and the named variable, and exposes events that a caller can consume.
Forking adds a second dimension. When the program forks, SGLang first synchronizes the current state. Each child receives a copy of the variables, text, messages, active role information, and image state at the fork point. The children then have their own executors and can issue generation work independently. A join can gather newly created variables, or it can concatenate branch output and append it to the source state with KV-cache handling.
The important design choice is that a branch has an explicit starting point. In the example with two candidate answers, both branches inherit the same preceding context. That makes the relationship available to the backend and to the cache, even if the branches later generate different tokens. It also makes the cost visible: each branch still requires model capacity, and the runtime may or may not share all computation depending on the backend and the way the join is performed.
The following view shows the control flow without pretending that every deployment has the same network boundary.
A zero-token commit may look odd if generation is understood only as producing text. In SGLang, it is a way to tell the runtime that the current prompt state matters even before a new token is requested. The frontend’s batch path can also trace a program to identify a common prefix and pre-cache it when the prefix is sufficiently substantial. These operations connect the program representation to reusable KV state; they are not claims that a cache hit eliminates all later work.
There is a corresponding limitation. SGLang’s frontend can express parallel branches, but it cannot make an expensive branch free. Forking copies logical state and may cause parallel requests to compete for the same GPU, scheduler slots, and memory. Joining variables is also different from joining KV state. If the application only needs the text of each branch, it can gather values. If it wants to continue from the combined generated context, the backend must perform the relevant cache and state operation. The syntax exposes an opportunity; the runtime decides what it costs.
Familiar APIs and richer control surfaces
An application does not have to adopt the frontend language to use SGLang. The quickstart documents an OpenAI-compatible chat endpoint, a native /generate endpoint, streaming, and an in-process Engine. This layered interface is an adoption strategy as much as an API choice.
The OpenAI-compatible endpoint is useful when an existing application already speaks that protocol. A client can point its base URL at an SGLang server and keep much of its surrounding code unchanged. That route is deliberately familiar: messages, sampling parameters, and streamed chunks cross a conventional HTTP boundary. It also creates a clear limit. Protocol compatibility does not mean identical model behavior, support for every provider-specific parameter, or identical latency. Chat-template handling and model capabilities still depend on the served model and the runtime configuration.
The native Engine moves the boundary inward. It can accept text or token IDs, multimodal data, sampling controls, logprob requests, LoRA-related inputs, reasoning settings, session information, priorities, streaming, and cache-related fields. It can run synchronously or asynchronously, and it can submit a batch of prompts without requiring an HTTP server between the caller and the engine. That is useful when an application needs to coordinate many requests or observe richer metadata, but it also means the caller is choosing an in-process systems interface rather than only a wire protocol.
The frontend language sits one level above that native control surface. A RuntimeEndpoint connects frontend programs to a running server. When it starts, it asks the endpoint for model information and uses the reported model path to select chat-template behavior unless the caller overrides it. Generation expressions become /generate requests; typed values can be translated into regular-expression constraints; streamed server-sent events are appended back into the executor’s variables. The same adapter can send a prefix-only request when a lazy operation must be committed.
Batching is another place where the boundary matters. The frontend can run a program over many argument sets, and SGLang can issue those programs concurrently. If tracing is enabled for precaching, the common part of the program can be extracted once before the batch runs. That is different from asking the application to build one enormous prompt containing all cases. The program remains a reusable description, while the backend handles the actual requests and scheduling.
Streaming changes the shape of the result but not the underlying dependency model. The executor maintains the accumulated text and variable while making new chunks available through events or iterators. An application can therefore display a response as it arrives without surrendering the ability to refer to the final value later. The trade-off is that partial output requires careful handling of completion, errors, and metadata; a stream is a lifecycle, not simply a string returned in smaller pieces.
Choosing how much structure to expose
The three surfaces—compatible HTTP, native Engine, and frontend program—should not be ranked as universally better or worse. They expose different parts of the same serving system.
Use the compatible API when the main requirement is to replace or add a serving backend with minimal application change. It offers the broadest adoption path and keeps the request boundary simple. Use the native Engine when the application needs in-process batches, multimodal inputs, richer request fields, or direct control of asynchronous generation. Use the frontend when the application’s value lies in relationships among generations: branches, named intermediate values, role-aware conversation, conditional control flow, streaming state, or deliberate common-prefix handling.
More control also means more responsibility. A frontend program can make a branch or prefix explicit, but the resulting behavior depends on the backend, model, tokenizer, and cache identity. The documented regular-expression example is for local models; a remote or merely protocol-compatible backend may not provide the same constraint path. Similarly, a prompt that is syntactically structured can still be semantically wrong, and a branch that is logically parallel can still be bottlenecked by one GPU.
That is the broader lesson of SGLang’s frontend. It does not optimize by guessing what the application meant after the fact. It gives the application a language for declaring enough intent that the runtime can preserve dependencies, schedule work, and recognize possible reuse. Once that intent reaches the serving runtime, another question takes over: how are requests admitted into prefill and decode steps when they have different lengths, phases, and memory demands? The next discussion follows the execution plan into that scheduler.