Memory, Batching, and the Speed of the Next Token

llama.cpp’s observed performance is shaped by how it stores state and schedules work, not only by the backend or model file. This chapter separates prompt processing from one-token generation, follows the effects of context and concurrency on memory, and explains when prompt caching, continuous batching, and speculative decoding can improve a real workload. The goal is a diagnosis method: identify the resource or phase that limits the run before changing a setting.

Prompt processing and generation are different workloads

A request begins with a prompt, but the runtime does not handle every token in the same way. Prompt processing can evaluate many input tokens in a batch. Autoregressive generation then produces the response one token at a time, because each newly selected token becomes part of the state used to predict the next one. The two phases use different computational shapes and therefore respond differently to batch size, thread count, memory placement, and backend kernels.

llama.cpp exposes that distinction directly. Its context parameters include a logical maximum batch size and a physical maximum ubatch size. The logical batch is the amount of work that can be submitted through the decoding interface; the physical ubatch is the amount processed as one smaller execution unit. The API also has separate thread settings for generation and for prompt or batch processing. These are not duplicate knobs. A machine can have enough parallel work to benefit from more threads while ingesting a long prompt and still become slower when too many threads compete during single-token generation.

This is why a single tokens-per-second number is a weak diagnosis. Prompt throughput answers how quickly the runtime absorbs input. Generation throughput answers how quickly it produces the continuing response. An interactive user may notice the delay before the first token and then the spacing between later tokens. A batch workload may care almost entirely about prompt processing. A server may need both numbers under several active sequences. The phase that looks strongest in isolation may not be the phase that controls the user’s experience.

The context carries the state that connects the phases. llama.cpp’s public API distinguishes the loaded model from a context, and the context can manage multiple sequences. The model weights remain the long-lived computational material; the context holds the active work needed to continue conversations or other sequences. As the context grows, the runtime retains more sequence state, including the key-value cache used by attention-based models. That cache is what lets later tokens reuse information from earlier ones instead of rebuilding the entire history each time.

Context size is therefore a resource promise with a cost. A larger limit permits longer prompts and responses, but it reserves or consumes more memory depending on the model and configuration. Multiple sequences multiply the active state in a server or parallel CLI run. The server’s parallel slots represent concurrent processing capacity, while unified KV settings determine how the cache is organized across those sequences. A model can fit comfortably for one conversation and fail to fit at the same context or concurrency when several conversations are active.

The result is a useful separation of memory questions.