Overview

This chapter explains why Transformers generation is an iterative loop rather than one ordinary forward pass. A language-modeling head supplies next-token scores, generation configuration selects how to decode them, and the key-value cache reuses past attention state. It covers stopping, prompt prefill, batching, cache variants, and the inference-only boundary, showing why speed, memory, and output changes must be evaluated separately.

Learning outcomes

Generation requires a language-modeling head; weights supply scores, while configuration determines how those scores become sequences.
The KV cache reduces repeated key/value projections during decoding, trading recomputation for memory that grows with retained context.
Cache choices are inference optimizations, not training shortcuts, and their value depends on request length, model support, hardware, and runtime.

Keywords

GenerationDecodingCacheAttentionSamplingInference

Generation: Decoding, State, and the KV Cache

A text-generation request looks like one function call, but the model usually performs a sequence of forward passes, each producing information needed for the next token. This chapter follows that iterative path in Transformers and separates three things that are often blended together: the model capability supplied by a language-modeling head, the decoding policy supplied by generation configuration, and the runtime state supplied by the key-value cache. The practical goal is to understand why generation behaves differently from ordinary pipeline inference and when a cache improves speed without becoming a training shortcut.

A bare backbone can return useful representations without being able to generate a sequence. Generation requires a model head that produces scores for possible next tokens and the surrounding configuration that defines how those scores become output IDs. The checkpoint supplies learned behavior; the generation settings choose how to use that behavior.

A token generator is an iterative model call

At the first step, the tokenizer turns a prompt into the model’s input IDs and attention mask. A generative model processes that context and returns scores for the next token. A decoding rule selects one token from those scores: it may choose the highest-scoring option, sample according to a distribution, search among several candidate sequences, or apply constraints. The selected token is appended to the sequence and becomes part of the context for the next step.

The loop ends when a stopping condition is reached, often an end-of-sequence token or a configured length limit. The final token IDs are then decoded by the tokenizer into text or another task output. The important point is that the model does not write the entire continuation in one undifferentiated operation. It repeatedly updates a growing sequence and a set of intermediate states.

flowchart LR A["Prompt inputs"] --> B["Model forward"] B --> C["Past key/value cache"] B --> D["Next-token scores"] D --> E["Next-token selection"] E --> F["Append token"] F --> G{"Stop condition?"} G -- "No" --> B G -- "Yes" --> H["Decode token IDs"]

Transformers exposes this behavior through GenerationMixin.generate(), which is intended for models with a language-modeling head. The method accepts a generation configuration and can override its settings with arguments supplied for a particular call. The configuration can come from a serialized generation file when one exists, from model configuration-derived defaults, or from the library’s defaults. Call-level values take precedence over the base configuration, which gives an application a way to change decoding without rewriting the learned model.

That precedence is useful only if the layers remain distinct. The model’s weights determine the scores it tends to assign. The tokenizer determines how prompts and generated IDs are represented. The generation configuration determines choices such as length, stopping, sampling, beam behavior, or output collection. A change from greedy selection to sampling can change the text without changing the checkpoint. A change to the prompt template can change the input distribution before decoding starts. A faster cache can reduce repeated computation without improving the model’s underlying judgment.

The separation also explains why generation quality cannot be attributed to one setting.

Published by DeepMox

FAQ

What does the KV cache do during generation?

The KV cache stores past key and value tensors so later decode steps can reuse earlier attention projections instead of recomputing them. Its memory use grows with retained context and model dimensions.

Can changing generation settings change output without changing model weights?

Yes. Changing sampling, beam behavior, length, stopping criteria, or other generation settings can change output sequences without changing the checkpoint’s learned weights.

Why should the KV cache not be used as a training shortcut?

The KV cache should not be carried into training because training has different gradient and dependency requirements from fixed-weight autoregressive inference. Reusing inference state can create incorrect dependencies or errors.

Previous10 minHugging Face Transformers Pipelines

Next8 minHugging Face Transformers Trainer Fine-Tuning

Hugging Face Transformers Text Generation

Generation: Decoding, State, and the KV Cache

A token generator is an iterative model call

FAQ