The Editor Problem: How Fill-in-the-Middle Works
Code Llama’s fill-in-the-middle capability addresses a specific editor problem: the code a developer needs belongs between a prefix and a suffix that already exist. This chapter explains why ordinary left-to-right continuation is not enough, how Code Llama trains an autoregressive model to predict the missing middle, and why prompt format and token boundaries affect the result. The goal is a practical judgment about when infilling is useful—and why a well-placed completion is still not the same as correct software.
Imagine a cursor inside a function. Above it are an argument check and a variable assignment; below it is a return statement that expects a particular value. A model that only predicts what would naturally follow the text above may continue in a plausible direction, but it has no direct requirement to make the insertion fit the text below. The editor needs a candidate that satisfies both sides of the gap.
That is the distinction between continuation and infilling. Continuation treats the visible prefix as the principal context and predicts the next tokens. Infilling treats the missing region as the object to recover from surrounding context. In a source editor, the difference is not cosmetic. It changes which information is available at the moment the model makes its prediction.
The suffix is useful because code is constrained in both directions. A closing parenthesis can reveal which arguments the missing call must produce. A later return statement can reveal the type or shape expected from an earlier expression. A following branch can rule out a completion that would otherwise look natural after the prefix. These clues do not make the answer logically determined, but they narrow the space of plausible insertions. The point of infilling is to let the model condition on those clues instead of asking it to guess the future from the past alone.
That extra context can also create new failure modes. If the suffix is unrelated, stale, or itself incomplete, the model may be guided toward a locally consistent but globally wrong insertion. If the gap is large, the task becomes more than a short cursor completion. And if the visible text is assembled in a format the model did not encounter during training, the model may spend capacity interpreting the wrapper rather than the code. The capability is valuable because it changes the available evidence, not because it removes ambiguity.
A missing middle becomes a trainable prediction task
Code Llama defines infilling as predicting a missing part of a program from its surrounding context. The intended uses include completion at the cursor, type inference, and generating documentation such as a docstring. The task is still implemented with an autoregressive language model: the model ultimately predicts a sequence from left to right. The training transformation changes what sequence it is asked to predict and how the visible context is arranged.
For a training document, the developers choose a prefix, a middle, and a suffix. The middle is the target that will be predicted. Instead of presenting the document only in its original order, the training example moves the target span to the end after marked context, so the model can learn to emit the missing region once it has read the surrounding pieces. This preserves the familiar next-token machinery while teaching it a new relationship among the parts of a file.
The boundaries must be explicit. Code Llama extends the tokenizer with special markers for the start and end of the relevant spans. These markers tell the model where the prefix ends, where the suffix begins, and where the predicted middle belongs. Without such signals, the same stream of tokens could be interpreted as an ordinary continuation, a suffix, or part of the answer. The markers are therefore part of the interface between the editor and the model, not incidental formatting.
The transformation is applied with probability 0.9 to training documents that meet the authors’ conditions. That high rate makes infilling a central training behavior rather than a rare augmentation, while still leaving ordinary autoregressive examples in the mixture. The released family also reflects a product decision: the paper trains the general 7B, 13B, and 70B models with an infilling objective, while the exact feature set is not identical across every size and branch. A wrapper should therefore identify the actual model variant instead of assuming that every model carrying the Code Llama name accepts the same infilling request.
The training data mixes two arrangements. In prefix-suffix-middle, or PSM, the prefix and suffix are presented in that order before the missing middle is generated. In suffix-prefix-middle, or SPM, the suffix comes before the prefix in the arranged context and the middle is still the target. Training on both arrangements gives the model more than one way to receive the same logical pieces. It also creates a condition that becomes important at the boundary between text and tokens.
The editor owns the file and decides what prefix and suffix to send. Code Llama owns only the prediction of the missing span; it does not automatically own the surrounding program’s state, dependencies, or test environment. That division matters when the completion is inserted. The model may produce a syntactically plausible middle, but the editor or a downstream tool must still determine whether the insertion fits the actual file.
The character-level split used to create training examples is also significant. A missing span can begin or end in the middle of what the tokenizer considers a subword unit. Human readers see characters; the model consumes tokens. If the training arrangement consistently avoids a kind of split that appears in an inference prompt, the model may understand the logical code situation while still struggling with the exact token sequence needed to spell the answer.
This is why infilling is not merely “give the model more context.” It changes the target, the ordering, and the boundary markers together. The desired behavior emerges only when the model has learned the relationship among those choices. A prompt that preserves the intended prefix-suffix relationship but uses the wrong markers or ordering may no longer look like the task the model was trained to solve.
The mechanism also explains why the feature has practical value beyond code generation benchmarks. An IDE can request a short insertion without asking the model to regenerate an entire file. A documentation tool can place a docstring between a function header and its body. A type-inference or code-assistance system can use the visible material on both sides of the cursor. These workflows reduce the amount of text that must be rewritten, but they do not remove the need to inspect the insertion’s effect on the program around it.
PSM, SPM, and the cost of crossing a token boundary
A token is the unit the language model predicts, and it is often smaller than a word but not identical to a character. That mismatch is usually invisible when a prompt ends between words. It becomes visible when a prefix ends inside a token. For example, the characters at the cursor may form only part of a word or identifier from the tokenizer’s perspective. The model then has to coordinate a character-level edit with a token-level vocabulary.
The paper reports a format-specific asymmetry. SPM training avoids breaking up tokens at the prefix-middle boundary, while PSM training exposes the model to such breaks. As a result, a random-span infilling prompt that ends inside a token can be out of distribution for SPM. The model may infer the intended code but produce the wrong continuation of the partial token. The paper gives the example of completing a fragment such as “enu” with “emrate” rather than the character sequence that would form “enumerate”: the logical direction is plausible, but token-to-character reconstruction fails.
The practical recommendation follows from that mechanism. When the prefix does not end in whitespace or a token boundary, PSM is the safer format in the paper’s evaluation. SPM can be useful in other arrangements, but the authors advise using it with token healing when the boundary requires it. Token healing is a way of correcting or reconsidering the tokenization at the edge before generation continues. It addresses the specific interface mismatch; it is not a guarantee that every arbitrary infilling prompt will be handled correctly.
The asymmetry is easy to miss because both formats can succeed on clean examples. Suppose the cursor falls after a space and the missing code begins with a complete identifier. The model’s token sequence is likely to align with what it saw during training. Move the cursor into the identifier, however, and the visible prefix may end with only a fragment of a token. A format that avoided such fragments during training has less direct experience with the exact boundary, even if the surrounding programming logic is familiar. The resulting error is not necessarily ignorance of the algorithm; it can be a failure to bridge the representation used by the editor and the representation used by the model.
That distinction matters for diagnosing an integration. If completions fail mainly at the middle of identifiers, changing the model size or adding more suffix text may not address the cause. The first question is whether the wrapper preserves the expected markers and uses a format appropriate to the boundary. Conversely, if a completion is well-tokenized but semantically wrong, a format change may do little. The same visible symptom—“the model produced bad code”—can arise from an interface failure or from a generation failure, and the two require different remedies.
The PSM/SPM distinction therefore belongs in system design, not just in a benchmark appendix. An editor integration may usually place the cursor after whitespace, or it may often place it in the middle of an identifier. Those are different input conditions. A wrapper that always chooses one format without inspecting the boundary can turn a model capability into a recurring, hard-to-diagnose failure. The model may appear inconsistent when the real inconsistency is in how the prompt crosses from visible text to the predicted span.
The infilling ablation shows the cost of learning this additional behavior. For 7B, adding the infilling objective reduces the reported average across the HumanEval and MBPP pass@1, pass@10, and pass@100 measures by 0.6 percentage points. For 13B, the reported average reduction is 1.1 points. The authors find no meaningful cost on autoregressive test loss, but the downstream code-generation metrics show a small cost, especially as the number of sampled candidates increases.
The word “average” should remain attached to these numbers. Individual benchmark cells move in different directions, and the cost is measured before the later long-context stage. The result is not that infilling makes a model broadly worse. It is that a multitask objective can slightly change the distribution of ordinary completions while adding a capability that ordinary next-token training does not provide. Looking only at the average hides the engineering question; looking only at the feature hides the opportunity cost.
That trade-off is easier to understand when the two workloads are placed side by side. A short benchmark asks for a completed function from a description. An editor asks for a local insertion that respects code already present on both sides. The first workload rewards the probability of a good standalone continuation; the second values the ability to condition on a gap. Optimizing for both can move the model slightly away from the best score on the first without making the second possible through ordinary continuation alone.
The paper’s infilling evaluations report strong performance against the compared open infilling models of similar size, including in multilingual settings. But the scope of that statement is bounded. The results depend on the benchmark’s masking method, the PSM or SPM format, the decoding setup, and the matching or test criterion. A single-line exact match demonstrates that the emitted span matches the reference under that task. It does not show that the model understood the whole program or that a longer insertion preserves the program’s intended behavior.
This distinction is especially important for documentation. A docstring can be grammatically polished and structurally appropriate while describing the wrong edge case. A completion can fit its immediate neighbors while violating an invariant established elsewhere in the file. Infilling uses surrounding text more effectively than ordinary continuation for the task it was trained to perform, but “uses both sides” is not equivalent to “verifies the program.” The missing middle remains a generated hypothesis.
The authors release infilling capability in selected Code Llama sizes because they judge the measured cost acceptable relative to the editor use cases it enables. That decision is not a claim that infilling dominates ordinary generation. It is a workload decision: if a system needs cursor-level completion or contextual documentation, a small decline on short left-to-right metrics may be worth paying. If the system only generates standalone functions and never fills gaps, the same trade may have less value.
The broader lesson is consistent with Code Llama’s specialization story. A capability becomes dependable only when the training objective, the input format, the evaluation, and the application boundary line up. Fill-in-the-middle changes the prediction problem in a way that makes editor workflows possible, and PSM/SPM behavior shows exactly where the interface can fail. The next question is what happens when the surrounding context is not just a few lines but an entire large file: whether Code Llama can make distant code useful, and what it sacrifices to do so.