Overview

Code Llama’s long-context capability combines 16,384-token fine-tuning with a RoPE base-period change from 10,000 to 1,000,000, rather than merely raising an inference limit. The paper reports perplexity continuing to improve beyond the adapted range and better full-file completion with LCFT, including a roughly 103,000-token example where all tested Code Llama models were syntactically correct and 7B was an exact match. However, synthetic key retrieval declines beyond 16,000 tokens and varies by position. Long-context tuning also slightly reduces short-input performance, so the results support useful large-file behavior under tested conditions, not reliable understanding of arbitrary repositories.

Learning outcomes

Long-context behavior depends on both 16,384-token fine-tuning and the RoPE base-period change; changing either alone failed in the reported ablations.
Perplexity and full-file completion improve under tested conditions, but key retrieval declines beyond 16,000 tokens and depends on position.
Evidence up to 100,000 tokens comes from particular tests; LCFT can sacrifice short-input performance, so workload fit matters more than the maximum.

Keywords

Code Llamalong contextLCFTRoPEretrievalperplexity

When the File Does Not Fit: The Promise and Price of Long Context

For Code Llama, long context is an attempt to solve a distance problem: useful code may be far away from the line the model must complete. This chapter examines why a longer input window requires dedicated fine-tuning and a change to the model’s positional representation, then reads the resulting evidence across perplexity, key retrieval, and full-file completion. The practical judgment is deliberately narrower than “the model understands long documents”: Code Llama can use more context under some tested conditions, but the benefit depends on the task, the position of the information, and the cost paid on shorter inputs.

A short completion can often rely on the few lines immediately around a cursor. A large source file is different. An import may determine which API is legal. A class definition may establish an invariant. A helper near the beginning of the file may explain a variable used thousands of tokens later. If those clues cannot enter the model’s context, no amount of clever next-token prediction can use them. If they can enter but the model cannot retrieve them or relate them to the current line, the larger window is mostly capacity on paper.

Why a longer window needs a different positional map

Llama 2 and Code Llama’s initial code-training stages use sequences of 4,096 tokens. Code Llama adds a dedicated long-context fine-tuning stage, or LCFT, using sequences of 16,384 tokens. This is more than changing an advertised maximum at inference time. The model has to receive training examples in the longer range so that its prediction behavior can adapt to information arriving at new distances.

The authors also change how the model represents position. Their transformer uses rotary position embeddings, commonly called RoPE. In broad terms, RoPE rotates parts of a token’s internal query and key representations according to the token’s position. The rate of that rotation determines how positional relationships appear to the attention mechanism. Code Llama changes the base period used to derive those rotation frequencies from 10,000 to 1,000,000 during long-context fine-tuning.

The purpose is not to make positions disappear. It is to reduce the model’s bias toward short-distance attention and make far-away tokens contribute more plausibly to a prediction. A model trained with one positional scale can behave unstably when it is suddenly shown much longer sequences. Changing the base period gives the model a different map of distance, while LCFT supplies examples that let it adjust to that map.

That pairing is important. The paper’s ablations report that changing the RoPE base period at inference time, without long-context fine-tuning, does not make far-away key retrieval work. Fine-tuning without changing the base period also fails to retrieve distant keys at shorter-than-expected lengths. In the authors’ earlier 7B experiments, both parts were needed for the long-context behavior they sought.

Published by DeepMox

FAQ

What changed to give Code Llama long-context behavior?

Code Llama combines long-context fine-tuning on 16,384-token sequences with a RoPE base-period change from 10,000 to 1,000,000. The paper’s ablations indicate that either change alone was insufficient for the sought retrieval behavior.

How does Code Llama’s long context perform beyond 16,000 tokens?

Beyond 16,000 tokens, Code Llama’s synthetic key-retrieval accuracy declines and varies with key position, even though perplexity continues to improve beyond the fine-tuning length. The paper reports only particular tested conditions, not universal success.

Does long-context fine-tuning improve every Code Llama task?

No. LCFT improves reported full-file completion, but the paper also reports a small decline on standard short-sequence coding benchmarks and up to two BLEU points lost for some sub-4,000-token groups. The benefit depends on workload shape.

Previous9 minCode Llama: Fill-in-the-Middle

Next9 minCode Llama Variants and Benchmarks

Code Llama: Long Context Trade-offs

When the File Does Not Fit: The Promise and Price of Long Context

Why a longer window needs a different positional map

FAQ