Overview

Code Llama’s base, Python, and Instruct branches encode different training goals, so no variant is universally best. The paper’s 70B results illustrate the condition: HumanEval pass@1 is 53.0% for Code Llama, 67.8% for Instruct, and 57.3% for Python, while MBPP is 62.4%, 62.2%, and 65.6%, respectively. These values use greedy pass@1, with HumanEval zero-shot and MBPP three-shot; pass@10 and pass@100 use nucleus sampling. Model selection therefore starts with the interaction and success test, not a leaderboard alone. Benchmark overlap and task scope also limit transfer to unfamiliar repositories.

Learning outcomes

Base, Python, and Instruct are branches with different objectives, not rungs on a universal quality ladder.
For 70B, Instruct leads reported HumanEval pass@1 while Python leads MBPP, showing that rankings change with benchmark and branch.
Pass@k is part of the product loop: pass@1 uses greedy decoding, while pass@10 and pass@100 rely on sampled candidates, so the scores are not interchangeable.

Keywords

Code LlamavariantsInstructPythonbenchmarkspass@k

Which Code Llama Are You Looking At? Variants, Benchmarks, and Conditional Wins

Code Llama’s model names encode different training histories, not just different packaging. This chapter compares the base, Python, and Instruct branches and then attaches the paper’s benchmark numbers to their actual protocols: model size, task, language, decoding, and amount of prompting. The practical value is conditional model selection. A reader should be able to ask which behavior a workload needs before asking which Code Llama score is highest.

The comparison begins with a warning about the word “best.” A code assistant may need a standalone function, an insertion inside an existing file, a response to a natural-language request, or help across several programming languages. The model that is strongest at one of these jobs may not be strongest at another. Even when two branches produce code, their training objectives can make them sensitive to different inputs and trade-offs.

Variants encode different objectives, not a quality ladder

The base Code Llama branch is the general code foundation. It is intended for code-generation tasks and is the branch from which the other principal specializations are easiest to understand. Some base sizes receive fill-in-the-middle training, and the family also includes long-context adaptation, but support is not identical across every size and variant. The label “base” therefore does not mean “missing all capabilities”; it means that the model has not received the same Python-focused or instruction-focused intervention as the other branches.

Code Llama - Python adds a further Python-heavy training stage after the general code stage. Its purpose is not to make every kind of coding better in equal measure. It tests whether denser exposure to one programming language improves selected Python tasks. The reported results show that it can: the 7B Python branch is able to outperform Code Llama 13B on HumanEval and MBPP in the paper’s comparisons. But the same branch is slightly weaker on some introductory and interview-level APPS problems while improving more clearly on competition-level problems. A Python label is therefore a statement about training focus, not a guarantee of universal superiority.

Code Llama - Instruct changes the interaction as well as the code distribution. It is based on Code Llama and receives additional instruction tuning with helpfulness and safety data, including a self-instruct coding set. The branch is intended for questions and code-assistant interactions, so its value includes following a requested format, interpreting a user’s wording, and responding more safely. Those goals can alter raw coding scores. An Instruct model may be preferable when the system must interact with people, even if a base model is a better fit for a narrow offline generation benchmark.

The 70B variants make the warning about training history especially important. The paper’s 70B base model was trained on 1 trillion initial code-training tokens rather than 500 billion, and the later Python and Instruct paths have their own ordering. Code Llama - Instruct 70B was trained from Code Llama - Python 70B, which the authors report as stronger on average than the base 70B model across the evaluated multilingual languages. A comparison among 70B labels can therefore combine parameter count, data volume, language specialization, and instruction tuning. It is not a clean test of one variable.

Published by DeepMox

FAQ

Why do Code Llama rankings change between HumanEval and MBPP?

They change because the branches and benchmarks emphasize different behavior and use different prompt conditions: 70B Instruct leads HumanEval pass@1 at 67.8%, while Python leads MBPP at 65.6%; HumanEval is zero-shot and MBPP three-shot.

What do pass@1, pass@10, and pass@100 mean for Code Llama?

Pass@1 measures one selected generation, while pass@10 and pass@100 measure finding a passing candidate among multiple samples. In the paper, pass@1 uses greedy decoding; pass@10 and pass@100 use nucleus sampling with top-p 0.95 and temperature 0.8.

Do Code Llama benchmark scores transfer to unfamiliar repositories?

The paper does not establish reliable transfer of Code Llama benchmark scores to unfamiliar repositories. Some evaluations overlap training data, and benchmark tasks omit the missing requirements, dependencies, and consequences of production work.

Previous8 minCode Llama: Long Context Trade-offs

Next9 minCode Llama - Instruct: Safety and Deployment

Code Llama Variants and Benchmarks

Which Code Llama Are You Looking At? Variants, Benchmarks, and Conditional Wins

Variants encode different objectives, not a quality ladder

FAQ