Quantization: Making a Model Fit Without Losing the Task
In llama.cpp, quantization is the decision about how much numerical precision a model should keep after it has become a usable GGUF artifact. This chapter explains what that decision buys and risks, how to read the project’s measurements without inventing a universal winner, and how to choose precision for the task rather than for the smallest file.
Smaller weights change more than storage
Model weights are numerical values used repeatedly during inference. In a high-precision representation, those values take more bits; in a quantized representation, they are stored using fewer bits and reconstructed or operated on with an approximation. llama.cpp describes quantization as reducing the precision of model weights—for example, moving from 32-bit floating-point values toward 4-bit integers—to shrink the model and potentially speed inference.
The first benefit is easier to see on disk. Fewer bits per weight generally mean a smaller artifact, which can reduce the storage needed to keep a model and the memory needed to load its weights. A smaller weight representation can also reduce the amount of data moved while the model runs. These effects are especially important for local inference because the boundary is often physical: a model either fits in available RAM or VRAM, or it must be partly offloaded, moved through a slower path, or not used at the desired context size.
The second benefit is speed, but it is conditional. Quantized values can be handled by kernels designed for particular formats, and moving less data can help. Yet speed depends on the operation, batch shape, processor, backend, and implementation. A lower-bit format is not automatically faster for every phase of inference. Prompt processing and one-token generation already have different computational shapes, so a quantization can look favorable in one measurement and less favorable in another.
The cost is approximation. The project notes that quantization may introduce accuracy loss, commonly evaluated with perplexity or Kullback–Leibler divergence. Those metrics are useful signals, but they do not answer every application question. A small change in next-token probabilities may be acceptable for casual conversation and harmful for code generation, extraction, structured output, or tool calling. “Quality” is therefore not one number detached from use. It is the behavior the application must preserve after the model has been made smaller.
This creates a three-way decision rather than a single optimization target. More precision usually spends more storage and memory in exchange for a closer numerical representation. Less precision can make a larger model available on a constrained machine, but may change its outputs and may not deliver the same speed across all workloads. The right choice depends on which constraint is binding: capacity, latency, throughput, or task fidelity.
Reading a benchmark without choosing a false winner
The quantization documentation includes a table for Llama 3.1 8B that illustrates the trade-off. It lists bits per weight, model size, prompt-processing speed at a batch of 512, and text-generation speed at a batch of 128 across many formats. The listed F16 entry is 14.96 GiB, while Q4_K_M is 4.58 GiB and IQ1_S is 1.87 GiB. Those figures make the capacity difference concrete: the smallest entry is about one eighth of the listed F16 size.
But the table does not say that IQ1_S is “best,” or that Q4_K_M is always the right compromise. It measures a particular model and set of conditions. Prompt processing and text generation are reported separately, and their rates do not move in a simple line with bits per weight. The F16 row has the highest listed prompt-processing rate in that table, while some lower-bit rows show higher listed generation rates. That is not a contradiction. It is evidence that a runtime’s work phase and kernels matter, and that the metric must match the experience being optimized.
The table also contains a more important lesson about comparison. A model that fits in memory may still be too slow for an interactive application. A model that generates tokens quickly may lose quality on a required task. A compact artifact may leave more memory available for context or concurrent users, improving the service-level result even if its isolated generation rate is not the highest. The denominator is always the intended workload: prompt length, response length, concurrency, context size, backend, and quality requirement. It is useful to separate at least two common scenarios. An interactive assistant may care most about the delay before the first response token and the quality of a long conversation. A batch job may care more about prompt-processing throughput. A shared server may accept a small change in isolated generation speed if the saved memory allows more active sequences. The same quantized artifact can therefore be a good choice for one workload and a poor choice for another without either result being surprising.
This also creates a staged option between “use the largest format” and “choose the smallest format.” A high-quality GGUF can serve as a reference for task outputs, while one or more quantized variants are tested for fit and speed. If the compact variant fails a representative extraction or tool-calling test, the response is not necessarily to abandon quantization altogether. The team can try a less aggressive format, preserve particular tensors at higher precision, use an importance matrix, or keep a more conservative KV-cache type. These choices spend resources selectively instead of treating precision as an all-or-nothing switch.
The repository’s measurements cannot decide among these paths in advance. They can show what changed under their recorded conditions, while a deployment test shows whether that change matters for the application. A disciplined comparison records the artifact, runtime settings, prompt shape, and result quality together. Without that context, a tokens-per-second number can encourage the wrong optimization: improving a phase users barely notice, or saving memory in a component that carries a quality-critical representation.
The documentation’s separate memory-and-disk examples show why this decision matters at larger scale. For Llama 3.1, it lists an 8B original size of 32.1 GB and a Q4_K_M size of 4.9 GB, then gives much larger original and Q4_K_M examples for 70B and 405B models. These are planning illustrations, not a promise about every model or file layout. They show the difference between a model family’s high-precision and quantized storage burden, while the detailed rate table shows why capacity alone cannot settle the choice.
There is also a cost before inference begins. The quantization tool currently loads models fully into memory, so the process needs sufficient RAM and disk for intermediate files. A model can be easy to run after quantization and still be difficult to quantize on the machine where the conversion is attempted. That distinction matters when a team assumes that reducing a file’s final size also reduces the resources needed to create it.
Spend precision where behavior depends on it
A uniform quantization applies one general policy across the model, but llama.cpp’s tool supports more selective choices. It can use an importance matrix, keep some tensors at different types, choose an output-tensor or token-embedding type, preserve shards, and apply tensor-specific overrides. These options reflect a simple engineering idea: not every part of a model contributes equally to the behavior the application cares about.
An importance matrix supplies information about how sensitive parts of the model are to quantization. The tool can use it for selected tensors or exclude selected tensors from the calibration scope. Mixed choices can then reserve more precision for parts that need it and use more compact formats elsewhere. This does not guarantee a quality outcome; it gives the quantization process a better-informed way to spend the available precision budget.
Selective precision also gives readers a safer way to think about “the model.” The output layer and token embeddings can be assigned separately. Layers can be pruned in advanced workflows. Already-quantized tensors can be requantized, but the documentation warns that doing so can severely reduce quality compared with quantizing from a 16-bit or 32-bit source. In other words, a smaller file is not always a neutral intermediate from which every later choice can be made. A high-quality source artifact remains valuable when producing and comparing variants.
The same principle applies to multimodal components, but with a different reason. A vision or audio-capable model commonly pairs the language-model GGUF with a projector or encoder. That component prepares the media representation delivered to the language model. The repository notes that these components are usually much smaller than the language model and that their output has a direct effect on generation quality. It therefore recommends formats such as BF16 or Q8 for them in many workflows. Aggressively shrinking a small component may save little memory while degrading the input representation on which the larger model depends.
This is a case where the right precision is determined by position in the information flow, not by a universal rule. The main language model may need aggressive quantization to fit a device. The projector may deserve higher precision because it sits before generation and its errors can change what the language model sees. Both files are part of one application, but they do not have the same role or the same useful trade-off.
Weight precision is not the whole memory decision
The weights are the most visible target for quantization, but llama.cpp also exposes data types for the key-value cache, or KV cache. The cache stores the state needed to continue processing a sequence. Its size grows with context and concurrency, so reducing its precision can be attractive when the model weights already fit but the active conversations do not.
That choice should not be folded into the weight-quantization decision as if the effects were identical. The project’s function-calling documentation warns that extreme KV quantizations can substantially degrade tool-calling performance. A deployment may therefore accept a lower-precision weight artifact while keeping a less aggressive cache type for an application whose protocol depends on reliable tool calls. The useful budget is not “the lowest number of bits everywhere.” It is a distribution of precision across the parts whose errors have different consequences.
The same reasoning applies to structured output. A JSON schema or grammar can restrict the shape of a response, but it cannot restore information lost through an unsuitable representation or guarantee that the content is correct. If an application requires valid structured records, the validation plan should check both the format and the task-level fields. If it requires tool use, it should test actual tool calls with the intended chat template and cache settings rather than relying on ordinary prose quality as a proxy.
A staged evaluation is more informative than one score. First establish that the candidate artifact loads and uses the intended model and optional components. Then compare task outputs against a higher-quality baseline on representative prompts. Measure prompt processing and generation separately, because users experience them differently. Finally, measure the resource behavior that motivated quantization: peak RAM or VRAM, context capacity, startup and load time, and throughput under the intended concurrency. The repository provides ingredients for these checks, but it does not supply a universal acceptance threshold.
The smallest file is not the decision
Quantization succeeds when it changes the resource equation without changing the task beyond what the application can tolerate. For a memory-bound experiment, a very compact format may be the difference between running a model and not running it. For a tool-enabled service, a slightly larger weight or cache representation may be worth the more reliable protocol behavior. For a multimodal workflow, preserving projector quality may matter more than saving a small amount of memory.
The most defensible selection process begins with the binding constraint. If the artifact cannot fit, compare formats or placement strategies. If it fits but is too slow, measure the relevant phase and backend rather than assuming fewer bits will solve it. If it is fast enough but fails the task, restore precision selectively, use calibration, or choose another model variant. If it passes single-user tests but fails under concurrency, budget cache and context state separately from the weights.
In llama.cpp, quantization is thus a bridge between model quality and machine reality. It makes deployment options possible, but it also moves part of model behavior into an engineering decision. Once that decision is made, the next question is where the resulting artifact should execute: which backends can cover its operations, how devices share the work, and why “supports GPUs” is not the same as “performs identically on every GPU.”