Memory Is More Than the Model File
This chapter follows how llmfit turns a model configuration into a memory requirement. The focus is not a model’s advertised size by itself, but the interaction among weight representation, context length, KV cache, runtime overhead, and the memory pool available on a particular machine. Understanding that interaction lets a reader predict what will change when they choose a different quantization or context setting.
A model file is the most visible part of local inference, so it is tempting to treat its size as the whole requirement. In practice, the file must share memory with the state created while the model reads a prompt and generates tokens. A configuration that fits its weights may become unusable at a long context. A lower-precision version may fit comfortably but trade away some quality. llmfit’s memory analysis exists to make those trade-offs explicit before a user starts a download or a server.
The weight footprint is only the first term
The first term is the model’s weights. A parameter is a learned value, and the parameter count gives a rough measure of how much learned data must be resident. The number of bytes used for each parameter depends on the representation. Full-precision weights need more memory than compressed weights, and different quantization formats make different quality and speed trade-offs.
Quantization means storing weights with fewer bits than a full-precision representation. In llmfit, the relevant consequence is straightforward: fewer bytes per parameter usually reduce the weight footprint, allowing a larger model or more headroom on the same memory pool. But quantization is not a free compression switch. The project assigns quality penalties and speed multipliers to known formats, so the memory-saving choice is part of the final ranking rather than an invisible implementation detail.
For GGUF-oriented analysis, llmfit searches a hierarchy from Q8_0 through Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, and Q2_K. The order runs from higher expected quality to greater compression. It tries the best quality that fits the selected budget. If the highest-quality option does not fit, it moves down the hierarchy instead of treating the model as simply unavailable. That behavior turns a yes-or-no memory question into a configuration question: which representation preserves enough quality while staying inside the pool?
The same search cannot be applied indiscriminately to every model. MLX-native entries use an MLX-specific hierarchy, ONNX entries use their own supported options, and AWQ, GPTQ, or AutoRound entries are pre-quantized. A pre-quantized model arrives with a fixed weight format; llmfit cannot pretend that it can turn an AWQ model into a sequence of GGUF choices merely because the numerical footprint would be convenient. Native low-precision formats likewise should not be relabeled with a GGUF quantization that the repository does not actually provide.
This is the first important boundary between a model record and a memory calculation. The model record identifies what representation the artifact uses. The fit engine can search among alternatives only when the runtime and format support those alternatives. If a fixed-format model fits, it should be recognized as fitting at its own footprint. If it does not fit, a lower entry in a generic hierarchy cannot rescue it unless the model genuinely has that alternate representation.
The weight footprint is still an estimate. llmfit expresses known quantization levels through bytes-per-parameter values and multiplies them by the model’s parameter count. That is useful for comparing configurations consistently, but it does not reproduce every tensor’s exact storage layout, metadata, allocator behavior, or file packaging overhead. The calculation is a common model for ranking candidates, not a promise that the resulting file will have exactly the displayed size.
The next term is state that grows while the model is used. A model does not only hold its weights; it also retains information about the prompt and generated sequence so that later tokens can attend to earlier context. That state is commonly called the key-value cache, or KV cache. Its effect is why a single “model needs X GB” statement is incomplete even when the weight format is known.
Context turns a model into a live memory problem
The KV cache stores intermediate key and value data for the attention layers. As the context becomes longer, more token positions must be represented. In a model with complete attention layers, the cache therefore grows with context length. Its size also depends on the number of layers, the number of key-value heads, the head dimension, and the number of bytes used for each cache element.
When llmfit has the relevant architectural metadata, it uses the precise relationship: two cache streams, multiplied by the number of layers, KV heads, head dimension, context tokens, and bytes per element. The factor of two reflects keys and values. A model with grouped-query attention can have fewer KV heads than attention heads, changing the cache slope. Two models with similar parameter counts can therefore have different context costs because their attention geometry differs.
The project also recognizes hybrid architectures. Some recent models mix full attention with linear or state-space layers. Only the full-attention portion carries a context-scaled KV cache; the recurrent portion maintains a fixed-size state instead. When the model record describes that layout, llmfit charges the growing part to the full-attention layers and treats the fixed recurrent state as part of runtime overhead. This avoids charging every layer as though it retained a full attention history.
Older or incomplete records may not include enough information for the precise formula. Rather than refusing to analyze every such model, llmfit falls back to a coarse linear approximation based on model parameters and context, adjusted for the available attention-layout information. That fallback keeps the catalog useful, but it changes the strength of the result. A number produced from complete layer and head metadata is not supported in the same way as one produced from a historical approximation.
Memory estimates combine the weight term, KV cache, and a fixed runtime allowance. In the current model, that allowance is 0.5 GB for runtime buffers and related overhead. For a selected weight quantization q and context c, the shape is therefore:
memory required = weight memory(q) + KV cache(c, layout, KV precision) + runtime overhead
The equation is simple enough to be useful, but each term carries a condition. Weight memory depends on the representation that the selected runtime can use. KV memory depends on the architecture and context. Runtime overhead is a project-level allowance rather than an exhaustive measurement of every backend. A result should be read as a consistent estimate under those assumptions, not as an allocator trace.
The equation also explains why memory decisions can feel unintuitive. Quantizing the weights does not automatically shrink every other term. At the same context, the KV cache still depends on the attention structure and the chosen cache precision. What the lower weight footprint changes is the amount of memory left over after the weights and fixed overhead are resident. That spare capacity can make a previously impossible context feasible, or it can provide the headroom needed to avoid a tight fit. The gain is therefore indirect but operationally important.
The reverse is true when context grows. Extending a prompt window adds cache state for each context-sensitive layer, even though the learned weights have not changed. A model can keep exactly the same identity and weight quantization while crossing a memory boundary because the active conversation is longer. This is why a model’s native window cannot be used as a universal memory requirement: it describes an upper capability, while the actual workload chooses how much of that capability must be resident at once.
Architecture determines the slope of that growth. Two models with similar parameter counts can have different numbers of layers, KV heads, or head dimensions. One may use grouped-query attention and retain fewer key-value streams; another may use more full attention. A hybrid model can devote part of its layers to a recurrent mechanism whose state does not grow with context in the same way. These differences are not cosmetic metadata. They determine how quickly memory pressure rises as the user asks for longer prompts.
The estimator must also decide what to do when it does not know those architectural details. Refusing every incomplete record would make a large model catalog unusable, so llmfit retains a coarse fallback. That fallback is valuable for relative comparisons, but it should not be read with the same confidence as a calculation based on explicit layer and head metadata. The missing field does not mean the architecture is simple; it means the current analysis has less information with which to model it.
This is also why llmfit normally caps the context used for estimation at 8,192 tokens when the user has not supplied another limit. Many model records advertise windows far beyond that size. Using the maximum advertised window for every ordinary recommendation would make the KV term dominate and could make typical configurations look dramatically larger than the contexts most runtimes use by default. The cap changes the scenario being estimated; it does not change the model’s native capability.
The distinction becomes especially important for a long-context model. Its catalog record may say that it supports a very large window. The fit calculation may still use 8,192 tokens by default, or a user may explicitly request a different cap. Separately, llmfit can estimate the usable context: how many tokens the selected memory pool can hold after the weights and fixed costs are resident. That number may be lower than the native window even when the selected configuration receives a favorable fit verdict.
Three context values answer three different questions. The native context is what the model claims to support. The effective context is the context used to calculate the current memory and speed figures. The usable context is what the current memory pool can actually accommodate alongside the weights. Confusing these values can turn a technically correct fit row into a misleading operational choice. A row that says a model has a large native window but only a smaller usable window is warning the reader about the real workload boundary.
The usable-context calculation is not a second benchmark. It derives the remaining pool after the fixed weight footprint and estimates how many context tokens fit along the KV-cache slope, capped at the model’s native window. If the architecture has no context-scaled cache, the result can behave differently because recurrent state is fixed. The output is valuable precisely because it tells the reader what the memory model implies, while still leaving runtime behavior and quality for later verification.
Quantizing the KV cache introduces another trade-off. The default is fp16-style cache storage, but llmfit can model fp8, q8, q4, and an experimental TurboQuant option where the runtime and backend support it. Lower KV precision reduces the cache term and can recover context capacity, especially for a model whose weights already occupy most of the pool. The reduction is not equivalent to changing the weight quantization: it affects the retained state, not necessarily the learned weights, and may carry its own quality or support caveats.
TurboQuant illustrates why a smaller number is not automatically a supported recommendation. The project treats it as experimental, limits it to the vLLM-plus-CUDA scenario for planning, and notes that recurrent state in hybrid models is not compressed in the same way as full-attention KV. A plan can show the possible memory savings and mark the support condition rather than silently presenting the option as available everywhere.
The complete memory path can be pictured as a sequence of contributions and constraints:
The diagram’s important direction is from configuration to pool, not from a model name to a universal requirement. Change the weight representation and the weight term changes. Change the context and the KV term changes. Change the architecture or the available metadata and the cache calculation may become more or less precise. Change the machine’s memory topology and the pool being compared may change entirely.
This explains why llmfit’s dynamic quantization search tries the best-quality option first and then retries at half context if nothing fits. A model can fail at the requested context without failing at every useful context. The fallback is still conditional: it does not claim that halving context preserves the same application behavior, only that the configuration may become feasible under a smaller memory demand. A long-document workload may genuinely need the larger context and therefore need more memory rather than a lower setting.
The practical reading rule is to carry the configuration along with every memory number. “This model needs 12 GB” is incomplete. A more meaningful statement identifies the weight quantization, context used for estimation, KV precision, and memory pool. It also distinguishes a precise architectural calculation from a coarse fallback. Only after those terms are understood can the next question be answered: whether the required memory should stay on the GPU, use an offload path, or fall back to the CPU—and how much headroom each path provides.