The VRAM Budget: Why Precision Alone Is Not Enough
This chapter builds Unsloth's resource model: what actually occupies memory during training, which settings move each cost, and how the project decides whether a fast path is safe on a particular backend. The practical result is a way to reason about a run before it fails. A 4-bit model may be the right starting point, but the decisive variables can be the context window, batch, optimizer, activations, embeddings, attention backend, or the hardware's native support.
A training run has several memory owners
The most visible memory consumer is the model's weight set. Quantization reduces the storage cost of many weight matrices by using fewer bits per value, while some components remain in a higher-precision form. In Unsloth's QLoRA-oriented path, the base model is mostly frozen, so the reader may reasonably expect the memory problem to be solved once the checkpoint loads. That expectation is incomplete because loading and training have different working sets.
The first additional owner is the adapter. LoRA adds small trainable matrices to selected projections. Their size grows with the chosen rank and with the number of modules being adapted. Adding attention and MLP projections creates a different cost from adapting only one family of layers. Training embeddings or the output head is more expensive still: Unsloth treats those as full matrices saved for training rather than as low-rank pairs. Tied embeddings can reduce duplication, but the choice remains a real part of the budget.
The second owner is the optimizer. A trainable parameter needs more than its current value. The optimizer may retain momentum-like state, higher-precision state, or paged state while updating it. With LoRA, the frozen base does not require optimizer state, but adapter parameters do. With full fine-tuning, the trainable set becomes the whole model and the optimizer term can change the feasibility of the run more than the difference between two storage formats.
The third owner is the gradient. Gradients must survive long enough to be accumulated and applied. Their size follows the trainable set, which is why full fine-tuning and LoRA have such different memory profiles even when they load the same base weights. A small adapter does not mean a zero-cost backward pass; it means the trainable portion of that pass is smaller.
The fourth owner is the activation set: the temporary values produced while tokens move through the layers. Activations depend strongly on batch size and sequence length. Increasing the context window from a short prompt to a long document changes the number of token positions present in many intermediate tensors. Gradient checkpointing can reduce how much of that intermediate state is retained by recomputing parts of the forward pass during backward. That trades memory for extra computation rather than making the work disappear.
There are also embeddings, dequantization buffers, communication buffers, allocator fragmentation, and fixed runtime overhead. Unsloth's estimator names these categories because a model that fits at load time can still run out of memory on its first backward step. The important mental model is a sum of owners, not a single model-size label.
Context length changes the shape of the problem
Suppose two users load the same quantized model with the same LoRA rank. One trains short instruction examples with a modest batch; the other trains long documents with a small batch. The second run can still require more memory because each layer carries larger token-dependent intermediates. A lower batch does not necessarily compensate for a much longer sequence, and gradient accumulation changes the effective batch seen by optimization without making every per-step activation disappear.
Attention adds another distinction. Attention compares token positions, so an implementation that stores a large pairwise score or workspace can make memory grow roughly with the square of sequence length. Memory-efficient Flash Attention, SDPA, or Flex Attention paths avoid retaining the same full score matrix in the usual way, so Unsloth's estimator charges them differently from eager attention. This is not only a performance preference. The attention resolver also considers whether a model family, head dimension, dtype, device capability, and installed kernel support make a backend numerically safe.
That coupling explains a common diagnostic trap. If a long-context run fails, reducing precision may help with weight storage but leave the dominant activation or attention term largely unchanged. Changing the attention backend may reduce memory, but the model may be excluded from that backend because its decode behavior or shape is known to be unsafe. Reducing sequence length may be the more direct remedy, even if the goal was to preserve long-context behavior. There is no universal “use more quantization” answer because different terms scale with different inputs.
Checkpointing has a similar trade-off. Unsloth's unsloth gradient-checkpointing mode uses a heuristic: below a relatively short sequence, its offloading overhead may not be worthwhile, so the implementation can use ordinary checkpointing instead. Above that region, the memory saved by retaining fewer activations can justify the extra work. The exact crossover is an implementation policy, not a law of hardware, which is why the effective mode should be recorded when comparing runs.
A budget becomes useful only after semantic gates
Memory-saving batching methods illustrate why fit is not enough. Sample packing places multiple short examples into a fuller sequence so less padding is processed. Padding-free batching removes padding-related work when the model and collator can interpret the resulting metadata correctly. Both can improve utilization, but both change the shape of the inputs that reach attention and loss computation.
Unsloth's trainer wrapper therefore checks the model and dataset before enabling them. It blocks automatic packing or padding-free behavior for vision-language models, processor-based inputs, custom collators, encoder-decoder models, and known model families whose attention behavior does not match the optimization. Hybrid linear-attention or state-space models receive special scrutiny because recurrent state and causal convolutions can leak across the boundary between examples after flattening. A structural check for those components is safer than relying only on a model name.
This gives the resource model a second dimension: semantic fit. A packed batch may consume less memory and still be wrong if positions, masks, recurrent state, or multimodal fields no longer align. The repository's response is to disable the optimization, try a safer configuration only when the maximum length remains enforceable, or let the original error surface. A slower correct batch is preferable to a faster batch that trains on cross-example state.
The same principle applies to attention. Unsloth can prefer a memory-efficient implementation, but model-specific exclusions prevent an explicit request from re-enabling a path known to produce incorrect outputs or illegal accesses. Float32 loads cannot use Flash Attention's lower-precision kernels. A model with a head dimension beyond the supported limit is redirected to SDPA, Flex Attention, or eager execution according to the available capabilities. The budget must therefore be computed from the attention path that can actually run, not from the fastest path mentioned in a configuration file.
The relationship can be summarized as a guarded sequence:
The arrows matter. Hardware detection does not validate a batch's semantics, and a semantic check does not prove that the resulting run fits. Unsloth keeps these gates separate because a failure at one boundary needs a different remedy from a failure at another.
Hardware support changes the available budget
Unsloth treats platform support as capability negotiation. Its device logic distinguishes CUDA, AMD HIP/ROCm, Intel XPU, and Apple MLX paths, then checks whether the native operations needed by a requested representation are usable. If bitsandbytes is absent or its native library cannot be used on the current setup, Unsloth disables the corresponding 4-bit or 8-bit route while retaining 16-bit LoRA or full fine-tuning where possible. That is a change in the budget and the kernel path, not an all-or-nothing verdict on the model.
The installer follows the same logic. Torch, companion libraries, ROCm architectures, Python versions, and platform wheels must agree. A package can be importable while its native extension is built for another accelerator or Torch minor version. The Studio hardware detector consequently distinguishes “no GPU,” “GPU hidden by an intentional visibility mask,” and “GPU present but unusable by this Torch build.” Those distinctions lead to different actions: accept chat-only mode, respect the mask, or repair the environment.
Multi-GPU placement adds another layer. Model weights and some trainable state can be divided across devices, but activations and runtime overhead still occupy the device executing a layer. Communication and synchronization also consume capacity. Unsloth's estimator discounts the nominal free memory of additional GPUs to account for that overhead and uses the result to select a minimum set of devices when automatic selection is enabled. An estimate can identify a plausible placement; it cannot guarantee that every shard, temporary buffer, or driver allocation will fit.
Modality matters here too. A generic decoder formula can estimate ordinary text layers, but the repository deliberately avoids inferring all vision-tower or multimodal LoRA modules from configuration shapes because those structures vary. A VLM may need processor state and image or video tensors that do not appear in a text-only estimate. An MLX run may be able to perform supervised fine-tuning while not supporting the same RL trainers available on CUDA or ROCm. “Supported hardware” therefore means a particular combination of operations, not equal feature parity.
A worked comparison: four runs, one model
Imagine holding the model family constant and changing only the job around it. In the first run, a frozen 4-bit base trains a modest LoRA adapter on short text examples. Weight storage is reduced, only the adapter needs optimizer state, and checkpointing can keep the activation term manageable. This is the case in which the QLoRA memory bargain is most visible.
Now keep the model and adapter the same but make the examples much longer. The base weights have not changed, yet the activation term grows with the number of token positions. If the selected attention implementation is memory-efficient, the run avoids retaining the same quadratic score workspace as eager attention; if the model or dtype forces a fallback, the longer context can become much more expensive. Reducing the batch may help the per-step working set, but it does not make a long sequence short.
In the third run, select 16-bit LoRA because the host cannot use the needed bitsandbytes native kernels. The adapter still limits trainable state, but the base weights occupy more memory and the fused low-bit matmul route is no longer the same. This may be the correct and supported outcome. The relevant comparison is not “4-bit failed, therefore Unsloth failed”; it is “the backend changed the representation and the budget, so the expected capacity and speed changed with it.”
In the fourth run, use a vision-language model with image inputs and a custom processor. Text-only formulas no longer capture every input or trainable module. Automatic packing or padding-free batching may be blocked, because the processor and multimodal fields must remain aligned. A configuration that looks less optimized can still be the more reliable one because it preserves the meaning of each example.
These cases illustrate why a single headline VRAM number is not transferable. The model is one input, but the effective budget is a function of representation, trainable scope, token count, batch, attention, checkpointing, modality, and backend. Unsloth's estimator makes those variables explicit; the trainer and loader guards decide which combinations are legal; the actual run verifies the estimate. Readers can use the same comparison method for their own workload without pretending that the four cases are benchmark results.
A disciplined way to make a run fit
Start by fixing the workload that must remain unchanged: model family, modality, dataset shape, desired context, and evaluation objective. Then identify the flexible terms. If the failure happens while loading weights, precision, a smaller model, or device placement is the likely lever. If it happens during backward, inspect trainable scope, optimizer state, activations, and checkpointing. If it appears only with long sequences, inspect context length and attention implementation before assuming that the weight format is responsible.
Next, verify the effective route rather than the requested route. Confirm that the checkpoint's quantization method matches the intended loader, that the selected LoRA modules are the ones being trained, that the attention backend is not a fallback, and that packing or padding-free batching was not disabled by a model or collator guard. On a heterogeneous machine, record the actual backend and selected devices. A configuration that looks identical in a notebook can have a different memory budget after capability negotiation.
Finally, treat the estimator as a decision aid and the run as a measurement. The repository's formulas include conservative floors and calibrated constants, but they remain estimates. They cannot establish quality equivalence, and they cannot know every optional module's runtime behavior. The most reliable conclusion is not “this model needs exactly a certain amount of VRAM.” It is “under this precision, trainable scope, context, optimizer, attention path, and backend, this configuration has a defensible chance of fitting and a clear fallback if it does not.”
That is the point at which Unsloth's operator-level optimizations become a route choice. Precision, adapters, batching, attention, and hardware now have visible relationships. The next judgment is to compare those choices together: not which flag sounds fastest, but which training route preserves the desired semantics within the budget and remains honest about its fallbacks.