Changing Representation: Export, Quantization, and Data

Unsloth's hardware-aware design does not end when training stops. A learned model still has to become an artifact that another runtime can load, and the data that produced it still determines what was learned. This chapter focuses on those two representation changes: how to choose among an adapter, merged weights, compressed formats, and GGUF, and how to prepare text or documents without losing the structure that training depends on. The practical judgment is whether an output is merely written to disk or is actually usable, compatible, and appropriate for the next stage.

An artifact preserves some choices and discards others

A LoRA adapter is the most reversible of the main outputs. It contains the learned low-rank changes while relying on a compatible base model. That keeps the base model separate, makes the adapter relatively portable, and preserves the possibility of loading it with a different base or continuing to adjust it—provided the base identity and adapter assumptions remain aligned. The trade-off is that deployment now has two dependencies: the adapter and the model it modifies.

Merged weights make a different choice. The base and learned update are combined into a self-contained model representation, which can simplify loading for a consumer that does not know how to attach a PEFT adapter. The merge also removes some flexibility. The result is larger than the adapter alone and does not preserve the same separation between the original model and the learned change. Merging is therefore convenient for a target runtime, not automatically better for future experimentation.

Unsloth Studio exposes these as distinct operations rather than hiding them behind one generic save button. It can export a merged model, a non-PEFT base model, or the adapter alone. Merged export can also route to compressed-tensors FP8/FP4 or portable TorchAO FP8/INT8 formats when the installed build and hardware support them. These paths have different compatibility requirements. The repository's compressed-tensors route requires an NVIDIA GPU and the corresponding support in the installed Unsloth environment; the portable route has a different hardware envelope. On macOS and MLX, the quantized export families described by that route are rejected rather than represented as universally available.

GGUF is a further representation change aimed at runtimes such as llama.cpp. It is not simply a renamed safetensors file. GGUF packages weights and metadata for a different loader, and its quantization methods trade storage and memory against approximation error. A small artifact can be useful when the deployment device is limited, but the smaller representation can change task behavior. The right output depends on whether the next user needs adapter flexibility, a self-contained model, a particular server, or the smallest practical local footprint.

Quantization is an approximation with a test attached

GGUF export can produce ordinary quantization methods and more aggressive low-bit choices. For some IQ formats, llama.cpp requires an importance matrix. For those formats, the importance matrix is a required calibration input to the export path. The repository establishes that IQ formats need it; it does not, by itself, establish that a particular matrix or quantization method preserves a particular workload. Unsloth's export API accepts an importance-matrix request or a supplied matrix path for the formats that need it.

That detail changes the meaning of “exported successfully.” A file can be syntactically valid and still be a poor representation for the task. A calibration matrix that reflects one prompt distribution may not preserve another.