The Model Package: From Architecture to Checkpoint
To understand how Transformers moves a model between tasks and systems, start with the package that makes the model identifiable. This chapter examines the relationship among an architecture, its configuration, learned weights, preprocessing, task heads, and outputs. The practical goal is simple: when a model identifier appears in a project, the reader should be able to ask what each part contributes and what would break if the parts were mixed carelessly.
A pretrained model is often spoken of as though it were a single object. In practice, “model” can mean at least three different things: the design of a neural network, a set of trained weights, or a ready-to-use task model. Transformers supports all three meanings, but its interfaces work best when the distinctions remain clear.
Architecture is the shape; a checkpoint is a learned state
An architecture is the model’s computational skeleton. It determines which layers exist, how information moves through them, what dimensions are expected, how attention or other operations are arranged, and which inputs and outputs the implementation supports. An architecture can be constructed with random weights. It then has a valid shape but no learned capability for a real task.
A checkpoint is a learned state for an architecture. It contains parameter values produced by training or conversion, usually accompanied by the information needed to reconstruct the model that those values belong to. A checkpoint is therefore not a replacement for the architecture. It is an answer to a narrower question: what values should this architecture use for this trained instance?
The distinction explains a common source of confusion. A model name may refer to a family, while a checkpoint identifier refers to one repository state and one set of artifacts. Two checkpoints can share an architecture and still differ in training data, fine-tuning objective, label mapping, vocabulary additions, or intended use. Conversely, a similar name does not establish that two architectures can accept one another’s weights. Shape compatibility is necessary, but it is not proof of semantic compatibility.
Transformers places the configuration between the abstract design and the instantiated model. A configuration stores structural and behavioral attributes such as vocabulary size, hidden dimensions, layer counts, attention choices, special token identifiers, and other model-specific settings. Those values are not the learned knowledge of the checkpoint. They are the instructions needed to build the right arrangement of modules and to interpret the weights and inputs correctly.
This makes configuration a kind of executable blueprint. When a model class receives a configuration, it can construct the layers implied by that blueprint. If the configuration says that the vocabulary has a particular size, the embedding and output structures must be consistent with it. If it describes an encoder-decoder model, the model must expose the corresponding input and generation behavior. If it carries special token identifiers or modality settings, preprocessing and generation need to use the same conventions.
The blueprint is also where some validation belongs. The repository’s configuration code checks or warns about architectural relationships and token identifiers, and it supports saving a configuration so the same structure can be reconstructed later. This does not make every configuration correct by itself. It means the model’s assumptions are represented in a durable place instead of being left implicit in a one-time script.
The model class turns that blueprint into computation. In the PyTorch implementation, PreTrainedModel is the common base responsible for configuration ownership and shared lifecycle behavior such as loading, saving, embedding access, and capability flags. Model-specific classes then define the actual architecture. This division is important: the base class supplies a stable route into the ecosystem, while the model implementation retains the mathematical decisions that make one architecture different from another.
A task head is a model built for a particular kind of answer. A bare backbone may return hidden states: numerical representations produced by the network for each input position or feature location. Those representations can support many uses, but they are not yet a classification result or a next-token distribution. A sequence-classification head adds a mapping from a suitable representation to class scores. A causal-language-modeling head adds the output projection needed to score possible next tokens. Other heads support other task contracts.
The head is not merely an ornamental layer added after the “real” model. It determines what labels the model can consume during training, what outputs a pipeline can post-process, and which public methods are meaningful. This is why Transformers’ training code rejects a representation-only base model when a task head is needed to compute a labeled objective. It is also why an Auto class for a task selects a task-appropriate model family rather than blindly returning the bare backbone.
The same backbone can therefore participate in several workflows without being the same complete object in each one. A researcher may inspect its hidden states. A classifier may attach a label-producing head. A generator may expose generate() because it has a language-modeling head and the required generation behavior. The shared architecture is valuable, but the head establishes the task boundary.
This is easiest to see when a team changes the question without changing the whole backbone. Suppose a pretrained encoder has learned useful representations of text. For sentiment classification, the downstream model needs a small output space whose entries correspond to labels such as positive and negative. For token classification, the output must instead preserve a score for each input position. The underlying representation can be shared, but the shape and meaning of the result differ. Training data, label alignment, loss calculation, and evaluation must follow the selected head. Calling both objects “the same model” hides the decision that matters most for the task.
The configuration participates in that decision in two directions. It supplies the values used to construct the shared body, and it records information that callers need to interpret the body’s behavior. A vocabulary size connects the preprocessor’s IDs to the embedding table. A label mapping gives scores names that a user can understand. Generation-related token identifiers tell a decoding routine when a sequence can begin, stop, or be padded. In a multimodal model, configuration can also describe how a text stream relates to image or audio features. The configuration is not a complete description of learned behavior, but it prevents the surrounding software from guessing structural facts.
That separation also makes debugging more disciplined. If a weight has the wrong shape, first ask whether the architecture and configuration agree. If the shape is valid but labels are nonsensical, ask whether the head and label mapping match the dataset. If hidden states look plausible but generated text is broken, inspect the language-modeling head, generation settings, and preprocessing conventions rather than assuming the backbone is at fault. Transformers cannot make these diagnoses automatically, but its package boundaries give each question a natural owner.
Outputs and saved artifacts complete the package
A model contract is not defined only by what enters the model. It also includes what comes out. Transformers uses structured output objects for common families of results. Depending on the model and requested options, an output may contain a last hidden state, pooled representations, logits, hidden states from several layers, attention values, cross-attention values, or past key/value state for faster sequential decoding.
This structure gives downstream code named places to find information. The main prediction remains distinguishable from optional diagnostics. A caller can request hidden states or attentions when they need to inspect behavior, while ordinary inference can use the primary output without learning the internal layout of every model implementation. The shapes also carry meaning: a hidden-state tensor has a different role from a class-logit tensor, and a cache has a different lifecycle from either one.
The contract is deliberately conditional. Not every model returns every field, and an option such as attention output may depend on the implementation chosen by the model. A vision model may produce spatial representations rather than token sequences. An audio model may use input features rather than token IDs. A multimodal model can combine several kinds of representations. Consistency means that supported fields have recognizable semantics, not that every architecture emits an identical record.
That distinction matters when a caller changes model families. Code that only asks for a supported task result can often use the common interface. Code that reaches into hidden states, attention maps, cache objects, or modality-specific fields must understand the capabilities of the selected architecture. The public surface lowers the cost of switching, but it does not remove the need to check what a particular model promises.
Saving is the point where these relationships become a reusable artifact. save_pretrained() is designed to create a directory that can later be passed back to from_pretrained(). The model saves its configuration together with the weights, and generative models can save generation configuration as well. A tokenizer or processor can be saved alongside the model so the next user does not have to reconstruct the input convention from memory.
For a small model, the learned parameters may fit into one safe tensor file. For a large model, the weights can be split into shards and accompanied by an index that maps parameter names to the shard containing them. The storage layout changes, but the logical package does not: configuration still defines the structure, and the index lets the loader recover the association between each parameter and its file. This is an example of portability through a stable contract rather than through one fixed file format.
The use of safetensors when available adds another boundary. The repository’s loading guide prefers it over traditional pickle-based weight files because it is safer and faster to load. That preference concerns how passive parameter data is serialized; it does not validate the model’s quality or make a repository’s custom Python code harmless. Serialization format and code trust are separate questions.
A saved package can also contain missing or newly initialized task-specific weights. For example, loading a base checkpoint into a model with a newly chosen task head may preserve the backbone while leaving the head to be trained for the downstream objective. Warnings about weights that were not initialized from the checkpoint or weights that were not used are therefore meaningful evidence about the relationship between the requested model class and the available artifact. They should be investigated rather than dismissed as routine noise.
The same applies to mismatched sizes. A checkpoint may have been trained with one number of labels while the new task requires another. A loader can expose options for handling some mismatches, but the resulting model still needs a meaningful training and evaluation plan. A mechanically successful load is not the same as a semantically complete model.
A useful inspection does not require reading every implementation file. It asks a small set of connected questions. What input names and modalities does the model accept? Which configuration values determine the shape of its layers and vocabulary? Is the selected class a bare backbone or a task head? Which output field represents the requested result, and which fields are optional diagnostics? Finally, what companion preprocessing and generation information must be saved so another environment can reproduce the same call? These questions follow the actual interfaces between components, so they remain useful when the architecture changes.
The answer may be distributed across several saved files, but the reader should understand it as one package. A shard index, for example, is not an extra model architecture; it is a storage map that lets the loader find the parameters required by the already-defined architecture. A generation configuration is not a second set of learned weights; it records defaults for how a generative model turns its output distribution into a sequence. Keeping these roles separate prevents storage details from being mistaken for model behavior.
Seen this way, a model package preserves a chain of decisions. The configuration says how to build the architecture. The weights say what the trained instance learned. The preprocessor says how raw inputs become the representation that training saw. The head says which task output and labels are meaningful. The structured output says how callers receive predictions and optional internal state. The save format and revision say which artifact should be reconstructed later.
This chain is why a model identifier should not be treated as a magic string. It names an attempt to recover a compatible set of objects. When the package is complete, the next stage can load it and follow the contract. When it is incomplete or mixed with the wrong companion files, the failure may appear much later as strange predictions, wrong shapes, missing labels, or a generation method that does not exist.
The next boundary is the input itself. Once the package has a shape, learned state, task head, and output contract, raw text or media still has to be converted into the exact tensors and aligned positions the model expects. That conversion is the work of tokenizers and multimodal processors, and it is why preprocessing assets belong beside the weights rather than in a forgotten caller script.