The Input Contract: Tokenizers and Multimodal Processors
Transformers can only run a model after raw human data has been turned into the numerical inputs that the model was built to interpret. This chapter follows that boundary from text to token IDs and masks, then extends it to images, audio, and video through multimodal processors. The practical goal is to tell an input-contract mismatch from a model failure: if the wrong vocabulary, special token, sampling rate, or modality alignment crosses the boundary, a perfectly loaded checkpoint can still receive the wrong problem.
The previous chapter separated a model’s architecture, configuration, head, weights, and outputs. Preprocessing is the part that connects that package to the outside world. It is not a disposable convenience layer. It determines what sequence or feature structure the model sees, and the model’s learned parameters assume that structure.
Why raw text must become a model-specific sequence
A neural model does not receive the meaning of a sentence directly. A tokenizer first turns the sentence into pieces, then maps those pieces to integer IDs from a vocabulary. The model consumes those IDs through an embedding table and uses additional tensors, such as an attention mask, to distinguish usable positions from padding or ignored positions. The tokenizer also knows how to reverse the representation, converting generated IDs back into text according to the model’s decoding rules.
The word “piece” matters. A tokenizer does not have to assign one ID to every complete word. Transformers supports several subword approaches because language creates a tension between vocabulary size and sequence length. A word-level vocabulary can keep common words intact, but it grows rapidly as spelling, inflection, names, and new words multiply. A character-level vocabulary is compact and can represent any word, but it makes sequences much longer and gives the model less semantic information per position.
Subword methods choose a middle ground. Byte-pair encoding starts with small units and repeatedly merges frequent neighboring pairs. Unigram begins with candidate pieces and removes those that contribute least to the overall tokenization likelihood. WordPiece also builds pieces iteratively, but chooses merges using a likelihood-oriented score rather than simple frequency. SentencePiece can apply related methods directly to raw text, which is useful for languages where spaces do not reliably separate words. These are different mechanisms, not interchangeable labels.
The shared consequence is that a tokenizer controls several practical variables at once. A larger vocabulary may shorten common inputs but increase the size of embedding-related parameters. A smaller or more decomposed vocabulary can represent unfamiliar words through known pieces but may produce longer sequences. Longer sequences increase the amount of context the model must process and can change memory and latency. An “unknown” token may be rare for a byte-level vocabulary while being more common for a vocabulary with limited character coverage. The right choice depends on the checkpoint and the data, not on a universal ranking of algorithms.
The trade-off is especially visible when the same application handles different languages, writing systems, or domains. A tokenizer designed around whitespace-separated English words may behave differently on Chinese or Japanese text, code, names, or technical strings. SentencePiece-style processing can work directly on raw character or byte streams, while byte-level vocabularies can cover arbitrary byte sequences. That does not make one approach universally better; it changes how much information is carried by each token and how many positions are needed for the same input. The model’s context limit and memory budget make this a systems decision as well as a linguistic one.
There is also a difference between training-time and application-time behavior. A tokenizer may offer normalization, truncation, padding, and batching options that change the tensors without changing the visible text. A training pipeline may pad each batch to its longest example, while an application may prefer a fixed or minimal shape for latency. Those choices affect efficiency and attention masks, but they must remain compatible with what the checkpoint’s model can accept. The caller is choosing how to present valid inputs, not redefining the vocabulary or the meaning of its IDs.
That dependence is why a tokenizer is associated with a model package. The IDs are not abstract numbers with a universal meaning. ID 412 in one vocabulary can represent a completely different piece in another, and the embedding row at that position was trained for the first vocabulary’s convention. Passing text through the wrong tokenizer can still produce a tensor of the expected type and shape. The model may run without an exception while interpreting every position incorrectly.
The attention mask is part of the same boundary. When examples have different lengths, padding makes them fit into a batch. The mask tells the model which positions are real input and which positions should not contribute in the same way. A model’s forward method generally expects named inputs such as input_ids and attention_mask, not the original text column. Training documentation therefore recommends tokenizing a dataset and removing columns that the model cannot accept. The change is not cosmetic: it ensures that the batch matches the model’s forward contract.
Special tokens make the contract more visible. Beginning-of-sequence, end-of-sequence, padding, unknown, separator, class, and mask tokens carry control information that ordinary text does not. A generation routine may use one token to begin a response and another to decide when to stop. A masked-language model may use a special position to indicate what should be predicted. A classifier may rely on a designated sequence position. If a special token is missing, remapped, or inserted in the wrong place, the model is not seeing the same task format it learned during training.
Prompt formatting can have the same effect. A conversation model may expect role markers or a particular arrangement of system, user, and assistant content. The visible words can look reasonable while the hidden token sequence differs from the checkpoint’s training convention. For this reason, prompt templates and extra special tokens belong with tokenizer configuration and saved assets rather than being treated as private habits in an application script.
The output side completes the transformation. Decoding is not simply joining arbitrary IDs with spaces; the tokenizer’s vocabulary, normalization, byte or whitespace conventions, and special-token rules determine how the sequence becomes text. A direct model call may return IDs, while a higher-level pipeline may perform decoding and task-specific cleanup. In both cases, the preprocessor owns the boundary between the model’s numerical representation and the user’s textual representation.
A useful debugging split follows from this mechanism. If the tokenizer cannot load its vocabulary, the problem is an artifact or package problem. If it loads but produces unexpected lengths, inspect truncation, padding, and the chosen vocabulary. If IDs have plausible shapes but a generation prompt behaves strangely, inspect special tokens and formatting. Only after those input assumptions match should the model’s architecture or weights become the primary suspect.
When one preprocessor is not enough
Text is only one input modality. A vision model needs pixels transformed into the size, channel order, normalization, and feature representation it expects. An audio model needs a waveform or feature array at a compatible sampling rate and often requires time-frequency processing. Video adds a temporal dimension and its own sampling and layout decisions. A multimodal model must combine these inputs with text while preserving the relationship between a textual placeholder and the features that replace it.
Transformers uses processors to make that relationship explicit. A ProcessorMixin can wrap a tokenizer together with an image processor, video processor, or audio feature extractor. The combined object presents one call to the user, but internally it coordinates several transformations. It normalizes the input layout, validates that a supported input is present, processes each modality, applies any model-specific placeholder replacements, merges the resulting fields, and returns a BatchFeature containing the keys the model’s forward method expects.
Consider a vision-language question. The user supplies an image and asks where an object is standing. The tokenizer handles the textual question, while the image processor turns pixels into image features. The processor may find an image placeholder in the text and expand it into the number or arrangement of positions required by the vision encoder. The model then receives text IDs, image values, masks, and any alignment information as one coherent input. The image is not merely attached as a separate blob; its representation has to occupy the positions the model’s architecture expects.
Audio illustrates a different dependency. A speech checkpoint may require a particular sampling rate. An audio processor can resample or prepare the waveform before extracting features, and a tokenizer can handle the text side for supervised training or sequence generation. If the waveform is left at an incompatible rate, the processor may produce a feature timeline whose relation to the model’s learned filters is wrong. The input can remain a valid array while its content is effectively shifted in time or frequency from what the checkpoint expects.
The processor therefore owns more than convenience. It owns the coordination rules between sub-processors. Those rules can include which attributes are required, how images are fetched or normalized, how audio is batched, how video metadata is handled, and how special modality tokens are replaced. The exact behavior is model-specific. One multimodal architecture may use a fixed number of image placeholders, another may derive the count from image patches, and an audio-language model may place acoustic tokens at a different position. The common interface makes the boundary discoverable; it does not make the internal alignment universal.
A BatchFeature is useful because it preserves the result of this coordination as named model inputs rather than leaving the caller to merge unrelated arrays. The text side may contain IDs and an attention mask; the vision side may contain pixel values; the audio side may contain input features and a feature mask; a video processor may also carry temporal metadata. The processor can return these values together while omitting helper data that the model does not consume. The important property is not that every batch has the same keys, but that the keys and their relationships match the selected model.
That distinction explains why AutoProcessor is useful. A caller can ask the checkpoint to provide its associated processor rather than manually guessing which tokenizer and image, audio, or video component belong together. A model-specific processor can also be constructed from separately loaded components when a user needs direct control. Either route should preserve the same model-facing conventions. Saving the processor with the model keeps those conventions available when the checkpoint is moved to another directory or shared on the Hub.
Alignment failures deserve special attention because they can look like model-quality failures. If the number of image features does not match the number of image placeholders, the model may reject the input or produce a result that associates visual content with the wrong text positions. If the audio sampling rate is wrong, a speech model may produce poor transcription even though tokenization is correct. If the video layout is unexpected, a model may receive a tensor with valid dimensions but the wrong temporal meaning. These are contract failures at the input boundary, not evidence that the learned architecture has failed.
Processors also validate some conditions without solving the user’s underlying problem. They can require at least one input, reject an unsupported argument, and ensure that a sub-processor has the expected class. They cannot decide whether a photograph answers the question, whether a prompt is semantically clear, whether an audio recording contains enough speech, or whether a checkpoint is suitable for a particular language or domain. Validation protects structure; it does not provide understanding.
The most reliable way to reason about preprocessing is to follow ownership. The tokenizer owns text-to-ID and ID-to-text conventions. The image, audio, and video components own modality-specific numerical preparation. The combined processor owns the order in which those transformations meet and the alignment metadata that connects them. The model owns the interpretation of the resulting tensors. When an output is wrong, locating the first boundary where the representation changes is often more useful than changing model parameters at random.
This is also why the input contract must stay attached to the checkpoint across the rest of the lifecycle. Pipelines use it before inference. Trainer uses it before assembling training batches. Generation depends on its special tokens and prompt format. Saving and Hub sharing need to preserve it so another environment sees the same model-facing representation. The next question is how Transformers reconstructs all of these matching pieces from a model identifier, a local directory, or a particular revision rather than relying on a caller to assemble them by hand.