The Data-to-Token Contract: When Format Becomes Behavior
In LlamaFactory, a dataset is not ready for learning merely because a file can be opened. The specific question is how an external record becomes the exact messages, tokens, labels, and media inputs that a chosen model will see. Following that path lets the reader diagnose a dataset or template mismatch before changing a learning rate, and distinguish structural validity—data that passes the loader—from meaningful supervision that teaches the intended behavior.
This distinction matters because language models learn from sequences, not from the conceptual labels humans attach to fields. A record with instruction, input, and output has to become a user turn and an assistant turn. A record with a list of conversations has to preserve roles and order. A preference example has to keep the chosen and rejected responses paired with the same prompt. A multimodal record has to keep placeholders synchronized with actual images, videos, or audio. Each transformation can preserve meaning, lose meaning, or quietly change the training signal.
LlamaFactory’s data pipeline is designed to put those transformations in named stages. The stages are reusable, but they are not neutral: the choices made before tokenization determine what the model is asked to imitate.
A dataset name is not yet a training example
The v0 workflow starts with a logical dataset name and a metadata description. The name might refer to a local file, a Hugging Face, ModelScope, or Modelers Hub dataset, a loading script, or a JSON resource in cloud storage. The metadata records how to find it and how to interpret it: formatting, whether it is a ranking dataset, the split, sample limits, column aliases, role tags, and optional fields for tools or media.
This layer separates identity from interpretation. A user can call a source support_dialogue, but the trainer should not have to know whether its prompts live in a column called instruction, question, or messages. The dataset description supplies that mapping. It can also select multiple sources, combine them by concatenation or interleaving, create a validation split, stream eligible data, and cache tokenized output. Those are not minor conveniences. They determine which examples enter the run, in what proportions, and whether the data can be revisited consistently.
Consider two records that express the same simple interaction. One has an instruction, an optional input, and an output. Another has a system message followed by a human message and a model response. LlamaFactory can describe these as different source formats, then align both to an internal representation. The data pipeline does not ask the later trainer to understand every possible spelling of a conversation. It gives the trainer a common structure containing prompt messages, response messages, system text, tool text, and media lists.
That normalization is performed by converters. The Alpaca-style converter can combine an instruction and an input into one user prompt, preserve a system message and history, and attach an assistant response. The ShareGPT-style converter maps source role tags such as human, gpt, observation, or function into the project’s internal roles. The OpenAI-style path handles role/content records and can turn tool-call structures and grouped tool responses into the same message vocabulary. Preference examples branch differently: the prompt is retained once, while chosen and rejected assistant responses become a pair. KTO examples carry a feedback tag that changes how the response representation is built.
The converter is also a gatekeeper. Conversation roles have to appear in an order that makes the turn structure meaningful. Ordinary supervised examples need a prompt followed by a response; ranking examples need the alternatives that the preference objective expects. If the role sequence or message count is abnormal, the converter marks the example as broken and skips it rather than passing malformed turns downstream. That is safer than allowing an invalid record to become a plausible-looking token sequence, but it creates an operational consequence: the effective dataset may be smaller than the file suggests.
The same point applies to media paths. A local image, video, or audio name can be resolved relative to the configured media directory, but a missing file is not repaired by the language model. The converter preserves the media association so a later multimodal plugin can process it. If the association is wrong, the text may still tokenize while the corresponding visual or audio content is absent, misplaced, or rejected later.
The data guide and implementation are not perfectly synchronized in every detail. The repository’s general format guide foregrounds Alpaca and ShareGPT as the supported dataset formats, while the converter registry also contains an OpenAI-specific converter. That is a useful warning about working with an evolving codebase: documentation can describe the common path while the implementation exposes a newer or narrower path. A reader should use the matching snapshot’s examples and converter behavior, rather than infer that every format has identical documentation, tests, or edge-case coverage.
The loader then preprocesses the aligned records according to the selected stage. A pre-training processor expects text. Supervised fine-tuning expects prompt/response turns. Preference stages use paired responses. KTO uses feedback metadata. The loader can reuse a tokenized dataset from disk, but doing so intentionally bypasses other data arguments; the cached representation becomes the effective input. This is another state transition worth noticing: once preprocessing is persisted, changing the original schema or template does not change the already-tokenized dataset until it is rebuilt.
The sequence makes a practical boundary visible. The source owns the raw record. Metadata owns the instructions for locating and interpreting it. The converter owns schema alignment and structural validity. The template owns model-facing message syntax. The stage processor owns how that syntax becomes a training example. The collator owns batching and the final tensor arrangement. The model receives the result, not the original JSON. A defect introduced early may only become visible after the later component has lost the information needed to explain it.
Templates decide what the model learns
After conversion, LlamaFactory still has not produced a model-ready learning signal. It has messages. The template determines how those messages become a sequence the checkpoint recognizes. A template specifies the formatting of system, user, assistant, function, observation, and tool messages; it can add beginning and ending markers, special tokens, stop words, and model-specific delimiters. If the tokenizer has no suitable chat template, LlamaFactory can construct one from the same formatting rules and attach it to the tokenizer.
This is why a template is part of the model contract, not a cosmetic wrapper around text. A base model trained with one role-marker convention may treat another convention as ordinary content. A stop token that is correct for one family may be meaningless for another. A system message may be fused into the first user turn for one model family but kept separate for another. The project’s model registry often associates a model family with a default template because the pairing is meaningful, but the existence of a default is not proof that it is correct for every checkpoint variant.
Supervised preprocessing makes the learning consequence explicit by constructing two related sequences: input IDs and labels. Input IDs contain the formatted conversation. Labels identify which positions contribute to the loss. By default, prompt positions are replaced with an ignore marker, while assistant response positions remain targets. The model reads the full context, but it is primarily trained to predict the response rather than to reproduce the user’s request.
That default can be changed. Training on prompts removes the prompt mask, making the model learn from both sides of the exchange. Masking history can preserve the context of earlier turns while assigning loss only to the final response. These options are not interchangeable ways of expressing the same intention. They alter which tokens receive gradient signal and therefore what behavior the examples reinforce. A dataset can be perfectly converted and tokenized while still teaching the wrong portions of the conversation for the intended task.
Length limits add another layer of meaning. The processor truncates examples to the configured cutoff length and may prioritize the most recent turns when history masking is enabled. A long conversation can therefore lose an earlier instruction, a system constraint, or the context needed to interpret the final answer. Sequence packing can combine several shorter examples into one training sequence, but the packed attention and position metadata must prevent one example from attending to another as if they were part of one conversation. Efficient packing is consequently both a memory decision and a semantic-boundary decision.
Reasoning-capable models make the coupling between formatting and loss even clearer. When a response lacks an explicit thought block, the reasoning template can add an empty one. With thinking enabled, the empty block is placed in the response and participates in loss computation. With thinking disabled, it is placed in the prompt so the model receives the control signal without being trained to emit the hidden reasoning content. The repository warns that the enable_thinking setting should remain consistent during training and inference. Otherwise, the same adapter can be trained under one response contract and prompted under another.
This does not mean that a visible reasoning trace is automatically a faithful explanation. It is still data supplied to the training process, and its quality depends on how the examples were produced and selected. The engineering point is narrower and more useful: the placement of thought markers changes both the rendered sequence and the positions that receive loss, so it must be treated as a deliberate data policy.
Multimodal examples add a second channel of alignment. The text contains placeholders such as an image, video, or audio marker, while separate fields contain the actual media. A plugin first checks that the selected model has the required processor and that the number of media items matches the number of placeholders. It then regularizes the inputs: images may be resized and converted to RGB, video may be sampled into frames, and audio may be converted to a common sampling rate. The placeholder is expanded according to the model’s expected visual or audio token sequence, and the processor produces the tensors consumed by the model.
The expansion cannot be universal because multimodal architectures do not consume media in the same way. One model family may represent an image as a repeated image token sequence; another may need image grids and patch counts; a video model may need frame or timestamp metadata; an audio-capable model may need feature masks. LlamaFactory keeps a common placeholder and media-field vocabulary at the data boundary, then delegates the model-facing expansion to specialized plugins.
The collator is where these branches meet batching. It gathers media from individual examples, pads text, creates attention masks and labels, and may calculate model-specific position information such as multimodal rotary positions or cross-attention masks. Packed multimodal sequences need boundaries that identify which image or video belongs to which sub-sequence. Some distributed configurations also need a dummy media input when a batch happens to contain only text, so every process follows a compatible model path. These details explain why “support images” is not equivalent to adding an images column.
The failure conditions are therefore concrete. If a field mapping points to the wrong column, the converter may create empty or malformed messages. If a chosen template does not match the checkpoint, token IDs can be valid but role boundaries wrong. If the prompt and response are both masked, the model receives no useful supervised target. If thinking settings differ between training and inference, the prompt contract changes. If media counts and placeholders disagree, the plugin should stop rather than guess. If a packed sequence loses its boundaries, examples can contaminate one another even though the batch has the expected tensor shape.
A disciplined operator can test these contracts in increasing cost. First inspect a normalized example and confirm its roles, response, tools, and media references. Then inspect the rendered text and which decoded tokens remain in the labels. For reasoning data, verify the placement of thought markers. For multimodal data, verify counts and processor outputs. Only after those checks should a small training smoke test be trusted as evidence that the intended supervision reached the model.
The central lesson is that LlamaFactory’s data layer does not merely make many file formats convenient. It preserves a chain of meaning from a human-designed example to a model-facing objective. The chain succeeds when source metadata, conversion, template, label ownership, reasoning policy, and media processing agree. Once that agreement is established, the next design question is no longer what the model should learn, but how much of the model needs to change—and what resource strategy can sustain that change.