When Training Becomes a System: Scaling, Objectives, and Serving
LlamaFactory becomes a full system when a valid configuration has to survive process launch, distributed execution, objective-specific training, evaluation, export, and inference. This chapter follows those state changes and separates two questions that are often conflated: how processes are started and how a model is partitioned, and how a loss is computed and how a resulting artifact is served. The reader should be able to identify which transition is being tested, what state persists across it, and why a successful launch or completed training run is not yet proof of a usable model.
The adaptation decision from the previous discussion determines the state that moves through this lifecycle. A full-tuning run carries broad model and optimizer state. An adapter run carries a base model relationship plus trainable adapter state. A quantized run carries an additional representation constraint. Those differences become visible as soon as the run leaves one device, changes objective, or crosses from training into serving.
Starting processes is not the same as partitioning a model
The launcher is the first state transition. In both the practical v0 path and the separate v1 path, LlamaFactory can relaunch a training command through torchrun when multiple devices are visible or when the user forces distributed execution. It prepares process-level information such as the number of nodes, the rank of each node, the number of processes per node, the master address, and a rendezvous port. Elastic settings can add a job identifier and a maximum restart count.
This automation solves a real usability problem. A person can begin with the same conceptual training command on one device or several, while environment variables supply the process topology when the job expands. The examples cover multi-node training, DeepSpeed ZeRO-3, Ray workers, elastic jobs, FSDP2, and other specialized paths. But the launcher only establishes the group of processes. It does not decide, by itself, which parameters each process owns, how activations or gradients are sharded, or whether the selected model and adaptation method are compatible with the backend.
That distinction is easiest to see in the v0 tuner. After arguments are parsed, it routes the stage to ordinary training or to a specialized backend such as Ray, HyperParallel, Megatron Bridge, or the Megatron-core adapter. Each route has conditions. Some backends support only particular stages. Some accept full or LoRA tuning but not quantized models. Some parallel dimensions must fit the world size exactly. Some optimizer modes are excluded under distributed execution. The same process launcher can therefore lead to different model-partitioning behavior depending on the rest of the configuration.
The v1 trainer makes the separation more explicit. It can fall back to DistributedDataParallel when multiple data-parallel processes exist without a named sharding configuration, or delegate sharding to a DeepSpeed or FSDP plugin. It creates batches, knows the data-parallel and context-parallel sizes, scales losses according to valid tokens, clips the effective global gradient norm, reduces metrics, and then advances the optimizer and scheduler. These are responsibilities below process creation. A run can reach the training loop while still failing to satisfy the memory, topology, or synchronization assumptions of the chosen implementation.
The physical topology matters because the model’s state is not the only thing being distributed. A batch may be split across processes, a model may be sharded across a device mesh, and a long sequence may be divided through context parallelism. Communication and synchronization then become part of the compute contract. If the world size does not support the requested parallel factors, the configuration should stop before the job is launched. If it does support them but the model’s attention implementation or quantization path does not, the failure may occur during initialization or the first step.