Checkpoint: The Shared-Base Mental Model

LoRA becomes clearer when its algebra and its systems behavior are considered together. This checkpoint connects the ideas already established: a pretrained model remains the shared base, a low-rank delta changes selected transformations, and the deployment representation determines which costs remain visible. The goal is a practical judgment: can you tell the difference between a compact update, a cheap inference path, and a complete guarantee that the adapted model will work?

One model, two representations

The most useful way to picture LoRA is not as a small model attached to a large one. It is one model with two representations of a task-specific change.

During training, the change is kept in factorized form. The frozen weight $W_0$ continues to process the input, while the trainable factors $A$ and $B$ produce a low-rank addition. The factors are small because their shared intermediate dimension is the chosen rank. Their product can influence many entries of the layer’s effective update, but only through the limited set of directions that the factors can express.

During deployment, those two representations can become one. The factors are multiplied and added to the frozen weight, producing $W = W_0 + BA$. The model then uses the resulting ordinary dense weight. The task-specific change has not vanished; it has changed form. Factorized form is useful for training, storage, and task-specific exchange. Merged form is useful when the service wants the simplest ordinary inference path.

That distinction explains several claims that might otherwise seem contradictory. LoRA can be called a small module because the trainable artifact is small. It can also be said to add no inference latency because the module can be absorbed into the weight before serving. Neither statement means that the base model is no longer needed. The shared weights still supply the general computation, whether they are held separately beside the factors or combined with them in a deployed matrix.

The same distinction applies to task switching. A service that keeps the base as its reference state can remove one low-rank delta and add another when the active task changes. This is far less data movement than replacing every parameter in a model-sized checkpoint. But a merged weight represents one active combination at a time. If different examples in one batch need different task deltas, the service cannot express that distinction merely by attaching a different label to each example. It must use a different execution arrangement, such as selecting unmerged factors dynamically, and that arrangement has its own cost.

The shared-base model therefore has an important invariant: the base is the stable owner of general capability, while the update is the variable owner of task-specific behavior. The transition from training to serving must preserve that relationship. If the current merged matrix is treated as the only source of truth, repeated switching can make it harder to know which base and which update are active. The mathematics makes addition and subtraction straightforward; the system still needs clear state ownership.