The LoRA Mechanism: A Frozen Model with a Trainable Low-Rank Delta
LoRA’s central mechanism is simple to state but easy to misunderstand: it changes selected weights through a small trainable update while leaving the pretrained weights themselves frozen. The question here is how that update can be both constrained enough to save resources and expressive enough to alter a large model’s behavior. Understanding the two paths, their dimensions, their initialization, and their eventual merge gives you the tools to judge LoRA without treating “low-rank” as a slogan.
Start with the unchanged computation
Consider one dense layer in a pretrained network. Its weight matrix is the part that turns an input vector into an output vector. Call that existing matrix $W_0$, and call the input $x$. Before adaptation, the layer computes:
$$ h = W_0x. $$
Full fine-tuning would change the entries of $W_0$ directly. After training, the new matrix could be written as $W_0 + \Delta W$, where $\Delta W$ is the accumulated task-specific change. If the layer has $d$ output dimensions and $k$ input dimensions, that change has $d \times k$ entries. A square layer with width $d$ therefore has $d^2$ possible entries in a full update.
LoRA leaves $W_0$ alone. Instead, it represents the change as a product of two smaller matrices:
$$ W = W_0 + \Delta W = W_0 + BA, $$
where $A$ has shape $r \times k$ and $B$ has shape $d \times r$. The middle dimension, $r$, is the rank selected for the LoRA module. The crucial condition is that $r$ is much smaller than both $d$ and $k$. The product $BA$ can still be a $d \times k$ matrix, but its rank cannot exceed $r$. In other words, the update is allowed to affect the full output space, yet only through a limited number of independent directions.
That constraint changes the trainable object. A full update has $dk$ parameters. The two factors have $rk + dr = r(d+k)$ parameters. For a square matrix, that becomes $2dr$ rather than $d^2$. When $r$ is small compared with $d$, the difference is large. LoRA is not compressing the entire pretrained matrix into a low-rank approximation; it is constraining the change applied to that matrix.
A small example makes the geometry less mysterious. If $r=1$, $A$ has one row and $B$ has one column. Their product is an outer product: one input-side pattern is paired with one output-side pattern. The update can affect many entries of the resulting matrix, but all of those changes are tied to that one pair of directions. With $r=4$, the product can be understood as a sum of four such paired contributions. The model is not limited to changing four individual weights; it is limited to combining four coordinated patterns across the layer. That is why a low-rank update can have a broad effect without having an independent parameter for every matrix entry.
The rank limit also explains what LoRA gives up. A full update can choose every input-to-output relationship independently. A rank-$r$ update must express those relationships through the same small intermediate space. If the downstream task’s useful change fits that space, the constraint removes a great deal of redundant freedom. If the task needs many unrelated changes, the bottleneck can become the reason the adapted model falls short. The method’s success therefore depends on the structure of the task-specific change, not only on the size of the base model.
The adapted forward pass makes the relationship clearer:
$$ h = W_0x + BAx. $$
The input goes through the original path and through the update path. The outputs are added coordinate by coordinate. This means that LoRA does not ask the small factors to recreate the model’s general language ability. The frozen path continues to perform the original transformation. The factors only need to supply the task-specific adjustment that the original transformation lacks.
The diagram also shows why initialization matters. LoRA initializes $A$ with a random Gaussian matrix and $B$ with zeros. Although $A$ is not zero, the product $BA$ is zero because $B$ is zero. The complete layer therefore begins with $W_0x$, exactly the pretrained computation. Training can then move the two factors away from that starting point. This is different from injecting a randomly chosen behavior into a model and hoping optimization will undo it.
The factors are scaled as well. The paper describes multiplying the update path by $\alpha/r$, where $\alpha$ is a constant associated with the chosen rank. The purpose is to make changing the rank less likely to require a complete retuning of the surrounding hyperparameters. In the authors’ Adam-based setup, changing this scale has a relationship similar to changing the learning rate when the initialization is adjusted appropriately. That does not make the choice universal, but it explains why the scaling is part of the design rather than an afterthought.
There is a useful operational way to read the equation. $A$ first maps the input from its original $k$-dimensional space into an $r$-dimensional bottleneck. $B$ then maps that compact representation back into the $d$-dimensional output space. The bottleneck is not a separate sequence of model layers added after the original layer. It is a factorized description of how the layer’s output should be nudged. The base path and the update path remain parallel until their outputs meet.
The two factors should also be understood as a pair, not as two independently meaningful miniature models. During optimization, the task loss changes them so that their product produces a useful update. A particular row of $A$ or column of $B$ does not automatically correspond to a named skill. What matters for the layer is the product $BA$ and its interaction with the rest of the network. This distinction matters when interpreting the phrase “low-rank”: it describes the shape of the allowed matrix change, not a promise that each of the $r$ components is separately human-readable.
For a natural-language-to-SQL task, for example, the model still uses the same pretrained transformations to read the question and table schema. The learned update does not contain a new database engine or a complete language model. It changes how selected transformations respond to that task’s patterns. The value of the update depends on whether the base model already contains enough general structure for the new behavior to be reached through a compact shift.
From training factors to a deployment weight
The dimensions explain why LoRA reduces trainable parameters, but the frozen status of $W_0$ explains the training-memory savings. The base matrix still participates in the forward pass; it is not removed. What is omitted is the gradient update and optimizer state for that matrix. With an adaptive optimizer such as Adam, a full fine-tuning job must maintain additional state for the huge set of parameters it updates. LoRA maintains that trainable state mainly for $A$ and $B$, whose size is controlled by $r$ and by the number of weight matrices selected.
Freezing the base therefore changes the training lifecycle, not just the final file format. At the start of a step, the model reads the fixed base weights and the current factors. The loss is computed on the downstream examples. The optimizer then updates the factors, while the base remains unchanged. At the next step, the same base is reused with a slightly different delta. This is why a task-specific checkpoint can describe the learned change without containing a second copy of the general model. It also explains the boundary: a task cannot be trained or served from the small factors alone, because the frozen computation is part of the result.
For a Transformer, the paper focuses mainly on the self-attention projections, especially the query and value matrices. If LoRA is applied to $\hat L$ matrices of model dimension $d_{model}$, the paper gives the trainable-parameter count as:
$$ |\Theta| = 2\hat L d_{model}r $$
for the square projection matrices used in its experiments. This formula exposes two independent choices. Increasing rank gives each adapted matrix more update capacity. Increasing the number of adapted matrices spreads capacity across more parts of the network. A small rank applied widely is not the same design as a large rank applied to one matrix.
The paper describes LoRA as having a qualified path back toward full fine-tuning. If the method were applied to all relevant weight matrices, included biases, and used a rank approaching the rank of the original matrices, it would roughly recover the expressive capacity of full fine-tuning. That is a statement about the flexibility of the parameterization as capacity expands. It is not the configuration behind the efficient headline results. In the main study, the MLP modules remain frozen, and the paper leaves systematic investigation of MLPs, LayerNorms, and biases for future work.
This qualification prevents a misleading interpretation of the parameter formula. The formula tells us how many trainable parameters a chosen configuration has; it does not tell us whether those parameters are attached to the right transformations for a particular task. Nor does a larger rank automatically guarantee better behavior. More capacity can be useful when the low-rank bottleneck is too narrow, but it can also change optimization and storage costs without producing a meaningful quality gain. The method’s generalization toward full fine-tuning is an upper-bound intuition, not a reason to skip configuration testing.
The deployment advantage follows from the same algebra. Once training is finished, compute the sum $W = W_0 + BA$ and store or use that ordinary weight. The model no longer needs to execute an additional adapter layer during inference. It simply performs the dense multiplication it would have performed with a fully fine-tuned weight. The low-rank factors were a convenient training and storage representation; the merged weight is a convenient inference representation.
This distinction between representations is the source of LoRA’s no-added-latency claim. A sequential adapter must remain an extra operation in the forward path unless the architecture is changed in some other way. A merged LoRA update is already part of the matrix being multiplied. The claim is therefore architectural and conditional: it applies when the factors are merged into the base weight. It is not a claim that every dynamic way of selecting adapters has zero additional work.
The merge also supports task switching. Suppose a service has a shared base and one update for summarization. To load a different task, it can recover the base by subtracting the old update and then add another update. The paper presents this as a quick operation with little memory overhead compared with replacing all model parameters. A service can therefore keep general weights resident and exchange small task-specific changes.
The reported GPT-3 example makes the representation difference concrete. With rank 4 applied to query and value projections, the LoRA checkpoint is about 35 MB, while the full model is about 350 GB. The small checkpoint does not make the 350 GB base disappear. It means that the base is stored once, while each task-specific change is stored separately. In the paper’s calculation, 100 adapted versions require roughly 350 GB plus 100 small checkpoints, about 354 GB in total, instead of roughly 35 TB for 100 separate full models.
This is also why a low parameter count should not be mistaken for low total deployment memory. The base model remains the dominant asset. LoRA changes the scaling of additional task variants and the cost of training their changes. It is especially valuable when many tasks share one foundation model, not because it turns a very large foundation model into a tiny one.
There is a boundary at the point where one request stream contains multiple tasks. If a merged weight represents one task at a time, examples needing different updates cannot simply share that same matrix without grouping or switching. Keeping factors unmerged makes dynamic selection possible, but then the cleanest ordinary-matrix execution is no longer the whole story. A production design must choose between the simplicity of merged weights and the flexibility of per-example selection.
The mechanism therefore yields a precise bargain. LoRA freezes the general transformation, restricts the task-specific change to a low-dimensional factorization, initializes that change at zero, and can later fold it into the base matrix. Those facts explain the savings in trainable state, checkpoint size, and merged inference. They do not by themselves prove that a chosen rank or target matrix will preserve task quality, nor do they settle how a service should handle heterogeneous traffic. Once the two paths are clear, the next comparison is about where other adaptation methods spend their scarce resources: extra forward computation, context positions, or optimization stability.