Overview

LoRA’s adaptation budget can be spent on higher rank in fewer matrices or lower rank across more matrices; those choices expose different paths through attention. GPT-3 experiments under an 18-million-parameter budget and rank sweeps show that coverage and rank interact, performance is metric- and task-dependent, and leading singular directions can be structured without making rank universal. Configuration needs controlled validation.

Learning outcomes

Rank controls capacity within an adapted matrix, while placement controls which transformations can change; equal parameter budgets do not make them equivalent.
Under the paper’s 18-million-parameter GPT-3 budget, query-plus-value rank 4 reached 73.7 WikiSQL and 91.3 MultiNLI, while all-four rank 2 reached 73.7 and 91.7.
Reported rank sweeps were non-monotonic, and singular-direction overlap in selected GPT-3 layers supports a structured-update hypothesis rather than a universal rank rule.

Keywords

LoRArankplacementattentionsubspacecapacity

Rank, Placement, and the Shape of a Useful Update

LoRA gives two main ways to spend a limited adaptation budget: increase the rank of an update, or attach updates to more kinds of weight matrices. The focus here is how those choices interact, and what the original GPT-3 experiments reveal about the geometry of a useful change. The practical result is not a magic rank; it is a way to form and test a configuration hypothesis without confusing parameter count with task capacity.

Rank and coverage are different knobs

A LoRA rank describes the size of the intermediate space through which one update is expressed. The higher the rank, the more independent directions that particular update matrix can combine. Coverage describes how many matrices receive an update in the first place. These choices can trade against one another under a fixed parameter budget, but they do not do the same job.

Suppose a service has enough budget for a certain number of trainable parameters. It could place most of that budget in one attention projection and give that matrix a higher rank. Or it could give several attention projections smaller ranks. The first option provides more capacity at one location. The second gives several locations a chance to change, with less capacity at each location. A parameter count tells you that the two configurations cost about the same; it does not tell you that they expose the same route through the Transformer.

The LoRA paper makes this comparison on GPT-3 with an 18-million-parameter budget spread across all 96 layers. If only one attention-matrix type is adapted, the budget permits rank 8. If two types are adapted, it permits rank 4 for each. If all four attention projections are adapted, it permits rank 2 for each. The four projections are the query matrix $W_q$, key matrix $W_k$, value matrix $W_v$, and output matrix $W_o$.

Under that shared budget, the reported WikiSQL and MultiNLI results are revealing. Updating $W_q$ alone at rank 8 gives 70.4 and 91.0. Updating $W_k$ alone gives 70.0 and 90.8. Updating $W_v$ alone gives 73.0 and 91.0, while $W_o$ alone gives 73.2 and 91.3. When two types are updated at rank 4, $W_q$ plus $W_k$ reaches 71.4 on WikiSQL and 91.3 on MultiNLI. $W_q$ plus $W_v$ reaches 73.7 and 91.3. Adapting all four at rank 2 reaches 73.7 and 91.7.

The result is not that every projection must be adapted. It is that distributing a sufficient amount of rank across useful locations can be better than concentrating the same parameter budget in one location. The value of $W_q$ and $W_v$ together is not explained by their combined parameter count alone. The two matrices participate in different parts of attention, so the same number of trainable parameters can create a different functional opportunity.

This is also why target selection should be treated as a hypothesis. The paper’s experiments focus on attention projections and find a favorable result for query and value updates in these GPT-3 tasks. They do not establish that query and value are universally optimal, nor do they settle whether the same allocation would hold for another architecture, dataset, or domain. The main study leaves the MLP, LayerNorm, and bias choices for future investigation.

Published by DeepMox

FAQ

How are rank and target placement different in LoRA?

Rank controls how many directions one adapted matrix can combine, while target placement determines which matrices receive that capacity. Under a fixed budget, a higher rank in one location and lower ranks across several locations expose different update paths.

What did the paper’s fixed-budget GPT-3 comparison show?

It showed that distributing rank across useful locations could outperform concentrating a similar 18-million-parameter budget in one projection, with results varying by task and metric. Query-plus-value rank 4 and all-four rank 2 were both strong but not universally best.

Do higher LoRA ranks always improve quality?

No. The reported rank sweeps were non-monotonic: higher ranks sometimes plateaued or slightly reduced a metric, and the apparent best rank differed by task and metric. Rank must be evaluated with the target placement and training conditions.

Previous9 minLoRA: The Shared-Base Mental Model

Next11 minLoRA: Benchmark Quality Across Models and Tasks

LoRA: Rank, Placement, and Useful Updates

Rank, Placement, and the Shape of a Useful Update

Rank and coverage are different knobs

FAQ