Overview

Across RoBERTa, DeBERTa, GPT-2, and GPT-3, the LoRA paper reports competitive downstream quality with far fewer trainable parameters, but scores remain task- and metric-specific. Low-data MultiNLI and LoRA-plus-prefix experiments show that data regime and composition can change results. Baseline provenance, hyperparameters, and reported variation limit universal claims.

Learning outcomes

Across reported RoBERTa, DeBERTa, GPT-2, and GPT-3 tasks, LoRA often matched or exceeded full fine-tuning with far fewer trainable parameters, but this is not universal.
In the reported GPT-3 MultiNLI experiment, LoRA scored 63.8% with 100 examples versus 60.2% for full fine-tuning, while rankings changed as data increased.
Baseline provenance, metric differences, task-specific settings, and reported variation limit how strongly small benchmark gaps should be interpreted.

Keywords

LoRAbenchmarksparametersGLUEMultiNLImetrics

What the Evidence Shows Across Models and Tasks

LoRA’s benchmark evidence answers a narrower question than “is LoRA better?” It asks whether a carefully chosen low-rank update can reach useful downstream quality while training and storing far fewer parameters. The focus here is how that pattern appears across language-understanding and generation tasks, and how data scarcity and method combinations change it. Reading the results carefully lets you claim that LoRA was often competitive in the reported settings without turning a set of experiments into a universal guarantee.

Across scales, the parameter-quality frontier is real

The original paper evaluates LoRA on several model families rather than on one convenient demonstration. It uses RoBERTa and DeBERTa for the GLUE natural-language-understanding benchmark, GPT-2 for generation, and GPT-3 175B for both understanding and generation-oriented tasks. The metrics differ across these settings, so the scores should be compared within a task and model family, not placed on one common leaderboard. What can be compared across the experiments is the relationship between downstream quality and the number of trainable parameters.

On RoBERTa base, full fine-tuning updates 125 million parameters and reports an average GLUE score of 86.4. LoRA reports an average of 87.2 while training 0.3 million parameters. The point is not that a smaller number automatically produces a higher score. It is that a change of roughly the same general model can be learned through a tiny fraction of the trainable state in this tested configuration.

RoBERTa large shows a similar relationship. Full fine-tuning uses 355 million trainable parameters and reports an average of 88.9. LoRA uses 0.8 million and reports 89.0. DeBERTa XXL extends the comparison to a 1.5-billion-parameter model: full fine-tuning reports 91.1 on the GLUE aggregate, while LoRA reports 91.3 with 4.7 million trainable parameters. These results make the efficiency pattern visible at increasing model sizes, but they do not imply that the same rank or target matrices will work unchanged in every model.

The GPT-2 generation results show why the comparison must preserve the task metric. On GPT-2 Medium’s E2E NLG Challenge, full fine-tuning uses about 354.92 million parameters. LoRA uses 0.35 million and reports BLEU 70.4, NIST 8.85, METEOR 46.8, ROUGE-L 71.8, and CIDEr 2.53. The full-tuning scores are 68.2, 8.62, 46.2, 71.0, and 2.47. On GPT-2 Large, LoRA uses 0.77 million compared with 774 million for full fine-tuning and reports a similar pattern across the listed generation metrics, with some metrics higher and one essentially tied.

A score such as BLEU is not directly interchangeable with an accuracy score on GLUE. The evidence is stronger when stated as a repeated pattern—LoRA can be competitive with much larger updates across both understanding and generation—than when expressed as one aggregate claim that ignores the measurement being used. The pattern also has a mechanism behind it: LoRA is reusing the base model’s general computation and learning a constrained change, rather than updating every model parameter independently for the downstream task.

The GPT-3 results are the paper’s most demanding scale test. Full fine-tuning uses about 175,255.8 million trainable parameters.

Published by DeepMox

FAQ

How does LoRA compare with full fine-tuning in the reported benchmarks?

In the reported benchmarks, LoRA often matched or exceeded full fine-tuning while updating far fewer parameters, although some task scores were slightly lower. The comparisons are specific to the tested models, tasks, metrics, and training conditions.

What happened when the MultiNLI training set was small?

In the reported GPT-3 MultiNLI experiment, LoRA scored 63.8% with 100 examples versus 60.2% for full fine-tuning, but the gap narrowed or changed as more data were used. This does not establish universal sample efficiency.

Do more trainable parameters or combined adapters always improve LoRA?

No. LoRA-plus-prefix improved reported WikiSQL results, did not improve MultiNLI in one combination, and performed worse with prefix-layer tuning despite more parameters. Added capacity or modules can introduce optimization conflicts.

Previous10 minLoRA: Rank, Placement, and Useful Updates

Next11 minLoRA in Apple’s 2025 Model Lifecycle

LoRA: Benchmark Quality Across Models and Tasks

What the Evidence Shows Across Models and Tasks

Across scales, the parameter-quality frontier is real

FAQ