LoRA, short for low-rank adaptation, freezes a pretrained model's weights and trains small added matrices in each transformer layer. Because only those small matrices learn, far fewer parameters need gradients and optimizer state, so training needs much less GPU memory than full fine-tuning.
The idea in plain words
Full fine-tuning updates every weight in the model. For a model with billions of parameters, that means storing a gradient and optimizer state for every one of them. That is where most of the memory goes.
LoRA leaves the original weights alone. Next to selected weight matrices it adds a pair of much smaller matrices whose product approximates the update you would have made. Only the small pair is trained. At inference you can either keep the adapter separate or merge it into the base weights.
What the paper claims
The original paper by Edward Hu and colleagues compares against fine-tuning GPT-3 175B with Adam. Its abstract states LoRA can reduce the number of trainable parameters by 10,000 times and cut the GPU memory requirement by 3 times. It also reports model quality on par with or better than full fine-tuning on several models, with higher training throughput and no added inference latency.
Those numbers are the authors' own, for that setup. Your savings will depend on model size, which layers you adapt and the rank you pick.
Why adapters are convenient to ship
An adapter is a small file. One base model can serve many tasks by swapping adapters, one for support replies, one for classification, one for a customer's style. Storage and distribution are cheap compared with keeping a full copy of the model per task.
The Hugging Face PEFT library describes this family of methods as fine-tuning only a small number of extra parameters, which lowers compute and storage costs while giving performance comparable to a fully fine-tuned model, and makes training feasible on consumer hardware. LoRA is one of several methods PEFT implements.
Where LoRA is not magic
- It still needs good data. A small adapter trained on noisy examples learns the noise.
- Very different target behaviour may need a higher rank or full fine-tuning. Test it rather than assume.
- The base model's flaws come along. If the base model is weak at the task, an adapter will not rescue it.
- Safety behaviour can shift after any fine-tuning. See how fine-tuning can weaken a model's safety training.
A practical starting recipe
- Pick a base model that is already close on your evals.
- Train an adapter at a low rank on a few hundred clean examples.
- Compare against your best prompt on the held-out eval set.
- Raise the rank or add data only if the gap justifies it.
For fitting larger models onto one GPU, read QLoRA explained.
Quick check
Does LoRA change the original model weights?
No. The base weights stay frozen. Only the small added matrices are trained, and they can be kept separate or merged later.
Does LoRA slow down inference?
The paper reports no added inference latency, since the adapter can be merged into the base weights.
Sources
- LoRA: Low-Rank Adaptation of Large Language Models, fetched 19 October 2026.
- Hugging Face PEFT documentation, fetched 19 October 2026.
Last reviewed: 19 October 2026.
