Training
LoRALoRA
Also known as: Low-Rank Adaptation
An efficient fine-tuning method that adapts a model by training a small set of extra weights while leaving the huge original model frozen.
LoRA is a cheap, fast way to customise a big AI model. Instead of retraining the whole enormous model (very expensive), you freeze it and train just a tiny add-on. The result behaves like a specialised version of the model, but you only had to train and store a small piece. It is like adding a lightweight adapter to a big machine rather than rebuilding the machine.
LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method. The original model's parameters are kept frozen; LoRA inserts small trainable matrices and only those are updated during training.
Why it is popular
- Cheap: trains far fewer parameters, so it needs much less memory and compute.
- Small to store and share: an adapter is tiny compared to the full model, and you can keep many task-specific adapters for one base model.
- Swappable: load different adapters on the same base model for different jobs.
Combined with quantization (as QLoRA), it lets people fine-tune large models on a single GPU.
LoRA freezes the pretrained weights W and learns a low-rank update ΔW = BA (with rank r ≪ dimensions), so only the small A, B matrices are trained, cutting trainable parameters and optimiser memory by orders of magnitude while approaching full fine-tuning quality (Hu et al. 2021). At inference the update can be merged into W (zero added latency) or kept separate for hot-swapping adapters.
QLoRA extends this by quantising the frozen base to 4-bit (NF4) and back-propagating through it into LoRA adapters, enabling fine-tuning of very large models on modest hardware (Dettmers et al. 2023). LoRA is the default of the broader PEFT family (alongside prefix/prompt tuning, adapters). Practical levers are rank r, which layers to adapt, and scaling factor; trade-offs include potential quality gaps on tasks needing large representational change and adapter-management complexity when serving many variants.