Freezes the pretrained weights and injects trainable low-rank matrices into each layer instead. The number of trainable parameters drops by several orders of magnitude, adapters are small enough to swap at runtime, and training cost falls with them.
@inproceedings{hu2021lora,
author = {Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},
title = {LoRA: Low-Rank Adaptation of Large Language Models},
booktitle = {International Conference on Learning Representations (ICLR) 2022},
year = {2021},
eprint = {2106.09685},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2106.09685}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2023
Cut the RLHF pipeline down to one training run. For teams without the infrastructure to run a reward model and PPO, this is usually the practical route into preference tuning.
2023
The reason fine-tuning stopped being a budget line item. Adapter-based tuning on one GPU is now the default first attempt; full fine-tuning has to justify itself.