2023
QLoRA: Efficient Finetuning of Quantized LLMs
The reason fine-tuning stopped being a budget line item. Adapter-based tuning on one GPU is now the default first attempt; full fine-tuning has to justify itself.
Reformulates preference learning so the reward model can be folded into the policy in closed form. The result is a single supervised objective over preference pairs, with no separate reward model and no reinforcement learning loop.
@inproceedings{rafailov2023direct,
author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Stefano Ermon and Christopher D. Manning and Chelsea Finn},
title = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
booktitle = {Advances in Neural Information Processing Systems 36 (NeurIPS 2023)},
year = {2023},
eprint = {2305.18290},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2305.18290}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2023
The reason fine-tuning stopped being a budget line item. Adapter-based tuning on one GPU is now the default first attempt; full fine-tuning has to justify itself.
2022
The "constitution" is just a written policy. That is the transferable idea: write your client refusal and escalation rules down, then have the model apply them.
2022
Read it for the preference-collection half. Most teams can gather a few thousand good comparisons; that dataset is the asset.