Fine-tunes language models on human demonstrations and then on human preference comparisons collected through reinforcement learning. The paper is the origin of the RLHF pipeline that produced the first widely used instruction-following assistants.
@inproceedings{ouyang2022training,
author = {Long Ouyang and Jeff Wu and Xu Jiang and Diogo Almeida and Carroll L. Wainwright and Pamela Mishkin and Chong Zhang and Sandhini Agarwal and Katarina Slama and Alex Ray and others},
title = {Training Language Models to Follow Instructions with Human Feedback},
booktitle = {Advances in Neural Information Processing Systems 35 (NeurIPS 2022)},
year = {2022},
eprint = {2203.02155},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2203.02155}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2023
Cut the RLHF pipeline down to one training run. For teams without the infrastructure to run a reward model and PPO, this is usually the practical route into preference tuning.
2022
The "constitution" is just a written policy. That is the transferable idea: write your client refusal and escalation rules down, then have the model apply them.