2020
Language Models are Few-Shot Learners
Where prompt engineering starts. The limitation matters as much as the result: few-shot performance is sensitive to the examples you pick and to their order.
Pre-trains a bidirectional transformer by masking tokens and predicting them from both directions, then fine-tunes the same weights on downstream tasks. The masked objective, and the idea of pre-train then fine-tune, set the template for the following years of NLP.
@inproceedings{devlin2018bert,
author = {Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova},
title = {BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the ACL (NAACL-HLT 2019)},
year = {2018},
eprint = {1810.04805},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1810.04805}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2020
Where prompt engineering starts. The limitation matters as much as the result: few-shot performance is sensitive to the examples you pick and to their order.
2017
The substrate. Everything on this page assumes it, and the parallelism is the reason the field could scale at all.