PublishedCurated

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

Advances in Neural Information Processing Systems 30 (NeurIPS 2017)12 June 2017

Abstract

Introduces the transformer: a sequence model built entirely from attention, with no recurrence and no convolution. Because every position is computed in parallel, the architecture scales to far larger training runs than the recurrent models it replaced.

arXiv

BibTeX

@inproceedings{vaswani2017attention,
  author    = {Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin},
  title     = {Attention Is All You Need},
  booktitle = {Advances in Neural Information Processing Systems 30 (NeurIPS 2017)},
  year      = {2017},
  eprint    = {1706.03762},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/1706.03762}
}

Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.

2020

Language Models are Few-Shot Learners

Where prompt engineering starts. The limitation matters as much as the result: few-shot performance is sensitive to the examples you pick and to their order.