Foundations

The architectures and pre-training results everything else stands on.

3 entries

  1. 2020NeurIPS 2020PublishedCurated

    Language Models are Few-Shot Learners

    Brown et al.

    Demonstrates that a sufficiently large language model can perform a new task from a handful of examples in the prompt, with no gradient updates. The paper established in-context learning as the default interface to a language model.

    Foundations
    arXiv
  2. 2018NAACL-HLT 2019PublishedCurated

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin et al.

    Pre-trains a bidirectional transformer by masking tokens and predicting them from both directions, then fine-tunes the same weights on downstream tasks. The masked objective, and the idea of pre-train then fine-tune, set the template for the following years of NLP.

    Foundations
    arXiv
  3. 2017NeurIPS 2017PublishedCurated

    Attention Is All You Need

    Vaswani et al.

    Introduces the transformer: a sequence model built entirely from attention, with no recurrence and no convolution. Because every position is computed in parallel, the architecture scales to far larger training runs than the recurrent models it replaced.

    Foundations
    arXiv

All research