BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Abstract

Pre-trains a bidirectional transformer by masking tokens and predicting them from both directions, then fine-tunes the same weights on downstream tasks. The masked objective, and the idea of pre-train then fine-tune, set the template for the following years of NLP.

@inproceedings{devlin2018bert,
  author    = {Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova},
  title     = {BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the ACL (NAACL-HLT 2019)},
  year      = {2018},
  eprint    = {1810.04805},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/1810.04805}
}