Research desk
The papers behind how we build
An annotated reading list. Every entry is credited to its original authors, with a note on what it changes once the idea meets a real system.
All entries
1 entry
- 2018NAACL-HLT 2019PublishedCurated
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Pre-trains a bidirectional transformer by masking tokens and predicting them from both directions, then fine-tunes the same weights on downstream tasks. The masked objective, and the idea of pre-train then fine-tune, set the template for the following years of NLP.
Browse by topic
Foundations
The architectures and pre-training results everything else stands on.
3 entries
Retrieval & RAG
Grounding a model in documents you actually own.
5 entries
Reasoning
Getting a model to show its work, and checking the work.
6 entries
Agents & Tool Use
Models that call tools, hold state, and finish multi-step jobs.
8 entries
Adaptation
Changing what a model does without retraining all of it.
4 entries
Alignment
Steering outputs toward what a person would accept.
6 entries
Evaluation
How you know it works before a customer finds out it does not.
11 entries