Research desk
An annotated reading list. Every entry is credited to its original authors, with a note on what it changes once the idea meets a real system.
3 entries
Demonstrates that a sufficiently large language model can perform a new task from a handful of examples in the prompt, with no gradient updates. The paper established in-context learning as the default interface to a language model.
Pre-trains a bidirectional transformer by masking tokens and predicting them from both directions, then fine-tunes the same weights on downstream tasks. The masked objective, and the idea of pre-train then fine-tune, set the template for the following years of NLP.
Introduces the transformer: a sequence model built entirely from attention, with no recurrence and no convolution. Because every position is computed in parallel, the architecture scales to far larger training runs than the recurrent models it replaced.
The architectures and pre-training results everything else stands on.
3 entries
Grounding a model in documents you actually own.
4 entries
Getting a model to show its work, and checking the work.
5 entries
Models that call tools, hold state, and finish multi-step jobs.
6 entries
Changing what a model does without retraining all of it.
3 entries
Steering outputs toward what a person would accept.
3 entries
How you know it works before a customer finds out it does not.
5 entries