Research desk
An annotated reading list. Every entry is credited to its original authors, with a note on what it changes once the idea meets a real system.
3 entries
A benchmark built from real issue reports and pull requests across popular Python repositories. A model is handed a codebase and a GitHub issue and has to produce a patch that makes the repository test suite pass, which is a far harder target than generating plausible-looking code.
Language models attend strongly to the beginning and end of a long context and weakly to the middle. The authors show accuracy degrading in a characteristic U-shape as the position of the relevant passage moves through the input, even in models explicitly built for long contexts.
An examination of whether a strong language model can stand in for a human rater on open-ended questions. The paper introduces MT-Bench and Chatbot Arena, then measures how often model judgements agree with human preference and where they systematically go wrong.
The architectures and pre-training results everything else stands on.
3 entries
Grounding a model in documents you actually own.
4 entries
Getting a model to show its work, and checking the work.
5 entries
Models that call tools, hold state, and finish multi-step jobs.
6 entries
Changing what a model does without retraining all of it.
3 entries
Steering outputs toward what a person would accept.
3 entries
How you know it works before a customer finds out it does not.
5 entries