How you know it works before a customer finds out it does not.
A benchmark built from real issue reports and pull requests across popular Python repositories. A model is handed a codebase and a GitHub issue and has to produce a patch that makes the repository test suite pass, which is a far harder target than generating plausible-looking code.
Language models attend strongly to the beginning and end of a long context and weakly to the middle. The authors show accuracy degrading in a characteristic U-shape as the position of the relevant passage moves through the input, even in models explicitly built for long contexts.
An examination of whether a strong language model can stand in for a human rater on open-ended questions. The paper introduces MT-Bench and Chatbot Arena, then measures how often model judgements agree with human preference and where they systematically go wrong.
Argues that accuracy alone is a misleading summary of a language model, and evaluates a set of models across many scenarios and many metrics at once, including calibration, robustness, fairness and efficiency, so the trade-offs stay visible.
A collection of retrieval datasets spanning very different domains and task types, used to test whether a retriever that works on one corpus still works on another without retraining. The headline finding is that sparse lexical baselines remain hard to beat out of domain.