The papers behind how we build

All entries

3 entries

1. 2023ICLR 2024PublishedCurated

   SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

   Jimenez et al.

   A benchmark built from real issue reports and pull requests across popular Python repositories. A model is handed a codebase and a GitHub issue and has to produce a patch that makes the repository test suite pass, which is a far harder target than generating plausible-looking code.

   EvaluationAgents & Tool Use

   arXiv
2. 2023TACL 2024PublishedCurated

   Lost in the Middle: How Language Models Use Long Contexts

   Liu et al.

   Language models attend strongly to the beginning and end of a long context and weakly to the middle. The authors show accuracy degrading in a characteristic U-shape as the position of the relevant passage moves through the input, even in models explicitly built for long contexts.

   Retrieval & RAGEvaluation

   arXiv
3. 2023NeurIPS 2023PublishedCurated

   Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

   Zheng et al.

   An examination of whether a strong language model can stand in for a human rater on open-ended questions. The paper introduces MT-Bench and Chatbot Arena, then measures how often model judgements agree with human preference and where they systematically go wrong.

   Evaluation

   arXiv

Browse by topic