A benchmark built from real issue reports and pull requests across popular Python repositories. A model is handed a codebase and a GitHub issue and has to produce a patch that makes the repository test suite pass, which is a far harder target than generating plausible-looking code.
@inproceedings{jimenez2023swebench,
author = {Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan and others},
title = {SWE-bench: Can Language Models Resolve Real-World GitHub Issues?},
booktitle = {International Conference on Learning Representations (ICLR) 2024},
year = {2023},
eprint = {2310.06770},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2310.06770}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2023
The single most useful result for anyone shipping retrieval. A bigger context window does not fix relevance: if the right chunk lands in the middle of twenty, it may as well not be there. Re-rank and put the best passage last.
2023
Model-graded evaluation is how you get past "it looks fine". The paper is equally clear about the failure modes, so read the limitations section before you wire an LLM judge into a release gate.
2023
More tokens and more latency per answer, so it earns its place only where a wrong answer is expensive. We do not reach for it on classification or extraction.