SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Abstract

A benchmark built from real issue reports and pull requests across popular Python repositories. A model is handed a codebase and a GitHub issue and has to produce a patch that makes the repository test suite pass, which is a far harder target than generating plausible-looking code.

@inproceedings{jimenez2023swebench,
  author    = {Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan and others},
  title     = {SWE-bench: Can Language Models Resolve Real-World GitHub Issues?},
  booktitle = {International Conference on Learning Representations (ICLR) 2024},
  year      = {2023},
  eprint    = {2310.06770},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2310.06770}
}