2023
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".
Argues that accuracy alone is a misleading summary of a language model, and evaluates a set of models across many scenarios and many metrics at once, including calibration, robustness, fairness and efficiency, so the trade-offs stay visible.
@article{liang2022holistic,
author = {Percy Liang and Rishi Bommasani and Tony Lee and Dimitris Tsipras and Dilara Soylu and Michihiro Yasunaga and Yian Zhang and Deepak Narayanan and Yuhuai Wu and Ananya Kumar and others},
title = {Holistic Evaluation of Language Models},
year = {2022},
eprint = {2211.09110},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2211.09110}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2023
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".
2023
The single most useful result for anyone shipping retrieval. A bigger context window does not fix relevance: if the right chunk lands in the middle of twenty, it may as well not be there. Re-rank and put the best passage last.
2023
Model-graded evaluation is how you get past "it looks fine". The paper is equally clear about the failure modes, so read the limitations section before you wire an LLM judge into a release gate.