2023
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".
An examination of whether a strong language model can stand in for a human rater on open-ended questions. The paper introduces MT-Bench and Chatbot Arena, then measures how often model judgements agree with human preference and where they systematically go wrong.
@inproceedings{zheng2023judging,
author = {Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric P. Xing and Joseph E. Gonzalez and Ion Stoica},
title = {Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena},
booktitle = {Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track},
year = {2023},
eprint = {2306.05685},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2306.05685}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2023
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".
2023
The single most useful result for anyone shipping retrieval. A bigger context window does not fix relevance: if the right chunk lands in the middle of twenty, it may as well not be there. Re-rank and put the best passage last.
2022
The antidote to leaderboard shopping. Multi-metric evaluation is more work up front and saves you from finding out in production that the model you picked is the one that falls over on perturbed input.