2023
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".
Agents that keep a natural-language memory stream, reflect over it to form higher-level conclusions, and plan their day from those conclusions. The paper studies what emerges when twenty-five such agents share a small sandbox world.
@inproceedings{park2023generative,
author = {Joon Sung Park and Joseph C. O'Brien and Carrie J. Cai and Meredith Ringel Morris and Percy Liang and Michael S. Bernstein},
title = {Generative Agents: Interactive Simulacra of Human Behavior},
booktitle = {Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST 2023)},
year = {2023},
eprint = {2304.03442},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2304.03442}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2023
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".
2023
More tokens and more latency per answer, so it earns its place only where a wrong answer is expensive. We do not reach for it on classification or extraction.
2023
Cheap, and it works. When an agent fails a task, a written critique plus one retry often beats a bigger model.