2023
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".
A model decides for itself when to call an external API, inserts the calls into its own training data, and keeps only the ones that actually reduce its prediction loss. No hand-written tool demonstrations are needed.
@inproceedings{schick2023toolformer,
author = {Timo Schick and Jane Dwivedi-Yu and Roberto Dessi and Roberta Raileanu and Maria Lomeli and Luke Zettlemoyer and Nicola Cancedda and Thomas Scialom},
title = {Toolformer: Language Models Can Teach Themselves to Use Tools},
booktitle = {Advances in Neural Information Processing Systems 36 (NeurIPS 2023)},
year = {2023},
eprint = {2302.04761},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2302.04761}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2023
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".
2023
More tokens and more latency per answer, so it earns its place only where a wrong answer is expensive. We do not reach for it on classification or extraction.
2023
The architecture to read if you want to understand long-lived agent memory. The memory stream, reflection and planning split is the part that transfers to production work.