Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, Michal Shmueli-Scheuer
Findings of the Association for Computational Linguistics (ACL)20 March 2025
Abstract
The first comprehensive survey of evaluation methods for LLM-based agents. The authors analyse the field across five perspectives: core LLM capabilities needed for agentic workflows such as planning and tool use; application-specific benchmarks including web and SWE agents; evaluation of generalist agents; the core dimensions of agent benchmarks; and the frameworks and tooling available to agent developers. Their analysis identifies critical gaps, particularly in assessing cost-efficiency, safety and robustness.
Most agent evaluation stops at "did it finish". That is a binary measured once,
usually on a benchmark that was published before the model was trained, and it
tells you very little about what happens on a Tuesday with your data.
The survey's useful contribution is naming what is missing: cost-efficiency,
safety and robustness are all listed as unresolved. In our experience
cost-efficiency is the one that kills projects, and it is the one least often
measured.
The cost axis nobody measures
A task completed in four attempts is not the same product as a task completed in
one, and it is four times the price. Yet the standard benchmark report counts
success and discards the attempt count.
Measure cost per successful task, not success rate. The two diverge sharply, and
they diverge most for exactly the cheap-model configurations that look best on
price.
What we take from it
Evaluation is not one number. It is a small dashboard: success rate, attempts
per success, cost per success, escalation rate, and the subset of cases a human
had to correct. Build those five before you argue about which model to use.
BibTeX
@inproceedings{yehudai2025survey,
author = {Asaf Yehudai and Lilach Eden and Alan Li and Guy Uziel and Yilun Zhao and Roy Bar-Haim and Arman Cohan and Michal Shmueli-Scheuer},
title = {Survey on Evaluation of LLM-based Agents},
booktitle = {Findings of the Association for Computational Linguistics (ACL)},
year = {2025},
eprint = {2503.16416},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2503.16416}
}
Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
Read this before you let an agent call anything a customer will see the output of. The failure mode is not a crash, it is an API returning 200 with a body that means "I did not understand you". There is no exception to catch and no status code to branch on. The result we quote most often is the last one: the fix was one line of schema, not a better model.
A useful dose of cold water for anyone quoting deep-research demos. Maximum F1 of 8.97 across eleven frontier systems means the gap between a slick demo and dependable multi-source analysis is still very wide, and the failure mode is hallucination under load rather than retrieval misses.
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".