PublishedCurated

Survey on Evaluation of LLM-based Agents

Abstract

The first comprehensive survey of evaluation methods for LLM-based agents. The authors analyse the field across five perspectives: core LLM capabilities needed for agentic workflows such as planning and tool use; application-specific benchmarks including web and SWE agents; evaluation of generalist agents; the core dimensions of agent benchmarks; and the frameworks and tooling available to agent developers. Their analysis identifies critical gaps, particularly in assessing cost-efficiency, safety and robustness.

Why this one is on the desk

Most agent evaluation stops at "did it finish". That is a binary measured once, usually on a benchmark that was published before the model was trained, and it tells you very little about what happens on a Tuesday with your data.

The survey's useful contribution is naming what is missing: cost-efficiency, safety and robustness are all listed as unresolved. In our experience cost-efficiency is the one that kills projects, and it is the one least often measured.

The cost axis nobody measures

A task completed in four attempts is not the same product as a task completed in one, and it is four times the price. Yet the standard benchmark report counts success and discards the attempt count.

Measure cost per successful task, not success rate. The two diverge sharply, and they diverge most for exactly the cheap-model configurations that look best on price.

What we take from it

Evaluation is not one number. It is a small dashboard: success rate, attempts per success, cost per success, escalation rate, and the subset of cases a human had to correct. Build those five before you argue about which model to use.

BibTeX

@inproceedings{yehudai2025survey,
  author    = {Asaf Yehudai and Lilach Eden and Alan Li and Guy Uziel and Yilun Zhao and Roy Bar-Haim and Arman Cohan and Michal Shmueli-Scheuer},
  title     = {Survey on Evaluation of LLM-based Agents},
  booktitle = {Findings of the Association for Computational Linguistics (ACL)},
  year      = {2025},
  eprint    = {2503.16416},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2503.16416}
}

Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.