Agents & Tool Use
Models that call tools, hold state, and finish multi-step jobs.
- 2026PreprintCurated
SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools
An audit of what happens when an agent calls a production API that answers 200 OK with a parsable body while having silently failed. Across 721,320 parameters in 2,501 independently published OpenAPI documents, 7.5% declare an enumeration and 15.2% declare any machine-checkable constraint at all, while 40.1% of documents state at least one constraint in prose their schema does not encode. Executing 219 schema-derived perturbations against live endpoints from 27 vendors, the authors find that constraint form rather than vendor identity predicts honesty: machine-checkable constraints produced an honest error in 111 of 111 cases, prose-only constraints failed silently in 44 of 61 (p = 2e-13). Twelve models across eight families then met these endpoints: a vocabulary the description merely exemplified was missed on 88 of 88 attempts, while vocabularies written out in full were used correctly 88 to 91% of the time. Running the full agent loop, models detected the silent failure in 12% of cases, repaired it in 0%, asserted a false negative to the user in 41%, and invented a figure in 12%. Promoting the vocabulary into the schema removes the failure, from 88 of 88 to 0 of 89.
- 2026PreprintCurated
Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents
A study of multi-party LLM agents, which act for a principal who briefs them and receives results while conversing in a separate channel with a counterparty whose interests may diverge, as in negotiating with a vendor, screening inbound requests or mediating between employees. The authors argue that "help whoever you are talking to" is the wrong objective and formalise the resulting loyalty problem. PrincipalBench is a 75-item multi-turn benchmark with leak probes, dual judges and an integrity-audit gate. Across 13 frontier subjects it exposes a sharp split, with one cluster at or below 20% harm and another at 53.6 to 75.3%, a split that single-turn safety evaluations do not surface: a selective cluster that declines adversarial probes while still following the principal's legitimate requests, and an over-refusing cluster that refuses broadly. A prompt-time loyalty scaffold of seven prioritised rules, open-coded from over 50 failure trajectories, holds Claude-Sonnet to 19.4% harm and all nine selective subjects at or below 20%. A per-token KL distillation recipe transfers a prompted Qwen3-32B teacher into 8B Qwen3 and Llama-3.1 students. Both mechanisms move along a common leak/over-refusal trade-off rather than crossing it.
- 2023ICLR 2024PublishedCurated
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
A benchmark built from real issue reports and pull requests across popular Python repositories. A model is handed a codebase and a GitHub issue and has to produce a patch that makes the repository test suite pass, which is a far harder target than generating plausible-looking code.
- 2023NeurIPS 2023PublishedCurated
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Generalises chain-of-thought into a search tree. The model proposes multiple partial solutions, evaluates them against each other, and can back out of a branch that is not working, rather than committing to one linear line of reasoning.
- 2023UIST 2023PublishedCurated
Generative Agents: Interactive Simulacra of Human Behavior
Agents that keep a natural-language memory stream, reflect over it to form higher-level conclusions, and plan their day from those conclusions. The paper studies what emerges when twenty-five such agents share a small sandbox world.
- 2023NeurIPS 2023PublishedCurated
Reflexion: Language Agents with Verbal Reinforcement Learning
Instead of updating weights, the agent writes down what went wrong after a failed attempt and carries that reflection into the next attempt. The reflection is stored in natural language and used as additional context, so no gradient step is required.
- 2023NeurIPS 2023PublishedCurated
Toolformer: Language Models Can Teach Themselves to Use Tools
A model decides for itself when to call an external API, inserts the calls into its own training data, and keeps only the ones that actually reduce its prediction loss. No hand-written tool demonstrations are needed.
- 2022ICLR 2023PublishedCurated
ReAct: Synergizing Reasoning and Acting in Language Models
Interleaves verbal reasoning traces with tool actions, so the model thinks about what to do next, does it, then reasons about what came back. The reasoning trace also becomes an auditable record of why the agent took each step.