PreprintCurated

SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools

Zongrong Li, Shengkun Ye, Feiyou Guo, Zuoyou Dang

29 August 2026

Abstract

An audit of what happens when an agent calls a production API that answers 200 OK with a parsable body while having silently failed. Across 721,320 parameters in 2,501 independently published OpenAPI documents, 7.5% declare an enumeration and 15.2% declare any machine-checkable constraint at all, while 40.1% of documents state at least one constraint in prose their schema does not encode. Executing 219 schema-derived perturbations against live endpoints from 27 vendors, the authors find that constraint form rather than vendor identity predicts honesty: machine-checkable constraints produced an honest error in 111 of 111 cases, prose-only constraints failed silently in 44 of 61 (p = 2e-13). Twelve models across eight families then met these endpoints: a vocabulary the description merely exemplified was missed on 88 of 88 attempts, while vocabularies written out in full were used correctly 88 to 91% of the time. Running the full agent loop, models detected the silent failure in 12% of cases, repaired it in 0%, asserted a false negative to the user in 41%, and invented a figure in 12%. Promoting the vocabulary into the schema removes the failure, from 88 of 88 to 0 of 89.

arXivDOICode

The failure has no error code

An agent calls a search API with a filter the endpoint does not support. The endpoint does not reject it. It returns HTTP 200 and a plausible looking body containing nothing, because an unrecognised filter is indistinguishable from a filter that matched no rows.

Every retry policy, every circuit breaker and every try/except in the codebase is aimed at a failure that announces itself. This one does not announce itself.

What the audit found

Across 721,320 parameters in 2,501 OpenAPI documents: 7.5% declare an enumeration, 15.2% declare any machine-checkable constraint at all, and 40.1% state at least one constraint in prose that the schema does not encode.

So the information needed to call most APIs correctly exists, is written down, and is in the wrong place. It is in the description, where a model reads it as prose, rather than in the schema, where a client can check it.

Then the live experiment. 219 schema-derived perturbations against endpoints from 27 vendors: machine-checkable constraints produced an honest error in 111 of 111 cases. Prose-only constraints failed silently in 44 of 61. The authors report p = 2e-13 and make the point that matters for anyone picking a vendor: constraint form predicted honesty, vendor identity did not.

What the models did about it

Twelve models, eight families. A vocabulary the description merely exemplified rather than enumerated was missed on 88 of 88 attempts. The same vocabulary written out in full was used correctly 88 to 91% of the time.

In the full agent loop, models detected the resulting silent failure in 12% of cases, repaired it in 0%, asserted a false negative to the user in 41%, and invented a figure in 12%.

Zero repairs is the number to sit with. Detection is rare, and recovery never happened once, because there is nothing in the loop to recover from. The model was never told anything went wrong.

The fix is a schema edit, not a model upgrade

Promoting the vocabulary into the schema moved the failure from 88 of 88 to 0 of 89.

That is the practical takeaway and it inverts the usual instinct. The team response to an agent getting an API wrong is normally to try a stronger model or write a longer prompt. Here the entire failure was one line of schema, and no amount of model quality would have found a constraint the schema never stated.

What we take from it

When we wire an agent to a client's internal API, the first job is not the prompt. It is reading the spec and moving every prose constraint into the schema where it can be validated. Where we do not control the API, we add the validation on our side of the call, because nobody upstream is going to do it, and the agent will never notice on its own.

BibTeX

@article{li2026silentprobe,
  author    = {Zongrong Li and Shengkun Ye and Feiyou Guo and Zuoyou Dang},
  title     = {SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools},
  year      = {2026},
  eprint    = {2609.00035},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2609.00035}
}

Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.

2026

Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead

The desk already carries SWE-bench from 2023, and this is the paper you should read immediately after it. A benchmark that can no longer separate its leaders has not stopped being useful, it has changed jobs: it is a regression floor now, not a ranking. The 29.8-point within-model scaffold spread is the number to remember, because it is larger than the whole spread across the top thirty entries. The harness you wrap around a model moves the score more than the model does.

2026

BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

Every RAG proposal we are handed assumes dense retrieval is the default and everything else is a downgrade. This is the study that says it depends on corpus size, and that above roughly 10 million tokens a lexical index built for nothing beats an agent that reasons its way through the corpus. It also costs 39 times fewer query tokens at the point where the agent is still winning.