Introduces DEEPSYNTH, a benchmark of 120 tasks across 7 domains and data sources covering 67 countries, built by a multi-stage pipeline in which annotators collect official data sources, create hypotheses, perform manual analysis and design tasks with verifiable answers. It targets realistic, time-consuming problems that combine information gathering, synthesis and structured reasoning rather than fact retrieval. The authors report that 11 state-of-the-art LLMs and deep research agents achieve a maximum F1 of 8.97 and 17.5 on an LLM-judge metric, and that current agents struggle with hallucination and reasoning over large information spaces.
A maximum F1 of 8.97. Not on adversarial inputs, not on an out-of-distribution stress set — on the benchmark as designed, by eleven state-of-the-art models and deep research agents.
Deep research is the most demo-friendly capability in the current generation because a single successful run looks extraordinary. This is what happens when you ask for 120 of them with verifiable answers.
Tasks were not scraped. Annotators gathered official data sources, formed hypotheses, did the analysis by hand, and only then wrote a task with a checkable answer. That ordering matters: it means the task cannot be solved by finding the sentence that contains the answer, because no such sentence exists anywhere. The model has to do the analysis.
Scoping follows from this. Multi-source synthesis that a human expert needs an afternoon for is not a solved problem, and it should not be sold as one. The honest version is assistance with the gathering and a human on the synthesis — which is still most of the value, and does not require the model to be right about something nobody can check.
@inproceedings{paul2026benchmark,
author = {Debjit Paul and Daniel Murphy and Milan Gritta and Ronald Cardenas and Victor Prokhorov and Lena Sophia Bolliger and Aysim Toker and Roy Miles and others},
title = {A Benchmark for Deep Information Synthesis},
booktitle = {International Conference on Learning Representations (ICLR) 2026},
year = {2026},
eprint = {2602.21143},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2602.21143}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2026
Read this before you let an agent call anything a customer will see the output of. The failure mode is not a crash, it is an API returning 200 with a body that means "I did not understand you". There is no exception to catch and no status code to branch on. The result we quote most often is the last one: the fix was one line of schema, not a better model.
2023
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".
2026
When a provider is bimodal, the mean is a fiction. Report the modes and the condition that separates them, or you will architect against noise.