A controlled scaling study of retrieval-augmented generation paradigms, usually evaluated on different benchmarks at one corpus size. Corpus size is varied along 28 strictly nested tiers spanning roughly 450-fold while questions and a fixed bedrock of relevant and adversarial documents are held constant, under one reader model and one judging protocol. The result is a scale-dependent crossover rather than an unconditional winner. File-System Agent leads at the smallest shared tiers but costs 39 times more query tokens at the bedrock and becomes less effective as the search space grows. Around 10 million corpus tokens BM25 overtakes it and leads at every larger shared tier, with a margin approaching 20 points at full scale, and anchors the low-cost end of the Pareto frontier without any LLM-based construction. Dense retrieval remains efficient but less accurate; graph-based RAG encounters construction walls before deployment scale and its scalable variants stay below BM25 at shared tiers. The authors conclude that corpus growth increasingly favours global candidate ranking, with agentic reasoning working best after ranked discovery rather than in place of it.
RAG paradigms are normally benchmarked against each other once, at one corpus size, on different datasets. That makes the results impossible to reconcile and very easy to over-read.
This study holds the questions, the reader model, the judging protocol and a fixed bedrock of relevant and adversarial documents constant, and varies only corpus size, across 28 strictly nested tiers spanning roughly 450-fold. That design is the contribution. It turns "which RAG is best" into "which RAG is best here".
At the smallest shared tiers the File-System Agent leads. It also spends 39 times more query tokens at the bedrock, and its advantage erodes as the search space grows.
Around 10 million corpus tokens, BM25 overtakes it and leads at every larger shared tier, by a margin approaching 20 points at full scale. BM25 also sits on the low-cost end of the Pareto frontier without any LLM-based construction step, which means no index build to pay for and nothing to rebuild when the corpus changes.
Dense retrieval stays efficient but less accurate throughout. Graph-based RAG hits construction walls before reaching deployment scale, and the variants that do scale remain below BM25 at the tiers where both can be measured.
The authors are not saying agentic retrieval is bad. They are saying it is in the wrong position. Corpus growth favours global candidate ranking, and agentic reasoning works best after ranked discovery rather than instead of it.
That is a pipeline shape, not a verdict on a component. Rank first, narrow the search space cheaply, then let the agent reason over a shortlist it can actually afford to read. The agent that explores the whole corpus sequentially runs out of budget before it runs out of corpus.
Ask how big the corpus will be in eighteen months, not today. Under about 10 million tokens the answer changes. Build the lexical index first, because it is close to free and it is the baseline every fancier approach has to beat. And budget the agent's token spend as a first-class cost, because at 39 times the query tokens the accuracy difference is not the expensive part.
@article{wang2026wins,
author = {Pengyu Wang and Benfeng Xu and Shaohan Wang and Mingxuan Du and Xin Zeng and Huarui Wu and Lei Zhang and Licheng Zhang},
title = {BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms},
year = {2026},
eprint = {2607.26497},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.26497}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2023
The single most useful result for anyone shipping retrieval. A bigger context window does not fix relevance: if the right chunk lands in the middle of twenty, it may as well not be there. Re-rank and put the best passage last.
2021
Run this before you commit to a retriever. A model that wins on one corpus and collapses on the next is the most common way a search feature disappoints after launch.
2026
The desk already carries SWE-bench from 2023, and this is the paper you should read immediately after it. A benchmark that can no longer separate its leaders has not stopped being useful, it has changed jobs: it is a regression floor now, not a ranking. The 29.8-point within-model scaffold spread is the number to remember, because it is larger than the whole spread across the top thirty entries. The harness you wrap around a model moves the score more than the model does.