A collection of retrieval datasets spanning very different domains and task types, used to test whether a retriever that works on one corpus still works on another without retraining. The headline finding is that sparse lexical baselines remain hard to beat out of domain.
@inproceedings{thakur2021beir,
author = {Nandan Thakur and Nils Reimers and Andreas Ruckl and Abhishek Srivastava and Iryna Gurevych},
title = {BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models},
booktitle = {Advances in Neural Information Processing Systems 34 (NeurIPS 2021), Datasets and Benchmarks Track},
year = {2021},
eprint = {2104.08663},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2104.08663}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2023
The single most useful result for anyone shipping retrieval. A bigger context window does not fix relevance: if the right chunk lands in the middle of twenty, it may as well not be there. Re-rank and put the best passage last.
2023
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".
2023
Model-graded evaluation is how you get past "it looks fine". The paper is equally clear about the failure modes, so read the limitations section before you wire an LLM judge into a release gate.