PublishedCurated

BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models

Nandan Thakur, Nils Reimers, Andreas Ruckl, Abhishek Srivastava, Iryna Gurevych

Advances in Neural Information Processing Systems 34 (NeurIPS 2021), Datasets and Benchmarks Track17 April 2021

Abstract

A collection of retrieval datasets spanning very different domains and task types, used to test whether a retriever that works on one corpus still works on another without retraining. The headline finding is that sparse lexical baselines remain hard to beat out of domain.

arXiv Code

BibTeX

@inproceedings{thakur2021beir,
  author    = {Nandan Thakur and Nils Reimers and Andreas Ruckl and Abhishek Srivastava and Iryna Gurevych},
  title     = {BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models},
  booktitle = {Advances in Neural Information Processing Systems 34 (NeurIPS 2021), Datasets and Benchmarks Track},
  year      = {2021},
  eprint    = {2104.08663},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2104.08663}
}

Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.

2026

BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

Every RAG proposal we are handed assumes dense retrieval is the default and everything else is a downgrade. This is the study that says it depends on corpus size, and that above roughly 10 million tokens a lexical index built for nothing beats an agent that reasons its way through the corpus. It also costs 39 times fewer query tokens at the point where the agent is still winning.

2023

Lost in the Middle: How Language Models Use Long Contexts

The single most useful result for anyone shipping retrieval. A bigger context window does not fix relevance: if the right chunk lands in the middle of twenty, it may as well not be there. Re-rank and put the best passage last.