PublishedCurated

Lost in the Middle: How Language Models Use Long Contexts

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang

Transactions of the Association for Computational Linguistics (TACL) 20246 July 2023

Abstract

Language models attend strongly to the beginning and end of a long context and weakly to the middle. The authors show accuracy degrading in a characteristic U-shape as the position of the relevant passage moves through the input, even in models explicitly built for long contexts.

arXiv

The result

Accuracy is highest when the relevant passage sits at the very start or the very end of the context, and drops when it is buried in the middle. The shape holds even for models built specifically for long contexts.

Why it matters more than context length

The tempting conclusion from a failing retrieval system is that the context window is too small. Usually every passage is already being passed in and the right one is simply not being read. Growing the window can make this worse, because it puts more distance around the answer.

What we do about it

Retrieve more than you need, re-rank, then place the strongest passage last, where the model is most likely to use it. Measure the position of the passage that actually answered the question, not just whether an answer came out.

BibTeX

@inproceedings{liu2023lost,
  author    = {Nelson F. Liu and Kevin Lin and John Hewitt and Ashwin Paranjape and Michele Bevilacqua and Fabio Petroni and Percy Liang},
  title     = {Lost in the Middle: How Language Models Use Long Contexts},
  booktitle = {Transactions of the Association for Computational Linguistics (TACL) 2024},
  year      = {2023},
  eprint    = {2307.03172},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2307.03172}
}

Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.

2026

BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

Every RAG proposal we are handed assumes dense retrieval is the default and everything else is a downgrade. This is the study that says it depends on corpus size, and that above roughly 10 million tokens a lexical index built for nothing beats an agent that reasons its way through the corpus. It also costs 39 times fewer query tokens at the point where the agent is still winning.