Language models attend strongly to the beginning and end of a long context and weakly to the middle. The authors show accuracy degrading in a characteristic U-shape as the position of the relevant passage moves through the input, even in models explicitly built for long contexts.
Accuracy is highest when the relevant passage sits at the very start or the very end of the context, and drops when it is buried in the middle. The shape holds even for models built specifically for long contexts.
The tempting conclusion from a failing retrieval system is that the context window is too small. Usually every passage is already being passed in and the right one is simply not being read. Growing the window can make this worse, because it puts more distance around the answer.
Retrieve more than you need, re-rank, then place the strongest passage last, where the model is most likely to use it. Measure the position of the passage that actually answered the question, not just whether an answer came out.
@inproceedings{liu2023lost,
author = {Nelson F. Liu and Kevin Lin and John Hewitt and Ashwin Paranjape and Michele Bevilacqua and Fabio Petroni and Percy Liang},
title = {Lost in the Middle: How Language Models Use Long Contexts},
booktitle = {Transactions of the Association for Computational Linguistics (TACL) 2024},
year = {2023},
eprint = {2307.03172},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2307.03172}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2021
Run this before you commit to a retriever. A model that wins on one corpus and collapses on the next is the most common way a search feature disappoints after launch.
2023
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".
2023
Model-graded evaluation is how you get past "it looks fine". The paper is equally clear about the failure modes, so read the limitations section before you wire an LLM judge into a release gate.