Research desk

The papers behind how we build

An annotated reading list. Every entry is credited to its original authors, with a note on what it changes once the idea meets a real system.

22 entries · RSS feed

All entries

Newest Oldest A to Z

All Foundations3 Retrieval & RAG4 Reasoning5 Agents & Tool Use6 Adaptation3 Alignment3 Evaluation5

Any 20239 20226 20212 20203 20181 20171

5 entries

2023ICLR 2024PublishedCurated
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez et al.
A benchmark built from real issue reports and pull requests across popular Python repositories. A model is handed a codebase and a GitHub issue and has to produce a patch that makes the repository test suite pass, which is a far harder target than generating plausible-looking code.
Evaluation Agents & Tool Use
arXiv
2023TACL 2024PublishedCurated
Lost in the Middle: How Language Models Use Long Contexts
Liu et al.
Language models attend strongly to the beginning and end of a long context and weakly to the middle. The authors show accuracy degrading in a characteristic U-shape as the position of the relevant passage moves through the input, even in models explicitly built for long contexts.
Retrieval & RAG Evaluation
arXiv
2023NeurIPS 2023PublishedCurated
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng et al.
An examination of whether a strong language model can stand in for a human rater on open-ended questions. The paper introduces MT-Bench and Chatbot Arena, then measures how often model judgements agree with human preference and where they systematically go wrong.
Evaluation
arXiv
2022PreprintCurated
Holistic Evaluation of Language Models
Liang et al.
Argues that accuracy alone is a misleading summary of a language model, and evaluates a set of models across many scenarios and many metrics at once, including calibration, robustness, fairness and efficiency, so the trade-offs stay visible.
Evaluation
arXiv
2021NeurIPS 2021PublishedCurated
BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
Thakur et al.
A collection of retrieval datasets spanning very different domains and task types, used to test whether a retriever that works on one corpus still works on another without retraining. The headline finding is that sparse lexical baselines remain hard to beat out of domain.
Retrieval & RAG Evaluation
arXiv

Browse by topic

Foundations

The architectures and pre-training results everything else stands on.

3 entries

Retrieval & RAG

Grounding a model in documents you actually own.

4 entries

Reasoning

Getting a model to show its work, and checking the work.

5 entries

Agents & Tool Use

Models that call tools, hold state, and finish multi-step jobs.

6 entries

Adaptation

Changing what a model does without retraining all of it.

3 entries

Alignment

Steering outputs toward what a person would accept.

3 entries

Evaluation

How you know it works before a customer finds out it does not.

5 entries

Loading research entries

Research desk

The papers behind how we build

An annotated reading list. Every entry is credited to its original authors, with a note on what it changes once the idea meets a real system.

22 entries · RSS feed

All entries

Newest Oldest A to Z

All Foundations3 Retrieval & RAG4 Reasoning5 Agents & Tool Use6 Adaptation3 Alignment3 Evaluation5

Any 20239 20226 20212 20203 20181 20171

5 entries

2023ICLR 2024PublishedCurated
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez et al.
A benchmark built from real issue reports and pull requests across popular Python repositories. A model is handed a codebase and a GitHub issue and has to produce a patch that makes the repository test suite pass, which is a far harder target than generating plausible-looking code.
Evaluation Agents & Tool Use
arXiv
2023TACL 2024PublishedCurated
Lost in the Middle: How Language Models Use Long Contexts
Liu et al.
Language models attend strongly to the beginning and end of a long context and weakly to the middle. The authors show accuracy degrading in a characteristic U-shape as the position of the relevant passage moves through the input, even in models explicitly built for long contexts.
Retrieval & RAG Evaluation
arXiv
2023NeurIPS 2023PublishedCurated
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng et al.
An examination of whether a strong language model can stand in for a human rater on open-ended questions. The paper introduces MT-Bench and Chatbot Arena, then measures how often model judgements agree with human preference and where they systematically go wrong.
Evaluation
arXiv
2022PreprintCurated
Holistic Evaluation of Language Models
Liang et al.
Argues that accuracy alone is a misleading summary of a language model, and evaluates a set of models across many scenarios and many metrics at once, including calibration, robustness, fairness and efficiency, so the trade-offs stay visible.
Evaluation
arXiv
2021NeurIPS 2021PublishedCurated
BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
Thakur et al.
A collection of retrieval datasets spanning very different domains and task types, used to test whether a retriever that works on one corpus still works on another without retraining. The headline finding is that sparse lexical baselines remain hard to beat out of domain.
Retrieval & RAG Evaluation
arXiv