Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Abstract

An examination of whether a strong language model can stand in for a human rater on open-ended questions. The paper introduces MT-Bench and Chatbot Arena, then measures how often model judgements agree with human preference and where they systematically go wrong.

@inproceedings{zheng2023judging,
  author    = {Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric P. Xing and Joseph E. Gonzalez and Ion Stoica},
  title     = {Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena},
  booktitle = {Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track},
  year      = {2023},
  eprint    = {2306.05685},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2306.05685}
}