Holistic Evaluation of Language Models

Abstract

Argues that accuracy alone is a misleading summary of a language model, and evaluates a set of models across many scenarios and many metrics at once, including calibration, robustness, fairness and efficiency, so the trade-offs stay visible.

@article{liang2022holistic,
  author    = {Percy Liang and Rishi Bommasani and Tony Lee and Dimitris Tsipras and Dilara Soylu and Michihiro Yasunaga and Yian Zhang and Deepak Narayanan and Yuhuai Wu and Ananya Kumar and others},
  title     = {Holistic Evaluation of Language Models},
  year      = {2022},
  eprint    = {2211.09110},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2211.09110}
}