The papers behind how we build

All entries

11 entries

1. 2026Published

   A 200 Is Not Proof: verifying a deploy you cannot look at

   Ejentic Research Desk

   A deploy that returns HTTP 200 has told you nothing. Three incidents in one project, each passing every status check while shipping the wrong thing.

   Evaluation
2. 2026Published

   Deleting the ROI Calculator

   Ejentic Research Desk

   We removed a working feature because it hardcoded a 75% improvement rate that nobody had measured. It converted well. That is exactly why it had to go.

   Evaluation
3. 2026AI4GOOD and EIML workshops, I…PublishedCurated

   Authority, Truth, and Citation Bias: A Large-Scale Multi-Domain Benchmark for Studying Epistemic Susceptibility in Large Language Models

   Khurana et al.

   Introduces AuthorityBench, a 220,564-prompt multi-domain benchmark that isolates how citation-based authority signals influence epistemic behaviour in LLMs. It uses a fully balanced 2x2 factorial design crossing claim veracity with citation veracity across four domains, with controlled variation over 40 prompt templates, four venue prestige tiers and a country-coded author name dataset. Evaluating seven models, the authors find that citation presence, whether real or fabricated, consistently increases hallucination rates relative to a no-citation baseline, with the effect strongest when fabricated citations accompany true claims.

   EvaluationAlignment

   arXiv
4. 2026PreprintCurated

   IH-Benchmark: A Conflict-Centered Benchmark for Instruction-Hierarchy Robustness in LLM Applications

   McCauley et al.

   A benchmark for what happens when a model receives conflicting instructions from different priority levels, built from a human-authored taxonomy of 44 constraint families across generic, health, finance, retail and coding settings. Across 37 evaluated models, hierarchy compliance ranges from 98.2% to 20.5%. The authors report that strong system-over-user compliance is not a reliable proxy for user-over-tool robustness: several models hold system constraints under direct user conflict but degrade sharply when the conflicting instruction arrives inside a tool output.

   EvaluationAlignment

   arXiv
5. 2026PreprintCurated

   SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools

   Li et al.

   An audit of what happens when an agent calls a production API that answers 200 OK with a parsable body while having silently failed. Across 721,320 parameters in 2,501 independently published OpenAPI documents, 7.5% declare an enumeration and 15.2% declare any machine-checkable constraint at all, while 40.1% of documents state at least one constraint in prose their schema does not encode. Executing 219 schema-derived perturbations against live endpoints from 27 vendors, the authors find that constraint form rather than vendor identity predicts honesty: machine-checkable constraints produced an honest error in 111 of 111 cases, prose-only constraints failed silently in 44 of 61 (p = 2e-13). Twelve models across eight families then met these endpoints: a vocabulary the description merely exemplified was missed on 88 of 88 attempts, while vocabularies written out in full were used correctly 88 to 91% of the time. Running the full agent loop, models detected the silent failure in 12% of cases, repaired it in 0%, asserted a false negative to the user in 41%, and invented a figure in 12%. Promoting the vocabulary into the schema removes the failure, from 88 of 88 to 0 of 89.

   Agents & Tool UseEvaluation

   arXiv
6. 2026PreprintCurated

   BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

   Wang et al.

   A controlled scaling study of retrieval-augmented generation paradigms, usually evaluated on different benchmarks at one corpus size. Corpus size is varied along 28 strictly nested tiers spanning roughly 450-fold while questions and a fixed bedrock of relevant and adversarial documents are held constant, under one reader model and one judging protocol. The result is a scale-dependent crossover rather than an unconditional winner. File-System Agent leads at the smallest shared tiers but costs 39 times more query tokens at the bedrock and becomes less effective as the search space grows. Around 10 million corpus tokens BM25 overtakes it and leads at every larger shared tier, with a margin approaching 20 points at full scale, and anchors the low-cost end of the Pareto frontier without any LLM-based construction. Dense retrieval remains efficient but less accurate; graph-based RAG encounters construction walls before deployment scale and its scalable variants stay below BM25 at shared tiers. The authors conclude that corpus growth increasingly favours global candidate ranking, with agentic reasoning working best after ranked discovery rather than in place of it.

   Retrieval & RAGEvaluation

   arXiv
7. 2023ICLR 2024PublishedCurated

   SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

   Jimenez et al.

   A benchmark built from real issue reports and pull requests across popular Python repositories. A model is handed a codebase and a GitHub issue and has to produce a patch that makes the repository test suite pass, which is a far harder target than generating plausible-looking code.

   EvaluationAgents & Tool Use

   arXiv
8. 2023TACL 2024PublishedCurated

   Lost in the Middle: How Language Models Use Long Contexts

   Liu et al.

   Language models attend strongly to the beginning and end of a long context and weakly to the middle. The authors show accuracy degrading in a characteristic U-shape as the position of the relevant passage moves through the input, even in models explicitly built for long contexts.

   Retrieval & RAGEvaluation

   arXiv
9. 2023NeurIPS 2023PublishedCurated

   Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

   Zheng et al.

   An examination of whether a strong language model can stand in for a human rater on open-ended questions. The paper introduces MT-Bench and Chatbot Arena, then measures how often model judgements agree with human preference and where they systematically go wrong.

   Evaluation

   arXiv
10. 2022PreprintCurated

    Holistic Evaluation of Language Models

    Liang et al.

    Argues that accuracy alone is a misleading summary of a language model, and evaluates a set of models across many scenarios and many metrics at once, including calibration, robustness, fairness and efficiency, so the trade-offs stay visible.

    Evaluation

    arXiv
11. 2021NeurIPS 2021PublishedCurated

    BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models

    Thakur et al.

    A collection of retrieval datasets spanning very different domains and task types, used to test whether a retriever that works on one corpus still works on another without retraining. The headline finding is that sparse lexical baselines remain hard to beat out of domain.

    Retrieval & RAGEvaluation

    arXiv

Browse by topic