Adding a reference answer to the prompt flips an LLM judge's correct-or-incorrect verdict by as much as 85% in some experimental settings, according to a new arXiv preprint from researchers at the University of Potsdam and Canada's National Research Council [S1]. That number exposes a flaw in one of the most common ways AI companies evaluate their models. If your automated grader is handing passing marks to wrong answers, every leaderboard it feeds might be built on sand.

The practice of using one language model to grade another, known as LLM-as-a-judge, has spread fast. It is cheaper than hiring human raters, scales to thousands of evaluations, and powers the benchmark scores that companies cite when they claim a model is state of the art. LLM-based verifiers have already hit 86.5% accuracy on Terminal-Bench V2, a coding benchmark. But that benchmark has reference answers. Many real-world deployments do not.

That gap is what Chalamalasetti Kranti of the University of Potsdam and Sowmya Vajjala of the National Research Council in Ottawa set out to measure [P2]. Their preprint, posted on arXiv on 15 July and categorised under cs.AI and cs.LG, has not been peer-reviewed [S1]. The findings are provisional but pointed.

How the study worked

The authors ran a two-stage pipeline across experiments covering three languages [S1].

The first stage, calibration, tested whether the judge model actually understood the task it was grading. The second stage, sensitivity, measured how the judge's performance changed when a reference answer was present or absent, and where in the prompt it sat [S1].

The design matters because most real-world deployments of LLM-as-a-judge skip both steps. A company picks a strong model, points it at a pile of responses, and trusts the scores.

The generosity problem

Without a reference answer, the judge models tended to over-credit incorrect answers [S1]. They marked wrong responses as correct more often than they should have.

When the researchers added reference answer information to the prompt, the judge's decisions flipped by as much as 85% in some settings [S1]. That is not a gentle nudge. It is the difference between a model that looks competent and one that looks broken.

A comparison with a subset of human annotations showed that the reference-driven changes generally aligned with human judgment [S1]. The judges were not being arbitrary. They were systematically too lenient without the answer key, and the reference answer corrected that bias in a direction humans agreed with.

What it means

The core finding is simple: LLM judges are unreliable graders when they have no ground truth to check against. They give wrong answers the benefit of the doubt.

This matters because reference-free evaluation is everywhere. When a company releases a chatbot and reports that a strong model judges prefer their output 72% of the time, that comparison often has no reference answer. The judge is picking the response it likes better, not the one that is correct. If the judge systematically over-credits plausible-sounding but wrong answers, the leaderboard rewards models that sound good rather than models that are good.

The authors recommend calibrating LLM judges with a reference-aware sample before deploying them in reference-free settings [S1]. Their methodology, they say, provides a blueprint for other researchers and practitioners to do this for their own tasks [S1].

This is not the first crack in the LLM-as-a-judge foundation. A separate preprint from researchers at the Institute of Science Tokyo found that training a model against its own reference-free judgments can produce outputs that are more convincing but not more correct [P3]. Another study from Kensho Technologies and MIT, titled "No Free Labels," documented the limitations of LLM-as-a-judge without human grounding [P4]. The Potsdam-Ottawa paper adds a concrete number to the warning.

What it means for business

A two-person AI consultancy running automated evaluations on client deliverables is the most exposed. If the firm uses an LLM to grade open-ended responses, support tickets, or creative drafts without a reference answer, the grading is likely inflated. Wrong answers are getting passing marks.

The fix is practical. Before trusting a judge model on reference-free tasks, run a calibration sample with reference answers and compare. The authors' two-stage pipeline, calibration plus sensitivity, is designed for exactly this [S1]. A small agency could run 100 to 200 items with and without reference answers, measure the flip rate, and decide whether the judge is reliable enough for production use.

For larger operators, the cost implications are real. If benchmark scores that inform model selection are inflated by lenient judges, a company might pick the wrong model for a production pipeline. The model that scores highest under a reference-free judge may not be the one that performs best against human-graded ground truth.

What we don't know yet

The 85% flip rate is a maximum observed in specific experimental settings, not an average across all conditions [S1]. The paper does not report a single headline number for how often judges over-credit wrong answers in general.

The human comparison used a subset of annotations, and the authors do not establish how representative that subset is [S1]. A larger human study could reveal cases where the reference-driven changes diverge from human judgment.

The study covers specific judge models and three languages [S1]. Whether the findings generalise to all LLM-as-a-judge deployments, across all languages and task types, remains an open question.

The paper has not been peer-reviewed [S1]. Its findings could change during review.

The next concrete signal to watch: whether the authors release their calibration dataset and code, which would let practitioners run the same two-stage pipeline on their own tasks. The methodology is described in the paper, but a public release would lower the barrier from blueprint to tool.

If this was useful, subscribe for more plain-English analysis of AI research that actually matters.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.