A new arXiv paper reports that large language models assign lower confidence to logically easier questions 31% of the time S¹. That number sounds small until you realise what it breaks: the assumption that when an AI says "I'm 90% sure," the 90% means anything at all. The paper argues the field's standard test for trustworthy confidence is checking the wrong thing, and that even the most popular fixes leave the real problem untouched.

My read: This is the first paper I've seen that attacks calibration not as a broken tool but as an incomplete one. The 31% figure is striking because it's not about edge cases or adversarial prompts. It's about basic logical structure. If a model is more confident on hard questions than easy ones nearly a third of the time, its confidence scores aren't behaving like probabilities. They're behaving like guesses that happen to average out. I don't buy the broad generalisation yet, because only the abstract is public and the full methodology could narrow the finding considerably. But the core argument, that passing a calibration test doesn't prove your probabilities are coherent, is hard to argue with.

Calibration checks the average, not the structure

Calibration is the field's go-to test for whether an AI model knows its own limits. The idea is simple: if a model says "90% confident" on a hundred questions, it should be right about 90 times. If the numbers line up, the model is well-calibrated and its confidence scores are considered trustworthy.

The new paper, posted to arXiv on 23 July 2026 S¹, says that test is necessary but not sufficient. Calibration can be gamed by what the authors call "trivially incoherent estimators" S¹. A model could be perfectly calibrated on average while being internally inconsistent, assigning high confidence to hard questions and low confidence to easy ones, so long as the errors cancel out across the test set. Calibration also depends on which questions you test with. Change the distribution and a well-calibrated model can start looking unreliable, or vice versa.

Most fundamentally, calibration never asks whether a model's confidence estimates behave like a real probability function. It checks whether the averages match. It doesn't check whether the underlying estimates are consistent with each other.

The 31% problem

Here's where the paper's most concrete finding lands. The authors report that models assign lower confidence to logically easier questions 31% of the time S¹. Think about what that means. If you ask a model "Is 2+2=4?" and it says 55% confident, then asks it a complex reasoning question and it says 90% confident, something is broken in how it estimates its own certainty. Not always. Not even most of the time. But nearly one in three instances is enough to make any downstream system that relies on those scores unreliable.

The authors formalise this problem along three axes: structural coherence (do the confidence scores follow the rules of probability), faithfulness (do they reflect the model's actual beliefs), and usefulness (do they help with downstream tasks) S¹. They turn these into a set of measurements they call the C1 metrics S¹.

When they run widely used confidence estimators through the C1 metrics, the estimators systematically violate the coherence conditions, even though they appear well-calibrated under traditional tests S¹. The paper concludes that calibration is orthogonal to probabilistic validity S¹. In plain terms: you can pass the calibration test and still fail the coherence test, and the reverse holds too.

The paper tests two of the most common interventions for improving model confidence: RLHF (reinforcement learning from human feedback, the training method behind ChatGPT and Claude) and chain-of-thought prompting (asking the model to reason step by step before answering). Both improve the usefulness axis S¹. The models produce confidence scores that are more helpful for downstream tasks.

But neither restores structural coherence S¹. The models still violate the basic rules of probability in their confidence estimates. The paper also reports that common interventions which reduce RMSCE, a standard calibration error metric, leave structural violations unchanged S¹. You can make the averages look better without fixing the underlying inconsistency.

The intuition here is straightforward: neural models can produce answers that look right without the underlying reasoning being sound. Confidence scores have the same problem. They can look calibrated without being coherent.

What to do about it

If you build systems that use LLM confidence scores to make decisions, this paper changes what you should check. Calibration alone is not enough.

Consider a legal tech startup that uses an LLM to scan contracts for risky clauses. The system flags clauses where the model's confidence exceeds a threshold, say 85%, for human review. If the model's confidence is well-calibrated but structurally incoherent, the startup might find that simple, obvious clauses sometimes get flagged at 60% confidence while ambiguous ones sail through at 90%. The calibration test would pass. The contract review would still be unreliable.

The practical step: if you rely on LLM confidence scores, test them for structural coherence in addition to calibration. The paper's C1 metrics offer a starting framework S¹. At minimum, check whether your model's confidence tracks logical difficulty. If it doesn't, treat the scores as rough signals rather than probabilities, and build your thresholds with that uncertainty baked in.

What we don't know yet

This is an arXiv preprint, not a peer-reviewed paper S¹. The 31% figure and other empirical results come from the authors' own experiments and have not been independently replicated. Only the abstract has been reviewed; the full methodology, including which models were tested and on what datasets, is not yet available in the evidence we have.

The claim that calibration is orthogonal to probabilistic validity is the authors' interpretation, and it is a strong one. It could turn out that some estimators or model architectures do better on coherence than the paper's sample suggests. Related work exists, including an ICLR 2025 submission on uncertainty estimation in natural language generation P² and a 2024 critical review of uncertainty in LLMs P⁴, but neither covers the specific C1 framework or the 31% finding.

The next signal: the paper's full text and methodology, which should clarify model scope and whether the C1 metrics generalise beyond the tested estimators. We'll check the coherence claims against the full experimental details when they're available.

If this was useful, subscribe for plain-English analysis of AI research that changes what you build.

Sources: S1 — Rethinking Uncertainty Evaluation in Large Language Models · P2 — openreview.net · P3 — Rethinking Uncertainty Estimation in Natural Language Generation · P4 — Rethinking the Uncertainty: A Critical Review and Analysis in the Era · P5 — Stability-AI/StableLM

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.

How often models assign lower confidence to easier questions