An arXiv preprint posted this week shows that large language models carry far richer confidence signals in their internal representations than they express in words — and that a lightweight predictor can extract those signals before the model finishes answering [S1]. The finding, from an un-peer-reviewed paper, implies models may "know" more about whether they're about to be wrong than they let on. If it holds up, it could reshape how every AI-powered system decides when to trust its own output — but the method carries a constraint that determines who can actually deploy it.

The confidence problem

Every LLM deployment faces the same uncomfortable question: when the model says something, how do you know it knows? Today's models produce fluent, confident-sounding answers that are flatly wrong — the well-documented hallucination problem. Verbalised confidence, where you simply ask the model how sure it is, has proven unreliable. Models tend toward overconfidence, and their stated certainty doesn't track actual accuracy.

Related work from Johns Hopkins, now under review at ICLR 2026, has been tackling this from a different angle: teaching models to express semantic confidence through uncertainty distillation [P4]. The new preprint takes a temporal approach instead, asking a deceptively simple question: does when you measure confidence matter as much as how?

Before vs after: the temporal gap

The authors borrow two terms from cognitive psychology. "Feeling-of-Knowing" (FOK) is the pre-solution gut sense — the model's internal state before it has generated an answer. "Judgement-of-Learning" (JOL) is the post-solution assessment — confidence measured after the answer exists [S1].

Their finding is consistent across both frontier and open-source models: post-solution confidence is better calibrated (it tracks actual correctness more faithfully) and more discriminative (it separates right from wrong more cleanly) than pre-solution confidence [S1]. This makes intuitive sense — it's easier to judge whether you got something right after you've done it than to predict whether you will.

But here's the catch. Post-solution confidence is useless for prevention. If you need to know the model is about to hallucinate before it does, a confidence signal that only works after the answer is complete arrives too late.

What the model knows but doesn't say

The paper's most striking finding sits in the hidden representations — the internal numerical states the model computes at each step. The authors trained linear probes (simple statistical classifiers) on these hidden states and found they recover substantially richer confidence-related information than the models themselves verbalise [S1].

In plain terms: the model's internal math contains far better signals about whether it's on the right track than anything it says out loud. The confidence is there. The model just doesn't surface it.

Future confidence distillation

This is where the method gets clever. The authors introduce "future confidence distillation" [S1]. The architecture works in two stages:

  • A teacher probe, trained on post-solution hidden states, produces high-quality confidence estimates — the gold standard.
  • A student predictor is then trained on pre-solution hidden states to match the teacher's output.

At inference time, only the student runs. It reads the model's early internal states and predicts what the teacher would have said, before the answer is generated.

The authors report that these distilled predictors recover much of the calibration benefit of post-solution confidence while needing only pre-solution representations at inference time [S1]. They're also highly sample-efficient and transfer across datasets within the same domain [S1].

The core insight: confidence-related information evolves throughout the answering process and can be anticipated before generation is complete [S1]. The model's early states already contain traces of whether it will produce a good answer.

What it means

This matters because the gap between "can this model tell you it's uncertain?" and "can this model's internals reveal it's uncertain?" is the gap between a chatbot and a reliable system.

Think of it like a doctor who won't admit doubt to a patient but whose vital signs — heart rate, pupil dilation — betray the uncertainty. The hidden representations are the model's vital signs. A linear probe is the stethoscope.

For regular users, this could eventually mean AI systems that flag their own likely errors in real time — not after the fact, but before the answer lands on your screen. For developers, it offers a path to confidence-aware routing: send high-confidence queries to a fast, cheap model and low-confidence ones to something more expensive, or to a human.

The FOK-versus-JOL distinction has a practical edge too. If you're building a system that needs to decide whether to act on a model's output, you need pre-solution confidence. Post-solution confidence tells you the model was probably wrong — after it already told your user the wrong thing.

What it means for business

For a two-person legal tech startup using an LLM to draft contract summaries, the cost of a confident hallucination is a lawsuit. Today they rely on post-hoc review — a human reads every output. If distilled confidence predictors become production-ready, that startup could flag low-confidence passages automatically, focusing human review where it matters.

For a suburban real estate agency using AI to generate property descriptions from listing data, the stakes are lower but the volume is higher. A confidence signal that works before generation completes could let them auto-publish high-confidence descriptions and hold back low-confidence ones for a quick human glance — cutting review time without increasing risk.

For any business routing customer queries through an LLM, the appeal is obvious: a pre-solution confidence gate could route uncertain queries to a human agent before the model sends a wrong answer to a customer. The inference cost — the cost of actually running the model — stays the same, but the downstream cost of errors drops.

None of this is available today. The paper describes a method, not a product. But the architecture is simple enough — linear probes on hidden states — that an engineering team could prototype it on an open-source model without waiting for a vendor.

What we don't know yet

This is a single-source, un-peer-reviewed preprint [S1]. Every substantive claim is the authors' report, not established consensus. Several questions remain open:

  • Cross-domain transfer: The authors show distilled predictors transfer across datasets within the same domain [S1], but they don't claim cross-domain generalisation. A probe trained on math problems may not work on medical questions.
  • Frontier model access: The method requires access to hidden representations. For closed models accessed via API — where you only get text in and text out — this is a non-starter unless the provider exposes internal states. The paper tests both frontier and open-source models, but deployment on closed systems depends on vendor cooperation.
  • Replication: No independent group has confirmed these findings. The calibration improvements, sample efficiency, and transfer claims all need external validation.
  • Production viability: Linear probes are lightweight, but the paper doesn't report latency overhead, failure modes under distribution shift, or performance on adversarial inputs.

The next concrete signal to watch: whether this paper surfaces in a peer-reviewed venue with reviewer comments, and whether any open-source model provider incorporates confidence probing into their releases. Related work on uncertainty distillation from Johns Hopkins is already under review at ICLR 2026 [P4] — if both threads converge, confidence-aware LLMs could move from preprint to practice faster than expected.

If your work depends on knowing when an AI is guessing, this is the research to track. Subscribe to keep reading as we follow it through review.

Sources: [S1] arXiv preprint, "Future Confidence Distillation in Large Language Models" (arXiv:2607.07626v1, cs.AI, cs.LG) · [P4] OpenReview, "Uncertainty Distillation: Teaching Language Models to Express Semantic Confidence," submitted to ICLR 2026

Sources

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.