A new study spanning 22 languages has found that prompting large language models to reason in English — even when the question sits in a low-resource language — substantially improves their ability to estimate their own uncertainty [S1]. The finding suggests models understand these languages far better than they can generate in them, narrowing the reliability gap between high- and low-resource tongues. But the study also reveals a deeper tension: the best method for detecting when an LLM is wrong flips depending on model size — and the rules that work for small models break at scale.

The study, from researchers at INSAIT in Sofia and Amazon [P2], was posted as an arXiv preprint on July 8 and has not yet been peer-reviewed [S1]. It is the first large-scale evaluation of uncertainty estimation — or UE, the science of getting a model to say how sure it is — across 22 languages spanning high-, mid-, and low-resource settings [S1]. The researchers used two human-curated Q&A datasets and compared nine different methods, some "open-box" (where you can inspect the model's internal probability calculations) and some "closed-box" (where you only get the model's own verbalised confidence), across different model sizes and architectures [S1].

The authors deliberately avoided two popular evaluation shortcuts: LLM-as-a-judge — using one model to grade another — and embedding-based scoring, which compares text similarity. Both, they argue, introduce evaluation noise that can mask the real signal [S1]. Instead, they elicited long-form reasoning from the models and measured uncertainty directly from that reasoning process.

The English-reasoning trick

Here is the finding that matters most. When models were prompted to reason in English while answering questions posed in low-resource languages, their uncertainty estimation performance improved substantially — enough to close the gap with high-resource languages [S1].

That is counterintuitive. You might expect a model to reason best in the same language it was asked in. But the data says otherwise: generation language matters more than question language for uncertainty estimation [S1].

The authors' interpretation is striking. They suggest that comprehension of low-resource languages is largely intact — the models understand what is being asked. The reliability bottleneck lies not in understanding but in generation: the models struggle to produce well-calibrated reasoning in languages they have seen less training data for [S1].

The model-size flip

The second finding complicates the picture. The best uncertainty estimation method depends on model scale:

  • At smaller model scales, open-box probability-based methods — which peek inside the model's internal probability distributions — outperform alternatives [S1].
  • At larger model scales, closed-box self-verbalised uncertainty — simply asking the model to state its own confidence — becomes superior [S1].

This means the toolkit for building reliable multilingual AI shifts as you scale up. What works for a 7-billion-parameter model will not necessarily work for a 70-billion-parameter one.

The paper also provides guidance on threshold selection for selective prediction — the practice of letting a model abstain from answering when its confidence is low — offering calibration advice for multilingual settings [S1].

What it means

Uncertainty estimation is the difference between a model that says "I don't know" and one that hallucinates with confidence. For any organisation deploying LLMs in production — especially across languages — knowing when to trust a model's output is the single hardest problem.

This study suggests a practical, near-zero-cost intervention: if you are running a multilingual system, prompt the model to do its internal reasoning in English, even if the user's question is in another language. The model likely understands the question fine; it just reasons more reliably in the language it was most trained on.

The model-size finding matters for architecture decisions. Smaller open-weight models — the kind a two-person firm might self-host — benefit from open-box methods that require access to internal probabilities. Larger proprietary models, accessed via API, may deliver better uncertainty signals simply by being asked to verbalise their confidence. The approach you choose should match the model you are actually running.

Reasoning gaps in LLMs are an active area of scrutiny, and this study adds a new dimension: not just whether models reason correctly, but whether they know when they are wrong.

What it means for business

For a suburban agency deploying chatbots in multiple languages, the English-reasoning trick is a prompt-engineering change, not an infrastructure one. If your bot serves customers in Vietnamese and Arabic, instructing the model to reason internally in English before responding in the customer's language could improve the system's ability to flag low-confidence answers — the ones most likely to be wrong.

For developers building selective-prediction systems — where the model abstains rather than guessing — the threshold-selection guidance offers a starting point for calibrating that abstention cutoff across languages. A single threshold will not work equally well for all languages; the paper's analysis suggests language-specific calibration matters [S1].

For teams choosing between open-box and closed-box methods, the decision now has a clear empirical anchor: match the method to the model size. A small self-hosted model should use probability-based uncertainty methods; a large API-based model may perform better with self-verbalised confidence.

What we don't know yet

This is a preprint, not yet peer-reviewed [S1]. The findings may change upon review or replication.

All substantive findings come from a single source with no independent corroboration. The authors' claim that this is the "first" such large-scale evaluation is unverified in the source material.

The abstract provides directional findings — "substantially improves," "closes the gap" — without releasing specific quantitative thresholds, confidence intervals, or accuracy figures. Without those numbers, it is hard to judge the practical magnitude of the improvement.

The study is limited to multiple-choice question answering. Whether the English-reasoning trick generalises to open-ended generation, summarisation, or other task types remains untested.

The authors' conclusion that comprehension is intact while generation is the bottleneck is interpretive — suggestive, but not independently validated. It is a hypothesis that fits the data, not a proven mechanism.

The next concrete event to watch: peer review and any code release. A related project, TokUR, appeared at ICLR 2026 [P3], suggesting the field is moving fast — but this specific paper's claims need replication before they are ready to build on.

If this kind of plain-English decode is why you subscribe, there is plenty more on the way.

Sources

  • [S1] arXiv preprint: "Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs," July 8, 2026.
  • [P2] arXiv HTML version of the same paper (author affiliations: INSAIT, Sofia; Amazon).
  • [P3] Wang-ML-Lab/TokUR, GitHub repository, ICLR 2026.

Sources

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.