Adi Krishnamoorthy and X.Y. Han ran ten frontier AI models through three shopping benchmarks and found that extra reasoning effort bought confidence, not correctness S¹. The premise behind reasoning models, the systems sold on thinking longer to solve harder problems, takes a direct hit. When the models thought harder, their product picks did not get measurably better. Their certainty that those picks were right did.
My read: The dangerous combination is confidence rising while accuracy stays flat. A model that is wrong but unsure prompts a human to double-check. A model that is wrong and certain lets the purchase go through. I would watch whether this pattern holds outside shopping, because the paper only tests product choice on three narrow benchmarks. The shared-error finding, where multiple models pick the same wrong product, points to a problem in the task or training data, not in any single model's reasoning chain.
What they tested
The authors built three product-choice benchmarks from existing e-commerce research platforms: WebShop, Amazon's SQID, and Wayfair's WANDS S¹. Each task gives a model multimodal product listings, images and text, and asks it to recommend a purchase on behalf of a consumer S¹. Ten frontier models were run through all three at varying levels of thinking effort, the experimental dial that controls how much reasoning a model does before answering S¹.
The concept of dialling reasoning effort up and down is not new. ThinkDial, an open-source framework published in August 2025, gave developers explicit control over how much reasoning a model produces P⁵. The question Krishnamoorthy and Han ask is sharper: does that extra reasoning actually help when the model is picking products?
Confidence up, accuracy flat
Across all three benchmarks, increasing thinking effort did not increase decision quality in a statistically meaningful way S¹. The same increase in effort raised the models' stated confidence at a statistically significant level S¹. Extra thinking buys confidence rather than correctness, the authors conclude S¹.
The gap between confidence and accuracy is a calibration problem. The authors measured how well each model's reported confidence percentage matched its true correctness percentage, and found that most models are not well-calibrated at any effort level S¹. A model can be highly confident and frequently wrong at the same time. Turning up the reasoning widens that gap rather than closing it.
This matters because agent reliability is already under pressure. The pattern echoes what AI email agents showed when they complied with 45% of adaptive phishing attacks: autonomous agents making decisions on behalf of humans can be steered toward bad outcomes. A shopping agent that feels more certain about a wrong purchase is the same failure mode in a commercial wrapper.
The same wrong product
The errors are not random. When one model errs on a task, several others tend to err on that same task S¹. On tasks where multiple models all fail, a large number of them pick the same wrong product S¹.
This shared-error pattern suggests the models are converging on the same bad answer, which points to common training data, common biases in how they read product listings, or tasks where the correct answer is genuinely ambiguous. The standard engineering fix of running multiple models and taking a majority vote would not help here, because the models fail together.
The evaluation gap this exposes is the one Microsoft and Hugging Face tried to close by grading AI agents against databases, not words: agent benchmarks that measure fluency and confidence rather than ground-truth outcomes will miss this failure entirely.
What would have to hold
The paper is an arXiv preprint, not peer-reviewed, and relies on a single source with no independent corroboration S¹. The findings are drawn from three specific product-choice benchmarks and may not generalise to other domains S¹. The abstract reports no statistically meaningful increase in decision quality, which is different from proving zero effect S¹.
The ten models are not named in the abstract, so the specific systems tested cannot be verified from the available evidence. The abstract reports no error bars.
Who feels this first
E-commerce platforms building autonomous shopping agents are the immediate audience. A retailer that lets an AI agent recommend purchases, process returns, or auto-buy on a consumer's behalf needs to know whether spending more on reasoning compute improves the quality of those decisions. This paper says it does not.
The broader signal reaches anyone deploying reasoning models in decision-making roles. More reasoning does not produce uniformly better outcomes, as OpenAI coding agents demonstrated when they accelerated AI research unevenly. The shopping-agent result adds a sharper edge: the model fails to improve and becomes more sure of itself while failing.
The paper is listed as arXiv:2611.22033v1, submitted on November 18, 2026 S¹. No peer review or independent replication has been announced.
Sources: S1 — Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Incre · P2 — Paper page - Thinking Harder Doesn't Lead to Better Choices: Reasoning · P3 — Fcr09/thinking-hard-not-smart · P4 — tgxs002/align_sd · P5 — Paper page - ThinkDial: An Open Recipe for Controlling Reasoning Effor
Related reading
- Microsoft and Hugging Face grade AI agents by database, not words — our technology desk, 2026-10-22
- AI email agents comply with 45% of adaptive phishing attacks — our technology desk, 2026-10-24
- OpenAI coding agents accelerate AI research, unevenly — our technology desk, 2026-11-19
Written from 5 sourced items, 4 of them primary.
