Four of the most capable LLM agents detect 84 to 88 percent of hidden supply-chain failures, usually within a week of onset [S1]. Yet when judged on the quality of their actions, two of those four score below a policy that ignores symptoms entirely. A new benchmark called STOCKTAKE, posted to arXiv on July 16, exposes why, and the reason should worry anyone building agents for real decisions.

The benchmark that separates seeing from doing

STOCKTAKE runs a simulated 26-week supply-chain replenishment problem [S1]. The agent manages inventory but never observes the true state of the system. Six hidden factor processes, things like supplier delays, demand shifts, or quality drops, drive the real costs. The agent sees only symptoms: stock levels and order arrivals.

This setup is a factored partially observable Markov decision process, or POMDP, a formal way to model decisions where the decision-maker never sees the full picture [S1]. The agent has to infer what is going wrong from indirect signals, then decide what to order.

The clever part is the scoring. STOCKTAKE builds a reference policy, an oracle, by applying a per-factor Bayes filter to the same observation stream that the agent receives [S1]. The oracle has no privileged information. It sees what the agent sees. Every run is then graded on a scale where a symptom-blind base-stock floor gets 0 and the oracle gets 1, yielding what the authors call a skill score [S1]. Score above zero and you are better than ignoring symptoms. Score below zero and you would have been better off not trying to read the situation at all.

What it means

The central finding is what the researchers call the knowing-doing gap. The four tested models, Claude Sonnet 5, GPT-5.4, DeepSeek-V4-Pro, and Grok 4.5, were run across fifty seeds with curated stress profiles [S1]. They detect 84 to 88 percent of hidden failures, typically within a week of onset [S1]. By any measure of perception, they are good at reading the world.

But their skill scores span from 0.62 down to -0.23 [S1]. Two of the four models end below the symptom-blind floor, worse than a policy that simply orders to a fixed level and ignores all signals [S1].

Here is the twist. The two models that score below the floor actually name the hidden factors slightly faster than the two that beat it [S1]. Faster detection does not produce better decisions. The gap has two faces. One is under-response: even when models correctly diagnose a stress week, 34 to 43 percent of those weeks still end in stockout, across every model tested [S1]. The models that score below the floor actually stock out least on diagnosed weeks. They see the problem but do not act forcefully enough [S1].

The other face is over-response: actions whose cost exceeds what they protect [S1]. A model might correctly detect a supplier delay and respond by doubling an order, paying rush fees and holding costs that outweigh the stockout risk it was trying to avoid. The benchmark catches this because it scores the full cost of the agent's decisions, rather than whether it noticed the problem.

Existing evaluations cannot separate these two failure modes [S1]. A benchmark that only checks whether the agent identified the right factor would rate all four models as performing well. STOCKTAKE's contribution is that it evaluates each week's written explanation to derive a detection lag and a knowing-doing rate, so that belief estimation and decision quality are scored independently [S1].

This matters because the AI industry is racing to deploy agents in real decision-making roles. STOCKTAKE adds a failure mode that prior benchmarks miss: the agent knows the right answer but still makes the wrong call.

What it means for business

For a two-person logistics firm or a suburban retail agency experimenting with AI-driven inventory ordering, the takeaway is concrete. An LLM agent can read your dashboards, spot a supplier delay in the data, and write you a confident paragraph explaining what went wrong. That paragraph will probably be correct. The problem is what happens next.

If the agent is also making the ordering decision, STOCKTAKE suggests the action may cost more than the problem it is solving. The model that detects the failure fastest is not the model that manages inventory best. A small operator who trusts the agent's confident diagnosis may not realise the agent is simultaneously over-ordering or under-ordering in ways that a simple fixed-level policy would avoid.

The practical step: separate the perception task from the control task. Let the LLM flag anomalies and explain them. Keep the ordering logic in a tested system, even a simple base-stock policy, until the agent's action quality is measured against a fair baseline. STOCKTAKE gives a template for that measurement.

What we don't know yet

All statistics in this study come from a single arXiv preprint that has not been peer-reviewed or independently replicated [S1]. The results apply only to the specific curated stress profiles and fifty seeds described in the paper. Generalisation to broader supply-chain settings or other agent deployments is untested.

The model names tested, Claude Sonnet 5, GPT-5.4, DeepSeek-V4-Pro, and Grok 4.5, may not correspond to currently released versions, and the paper does not detail how each model was prompted or configured. Prompting strategy, tool-use scaffolding, and system instructions can all shift action quality substantially, and none of that is reported here.

The next event to watch: independent replication on different supply-chain scenarios, and testing with explicit prompting interventions designed to close the knowing-doing gap, for instance forcing the agent to justify the cost of each action before taking it. Until that work appears, STOCKTAKE is a warning shot, not a verdict.

If this kind of evidence-based AI reporting is useful to you, subscribe to keep reading.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.