A new black-box audit applied to GPT-4o found that 66% of correctly-solved reasoning problems contained at least one step where the model's conclusion did not actually depend on the premise it claimed to use [S1]. The method, detailed in an arXiv paper announced July 16, works without any access to model weights or internals. What it exposes about how AI models actually reason, and why existing checks miss the problem, raises hard questions for anyone deploying chain-of-thought systems in production.

How the audit works

The technique is called interventional grounding audits, introduced by independent researcher Hironao Nakamura [P4]. The approach works by taking a reasoning problem, replacing one premise's predicate with a novel meaningless token, running the model again, and then comparing whether each step's output changes accordingly [S1].

If the model's reasoning genuinely depends on that premise, the conclusion should shift when the premise changes. If it does not shift, the model is producing a conclusion that ignores the very input it claims to use.

This is a black-box test. No weight inspection, no activation probing. The auditor feeds prompts in and reads outputs out, which means it works on any model accessible through an API, including closed systems like GPT-4o [S1].

The numbers on 50 problems

The audit was tested on 50 problems from ProntoQA, a synthetic multi-hop deductive reasoning benchmark where the correct step-by-step proof is known in advance [S1]. On these problems, the method achieved an F1 score of 0.806 for detecting which reasoning steps genuinely depend on their stated premises [S1].

For a specific category called predicate-determining dependencies, the method reached F1 = 0.885 with 100% recall [S1]. A self-consistency baseline, a common existing approach that checks whether a model gives the same answer across multiple runs, scored just F1 = 0.343. The 95% confidence intervals for the two methods did not overlap [S1].

The most striking finding: 66% of problems GPT-4o solved correctly contained at least one reasoning step that was insensitive to a direct proof-tree dependency [S1]. The authors call this a "right answer, wrong reasoning" signal that passive methods cannot detect [S1].

All of these insensitive steps involved entity-introduction premises, which the authors identify as a known blind spot of their own evaluator [S1]. They disclose this limitation openly.

What it means

Chain-of-thought prompting, where a model is asked to show its work step by step, has become one of the most widely used techniques in AI deployment. The assumption is simple: if the steps look logical and the answer is correct, the reasoning is sound.

This audit challenges that assumption directly. A model can produce a correct final answer while individual reasoning steps ignore the premises they claim to rely on. The logic looks right. The answer is right. But the connection between them is broken.

The predicate substitution method offers a different angle. Instead of checking whether the model's reasoning is consistent across runs, it checks whether the reasoning is causally connected to the inputs. It asks whether the model's conclusion would change if the premise changed, rather than whether the model says the same thing twice. That is a stronger test.

What it means for business

For a two-person AI consultancy building agents that make decisions based on chain-of-thought reasoning, this method offers a way to audit model outputs without needing access to model internals. The audit runs on API outputs alone [S1].

The practical use case is narrow but real. If your agent solves multi-step logical problems, say eligibility checks or compliance reasoning where each step should depend on a specific input, you can run the predicate substitution test to verify that dependency. Steps that fail the audit are producing conclusions that ignore their stated premises, even when the final answer is correct.

The code, raw outputs, audit certificates, and reproduction scripts are publicly available on GitHub [P3], which means a developer can run the audit today on their own prompts. Adapting it beyond the ProntoQA benchmark format requires work the authors flag as a scope limitation [S1].

The cost is modest in API terms: each audit requires re-running the model with substituted predicates, which doubles or triples the calls per problem. For a firm running hundreds of reasoning queries daily, that is a real but manageable overhead.

What we don't know yet

The method has been tested on a single model, GPT-4o, and a single benchmark, ProntoQA, which is synthetic and designed for deductive reasoning [S1]. The authors explicitly discuss scope limits beyond formal, parsable benchmarks [S1], but the paper does not yet demonstrate that the approach works on real-world reasoning tasks.

The paper is an arXiv preprint and has not been peer-reviewed [S1]. An OpenReview version exists [P4], suggesting it may be under review at a venue, but no acceptance decision is confirmed.

All performance figures are self-reported by the author without independent replication [S1]. The 66% finding, while striking, applies specifically to correctly-solved problems and involves a known evaluator blind spot that the authors disclose [S1]. The 100% recall figure applies only to predicate-determining dependencies, not to overall performance [S1].

The next concrete signal to watch: whether independent researchers replicate the method on other models and benchmarks, and whether the OpenReview submission progresses through peer review. If you want to catch that when it lands, subscribe and we will have it in your inbox.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.