A new arXiv preprint treats Codex and Claude Code as "stochastic model-discovery operators" — and argues that a single benchmark run is not enough to judge them [S1]. If every run produces a different result, the leaderboard you've been reading might be measuring noise as much as skill. The paper proposes a statistical framework to find out — but whether it holds up outside a word-game testbed is the question that could decide whether it matters for your team.

The randomness hiding in your agent run

The paper's core claim is simple but uncomfortable: large language model coding agents are "stochastic and adaptive," meaning they don't produce the same output twice, even on the same task [S1]. That makes the standard evaluation method — run the agent once, check the score — fundamentally unreliable. You might catch a good day or a bad one.

The authors, whose preprint appears in arXiv's cs.AI and cs.LG categories and is explicitly marked as not peer-reviewed [S1], propose treating agents not as programs with fixed outputs but as statistical operators. In their framing, an agent like Codex or Claude Code takes task-specific data and an optimisation target and produces a fitted model — a "stochastic model-discovery operator" in their language [S1]. The output varies. The question is how much, and why.

The experimental design framework

To answer that, the authors borrow from classical experimental design — the same statistical toolkit that drug trials use to separate signal from noise. They vary four factors: the agent's reasoning effort (how hard it thinks before acting), the task itself, the optimisation metric, and the composition of training data [S1]. For each combination, they run regression models across four response variables: output quality, dollar cost, wall-clock time, and process complexity [S1].

They also introduce what they call a "utility-aligned canonical decomposition" — a method to identify the dominant direction of the reasoning-effort effect and check whether it aligns with a performance-cost trade-off that actually matters to users [S1]. In plain terms: does thinking harder make the agent better in a way that justifies the extra spend?

The testbed is a set of networked word-forming games [S1] — a controlled environment that lets the researchers isolate factors without the messiness of real-world codebases.

What it means

The paper's central insight is one that anyone deploying AI agents should internalise: a single benchmark score tells you almost nothing about an agent's real-world reliability. If the same agent on the same task can produce different outputs, then the gap between a 90% score and a 70% score might be luck, not capability.

This matters because the AI industry runs on leaderboards. Models are ranked, purchased, and deployed based on benchmark performance — often from a single run. If agents are as stochastic as this paper suggests, those rankings could be misleading. A model that wins a benchmark by three points might be sitting inside the margin of its own randomness.

The experimental design approach offers a corrective: run the agent many times under controlled conditions, vary the factors that matter, and use statistics to separate genuine effects from noise. It's the difference between asking "how good is this agent?" and asking "how good is this agent, on average, under these conditions, and how much does it vary?"

The paper also connects to a growing body of work on agent evaluation. BoxingGym, another arXiv preprint, benchmarks progress in automated experimental design and model discovery [P4]. Hugging Face's "is-it-agentic-enough" repository measures how coding agents use iterations and environment feedback [P5]. And the GBOED framework on GitHub explores experimental design that's robust to model misspecification through generalised Bayesian inference [P3]. Together, these point to a field waking up to the fact that evaluating agents requires fundamentally different methods than evaluating static models.

What it means for business

For a two-person dev shop or a suburban agency experimenting with AI coding agents, the practical implication is immediate: don't judge an agent by one run.

If you're deciding between Codex and Claude Code for a workflow — say, auto-generating data models from client briefs — run the task five or ten times, not once. Track not just whether the output is correct, but how much it costs each time, how long it takes, and how complex the agent's process gets [S1]. The variance itself is information. An agent that produces excellent work 60% of the time and garbage 40% of the time is a different proposition from one that produces solid work 90% of the time — even if their average scores look identical.

The paper's focus on reasoning effort is also directly relevant to cost management. If you're paying per token, the difference between "think hard" and "think fast" could be the difference between a $2 run and a $20 run. The authors' utility-aligned decomposition is essentially asking: is the extra thinking worth the extra money? That's a question every operator should be asking before cranking reasoning effort to maximum.

What we don't know yet

The findings come with significant caveats:

  • Only two agents were tested — Codex and Claude Code [S1]. Conclusions about "agentic AI" broadly should be scoped to these specific systems.
  • The testbed is networked word-forming games [S1], not real-world codebases or production tasks. Whether the framework's findings transfer to, say, a Python web app or a SQL pipeline is untested.
  • The preprint is not peer-reviewed [S1]. The methodology and findings have not undergone independent academic scrutiny.
  • The abstract characterises findings as "insightful" [S1] — an author self-assessment without independent corroboration.
  • The claim that LLM coding agents "increasingly perform" open-ended modelling is presented without supporting citation in the provided text [S1].

The next concrete event to watch: whether this framework gets applied to real-world coding tasks, and whether peer review confirms the methodology holds up. Until then, the paper is a promising method in search of a broader testbed.

If this kind of clear-eyed analysis is what you want before deploying AI in your own work, subscribe — there's more where this came from.

Sources

  • [S1] arXiv preprint, "An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery" (cs.AI, cs.LG) — not peer-reviewed
  • [P3] yasirbarlas/GBOED, GitHub — Generalised Bayesian Optimal Experimental Design
  • [P4] BoxingGym, arXiv — Benchmarking Progress in Automated Experimental Design and Model Discovery
  • [P5] huggingface/is-it-agentic-enough, GitHub — agent-eval framework

Sources

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.