The ICML 2026 Open Reproductions challenge set AI coding agents loose on roughly 2,200 accepted papers from the conference. Twenty-three percent of them, 496 papers, had at least one claim falsified or contested when independent teams tried to reproduce the results S¹. Forty-nine papers had every claim fall apart, with nothing verifiable at all S¹. What that means for a conference that doubled its intake to 6,352 accepted papers this year, partly because those same agents are now running experiments, is the question this audit begins to answer S¹.

My read: This is the first large-scale reproduction audit I've seen that uses the same tools causing the problem. The coding tools now available, from Claude Code to Codex, Cursor, and Pi, can take a paper, produce the code, execute the experiments, and report results S¹, which means the barrier to producing and checking research has collapsed at the same time. I don't think the 23% falsified rate means a quarter of ICML is junk. The 242 papers where independent teams reached opposite verdicts on the same claims S¹ tells me reproducibility itself is noisy, and "falsified or contested" mixes genuine errors with measurement variance. But 49 papers where nothing could be verified is a real signal, and the fact that a spotlight paper was accepted despite a reviewer admitting they didn't check the proofs S¹ is the kind of detail that should worry every conference organiser.

What 2,962 cloud jobs found

The challenge ran from July 15 to August 2, 2026 S¹. Participants received $20 in Hugging Face compute credits and launched 2,962 cloud jobs S¹. They confirmed 3,978 individual claims with real experiments S¹.

The reported outcomes:

  • 266 papers were fully reproduced, with every extracted claim verified S¹
  • 632 papers were partially reproduced with nothing falsified S¹
  • 502 papers had toy-scale evidence only, where a full reproduction was not feasible and participants created synthetic data matching the original's properties to run smaller experiments S¹
  • 496 papers had at least one claim falsified or contested S¹
  • 49 papers had all claims falsified and nothing could be verified S¹
  • 242 papers had independent reproduction teams reach opposite verdicts on the same claims S¹

ICML 2026 reproduction outcomes (share of ~2,200 examined papers)

Fifty-one percent of examined papers, 1,103 in total, had at least one claim independently verified S¹. That is the headline success. But the gap between that number and the 23% with falsified or contested claims is where the 502 toy-scale papers and 242 conflicting-verdict papers sit, and those categories tell a more complicated story about what "reproduced" actually means.

The spotlight paper nobody checked

One accepted ICML 2026 spotlight paper received strong scores and was accepted as a spotlight S¹. A reviewer wrote: "My low confidence score is because I did not check all the proofs carefully" S¹. The paper sailed through anyway.

This is the peer review problem at scale. ICML 2026 received 23,918 submissions and accepted 6,352, roughly double the previous year S¹. Conference reviewers are typically volunteers, and many lack the time or expertise to thoroughly assess a paper S¹. The submission growth is continuing an exponential trend, driven at least partly by AI agents that accelerate both running experiments and writing them up S¹. The same tools flooding the inbox are now the tools checking it.

What to do about it

Consider a small logistics startup that wants to use a new graph optimisation method from an ICML 2026 paper. Before this challenge, they would either trust the paper or spend weeks reproducing it themselves. Now they can check the public reproduction logbooks mirrored on Hugging Face P⁴ to see whether independent teams verified the specific claim they care about. If two teams reached opposite verdicts on that paper, the logbook will show both attempts, and the startup can read the disagreement and decide which setup matches their use case.

The practical step: before building on any ICML 2026 paper, search the ICML-2026-agent-repro dataset on Hugging Face for its reproduction logbook P⁴. If the paper has no logbook, treat its claims with the same caution you would any unreviewed preprint.

What we don't know yet

The aggregate statistics come from a single self-reported blog post by the challenge organisers on Hugging Face, with no independent corroboration S¹. The absolute counts and percentages imply slightly different totals, around 2,160 versus 2,200, suggesting rounding or overlapping categories. "Falsified or contested" mixes distinct outcomes: the 242 papers with opposite verdicts point to reproducibility noise rather than clear error. The causal claim that AI agents are driving submission growth is observational and not proven S¹. And the final accounting category in the blog post is cut off, leaving the full breakdown incomplete.

The next signal: a formal methodology writeup from the organisers, which would let the community audit the audit. We'll check the aggregate claims against it when it appears. If this kind of scrutiny matters to you, subscribe for the follow-up.

Sources: S1 — What We Learned by Reproducing 2,200 papers from ICML · P2 — ICML-2026-agent-repro (ICML 2026 Agent Reproductions) · P3 — UKPLab/arxiv2026-controllable-reasoning-models · P4 — qy2100/icml-2026-reproductions · Datasets at Hugging Face

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.