SWE-bench Verified, a leading benchmark for AI coding agents, needs 90% of its tasks run before a partial evaluation reliably matches the full result, according to a new analysis on arXiv [S1]. The same study found AppWorld needs just 15%. That six-fold gap means anyone cutting corners on agent evaluation is gambling blind, with no universal rule to fall back on.

What the replay found

The paper, by independent researcher Wei-Jung Huang, replayed completed public task-level records from three widely used agent benchmarks: SWE-bench, AppWorld, and tau-bench [S1][P2]. Rather than running new agents, the study re-sampled existing results at different task fractions to see how little you could get away with running and still reach the same conclusion the full benchmark would give.

The method is deliberately strict. A partial budget only qualifies as sufficient when it yields the same head-to-head ranking as the full benchmark, includes every required task group, and keeps the share of unsettled comparisons below a chosen limit [S1]. The analysis uses a strict 0 percentage point threshold on a 5 percentage point budget grid. The partial run must match the full run exactly, with zero tolerance for disagreement [S1].

The results vary sharply. AppWorld first meets all targets at 15% task fraction. tau-bench reaches reliability at 25%. SWE-bench Verified needs 90% [S1]. And SWE-bench Lite, a smaller variant, does not meet all targets even at 95% under the primary coverage rule [S1].

In plain terms: for AppWorld, you can skip 85% of tasks and still trust the ranking. For SWE-bench Verified, you need to run almost everything. For SWE-bench Lite, running almost everything still is not enough.

What it means

The core finding is that a task fraction alone tells you nothing about whether a partial run supports the same conclusion as the completed benchmark [S1]. This matters because running agent benchmarks is expensive. Each task can involve multiple API calls, tool interactions, and long reasoning chains. A team comparing two models on SWE-bench might spend thousands of dollars on inference, and the temptation to run a subset is obvious.

But this paper shows the safe subset size depends entirely on which benchmark you are using, and on details that most evaluation reports never mention. The study argues partial-evaluation reports should disclose five things: the minimum performance gap between agents, the method for picking tasks, the coverage threshold needed, the decision criterion applied, and the allowable share of unresolved head-to-head comparisons [S1]. Most published benchmark scores state none of these.

The SWE-bench Lite result is the sharpest warning. Even at 95% of tasks, under the strict threshold, the partial run cannot reliably reproduce the full benchmark's pairwise decisions [S1]. That does not mean SWE-bench Lite is broken. It means the benchmark's task-level results are noisy enough that even a near-complete run can flip a head-to-head comparison. Anyone using SWE-bench Lite to decide between two closely matched agents should treat the ranking with caution.

The benchmarks themselves are still poorly understood as measurement instruments. This paper is one of the first to ask how much of a benchmark you actually need to trust.

What it means for business

For a two-person AI startup choosing between two coding agents, the practical takeaway is blunt. If you are using SWE-bench Verified, plan to run the full benchmark. A 90% task fraction means a partial run saves you almost nothing. If you are using AppWorld, a 15% sample may suffice and cut inference costs by roughly 85%.

For evaluation teams inside larger firms, the paper's reporting checklist is the actionable part. Any internal benchmark report should now state the five parameters the study identifies: the performance gap threshold, task selection method, coverage rule, decision rule, and the fraction of unresolved comparisons [S1]. Without those, a partial-run score is a number without a confidence interval.

For vendors publishing benchmark scores, the risk is reputational. A model that tops a partial SWE-bench Lite run may not hold that ranking on the full benchmark. The paper does not name specific models or vendors, but the methodology gives buyers a way to ask harder questions of any benchmark claim that does not report its task fraction or decision rule.

What we don't know yet

The paper is an arXiv preprint and has not been peer-reviewed [S1]. The percentages are conditional on the strict 0 percentage point threshold and the 5 percentage point budget grid. They do not generalise to looser decision thresholds or different coverage criteria [S1].

The study covers three benchmarks. Whether other widely used agent benchmarks, such as AgentBench [P3], show similar or different task-fraction requirements is unknown. The paper also does not propose a universal rule for predicting the safe fraction from benchmark properties. Each benchmark must be tested individually.

The GitHub repository for the study was created on 13 July 2026 [P4], and the paper appeared on arXiv on 15 July [S1]. The next concrete event to watch is whether the KDD workshop on Evaluation and Trustworthiness of Agentic AI, where this work is submitted [P4], publishes peer-reviewed proceedings that confirm or revise these numbers.

If you want the next instalment of this benchmark audit series in your inbox, subscribe to keep reading.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.