GPT-5.5-xhigh, the strongest model tested, scored 66.1 on doctoral qualifying-exam proofs in a benchmark released this week on arXiv [S1]. It managed 75.8 on the undergraduate split. Both figures come from AdvancedMathBench, a suite built to test what happens when a model has to construct a full mathematical argument, rather than pick the right answer. The deeper problem sits one layer below: can these models at least tell a valid proof from a broken one?

A benchmark that refuses to grade on the final answer

Existing math benchmarks tend to check whether the model lands on the right number [S1]. That works for calculus drills and olympiad short answers. It falls apart when the task is a proof, because a proof is a sequence of logical steps where the answer is the argument itself.

AdvancedMathBench, posted as a preprint on 14 July, tries to fill that gap [S1]. The suite has two parts. ProverBench contains 296 problems drawn from undergraduate coursework and doctoral qualifying exams [S1]. VerifierBench holds 888 model-generated proof trajectories, each paired with expert ground truth, to test whether a model can correctly judge whether a proof holds up [S1].

The authors also built an automatic verification pipeline, trained on large-scale expert annotations, that produces both correctness verdicts and fine-grained assessments of where a proof goes wrong [S1]. They report that this pipeline shows strong agreement with human experts on held-out proof trajectories, though the evaluation is the authors' own [S1].

The numbers that matter

On proof generation, GPT-5.5-xhigh scored 75.8 on the undergraduate and doctoral (UGD) split and 66.1 on the qualifying-exam (QE) split [S1]. The authors describe these results as showing substantial room for improvement on advanced mathematical proof construction [S1]. The scores are presented without baselines or a stated maximum, so 66.1 is best read as a raw figure rather than a percentage of perfection.

The verification results are more telling. The best model achieved a Balanced F1 of 65.1 [S1]. More striking: models generally showed low true negative rates. They struggled to correctly identify proofs that were actually wrong [S1]. The authors flag critical error detection as a major bottleneck [S1].

The models that can sometimes write a passable proof are the same models that have trouble spotting a flawed one.

What it means

AdvancedMathBench exposes a ceiling on frontier models: even at the frontier, models that ace high-school and olympiad mathematics still falter when the task demands sustained logical construction at the graduate level [S1].

The verification gap matters more than the generation gap. A model that writes a flawed proof but cannot detect its own errors is a tool that cannot self-correct. For anyone hoping to use AI as a research assistant in mathematics, that is the limitation that determines whether the output is usable or actively misleading.

The benchmark also arrives amid a wave of similar efforts. FormalProofBench, on OpenReview, asks whether models can write graduate-level proofs that are formally verified [P2]. IMProofBench, another arXiv preprint, benchmarks research-level proof generation [P4]. The proofQED/QED repository on GitHub, created in March 2026, is building open tooling in the same space [P5]. Hugging Face's yourbench framework, open-sourced in January 2025, lets organisations build custom benchmarks on their own data [P3]. The field is converging on the same question from different angles: can models reason at the level of a working mathematician?

What it means for business

For a two-person data science consultancy or a financial modelling firm, the practical takeaway is narrow but real. If your workflow involves asking a language model to produce or check mathematical arguments, the current frontier cannot be trusted to verify its own work. A Balanced F1 of 65.1 on proof verification means the best model's combined precision and recall leave a wide error margin [S1]. That is not a threshold you can ship to a client without a human checking the output.

For education technology companies building tutoring or assessment tools around advanced mathematics, the benchmark sets a ceiling on what is currently possible. A model that scores 66 on doctoral qualifying exams is not ready to replace a graduate teaching assistant. It might be useful as a first draft that a human then checks, provided the human has the expertise to catch what the model misses.

For research labs and AI teams, the low true negative rate is the signal to watch. A model that approves broken proofs is worse than one that refuses to answer, because it produces confident errors. Any pipeline that uses a language model to verify mathematical or logical reasoning needs a human in the loop until that true negative rate climbs substantially.

What we don't know yet

The paper is a preprint that has not been peer-reviewed [S1]. All results are self-reported by the authors, including the evaluation of their own verification pipeline, which creates a potential conflict of interest [S1]. The model identifier GPT-5.5-xhigh is not a widely recognised commercial product, and the source does not explain its nature or availability [S1]. The scores of 75.8, 66.1, and 65.1 are presented without baselines or clear maximum values, making it difficult to calibrate how far substantial room for improvement actually extends [S1].

Whether the automatic verification pipeline generalises beyond the held-out trajectories the authors tested is an open question. Independent replication by a separate team, using different models and different proof corpora, would settle whether the pipeline's agreement with human experts holds up under scrutiny.

The next event to watch is peer review and community adoption. If other research groups adopt AdvancedMathBench and report their own numbers, the benchmark's difficulty claims gain external validation. Until then, the scores are a single team's reading of a problem the field is only beginning to measure.

If this was worth your time, subscribe to get the next benchmark decoded before the hype settles.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.