Researchers behind an October 30 arXiv preprint report that large language models can land near the optimal answer to numerical optimization problems while breaking the very constraints that would make those answers usable S¹. That paradox, where the answer is both close and broken, cuts against the hope that general-purpose models might replace specialised mathematical solvers. The preprint, listed as 2610.37932v1, measures how often this happens, and the conclusion is blunt: recurring mistakes in both computation and problem structure mean the models cannot yet serve as independent solvers S¹. No specific error rates, benchmark scores, or dimension thresholds appear in the available abstract, so these qualitative claims have not been independently verified S¹.
My read: The paper's most useful contribution is the evaluation framework that separates solution quality from execution failures. Too many studies conflate "the model got the wrong number" with "the model doesn't understand the problem." If LLMs are capturing optimization structure but failing on formatting and numerical precision, that is a fixable engineering problem, not a fundamental capability wall. What I cannot judge from the abstract alone is the scale of the gap: no specific benchmarks, dimension thresholds, or error rates are disclosed, so the qualitative claims about "near-optimal" and "substantially improves" need the full paper to evaluate. The direction aligns with prior work from Allen Nie, Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan at Microsoft Research, who showed in 2023 that directional feedback matters for LLM-based optimizers P².
The evaluation framework distinguishes three distinct failure modes
The authors built an evaluation framework that splits failures into three buckets: numerical inaccuracies, constraint violations, and output-format errors S¹. This matters because a model that returns the right optimal value in the wrong format, or a solution that satisfies the objective but breaks one constraint, is doing something fundamentally different from a model that has no grasp of the problem at all.
They tested LLMs on representative problems from convex optimization, fractional programming, and constrained projection S¹. Convex optimization is the workhorse of operations research and machine learning training. Fractional programming appears in finance and signal processing. Constrained projection is the maths behind finding the closest valid point to an invalid one. Each LLM answer was assessed against deterministic reference solvers, the kind of specialised software, like the optimagic Python package that wraps SciPy and NlOpt solvers P⁵, that has handled these problems reliably for years.
The distinction the authors draw gets lost when people ask whether LLMs can "do maths." The question this paper asks is narrower and more useful: can an LLM find a near-optimal point, even if it cannot reliably express that point in the right format with the right precision?
Near-optimal answers routinely violate the constraints that make them valid
The answer is yes, often, but with a catch that grows with scale.
The models demonstrate an ability to grasp the underlying structure of optimization tasks and land on answers close to the optimum, despite routinely producing outputs that miss numerical precision targets or violate required formatting conventions S¹. The model may understand the shape of the problem well enough to land near the optimum, but stumble when asked to express that answer with the precision and constraint-satisfaction that a solver demands.
As the size of the problem grows, reliability drops off, and the degree of degradation tracks closely with the type of optimization structure involved S¹. The paper does not disclose the specific dimension thresholds at which performance collapses, which limits how directly practitioners can apply the findings. This is a recurring limitation: the abstract describes qualitative patterns without reporting the quantitative metrics, error rates, or benchmark scores that would let a reader calibrate the claims.
The tension here echoes a broader pattern in AI research, where models show surprising competence on structured tasks before hitting a wall that specialised tools do not face.
Iterative refinement narrows but does not close the reliability gap
The researchers set up a multi-round refinement process that passes specific details about which constraints were violated and how far the answer was from optimal back into the model S¹. The approach mirrors the directional feedback that Allen Nie and colleagues at Microsoft Research identified as important for LLM-based optimizers in their NeurIPS 2023 workshop paper P². Tell the model which constraint it violated, and it can often correct course.
Feeding targeted feedback back to the model across multiple rounds meaningfully boosts reliability and pushes the ceiling on problem size before solutions break down S¹. But "substantially" is a word the abstract uses without a number attached, and no one outside the authors has checked these claims: the paper is a preprint, not peer-reviewed, and the evidence pack contains only the arXiv abstract S¹.
Even with refinement, recurring numerical and structural mistakes persist S¹. The remaining errors constrain the models' usefulness on their own and point toward architectures that pair LLM-generated reasoning with traditional numerical checking S¹. In a hybrid system, the LLM would propose a solution structure or initial guess, and a deterministic solver would verify feasibility and correct precision errors.
For an operations research analyst running daily delivery-route optimization, the practical shift would be modest: an LLM could draft the initial problem formulation and suggest a starting point, but a deterministic solver would still need to verify every constraint before the solution ships to drivers S¹P⁵.
Operations research teams face a practical calibration question
The immediate audience is operations research teams and quantitative finance groups that already run deterministic solvers but want to explore whether an LLM can accelerate problem setup, suggest solution strategies, or handle simpler instances without invoking a full solver. A logistics company routing deliveries, for instance, might use an LLM to draft an initial optimisation formulation, then pass it to a verified solver rather than trusting the model's raw output.
The practical test for any team reading this: hand your LLM a constrained optimisation problem you already have a verified answer for, and check whether the output looks right and whether it satisfies every constraint to numerical precision. That is the gap this paper exposes, and the gap that hybrid architectures would need to close. The preprint, posted October 30 on arXiv, does not yet carry the specific dimension thresholds and error rates that would let practitioners calibrate exactly where that gap opens.
Sources: S1 — Understanding Large Language Models as Numerical Optimizers · P2 — Importance of Directional Feedback for LLM-based Optimizers - Microsof · P3 — CogSciPrag/Understanding-LLMs-course · P4 — [20230910] Weekly AI ArXiv 만담 시즌2 - 25회차 · P5 — optimagic-dev/optimagic
Written from 5 sourced items, 4 of them primary.
