The strongest AI models available today fail more than half the time when asked to look at an image and call a tool based on what they see, according to a new benchmark released this month. The framework, called MM-ToolSandBox, tests agents across 500-plus tools and 16 application domains, and the best-performing model among 12 evaluated still scored below 50% success [S1]. The question is why, and the answer changes depending on how big your model is.

A benchmark that talks back

MM-ToolSandBox, published as an arXiv preprint on 14 July [S1], is built for a specific kind of AI agent: one that receives images during a conversation, extracts information from them, and then calls external tools to complete a task. Think of a user uploading a screenshot of a spreadsheet and asking the agent to extract a figure and run a calculation. The agent has to see the number, pick the right tool, feed it the right input, and handle follow-up corrections if the user changes their mind mid-conversation [S1].

The framework supports multi-image, multi-turn tasks where visual inputs arrive progressively. Agents must cope with goal revisions, error corrections, and state mutations, the kind of messy back-and-forth that real conversations produce [S1]. An automated pipeline generated 258 human-verified scenarios, plus 50 additional variants targeting interactive UI applications [S1]. The code is public on GitHub at apple/ml-mmtoolsandbox, created 6 July [P2].

This is the multimodal successor to Apple's earlier ToolSandbox, a text-only benchmark for LLM tool use presented at NAACL in March 2025 [P3][P4]. The original repo has drawn 263 stars on GitHub [P3]. The new version adds the visual dimension that the older one lacked.

The number that should make you pause

The research team evaluated 12 state-of-the-art models, ranging from 4-billion-parameter open-weight systems to frontier proprietary models [S1]. The strongest performer still couldn't clear 50% success [S1].

Vision is increasingly baked into how models learn, and visual pretraining has been shown to improve language model performance over text-only training. But this benchmark suggests that seeing and acting on what you see are two very different skills.

The most telling finding: 53% of failures in capable models came from incorrect information extraction from images, even when the agent's task workflow was otherwise correct [S1]. The model knew what to do. It just couldn't read the screen properly.

What it means

There is a pattern in the failure data that the researchers call a "planning-to-precision crossover." Smaller models tend to fail at deciding what to do. They struggle with the reasoning step: which tool to call, in what order, with what arguments. Larger models handle that part well but fail at perceiving what they see. They plan correctly, then misread the image [S1].

This matters because the AI industry has been betting that scale fixes everything. Make the model bigger, the thinking goes, and it gets better at every task. This benchmark suggests that at some point on the size curve, the bottleneck shifts. The model stops failing at thinking and starts failing at seeing. Throwing more parameters at the problem doesn't fix a perception error. It just produces a very expensive model that confidently misreads a chart.

For a regular person using an AI assistant to, say, upload a photo of a receipt and ask the agent to categorise the expenses and log them in a spreadsheet, this means you should expect errors. Not occasional errors. Frequent ones. The agent might pick the right tool and still transcribe the wrong amount from the image. The failure is invisible to you until you check the output.

What it means for business

A two-person accounting firm experimenting with AI agents to process client documents should treat visual tool-calling as a half-solved problem. The workflow design matters more than the model choice. If 53% of failures in capable models come from image extraction errors [S1], then the highest-leverage fix is not upgrading to a bigger model. It is adding a verification step between the image read and the tool call.

For a suburban real estate agency that wants an agent to read property photos and pull listing data into a CRM, the practical implication is that human review of the extracted data is still mandatory. The agent can handle the routing and the tool selection. It cannot yet be trusted with the numbers it pulls from an image.

Developers building visual agents should note that the benchmark and its 258 scenarios are publicly available [S1][P2]. Running your model against this test before shipping a visual tool-calling feature would give you a concrete success rate to compare against the sub-50% baseline the researchers established.

What we don't know yet

The paper is a preprint and has not been peer-reviewed [S1]. The failure and success statistics are author-reported and describe performance on this specific benchmark. They may not generalise to other visual tool-calling tasks or to production environments with different tool sets.

The 12 evaluated models are not individually named in the provided source text, so we cannot tell you which specific model came closest to 50% or which open-weight system performed best. The GitHub repository may contain model-level results, but we have not verified this.

The planning-to-precision crossover is a pattern the researchers observed across the models they tested. Whether it holds as the next generation of models ships, or whether new training methods close the perception gap, is an open question. The next concrete signal will come from independent teams running their models against the public benchmark and reporting results.

If you want to know whether visual AI agents are ready for your workflow, this benchmark says: not yet, and the problem is eyes, not brains.

Subscribe for the next instalment, where we'll track who runs this benchmark and what they find.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.