> ## Content Index
> Fetch the complete content index at: https://www.notatechguy.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Google DeepMind benchmark: AI agents score below 57% on visual work
- URL: https://www.notatechguy.com/google-deepmind-benchmark-ai-agents-score-below-57-on-visual-work/
- Published: 2026-11-21T23:18:38.000Z
- Updated: 2026-11-21T23:18:39.000Z
- Description: Google DeepMind's MKA benchmark tests AI agents on CAD files, X-rays and schematics, with the best model scoring 56.6% on the hardest criteria.
- Author: Marcello Babbili
- Tags: Technology & AI, Google, OpenAI, AI Agents

Google DeepMind researchers built a benchmark that asks AI agents to do what a radiologist or an electrical engineer does every working day: read a visually dense file, cross-reference it with others, and produce a finished deliverable. Even the strongest model they tested, Claude Fable 5.2, satisfied just 56.6% of the hardest grading criteria [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com). That number matters because the benchmarks already in use barely test visual reasoning: when the researchers removed image-viewing tools from Gemini 4 Carbon on GDPval, its score did not change [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com).

**My read:** The GDPval detail is the tell. If stripping image tools doesn't move the score, the benchmark wasn't requiring the model to look at anything. The 56.6% Climb-WSR figure is honest about how far multimodal agents still have to go, but I'd want to see absolute token counts behind the 5.7× efficiency claim before calling Gemini 4 Carbon the practical pick. The autorater's 96.2% agreement with human experts is measured on individual criterion judgments, not full deliverables, so the grading pipeline is less proven than the headline number suggests.

## The gap MKA fills

Single-format evaluations like GDP.pdf and GDP.xlsx test whether a model can work within one file type. They do not require reasoning across files or across modalities [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com). An agent that reads a CAD blueprint, pulls a specification from a separate PDF, and exports a cost estimate as a spreadsheet is doing something fundamentally different from answering questions about a single document. That gap is what Anima Singh, Suhas Yogin and Weicheng Kuo, researchers at Google DeepMind, set out to close with the Multimodal Knowledge Agent benchmark [P³](https://arxiv.org/abs/2611.25228v1?ref=notatechguy.com). Google's announcement, from senior product manager Rohan Doshi and principal engineer Ben Caine, says the benchmark was developed with Artificial Analysis [P²](https://blog.google/innovation-and-ai/models-and-research/gemini-models/multimodal-knowledge-agent-benchmark/?ref=notatechguy.com).

## What the numbers say

MKA contains 200 tasks spanning 10 economic sectors [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com). Each task feeds an agent multiple visually dense files: CAD blueprints, electronics schematics, radiographs, complex charts and geospatial maps [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com). The agent must extract and combine information across those files, then produce a standalone deliverable in a format a professional would actually use, whether .xlsx, .pptx, .pdf, .docx, .html or .png [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com). Deliverables are graded against expert-weighted rubrics by a sandboxed agentic autorater, which agreed with human domain experts on 96.2% of criterion judgments in an expert-graded study, corresponding to a Cohen's κ of 0.91 [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com). Tasks are scored on two metrics: weighted satisfaction rate (WSR), the share of rubric criteria the agent satisfies, and Climb-WSR, which focuses on frontier criteria that prior-generation models found difficult [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com).

The researchers evaluated four frontier models: Gemini 4 Carbon, Claude Fable 5.2, Claude Opus 5.5 and GPT-6 Astra [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com). Claude Fable 5.2 led with 56.6% Climb-WSR and 73.3% Overall WSR, edging Gemini 4 Carbon at 54.3% Climb-WSR and 71.9% Overall WSR [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com). Gemini 4 Carbon, however, used 5.7× fewer tokens than Claude Fable 5.2 to reach a near-identical score [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com). Even the strongest model stayed below 57% on Climb-WSR, which the authors describe as substantial headroom for multimodal agents [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com).

![Frontier model scores on the MKA benchmark](https://storage.ghost.io/c/6e/89/6e896869-22ef-4281-a213-b4c462c17cff/content/images/2026/11/chart_c72839703b9f780c271c.png)

The paper is not peer-reviewed.

It appeared on arXiv on November 20, 2026 [S¹](https://arxiv.org/abs/2611.25228?ref=notatechguy.com). Ten sample examples are available on HuggingFace as google/multimodal-knowledge-agent, under an Apache 2.0 licence [P⁴](https://huggingface.co/datasets/google/multimodal-knowledge-agent?ref=notatechguy.com).

## Who should pay attention

A construction firm evaluating whether an AI agent can draft a quantity survey from CAD drawings and a separate bill of materials would find MKA more relevant than any text-only benchmark. The same goes for a medical imaging team testing whether an agent can pull findings from a radiograph and format them into a structured report. The benchmark's insistence on real deliverable formats, rather than multiple-choice answers, is what separates it from the agent evaluations we have seen so far, as we found when [OpenAI's BioBasicsBench tested agents on 100 wet-lab tasks](https://www.notatechguy.com/openai-s-biobasicsbench-tests-ai-agents-on-100-wet-lab-tasks/) and when [OpenAI coding agents accelerated AI research, unevenly](https://www.notatechguy.com/openai-coding-agents-accelerate-ai-research-unevenly/).

The practical first step: download the 10 sample examples from HuggingFace [P⁴](https://huggingface.co/datasets/google/multimodal-knowledge-agent?ref=notatechguy.com) and run one against your own agent stack. If it cannot produce a valid .xlsx from a CAD file and a PDF spec sheet, the 56.6% ceiling is not your problem.

Your agent is not even reaching the grading stage.

---

*Sources: [S1 — MKA: Evaluating Multimodal Knowledge Agents on end-to-end professional](https://arxiv.org/abs/2611.25228?ref=notatechguy.com) · [P2 — Multimodal Knowledge Agents benchmark: November 2026](https://blog.google/innovation-and-ai/models-and-research/gemini-models/multimodal-knowledge-agent-benchmark/?ref=notatechguy.com) · [P3 — MKA: Evaluating Multimodal Knowledge Agents on end-to-end professional](https://arxiv.org/abs/2611.25228v1?ref=notatechguy.com) · [P4 — google/multimodal-knowledge-agent · Datasets at Hugging Face](https://huggingface.co/datasets/google/multimodal-knowledge-agent?ref=notatechguy.com) · [P5 — FastLM/MKA](https://github.com/FastLM/MKA?ref=notatechguy.com)*

## Related reading

- [OpenAI's BioBasicsBench tests AI agents on 100 wet-lab tasks](https://www.notatechguy.com/openai-s-biobasicsbench-tests-ai-agents-on-100-wet-lab-tasks/) — our technology desk, 2026-11-07
- [AI email agents comply with 45% of adaptive phishing attacks](https://www.notatechguy.com/ai-email-agents-comply-with-45-of-adaptive-phishing-attacks/) — our technology desk, 2026-10-24
- [OpenAI coding agents accelerate AI research, unevenly](https://www.notatechguy.com/openai-coding-agents-accelerate-ai-research-unevenly/) — our technology desk, 2026-11-19

---

*Written from 5 sourced items, 4 of them primary.*