Security scanners report that 96.89% of MCP servers are risky, according to a new large-scale study published this month on arXiv [S1]. But when researchers manually checked a sample of those alerts, fewer than half were true positives [S1]. If you are a developer trusting a scanner to tell you which MCP servers are safe to plug into your AI agent, that gap between alarm and reality is where the trouble starts.

The number that should make you nervous

MCP, or Model Context Protocol, has become the standard way that AI agents talk to external tools and services [S1]. An MCP server can give a language model access to a database, a file system, a shell prompt, or a web API. The protocol has spread fast because it solves a real problem: models are useless if they cannot act on the world. But that same access makes MCP servers a serious security surface. They handle privileged operations like shell execution, network access, and file-system manipulation [P2].

Until now, the way developers assessed MCP server risk was to run a security scanner. These tools inspect server code or configuration and flag potential issues. The problem, according to the new preprint, is that nobody had checked whether the scanners themselves were any good [S1].

What the study actually did

The researchers built MCPZoo, which they describe as the biggest MCP server dataset created so far for dynamic analysis [S1]. It contains 64,611 unique MCP servers, drawn from 113,927 total entries [S1]. More than 37,288 of those servers support dynamic analysis, meaning they can actually be run and tested rather than statically inspected [S1].

Building MCPZoo was not trivial. The team used a multi-agent framework that mimics how a human expert would build, diagnose, and fix deployment problems: it infers the environment a server needs, tries to run it, reads the error messages, and iterates until the server works [S1]. The servers were then validated through real protocol interactions, rather than code inspection alone [S1].

No previous study has measured the MCP ecosystem and its scanning tools at this scale [S1].

The 96.89% number and why it misleads

Here is where the story turns. Existing scanners, when run across the MCPZoo collection, report that 96.89% of servers are risky [S1]. That is a number that would stop any security team in its tracks. Nearly every MCP server in the wild, flagged as dangerous.

But the researchers manually validated a sample of those scanner alerts. Fewer than half were true positives [S1]. The scanners also disagreed with each other: outputs showed clear inconsistency across different tools [S1]. One scanner's critical risk was another scanner's clean bill of health.

The risk from MCP servers is real. But this new study suggests the tools people rely on to find that risk are crying wolf more often than they are catching genuine threats.

What it means

The core finding is simple and uncomfortable. The security tooling around MCP has not kept pace with the protocol's adoption. MCP servers are increasingly trusted with security-sensitive operations [S1], and the scanners used to vet them are producing results that are inconsistent and largely unverified.

For a developer, this means a scanner flag is a starting point, not a verdict. A server flagged as risky might be risky, or it might be a false alarm from a tool that cannot tell the difference. The study's authors have released a public query interface for MCPZoo so developers can look up servers and run their own assessments [S1].

The broader issue is that MCP security has been measured at the wrong resolution. Previous work looked at small numbers of servers [S1], or focused on specific vulnerability classes like taint flaws [P2] or maintainability issues [P4]. Risk assessment frameworks exist [P3], but they had not been tested at ecosystem scale until now.

What it means for business

For a two-person dev shop building an AI agent that connects to external tools, the practical impact is immediate. If you are running a security scanner on an MCP server before deploying it, treat the output as a lead, not a conclusion. A flagged server needs a human to look at what the scanner found and decide whether the alert is real. That takes time, and right now the scanners are generating more noise than signal.

For larger teams, the inconsistency across scanners means you cannot pick one tool and trust it. Running two scanners and comparing results is the minimum sensible approach, but even then, agreement does not guarantee accuracy.

The public MCPZoo query interface [S1] gives teams a way to check whether a server they are considering has been analysed at scale, which is more than most developers had access to before this week.

What we don't know yet

The study is a preprint and has not been peer-reviewed [S1]. All findings are provisional and could change after review. The manual validation that found fewer than half of alerts were true positives was based on a sample, which may not generalise to every scanner or every server [S1].

MCPZoo itself was built using a novel multi-agent framework that could introduce its own biases or errors [S1]. The framework emulates human expert behaviour, but emulation is not the same as human verification.

The distinction between 64,611 unique servers and 113,927 total entries means there are duplicates in the collection, which is normal for a dataset scraped from public repositories but worth keeping in mind when reading the scale numbers.

The next concrete event to watch is whether this preprint survives peer review and whether scanner vendors respond to the inconsistency findings. The public query interface is live now, so developers can start testing their own servers against it immediately.

If this story was worth your time, subscribe to get the next one delivered.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.