Twenty-three frontier AI models were set loose on ten real compromised cloud hosts, each riddled with attacker techniques, and not one could fully detect the intrusion and produce a working fix S¹. The result comes from SecRespond, a new arXiv preprint published July 29 that claims to be the first benchmark for post-breach incident response, the messy work security teams do after an attacker is already inside S¹. The agents could follow alerts. They could not find what the alerts missed.

My read: This is the first benchmark I have seen that tests AI agents in the scenario that actually matters, the moment after a breach when the disk is dirty and the attacker has already moved. Most AI security research tests prevention, the clean-room phase before an attack lands. The zero-for-ten result is sobering but not shocking. The harder question is whether this is a fixable engineering gap or a ceiling on what language models can do with a filesystem.

What the benchmark actually does

SecRespond gives an AI agent a forensic disk snapshot from a compromised host, plus the alerts, vulnerability scans, and baseline checks that a host security product would produce S¹. The agent then has to write forensic reports covering the intrusion, baseline risks, and vulnerability risks, and produce a remediation plan S¹.

Ten cyber ranges, each built from a distinct compromised cloud host, span four entry-point types, 21 ATT&CK techniques (the MITRE framework's catalogue of attacker tactics), and five operating systems S¹. The 23 models were tested on the OpenCode agent harness, which gives them command-line access to investigate the disk S¹.

The authors argue that existing cybersecurity benchmarks focus on pre-compromise settings, where agents sit in a clean, idealised environment before an attack occurs S¹. That leaves the post-compromise setting, the one where stakes are highest and data is messiest, largely untested. SecRespond tries to close that specific gap.

Why zero out of ten is the number that matters

The results split cleanly. Current agents can reliably uncover problems that alerts already flag S¹. Give them a blinking red light and they will trace it. But they struggle to proactively investigate the disk for silent intrusions, the attacker who left no alarm but did leave traces in log files, scheduled tasks, or modified binaries S¹. They also struggle to produce comprehensive, verified remediation plans S¹.

No tested model achieved complete detection and remediation on any single cyber range S¹. Not one host, out of ten, was fully solved by any of the 23 models.

The authors call this a "fundamental bottleneck in building agents for real-world incident response" S¹. That is their interpretation, not an industry consensus, and the paper has not been peer-reviewed S¹. But the pattern is clear: agents that follow alerts are useful triage assistants. Agents that must hunt without a prompt are not yet ready.

This matters because LLM agents are already being adopted in real-world security operations with access to host artifacts and command-line interfaces S¹, as the authors note. A model that can read an alert but cannot find a hidden persistence mechanism is a tool with a dangerous blind spot. The agent that misses the silent intrusion gives its operator false confidence that the host is clean.

The timing sharpens the point. Two days before SecRespond appeared, a technical timeline of a July 2026 intrusion at a frontier AI lab was published on HuggingFace, detailing how an agent-based system was compromised P⁵. In April, SIR-Bench, a separate preprint, began evaluating investigation depth in security incident response agents P². OpenSec, another recent preprint, measures how well incident-response agents hold up under adversarial evidence, noting that frontier agents can now generate working exploits for under $50 in compute P⁴. The research community is converging on the same uncomfortable question from different angles.

What to do about it

For a security operations team running an LLM agent in their incident-response pipeline today, the practical takeaway is narrow but important. Use the agent for what it does well: triaging alerts, summarising flagged events, and drafting initial reports from known indicators. Do not rely on it to clear a host as clean. The SecRespond results show that agents miss silent intrusions, the traces an attacker leaves that never trigger an alarm.

Consider a mid-sized fintech whose security team has started using an LLM agent to investigate compromised cloud instances. The agent receives an alert about unusual outbound traffic, traces it to a modified cron job, and drafts a report. That is useful work, and it saves the team an hour. But the same agent, asked to verify the host is now clean after the cron job is removed, may miss a second persistence mechanism hidden in a systemd timer or a modified SSH key. The team signs off, the attacker returns through the back door, and the breach cycle restarts.

The fix is process, not model choice. Treat agent output as a first draft, not a verdict. Pair every agent-generated clearance with a human review of the disk for persistence mechanisms the agent was not asked about specifically. And if you are evaluating agents for this workflow, the SecRespond benchmark is publicly available on GitHub under the Alibaba-NLP namespace, so you can test your own models against the same ten ranges S¹.

One thing to check this week: if your team uses an LLM agent in incident response, run it against a host where you have planted a persistence mechanism that does not trigger any alert. See whether it finds it.

What we don't know yet

The preprint does not publish individual model scores, rankings, or confidence intervals S¹. We know 23 models were tested and none fully solved any range, but we do not know which models came closest, whether some operating systems or ATT&CK techniques proved harder than others, or how wide the gap was between the best and worst performers.

The "first benchmark" claim is an author assertion S¹. SIR-Bench, published in April 2026, also evaluates security incident response agents, though it focuses on investigation depth rather than the full detection-to-remediation workflow P². Whether SecRespond is genuinely first in its specific niche may be disputed as the field grows.

The benchmark covers ten cyber ranges. That is enough to expose a pattern but small enough that a broader evaluation could shift the picture. The authors' "fundamental bottleneck" framing is their interpretation, and the paper has not undergone peer review S¹.

The next signal: peer review and community replication. If independent teams run the benchmark and reproduce the zero-for-ten result across a larger set of cyber ranges, the bottleneck claim hardens from interpretation to evidence. Watch for a follow-up paper or a workshop submission at a major security conference. We will check the claim against it.

If this kind of reporting helps you make better decisions, subscribe to keep reading.

Sources: S1 — SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Inci · P2 — SIR-Bench: Evaluating Investigation Depth in Security Incident Respons · P3 — THUDM/AgentBench · P4 — OpenSec: Measuring Incident Response Agent Calibration Under Adversari · P5 — Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.

SecRespond benchmark: models tested vs ranges solved