A new arXiv preprint reports that large language models can read cyber threat intelligence reports, generate attack playbooks, run them automatically, and fix their own failures, reaching 84.22% execution success with Claude Sonnet 4.5 [S1]. The previous best system, AURORA, needed human intervention at key steps and could not revise playbooks when attacks failed mid-execution [S1]. Close that gap and the question becomes: how much of adversary emulation, the practice of simulating real attacks to test defences, can now run without a human in the loop?

The problem with prewritten playbooks

Adversary emulation is how security teams test whether their defences actually hold up. You take a real threat actor's tactics, techniques and procedures, write them into a playbook, and run that playbook against your own network to see what breaks. The MITRE ATT&CK framework provides the shared vocabulary, and Caldera is the open-source automation platform that executes the steps [S1].

The bottleneck has always been the playbook itself. Security analysts read cyber threat intelligence reports, manually map each technique to MITRE ATT&CK identifiers, and hand-code the execution steps. Prior work either ran prewritten playbooks that quickly went stale, or partially automated the generation but still needed a human to fill gaps and handle failures [S1]. AURORA, which the paper identifies as the leading existing system, was able to produce playbooks from CTI reports but still relied on manual input at certain stages and lacked any way to adjust a playbook after an execution failure [S1].

How the closed loop works

The framework from researchers at Chonnam National University in Gwangju, South Korea, unifies three steps into a single workflow: playbook generation, execution, and failure recovery [S1] [P2].

First, an LLM reads a CTI report and extracts the attack techniques, mapping them to MITRE ATT&CK identifiers. It then generates a Caldera playbook, a structured set of "Abilities" (individual attack steps) ready for automated execution [S1].

Second, the framework executes the playbook automatically through Caldera.

Third, when an Ability fails, a failure-type-aware recovery mechanism kicks in. The system identifies what type of failure occurred and prompts the LLM to revise the failed step. This is the piece AURORA lacked: the ability to learn from execution errors and try again [S1].

The code is public. The GitHub repository, created in November 2025, has 32 stars and includes Python, PowerShell, and C++ components [P4].

The numbers that matter

The researchers evaluated the framework on 11 CTI reports using four LLMs: Claude Sonnet 4.5, GPT-4o, Gemini 2.5 Pro, and Grok 4 Fast [S1].

Claude Sonnet 4.5 produced the best results: 27.3 Abilities per playbook (meaning each generated playbook contained an average of 27.3 attack steps), and 84.22% execution success after the failure recovery mechanism ran [S1]. The model also achieved CTI Precision of 73.95%, Recall of 52.48%, and an F1 score of 60.50% [S1]. Precision measures how many of the techniques the LLM extracted were actually in the report. Recall measures how many real techniques it found. F1 balances the two.

The failure recovery mechanism was the single biggest contributor. It improved execution success across all four models by between 14.59 and 17.23 percentage points [S1]. On 10 CTI reports drawn from AURORA's own dataset, the final execution success rate surpassed AURORA's [S1].

What it means

For security teams, the core shift is from manual translation to automated iteration. Today, a security analyst might spend hours reading a threat report, mapping techniques, and writing a playbook. If a step fails during execution, the analyst debugs it by hand. This framework collapses that cycle into minutes, and the failure recovery means the system can fix its own mistakes without a human intervening.

The 52.48% Recall figure is the honest limitation in the results. It means the LLM missed nearly half of the attack techniques present in the CTI reports. Precision at 73.95% is stronger, meaning when the model did identify a technique, it was usually correct. For a security team, a playbook that covers half the known techniques but runs them reliably is still useful, especially when the alternative is a stale prewritten playbook that covers none of the current threat.

The multi-model comparison is also worth noting. The choice of model materially affects adversary emulation outcomes, with Claude Sonnet 4.5 clearly ahead of the other three tested models on this task [S1].

What it means for business

A two-person security consultancy that currently writes adversary emulation playbooks by hand could use this framework to process threat reports at a volume that was previously impractical. The GitHub repository is public, and the framework works with Caldera, which is free and open-source [P4]. The main cost is LLM API calls, which scale with the number of CTI reports processed and the complexity of each playbook.

For a mid-sized company with a small security team, the practical workflow changes like this: instead of an analyst reading a threat report and spending a day building a playbook, the analyst feeds the report into the framework, reviews the generated playbook for gaps (watching for that 52% Recall shortfall), and lets the system execute and self-correct. The analyst's role shifts from author to reviewer.

Security vendors that sell adversary emulation as a service face a different pressure. If an open-source framework can match or exceed the state-of-the-art system's execution success rate, the value proposition of commercial tools needs to move beyond playbook generation alone. The differentiator may shift to reporting, integration with specific security stacks, or coverage of threat types the LLM-based approach handles poorly.

What we don't know yet

This is a preprint. It has not been peer-reviewed, and all empirical claims are self-reported by the authors without independent verification [S1]. The comparison against AURORA was conducted by the authors of the new framework, on a subset of 10 reports from AURORA's dataset, which is a limited and potentially favourable test set [S1].

The evaluation covered only 11 CTI reports and four LLM models. Whether the framework generalises to other report styles, threat actor profiles, or models outside the four tested is unknown. The 52.48% Recall figure suggests the LLM may struggle with certain technique types, but the paper does not break down performance by technique category in the extracted evidence.

The framework has not been deployed in live enterprise production environments, and there is no evidence it has been used to prevent or detect real-world attacks. The failure recovery mechanism's improvement of 14 to 17 percentage points is measured only on the four tested models. Whether it helps with other LLMs is unverified.

The next concrete signal to watch: whether the framework gains traction in the Caldera community, and whether an independent team reproduces the AURORA comparison on a larger dataset. The GitHub repository's star count and fork activity, currently at 32 stars and 17 forks, is an early indicator of adoption [P4].

If this was useful, subscribe for more plain-English analysis of AI research that changes how people actually work.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.