A July 17 arXiv preprint by Paul Kassianik, Blaine Nelson and Yaron Singer [P2] assesses AI security agents based on their operational costs and success rates [S1]. The main finding: offensive hacking tasks scale with increased compute spending, but defensive investigations do not [S1]. For any team deciding between an inexpensive open-weight model and a costly frontier system, this asymmetry fundamentally changes the purchasing decision, and the reason lies in how defense actually functions.

The cost-success lens

The unreviewed arXiv preprint, titled "Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents," appears in the cs.AI and cs.LG categories [S1]. The research team tested language-model security agents using two different benchmark suites: Cybench for offensive capture-the-flag exercises, and Splunk BOTS v1 for defensive SOC (security operations centre) investigations [S1].

Rather than simply publishing top success rates, the researchers evaluated models at set cost thresholds and split performance into two spending categories: inference spend (the price of executing the model's reasoning) and tool spend (the expense of invoking external tools and APIs) [S1]. This matters because those two line items determine whether an AI security agent is viable in production. A model that solves 90% of challenges but costs $50 per investigation is useless to a SOC team processing thousands of alerts daily. A model that solves 70% at $0.50 might be deployable.

Why offense scales but defense doesn't

The findings reveal different scaling patterns for offensive and defensive operations [S1]. For offensive tasks, CTF performance gets better as more test-time compute is applied to the challenge [S1]. When scaled up, open-weight models can rival frontier proprietary systems on offensive tasks while remaining cost-effective [S1]. More reasoning steps means more attack vectors tried, more backtracking, more iteration. It scales.

Defensive operations show a different pattern. SOC investigations do not improve similarly when more compute is added [S1]. Positive outcomes rely much more on careful tool usage, telemetry analysis, and selective enrichment rather than simply increasing the reasoning budget [S1]. An agent that thinks harder about a poorly chosen log query is still looking at the wrong log.

The researchers suggest that evaluations for security agents need to account for economic efficiency and operational suitability in addition to task completion rates [S1]. They have published an interactive website with their results at evals.frontier.security [S1].

What it means

The obvious assumption is that a smarter model, given more time to think, will be better at any security task. The paper says that holds for offense but breaks for defense.

Think of it this way. An offensive agent cracking a CTF challenge benefits from more reasoning because it can try more approaches, fail, and try again. More compute means more attempts. That is a direct return on spend.

A defensive agent investigating a Splunk alert faces a structurally different problem. It needs to query the right logs, follow the right telemetry trail, and decide which signals matter. Throwing more reasoning at a poorly directed investigation does not help. The agent needs to know which tool to call and when, rather than think harder [S1].

This paper steps back and asks a different question: even when agents work, what do they cost, and does spending more help?

The answer, at least for defense, is that spending more on reasoning is not the lever. Tool discipline is.

What it means for business

For a two-person security consultancy or a suburban MSP running automated pentesting, the offensive findings are encouraging. When scaled up, open-weight models can rival frontier proprietary systems while remaining cost-effective [S1]. A small firm can potentially run offensive security agents on cheaper open-weight infrastructure without paying frontier-model API rates, at least for CTF-style tasks.

For a SOC team at a mid-sized company, the defensive findings are a warning. Buying the most expensive model will not automatically produce better investigations [S1]. The bottleneck is not reasoning capacity but tool discipline: which logs to query, which telemetry to follow, which alerts to enrich. A cheaper model with well-designed tool-calling workflows could outperform a frontier model with poor tool integration.

This has hiring implications. If defensive outcomes rely on careful tool usage and telemetry analysis [S1], the relevant skills shift from prompt engineering toward security operations expertise: knowing which data sources to connect to the agent and how to structure investigation workflows.

For vendors selling AI security agents, the researchers' suggestion that evaluations need to account for economic efficiency and operational suitability [S1] sets a challenge. Customers will start asking "does it work?" and "what does it cost per investigation, and does it scale?"

What we don't know yet

The paper is a preprint, not peer-reviewed [S1]. Its claims about scaling differences and cost-competitiveness come from a single unreviewed study.

The benchmarks used, Cybench and Splunk BOTS v1 [S1], are controlled challenge suites. They may not generalise to live operational environments, where alert noise, false positives, and adversarial adaptation look very different from a curated benchmark.

No specific dollar figures, success rates, or model names are available in the evidence pack. The paper's interactive website at evals.frontier.security [S1] may contain more granular data, but its accessibility and permanence are not guaranteed.

The broader research trend is moving toward cost-aware evaluation. Deep research surfaced related work including BeyondBench, an ICLR 2026 accepted paper on contamination-resistant evaluation of reasoning in language models [P3], CARE-Bench, a cost-aware benchmark for agentic software-security detection [P4], and CostNav, a cost-aware benchmark for embodied agents [P5]. Whether findings from this paper align with or contradict those frameworks is not yet established.

The next concrete event to watch: whether peer review accepts or modifies these claims, and whether major security-agent vendors adopt cost-aware benchmarking in their own evaluation disclosures.

If this kind of cost-aware analysis is useful, subscribe for more stories that decode what AI research actually means for the people who have to deploy it.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.