A new arXiv preprint from researchers at the University of Minnesota and Ant Group proposes a framework called HARD that lets AI agents automatically build and improve their own security defenses by learning from their failures S¹. If the approach holds, it could replace the manually-built safety rules that today's AI agents depend on, rules the authors say are already struggling to keep pace with sophisticated attacks S¹. But the paper, posted on 13 August and not yet peer-reviewed, discloses no specific performance numbers to back its central claim S¹.

My read: This is the third self-evolving defense idea I've seen this month, and the pattern is familiar: researchers recognise that handcrafted security can't scale with agent capabilities, so they propose letting the system learn from its own mistakes. The concept is sound, but I don't buy the "extensive experiments" claim yet. The abstract discloses zero quantitative metrics. Until those numbers appear and survive review, HARD is a promising architecture sketch, not a proven defense.

Why handcrafted defenses are breaking

The problem starts with what AI agents can now do. As LLM agents gain the ability to browse the web, execute code, call APIs, and chain tools together, the attack surface grows faster than any team of security engineers can write rules for S¹. The researchers note that current runtime protections, the guardrails active during an agent's task execution, depend significantly on human-crafted countermeasures S¹. Each new attack pattern requires a new rule. Each new rule needs testing. And the rules interact in ways that are hard to predict.

How HARD works

The paper's contribution has two parts. First, the paper introduces a "harness-level formulation" for runtime defense, providing a standardized method to describe how the software framework surrounding an LLM agent facilitates security measures S¹. Think of this scaffolding as the cockpit where inputs are checked, outputs are filtered, and tool calls are gated. Today, each defense is bolted on ad hoc. The formulation gives engineers a shared vocabulary.

Second, they propose HARD, short for Harness-based Autonomous Runtime Defense Evolution S¹. The HARD system monitors the agent's live operations, detects defense failures, such as successful attacks or blocked legitimate actions, and autonomously chooses and improves the appropriate countermeasure using those failure logs S¹. Instead of a human writing a new rule each time, the system learns from what went wrong and adjusts its own defenses.

The researchers state that "extensive experiments" demonstrate HARD's superiority over traditional manual defenses in security scenarios without disrupting valid tasks S¹. But they don't say by how much, on which benchmarks, or against which attacks.

A growing field with the same gap

HARD is not alone. A June preprint named Membrane introduced a "self-evolving contrastive safety memory" for protecting LLM agents, also appearing on arXiv and currently lacking citations P⁴. SelfDefend, published at USENIX Security 2025 with accessible code on GitHub, explored a similar strategy for enabling LLMs to protect themselves against jailbreaks P⁵. The self-improvement pattern is spreading across agent tasks, including security.

The common thread: researchers are betting that static, human-authored defenses will lose against adaptive attackers. The fix they propose is making defenses adaptive too. The common gap: most of these papers are preprints with limited or no independent verification.

What to do about it

Consider a fintech company that deploys an AI agent to handle customer transaction disputes. The agent reads account histories, processes refunds, and can initiate wire transfers. Security engineers currently write rules: cap transfers at a threshold, flag requests matching known social engineering patterns, block calls to unverified external APIs. Every new fraud tactic means a new rule, a new test cycle, and a new risk of blocking legitimate customers.

HARD's pitch is that the agent's runtime scaffolding could learn from each blocked attack and each false positive, then rewrite its own guardrails without a human in the loop. That sounds appealing, but no team should deploy an unpeer-reviewed preprint's framework into production without their own evaluation. The practical step this week: audit your current agent's runtime defenses and count how many are manually maintained. If that number is growing faster than your team can test them, the self-evolving approach is worth tracking, not adopting yet.

What we don't know yet

The preprint discloses no specific experimental metrics, datasets, or attack models S¹. The "extensive experiments" claim is self-assessed, and the source is explicitly marked as not peer-reviewed S¹. The cross-listing under q-fin.GN (general finance) might suggest financial-domain testing, but the abstract makes general claims about LLM agents, so we can't assume domain specificity.

We also don't know whether HARD has been tested against adaptive attackers who know the defense is self-evolving, or whether the system can be poisoned by feeding it manipulated failure traces. The Membrane paper P⁴ and SelfDefend P⁵ face the same open question: self-evolving defenses that learn from data can also learn the wrong lessons.

The next signal: a revised version of this preprint on arXiv, which would typically appear within weeks of the initial posting if the authors respond to community feedback. We'll check the listing for v2 and compare any new metrics against the claims here.

If this analysis helped you think differently about AI agent security, subscribe for weekly breakdowns of the research that matters.

Sources: S1 — Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM Age · P2 — Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM Age · P3 — scthornton/securecode · P4 — Membrane: A Self-Evolving Contrastive Safety Memory for LLM Agent Defe · P5 — selfdefend/Code

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.