A new arXiv preprint, published 28 August 2026, surveys the peer-reviewed literature on LLM-based security agents from 2023 to 2026 and reaches one blunt conclusion: researchers can create systems that execute tasks, but they have not yet figured out how to limit their permissions or verify their actions S¹. This problem is compounded by the lack of a shared definition for the term "agent" among researchers. Evaluation methods seldom align. Use cases span everything from simple code checks to fully automated vulnerability fixes S¹. What happens when the tools guarding your systems have no guardrails of their own?

My read: This is the first survey I have seen that frames the AI-agent security problem as a governance gap rather than a capability gap. I am skeptical of the 2023 to 2026 literature span, because the paper is itself a 2026 preprint, so some of those "2026" citations may be forthcoming work the authors had early access to. The authors are not saying these agents do not work. They are saying nobody can check whether they worked correctly, and that is a different and more dangerous failure mode.

What the review actually covers

The paper, filed under cs.AI and cs.LG on arXiv, is a systematic literature review, not a new agent system or benchmark S¹. It looks at technical methods (such as system design, memory, reasoning, action space, and orchestration), the use cases for these systems, and how they are evaluated (including datasets, metrics, safety protocols, and baselines) S¹.

The literature span, 2023 to 2026, captures the full arc of this subfield from its first stirrings through to work that may still be in press S¹.

The word "agent" is doing too much work

One of the review's sharpest findings is terminological. The authors state that the term "agent" lacks a consistent definition in the research S¹. In some papers it means a chatbot with tool access. In others it means a multi-step autonomous system that can plan, execute, and self-correct. The difference matters: a chatbot that suggests a patch is a tool a human reviews. An autonomous system that applies the patch and moves on is a different risk entirely.

The review also finds that applications differ sharply in risk S¹. An agent that triages vulnerability reports operates in a different threat environment than one that executes exploit code in a sandbox. Yet the evaluation methods used to assess them rarely align S¹, featuring disparate datasets and metrics without common baselines. You cannot rank systems you cannot compare.

The gap between acting and accounting

The review's central synthesis is this: researchers have created systems capable of taking action, but they have not yet developed systems with restricted permissions or verifiable action logs S¹. That sentence does a lot of work. It says the capability exists. It says the control does not.

This connects to a broader thread in the literature. A May 2026 preprint by Christodorescu, Fernandes, Jha and colleagues, "Agent Security is a Systems Problem," argues that protecting LLM agents means viewing them as complete systems with strict limits, rather than just language models with instructions P⁴. A separate paper, "Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation," outlines the vulnerabilities, showing how an agent's data sources, utilities, or stored knowledge can be compromised P². The review synthesises work like this and finds the field has not yet absorbed its own lessons.

On the tooling side, the open-source ecosystem is moving fast. HKUDS/AutoAgent, a zero-code LLM agent framework on GitHub, has accumulated 9,753 stars since its creation in February 2025 under an MIT licence P³. SecCoderX, a smaller project focused on secure code generation through online reinforcement learning, launched in February 2026 with 41 stars P⁵. Both are building agents. Neither, by the review's criteria, has solved the authority or auditability problem.

What to do about it

For a security operations team at a mid-sized financial firm, say, a team of six analysts who triage 200 vulnerability reports a week, the review's findings translate into a concrete caution. The authors note that LLM-based agents are being rapidly adopted to automate procedural security workflows, the repetitive, rule-bound tasks that consume hours of analyst time S¹. An LLM agent that auto-classifies and prioritises those reports can save 30 hours a week. But if that agent's classification logic cannot be audited, if you cannot reconstruct why it flagged report A as critical and report B as noise, then you have traded transparency for speed. A regulator will eventually ask for the audit trail.

The practical step: before deploying any LLM-based security agent, ask the vendor two questions the review says the field cannot yet answer well. Can you bound what this agent is authorised to do? Can you produce a record of every decision it made and why? If the answer to either is no, treat the agent as an advisory tool, not an autonomous one.

What we don't know yet

The review is a single preprint, not peer-reviewed, and its synthesis is interpretive S¹. The 2023 to 2026 literature scope is unusual. Some cited 2026 work may not yet be publicly available, which makes independent verification harder. The authors' claim that assessment protocols are "often incomparable" is their reading of the field, not a finding confirmed by a second review or meta-analysis.

The next signal: the peer-reviewed version of this paper, if accepted, would carry the weight of editorial review. Watch also for a response from the "Agent Security is a Systems Problem" group P⁴, whose May 2026 preprint shares the review's framing but predates its synthesis. If those authors cite this review, the governance-gap thesis gains traction. We will check both against the next major AI security venue, likely the ACM CCS 2026 program, if the paper is submitted there.

If this kind of decode is useful, subscribe and you will get the next one before the preprint hits your feed.

Sources: S1 — LLM-Based Agents for Software and Systems Security: Approaches, Applic · P2 — Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Eval · P3 — HKUDS/AutoAgent · P4 — Agent Security is a Systems Problem · P5 — AndrewWTY/SecCoderX

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.

GitHub stars: LLM agent frameworks for security