Zimo Xu at the University of Tokyo and colleagues blocked all 13,935 prompt-injection attacks against small AI agents in a new arXiv preprint S¹. The result challenges a core assumption in agent security: that the model itself should write the rules that keep it safe.
My read: The headline number is striking, but it comes from a single benchmark and a preprint no one outside the authors has checked. What I find convincing is the failure data for existing defenses. When 14 of 15 small models produce vulnerable security artifacts under Progent, the problem isn't tuning, it's structural. Deriving policy from tool schemas rather than model output sidesteps that failure mode entirely. What I'd watch is whether the 'write integrity' guarantee holds outside AgentDojo's attack set, and whether 29.0% utility under attack is enough for real workflows.
The model can't forge its own padlock
Current enforcement-based defenses — CaMeL, Progent, IPIGuard, and DRIFT, rely on the same fundamental mechanism: the agent's language model generates some form of security artifact, such as a program, policy, graph, or constraint set, which downstream code then executes S¹. The approach works when the model is large enough to produce correct security logic. On small models, it breaks.
Xu and co-authors Wenzhuo Liu of the Institute of Automation, CAS, and Hideki Nakayama of the University of Tokyo evaluated 15 open-weight models, 13 of them between 1.7 billion and 12 billion parameters S¹. The security artifacts those models wrote were riddled with holes. Under Progent, attacks succeeded on 14 of the 15 models. Under IPIGuard, 13 of 15. Under DRIFT, 11 of 15 S¹. CaMeL, which asks the model to write executable programs, had a different failure: its programs did not run in up to 68% of episodes S¹.
This echoes what Tencent researchers found when testing AI coding agents: small models trigger their own safety mechanisms in only 27 to 36% of runs. When a model is too small to reason reliably about security, asking it to write the security rules is asking for trouble.
A policy the model never touches
CapLite takes a different path. Rather than having the model generate a security artifact, it extracts its security policy directly from the tool schemas — the structured descriptions of each tool's inputs and behavior S¹. When the agent wants to make a write call, CapLite finalizes that call prior to reading back any output from the tool S¹. A write operation is permitted only when all of its decision-critical arguments, such as the recipient, monetary value, target object, or destination, appear verbatim in what the user originally asked for S¹.
The authors claim this guarantees what they call 'write integrity': malicious injected text is unable to introduce new write operations, and every write that does execute targets only the recipients, amounts, objects, and destinations that the user explicitly specified S¹.
If the user didn't say it, the agent can't do it.
No attack succeeded across 13,935 attacked AgentDojo episodes testing CapLite S¹.
Utility under attack, the fraction of episodes where the agent still completed the user's task while under attack, reached 29.0% for CapLite versus 27.2% for CaMeL S¹. CapLite consumed 17,100 tokens per episode — a figure the authors describe as lower than all rival enforcement defenses and even lower than running with no defense at all, amounting to just 18% of CaMeL's token cost S¹. The authors report CapLite achieved the highest utility per token of all enforcement defenses tested S¹.
That token efficiency matters for the models CapLite targets. A 1.7B-parameter model running on a phone has no budget for the multi-step program generation that CaMeL requires. SqueezeAILab's TinyAgent project, demonstrated at EMNLP 2024, showed function calling at the edge with small models P⁴. CapLite's low overhead fits that same niche.
What would have to hold
The paper is a preprint and has not been peer-reviewed S¹. All security and performance claims derive from a single unreviewed source. The evaluation is limited to the AgentDojo benchmark, and real-world robustness against novel prompt-injection techniques is unknown. The authors' 'write integrity' guarantee is an unverified claim, not a formally proven or third-party audited property. The paper does not indicate that CapLite's code is publicly available or deployed in production.
Other defense approaches are also evolving. A recent jailbreak defense that tunes attention heads without retraining shows the range of strategies researchers are trying. CapLite's contribution is narrower and more specific: it targets write integrity for small-model agents, and on that metric, the benchmark numbers are clean.
For a developer building an agent on a 3B or 7B model that handles email, calendar or file operations, the first question is whether CapLite's schema-derived policy covers their actual tool set. If a tool's schema doesn't cleanly separate 'who' and 'what' arguments from content arguments, the policy may block legitimate writes or miss attack vectors the benchmark didn't test. The next checkpoint is whether the authors release code and whether independent researchers reproduce the zero-attack result on a broader suite.
Sources: S1 — CapLite: Prompt-Injection Defense for Small Language Model Agents with · P2 — CapLite: Prompt-Injection Defense for Small Language Model Agentswitho · P3 — liu00222/Open-Prompt-Injection · P4 — SqueezeAILab/TinyAgent · P5 — CapLite: Prompt-Injection Defense for Small Language Model Agentswitho
Written from 5 sourced items, 4 of them primary.
