When researchers at Purdue University handed AI coding agents a set of instructions most developers would consider straightforward — do this, but only if that condition is met; if it fails, run this recovery step — the agents ignored the rules up to 70% of the time [S1]. The violations were not abstract. They produced privacy leaks, unsafe configuration changes, and incomplete cleanup of temporary files [S1]. The question now is whether the fast-growing market for agent skills — reusable procedure packs that extend what LLM agents can do — is building on a foundation that quietly fails when the instructions get complicated.

Skills are the new app store for AI agents

Agent skills are bundles of reusable procedures, tools, and domain-specific workflows that tell an LLM agent how to accomplish a task [S1]. Think of them as recipes: step-by-step instructions an agent reads and executes. The Hugging Face skills repository alone has drawn over 10,700 stars on GitHub [P5], and the broader ecosystem is expanding fast.

The problem is that real skills are not simple linear recipes. They contain logical relations — rules that govern when, how, and under what conditions each step should run. The Purdue team built a taxonomy of eight relation types, including preconditions that gate valid actions, constraints that limit how an allowed action may be performed, and fallbacks that specify recovery behaviour after a failure [S1].

When the researchers scanned more than 5,000 public skills, they found that 70% contained at least one such logical relation [S1]. In other words, the majority of skills in the wild are not plain instructions. They carry conditional logic that agents must parse and respect.

SLBench: 86 cases that expose the gap

To test whether agents actually follow that logic, the authors built SkillLogic, a framework for analysing the relations embedded in skill files and turning them into executable tests [S1]. From that framework they constructed SLBench — an 86-case benchmark drawn from relations the authors classified as high-confidence, high-impact, and locally testable [S1].

They then ran two agent platforms, Codex and Claude Code, across six different LLM backbones [S1]. The results were stark: unsafe rates reached as high as 70% on the worst-performing combinations [S1]. The phrase "up to" matters here — that figure is a ceiling across tested backbones, not an average. But even the best-performing combinations left a meaningful gap between what the skill instructed and what the agent did.

The violations fell into three concrete categories: privacy leaks (agents exposed data the skill said to protect), unsafe configuration changes (agents altered settings the skill said to leave alone), and incomplete cleanup (agents abandoned temporary files or processes the skill told them to remove) [S1].

Why agents break the rules

A human audit of the failures pointed to two causes working in tandem [S1]. First, agent capability gaps — the models simply failed to reason through the conditional logic. Second, low-salience skill text — the instructions were written in a way that buried the critical rule among less important details, so the agent skimmed past it.

That second finding is quietly significant. It means the problem is not purely a model limitation. Skill authors who write clear, prominent conditionals may get better compliance than those who tuck a safety rule into paragraph four. The benchmark code and related tooling are public on GitHub [P4], which means skill authors can test their own instructions before publishing.

What it means

The core finding is that logical-relation following is a distinct reliability challenge for skill-guided agents — separate from the model's raw coding ability or tool-use accuracy [S1]. An agent can be excellent at writing code and still blow through a precondition because the instruction was phrased as a footnote.

For anyone building or buying agent systems, this reframes the safety question. It is not enough to ask whether the agent can perform the task. You have to ask whether the agent can read a multi-step instruction with conditions, constraints, and fallbacks, and execute each branch correctly. The 70% unsafe rate on SLBench suggests that for many current combinations of agent platform and model backbone, the answer is no — at least not reliably.

The fix the authors propose is SLGuard, a lightweight inference-time scaffold — a layer that runs alongside the agent and checks its actions against the skill's logical relations before execution [S1]. In tests, SLGuard reduced violations by 63% on targeted cases [S1]. That is a meaningful improvement, but it is not elimination. More than a third of targeted violations still got through.

What it means for business

A two-person dev shop that ships agent skills to customers — or a suburban agency that builds custom workflows on top of Claude Code or Codex — now has a concrete risk to manage. If your skill says "delete the temporary database only after confirming the backup completed," there is a real chance the agent skips the confirmation and deletes it anyway.

The practical steps are grounded in what the study found. First, audit your skill files for logical relations and make the critical rules prominent — the human audit showed low-salience text contributed to failures [S1]. Second, if you are running an agent platform that supports inference-time guards, test whether a scaffold like SLGuard catches your highest-risk cases. Third, do not assume a strong model solves this. The study tested across six backbones and found unsafe behaviour in all of them [S1].

For teams selling agent-based services, the benchmark also offers a due-diligence tool. You can run a prospective agent platform against SLBench's 86 cases before committing [P4]. If a vendor's platform fails 50% of them, that is a number you can put in a procurement decision.

What we don't know yet

This is an arXiv preprint that has not been peer-reviewed [S1]. The benchmark, the 5,000-skill scan, and the SLGuard results are all self-reported by the same team that built the framework. Independent replication has not yet appeared.

The 63% violation reduction from SLGuard applies only to targeted cases — the specific relation types the scaffold was designed to catch [S1]. Whether it generalises to the full range of logical relations in real-world skills is an open question.

The 86 cases in SLBench are deliberately curated for testability, which means they may not represent the messier, ambiguous instructions found in production skills. A skill that says "use your best judgement" is harder to benchmark than one that says "check the return code before proceeding."

The next thing to watch is whether agent platforms — OpenAI's Codex, Anthropic's Claude Code, and others — integrate relation-checking scaffolds natively. If they do, the 70% unsafe rate becomes a baseline they will need to beat. If they do not, skill authors are on their own.

If this kind of grounded, evidence-first analysis is what you want from AI coverage, subscribe — there is more where this came from.

Sources

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.