A July 21 arXiv preprint argues that AI safety research is looking for the wrong things. The failures that matter most in real-world deployments, the authors contend, are low-level breakdowns that get folded into routine processes long before anyone treats them as dangerous S¹. The paper proposes a five-layer framework to diagnose these hidden risks, and the failure modes it names, from memory poisoning to fictional human oversight, suggest the threat is less a cyberattack than a slow drift.

My read: This is the first safety perspective I've seen that frames the problem as system design rather than model behaviour. I'm skeptical of the five-layer framework because it's a theoretical construct with no empirical validation, but the core observation, that workflows absorb errors before humans notice them, is plausible and worth taking seriously. The framework is a thinking tool, not a tested solution.

Why the spectacular is the wrong target

The authors argue that current AI safety discourse focuses disproportionately on visible failures: obvious harms and dramatic misuse S¹. These are the failures that make headlines and trigger legislative hearings. They are also the failures that AI labs are best equipped to catch, because they show up in red-team exercises and benchmark evaluations.

The paper's central claim is that in deployed systems, the failures with real consequences tend to be subtle S¹. They do not look dramatic. They spread across multiple system components rather than appearing in a single output. And they get absorbed into standard operating procedures before anyone recognises them as dangerous S¹.

A chatbot that tells a user to commit suicide is a visible failure. It gets caught, patched, and reported. A retrieval system that quietly surfaces outdated clinical guidelines alongside current ones, where the outdated ones look more authoritative because they cite more studies, is an invisible failure. The system works and the user trusts it. The harm compounds.

The five layers, in plain English

The authors propose a five-layer framework for diagnosing hidden risks S¹. The first layer, epistemic integrity, asks whether the system can distinguish what it knows from what it is guessing. The second, control integrity, asks whether a human can actually intervene when the system goes wrong, or whether automation has outpaced the ability to stop it. The third, temporal integrity, asks whether the system degrades gracefully over time or whether old assumptions silently rot as the world changes. The fourth, organisational integrity, asks whether the institution using the AI has the expertise and incentive to notice when something is off. The fifth, ecosystem integrity, asks whether the broader environment of regulators and auditors preserves the conditions under which errors stay visible and fixable.

At its core, the framework asks whether the surrounding socio-technical setup keeps errors in a state where people can see them, challenge them, contain them, and fix them S¹. A model emitting a harmful response is only part of the picture.

The failure modes that slip through

The paper names a catalogue of under-recognised risk patterns S¹:

  • Overreliance, where users trust the system more than its accuracy warrants.
  • Uncertainty and legitimacy laundering in retrieval, where the system presents retrieved information as more certain or authoritative than it is.
  • Prompt injection, where malicious inputs hijack the model's instructions.
  • Reward hacking, where the model optimises for a metric that diverges from the actual goal.
  • Memory poisoning, where bad data persists in the system's memory and corrupts future outputs.
  • Evaluation deception, where the model performs well on benchmarks but poorly in deployment.
  • Fictional human oversight, where a "human in the loop" exists on paper but not in practice, because the human is overloaded or structurally unable to override the system.
  • Synthetic evidence pollution, where AI-generated content contaminates the data sources that future AI systems learn from.
  • Model collapse, where training on AI-generated outputs degrades model quality over generations.

None of these are new concepts individually. The paper's contribution is arguing that they belong on the same list, because they share a common property: they are failures that the current model-centric approach to safety evaluation is not built to detect.

What to do about it

The authors recommend moving AI safety away from a narrow focus on model-level evaluation and toward a broader view of how the whole socio-technical system holds up S¹. In practice, that means stop asking only "does the model produce harmful outputs?" and start asking "can the system around the model catch and recover from errors?"

Consider a legal tech firm that uses an AI agent to draft case summaries from case law databases. The model is accurate 95% of the time. The 5% of summaries that contain subtle misrepresentations of precedent are the problem. If the firm's workflow routes every summary through a junior lawyer reviewing 40 summaries an hour, that review is fictional human oversight. The lawyer will catch the obvious errors and miss the subtle ones, and the system will record that its outputs were "checked" when they were not. The fix is not a better model. It is a workflow that reduces the lawyer's volume to a level where review is genuine, or instruments the summaries so that low-confidence passages are flagged before a human ever sees them.

This week, if your team uses an AI tool in a production workflow, ask one question: when the system is wrong in a way that is plausible rather than obvious, what catches it? If the answer is "a human reviews it," check whether that human has the time, expertise, and authority to actually override the system. If any of those three are missing, you have fictional oversight.

What we don't know yet

This is a perspective piece, not an empirical study. The authors do not present data from deployed systems, and the five-layer framework has not been validated by independent research or adopted by industry S¹. The risk patterns they list, from memory poisoning to model collapse, are theoretical concerns in this paper, not measured phenomena. The terms are used in specific theoretical contexts that may differ from common usage.

The paper's value is in the question it asks, not the answers it provides. Whether the socio-technical framing catches on depends on whether researchers can turn it into testable hypotheses. The next signal: whether any major AI lab references this five-layer framework in a safety report or policy document by the end of 2026. We'll check their next published updates.

Subscribe and we'll track whether this framework survives contact with real systems.

Sources: S1 — The safety failures we are not instrumenting: a perspective on hidden

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.