A new scanner from Google Cloud, posted to arXiv on 6 August, detects tampered safety training in 10 of 14 language model configurations spanning Llama, Gemma, Qwen and Mistral S¹. The tool, called AMS, reads the internal geometry of a model to check whether its safety guardrails are still intact. But the paper documents its own blind spot: the stealthiest tampering technique leaves no fingerprint in the signals AMS reads. Whether 71% is good enough depends entirely on which 29% gets through.

My read: This is the first safety-verification tool I've seen that maps its own failure mode instead of burying it. The 71% figure is modest, the p-value (0.043) sits right at the edge of conventional significance, and 14 models is a small sample. But the four-class taxonomy is the real contribution: it tells you exactly which tampering techniques you can catch and which you can't. I don't buy the compliance correlation yet, because r = -0.546 on 14 data points with the authors noting meaningful noise is thin evidence. What I'd watch is whether independent labs replicate the sigma signal on larger samples, and whether someone closes the behavioural fine-tuning gap with a different probing method.

How safety training writes itself into a model's geometry

When you train a language model to refuse harmful requests, the training reshapes the model's internal representations. In activation space, the high-dimensional map of neuron firing patterns, safety training creates a measurable gap between how the model processes harmful content versus benign content S¹. Think of it as the model learning to sort dangerous and safe requests into two separate piles inside its own brain.

AMS, short for Activation-based Model Scanner, measures that separation. The author, Glen Messenger of Google Cloud, built the tool to check whether a model's safety training is still in place by reading the geometric structure of safety-relevant concepts in activation space P². The code is public on GitHub under the GoogleCloudPlatform organisation, licensed Apache 2.0, written in Python P³.

The logic is simple. If safety training creates separation between harmful and benign content, removing that training should collapse the separation. AMS looks for that collapse.

The four ways to strip safety, and which ones AMS catches

The paper identifies four distinct ways someone can modify a model's safety training, each leaving a different fingerprint in activation space S¹.

Training removal is the bluntest approach. Strip the safety training out, and the separation between harmful and benign content clusters collapses. AMS catches this with its Tier 1 check, a threshold on a metric called sigma that measures how far apart the clusters sit S¹.

Weight-orthogonalisation abliteration is more surgical. Instead of removing training, you mathematically neutralise the refusal direction in the model's weights. This both collapses the cluster separation and rotates the refusal direction, the internal vector that points the model toward saying no. AMS catches this too, through the same Tier 1 sigma threshold S¹.

Rotation-without-collapse abliteration is subtler. It preserves the separation between harmful and benign clusters but rotates the refusal direction so the model no longer acts on it. The clusters are still apart, but the model's steering wheel has been turned. AMS needs its Tier 2 check, a direction-similarity verification, to catch this S¹.

Behavioural fine-tuning is the one that slips through. It preserves both the magnitude and direction of safety-relevant signals in activation space. The model's internal geometry looks intact, but its behaviour has changed. AMS cannot detect this by activation-only probing, and the paper says so explicitly S¹.

What the numbers actually say

The 71% accuracy figure comes from leave-one-out cross-validation across the 14 model configurations S¹. That means the tool correctly flagged 10 of 14 models as tampered or intact. The bootstrap 95% confidence intervals on the sigma estimates have a median width of 3.4 sigma, which is wide enough to signal real uncertainty in the threshold S¹.

AMS detection results across 14 model configurations

The paper also tested whether sigma, the separation metric, predicts actual model behaviour. On 20 stratified JailbreakBench prompts per model, sigma on the harmful-content concept correlated with behavioural compliance at Pearson r = -0.546, with p = 0.043 S¹. That correlation is negative, meaning more separation means less compliance with harmful requests, which is what you'd expect. But p = 0.043 is barely under the 0.05 threshold, and the authors note meaningful noise in the relationship S¹.

This connects to a wider problem in AI safety verification: the question of who checks whether models are safe, and how, is becoming a policy fight. AMS is one answer: look inside the model. But it is an answer with a hole in it.

What to do about it

If you are deploying a fine-tuned open-weight model, Llama, Mistral, Qwen or Gemma, in a setting where safety matters, AMS gives you a partial check. The tool is open source and runs in Python P³. You can clone it, point it at your model, and get a sigma score that tells you whether the safety-relevant clusters in activation space are still separated.

Consider a hospital deploying a fine-tuned Llama model for clinical note summarisation. The hospital's vendor hands over a model file and says the safety training is intact. Running AMS would catch the case where the vendor stripped safety training entirely or used weight orthogonalisation to disable refusals. It would not catch the case where the vendor did behavioural fine-tuning that changed how the model responds to harmful prompts while leaving the activation geometry untouched. For that, you still need behavioural testing, sending the model adversarial prompts and checking what comes back.

The practical takeaway: run AMS as a first-pass filter, but do not treat a clean result as proof that safety training is intact. Pair it with behavioural red-teaming, especially JailbreakBench-style prompts, which the paper itself used to validate the correlation between internal geometry and actual behaviour S¹.

What we don't know yet

The paper is a preprint that has not been peer-reviewed S¹. The 14-model sample is small, and the four architecture families tested, Llama, Gemma, Qwen and Mistral, do not cover every major model line S¹. Whether AMS generalises to larger parameter counts or to architectures outside those four is untested.

The compliance correlation, r = -0.546 with p = 0.043, is borderline. The authors acknowledge meaningful noise, and 14 data points is too few to call the relationship settled S¹. A related paper on measuring alignment-induced activation shifts, by Yuki Nakamura at the Open University of Japan, flags methodological pitfalls in comparing model internals that could bear on AMS's approach P⁴. Whether AMS's template-controlled comparisons avoid those pitfalls is an open question.

The biggest gap is behavioural fine-tuning. The paper names it as a documented failure mode and an open problem S¹. If someone can fine-tune a model to drop its guardrails while preserving the activation-space geometry that AMS reads, the scanner gives a false clean bill of health. Closing that gap likely requires a different probing method, not a better threshold on the same signal.

The next signal: whether independent labs replicate the sigma-compliance correlation on larger model sets, and whether the GoogleCloudPlatform repository gains community contributions extending AMS beyond the four architecture families tested. We'll check both against the repo's activity and any citing papers by October 2026.

If this kind of reporting helps you think clearly about what AI tools can and can't do, subscribe to keep reading.

Sources: S1 — Detecting Safety Training Modification in Language Models via Activati · P2 — Detecting Safety Training Modification in Language Models via Activati · P3 — GoogleCloudPlatform/activation-model-scanner · P4 — Measuring Alignment-Induced Activation Shifts Correctly: A Template-Co · P5 — Stability-AI/StableLM

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.