A preprint on arXiv has run two paired audits across five large language models, zeroing out neuron rows one at a time to test whether the neurons we think drive safety refusals actually do [S1]. The attributed rows are sufficient to install refusal on hate and crime while keeping the model fluent — but the same paper reveals something that should make anyone editing model safety pause before they celebrate. The question is: if these are the "refusal neurons," why do completely different methods find completely different ones that work just as well?

The audit nobody was running

Attribution scores — numerical rankings that claim to tell you which neuron rows in a language model matter most — are now standard tools for pruning, interpretability, and safety editing [S1]. If you want to strip out a model's ability to generate harmful content, or understand why it refuses a benign request, you reach for these scores to point you at the right neurons.

What's rarely asked: do the neurons these scores highlight actually cause the behaviour, or do they just correlate with it?

The paper, "Faithfulness to Refusal: A Causal Audit of Neuron Selectors," tackles this gap with two paired audits built on one-shot neuron-row zeroing — a technique that silences individual neuron rows and watches what breaks [S1]. The approach is brutally direct: instead of trusting a ranking, you pull the plug and see if the lights go out.

Attribution beats the baselines — but that's not the headline

Across five LLMs, attribution-based methods substantially outperformed activation and magnitude-based baselines at identifying dispensable rows [S1]. When the authors zeroed the rows flagged by attribution scores, the model's refusal behaviour changed. When they zeroed rows flagged by simpler heuristics — how active a neuron is, or how large its weights are — the effect was weaker.

So far, so reassuring. The attributed rows were sufficient to install refusal on hate and crime prompts while keeping benign over-refusal low, and the intervention preserved the model's fluency [S1]. Layer-matched random controls at the same depths failed, meaning you can't just zero any neurons at the right layer and get the same result [S1].

But here's where the story turns.

The redundancy problem

Refusal, the paper found, lives in a redundant subspace [S1]. Different attribution methods install refusal through largely disjoint row sets — meaning Method A and Method B both work, but they're pointing at different neurons. The recovered edit is one realisation of a sufficient set, not a unique mechanism [S1].

Think of it like a fuse box. You can cut power to the kitchen by pulling the fuse labelled "Kitchen," or by pulling the main breaker, or by disconnecting the right wire in the junction box. All three work. None of them is the kitchen switch. The paper is saying something similar about refusal: there are multiple, overlapping sets of neurons that can each produce the behaviour, and the set any one method finds is just one that works — not the mechanism.

This matters because the interpretability field has been building toward "we found the refusal neurons" as a goal. A related preprint, "Targeted Neuron Modulation via Contrastive Pair Search," notes that popular steering methods operate on the residual stream and degrade output coherence at high intervention strengths [P4] — a sign that the field already suspects current methods are blunt instruments. And a separate mechanistic audit of Qwen2.5-1.5B-Instruct, run on a laptop in roughly 4.5 hours of CPU time, shows how accessible this kind of investigation has become [P2].

The rank-stability trap

The paper's most counterintuitive finding: highly rank-stable selectors — methods that consistently flag the same neurons across different inputs or runs — can be among the least causally valid [S1].

Rank stability is a proxy. It says: "this method keeps pointing at the same neurons, so those neurons must be important." But the audit shows that consistency and causal importance are different things. A method can reliably identify the same neurons, and those neurons can reliably do nothing when you silence them. Rank-stability proxies miss the kinds of selector failures a direct causal audit can surface [S1].

For anyone building safety edits on top of attribution scores, this is the detail that should reframe the workflow. The metric you've been using to validate your neuron selection might be validating the wrong thing.

What it means

The core finding isn't that attribution methods don't work — they do, better than the alternatives. It's that "works" means "finds a sufficient set," not "finds the mechanism." For regular people, this means the safety behaviours baked into AI models are more distributed and harder to surgically modify than the field has assumed. You can install refusal. You can remove it. But you can't point at a small, unique cluster of neurons and say "that's where the refusal lives."

For a smart reader with no background: imagine being told a city's traffic flows through one intersection, only to discover there are five different intersections that each handle the full load. Removing any one of them doesn't stop traffic. The map you were given wasn't wrong, exactly — it was just one of several valid maps.

What it means for business

For a two-person AI startup building custom model behaviour — say, a suburban agency fine-tuning an open-weight model to refuse certain query types — the practical implication is caution. Attribution-based neuron editing will likely work for installing or removing targeted behaviours, but the redundancy finding means you shouldn't assume your edit is targeting a unique mechanism. If a competitor or an adversary uses a different attribution method, they may identify a completely different set of neurons to manipulate — and your safety edit won't catch it.

For teams doing model pruning — cutting neurons to shrink model size and inference cost — the rank-stability finding is a warning light. If you've been using rank stability to decide which neurons are safe to cut, this paper suggests that metric can fail precisely where it matters most. A direct zeroing test — silence the neuron, check if behaviour changes — is the more reliable validation, though it costs compute.

For compliance teams: the finding that refusal is redundant, not unique, complicates any audit framework that assumes safety behaviours can be localised to specific, inspectable components. Regulators expecting a "show me the safety neurons" approach may need to grapple with the reality that there is no single such set.

What we don't know yet

The paper is a preprint, not peer-reviewed [S1]. No independent replication studies are cited, and the technical claims about causal neuron-level interventions are contested within the interpretability field. The five LLMs tested are not named in the available abstract, so we can't yet assess whether the findings generalise across model families, sizes, and training regimes.

The redundancy finding raises a question the paper doesn't fully answer: if multiple disjoint neuron sets can each install refusal, what determines which set a given attribution method finds? Understanding that selection pressure would tell us whether different methods are exploring the same redundant subspace or sampling from genuinely different mechanisms.

The rank-stability result also needs explanation: why do selectors that are consistent across inputs fail causally? One possibility is that rank-stable neurons are involved in general processing rather than refusal-specific circuits — but this is speculation the paper doesn't confirm.

The next concrete event to watch: peer review and independent replication. Until another lab reproduces the zeroing audits on different models and confirms the redundancy finding, this is a strong signal, not a settled result.

If you want to keep reading work that decodes AI research before the hype cycle catches up, subscribe — the next audit might be the one that changes how you deploy.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.