A new arXiv benchmark runs 10 AI agent configurations through 20 tasks that ask them to reverse-engineer concepts inside Google's Gemma-2-9B-IT language model, and the agents show real discovery ability but remain well behind expert researchers S¹. The strangest finding: agents can design experiments to rule out wrong answers, but they routinely misread what their own measurements tell them. That gap, between setting up the test and understanding the result, is what separates a lab assistant from a scientist, and the paper doesn't resolve whether agents can close it.

My read: This is the first benchmark I've seen that treats interpretability research as a closed-loop skill rather than a pattern-matching exercise. The agents aren't failing at the hard part and passing the easy part. They're failing at the part that feels like it should be straightforward: reading a number and knowing what it means. I don't buy the authors' framing around recursive self-improvement yet, because a system that misreads its own experiments can't reliably build on its own findings. I'd watch whether that measurement problem is specific to SAE tools or generalises across every autonomous research loop.

What the benchmark actually tests

The paper, from Yuqiao Tan and colleagues at the Institute of Automation, Chinese Academy of Sciences, introduces SAEScientist-Bench P². The setup is specific: agents receive a target concept and must find the exact feature inside Gemma-2-9B-IT that represents it S¹.

The search space is vast. Gemma Scope, Google's open dictionary of sparse autoencoder features for Gemma-2, contains more than 131,000 individual features S¹. A sparse autoencoder, or SAE, is a tool that decomposes a neural network's internal activations into individual, human-interpretable units, each roughly corresponding to a single concept. Finding the right one for a given concept is the core labour of mechanistic interpretability, the field that tries to reverse-engineer how neural networks actually work.

The benchmark scores agents on three dimensions: activation rank (does the feature fire on the right inputs?), concept selectivity (can it distinguish the target concept from contrastive examples?), and causal steering (if you turn the feature up or down, does the model's behaviour change in the expected direction?) S¹. Expert researchers have curated reference features for each task, hosted on Neuronpedia, an open platform for browsing SAE features S¹.

The strange split in results

Across all 10 agent configurations and 20 tasks, frontier agents show what the authors call genuine discovery capabilities S¹. On concept selectivity, the narrowest measure, agents approach expert levels at separating a target concept from contrastive controls S¹. That means they can look at a batch of text and tell which feature lights up for the right reason.

But on causal steering, the test that matters most, agents lag substantially behind experts S¹. Causal steering is the proof step: you take the feature you found, amplify it inside the model, and check whether the model's output shifts in the predicted direction. If you claim you found the sycophancy feature, turning it up should make the model more sycophantic. Agents struggle to do this reliably.

The most telling detail sits in the middle of the process. Agents can design contrastive experiments, feeding the model pairs of texts to rule out spurious candidate features S¹. But they frequently misinterpret the experimental measurements those tests produce S¹. They set up the right test and then read the wrong answer from the results.

The SAEScientist-Bench results suggest a ceiling on agent autonomy: agents can do the mechanical work of research, but the interpretive step, deciding what an experiment actually proves, remains where human judgement earns its keep.

The authors position their work as a way to quantify how well agents can understand models through experimentation, framing it as a step toward measuring closed-loop autonomous AI research capability S¹. That is a careful claim. They are not saying agents can do science. They are saying we can now measure how far they are from doing it.

A related GitHub repository, SAGE (SAE Automated Generation of Explanations), exists with 3 stars and was last updated in November 2025, suggesting the tooling around automated SAE interpretability is still in its earliest stages P⁵.

What to do about it

For AI safety teams and interpretability researchers at mid-size labs, this benchmark offers a concrete way to test whether agent pipelines can handle the tedious first pass of feature discovery. Consider a three-person safety team at a frontier lab that needs to audit a new model for deceptive alignment before deployment. Today, that team manually searches through tens of thousands of SAE features, testing each one against contrastive text pairs and running steering experiments. SAEScientist-Bench suggests an agent could handle the first filter, narrowing 131,000 features down to a shortlist of candidates that fire on the right inputs. The humans then take over for the causal steering and the interpretation of results, the two steps where agents still fail.

That division of labour is the practical takeaway: use agents for the search, use humans for the proof. Anyone building an agent pipeline for interpretability work should build the human checkpoint at the measurement-interpretation step, not at the experiment-design step, because that is where the agents break.

One thing to check this week: browse the Neuronpedia platform referenced in the paper S¹ and try searching for a concept in a Gemma-2 SAE dictionary yourself. The scale of the search space, 131,000 features, is the best way to feel why agents are useful here even when they can't close the loop alone.

What we don't know yet

The paper is a preprint and has not been peer-reviewed S¹. The benchmark, the expert baselines, and the evaluation metrics are all designed by the same team, and no independent group has validated the results. The claim that agents approach expert levels on concept selectivity is relative to the authors' own expert reference features, which may not represent the full range of possible correct answers.

The 10 agent configurations and 20 tasks are a narrow sample. We don't know how performance changes with different language models, different SAE dictionaries, or more complex target concepts. The measurement-misinterpretation problem could be a fundamental limit on autonomous research loops, or it could be a prompt-engineering issue that better tool descriptions fix.

The recursive self-improvement framing in the paper extends beyond what the data shows. A system that misreads its own experiments cannot reliably build on its own findings, so the path from genuine discovery capabilities to closed-loop autonomous R&D is proposed here, not demonstrated.

The next signal: whether an independent lab reproduces the benchmark with different expert baselines in the weeks following the paper's 8 September release. We'll check this claim against any replication that surfaces on arXiv.

If you want to follow this thread as it develops, subscribe and we'll track the replication efforts as they land.

Sources: S1 — SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretabil · P2 — SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretabil · P3 — [2609.09113] SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE · P4 — Paper page - SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE · P5 — jiujiubuhejiu/SAGE

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.

Agent performance vs expert baseline by evaluation dimension