Hanming Yang and colleagues at Columbia Business School found a way to estimate one-in-a-billion AI agent failures 800× more cheaply than brute-force simulation S¹. The received view in AI safety testing says run enough simulations and the rare failures eventually surface. For events with probabilities near 10^-9, that approach needs billions of runs, which nobody can afford S¹.
My read: This is the first method I've seen that attacks rare-event estimation for language-model agents by perturbing the model's own weights rather than searching over output trajectories. The 800× figure is impressive but explicitly scoped to the authors' "most verifiable settings" and events below 10^-7 probability, so I would treat it as an upper bound until independent replication confirms it. The fact that they tested only up to 2.6 billion parameters leaves open whether this scales to the 70B-plus models that production agents actually run on.
Why "can it happen" is the wrong question
The authors, Yang, Daksh Mittal, Jing Dong and Hongseok Namkoong from Columbia Business School's Decision, Risk, and Operations Division P², argue that safe deployment of autonomous agents depends on how often rare events might occur, not just whether they can occur S¹. An agent that crashes a system once in a million attempts is a different deployment decision from one that does it once in a thousand, even though both "can" fail.
The rare events they study come from stochastic variation in the agent's own actions, not from external system failures S¹. Every time a language-model agent samples a token, there is a small chance it heads down a path that leads to a harmful action. Those paths are real but vanishingly rare, and the combinatorial space of possible trajectories is too vast for naive Monte Carlo to search efficiently S¹.
Perturbing weights instead of searching outputs
Standard importance sampling, a technique that reweights simulations to make rare events more frequent and then corrects for the bias, requires coordinated changes to a chain of conditional distributions that depend on context S¹. The authors take a different route: they perturb the original model's weights to construct a new proposal distribution, which is itself a differentiably parameterized language model S¹. That means they can use gradient-based search over weight space to find perturbations that amplify the rare events, rather than trying to search over the exponentially large space of possible outputs.
They formulate an objective that combines a differentiable surrogate for event amplification with an adaptive regularization scheme that balances amplification against estimator stability S¹. In practical terms, the method nudges the model toward the rare event without distorting the probability estimates so badly that they become meaningless.
The numbers, and their limits
The team evaluated the method on models with approximately 120 million and 2.6 billion parameters S¹. They tested across three event families spanning more than 300 rare events, with probabilities as low as 10^-9 S¹. Reference probabilities were computed with less than 10% relative standard error S¹.
In their most verifiable settings, the importance sampling estimator achieved over 800× compute-weighted efficiency gains over naive Monte Carlo for events with probabilities lower than 10^-7 S¹. That 800× figure is scoped: it applies to the authors' most verifiable configurations and to the rarest event regime, not across all probability levels or model sizes.
The paper is a preprint and has not been peer-reviewed S¹. Nobody outside the authors' lab has independently replicated the results, and the method has not been tested on models larger than approximately 2.6 billion parameters S¹. The code is publicly available at github.com/namkoong-lab/iterative-unalignment, though the repository has not been independently audited S¹.
Who would use this first
An AI safety team evaluating whether a language-model agent is safe to deploy in a customer-facing setting could use this method to estimate the probability of specific failure modes, like the agent leaking sensitive data or executing a harmful tool call, without needing to run billions of trial episodes. A deployment engineer building autonomous agents could pull the repository, run the method on a smaller model to validate the failure-rate estimates against known outcomes, and then assess whether the approach scales to their production model size.
The next checkpoint is whether other labs can reproduce the 800× efficiency gain on different model architectures. The code is available now at github.com/namkoong-lab/iterative-unalignment S¹.
Sources: S1 — Rare Event Estimation via Iterative Unalignment · P2 — Rare Event Estimation via Iterative Unalignment · P3 — AlthausKonstantin/rareeventestimation · P4 — princeton-nlp/unintentional-unalignment
Written from 4 sourced items, 3 of them primary.
