Martin Marek and Max Ryabinin at Together AI published a preprint showing a single correction term matches the standard fix for RL instability in models from 0.6B to 30B parameters S¹. The standard fix, importance sampling, is what most practitioners reach for when training and inference engines disagree. The authors claim their "score centering" approach matches it and pulls ahead as the mismatch gets worse S¹.

My read: This is the first clean decomposition I have seen that separates drift from noise in the training-inference mismatch problem. I am skeptical of the claim that drift is the primary cause, because the paper offers no independent replication and the authors' own experiments are the only evidence. But the additive property is elegant: if score centering truly composes with importance sampling rather than replacing it, the downside risk of trying it is close to zero. I would watch for whether OpenRLHF or any other major RL framework picks it up.

The problem the paper attacks is real and widely felt. When you train a language model with reinforcement learning, you need two engines: one that updates weights and one that generates text for rollouts. In practice, teams use different engines for each. The training engine might run in full precision while the rollout engine uses quantization to save cost. These small differences create what the field calls the training-inference mismatch, or TIM S¹.

You cannot simply eliminate TIM. Running the rollout engine in full precision to match the trainer would, as the authors note, come at a major cost to rollout efficiency S¹. Rollouts are already the bottleneck in RL training pipelines. This is the kind of compounding system error that reinforcement learning faces across the stack, from power consumption to numerical precision.

Why drift accumulates

Marek and Ryabinin argue that the instability RL exhibits under TIM is primarily caused by drift, which they define as a persistent bias between the training and inference engines that accumulates with every training step S¹. Think of it like a scale that reads one gram heavy every time you step on it. A single measurement is off by a gram. Over a thousand measurements, the error compounds and the scale becomes useless.

Their fix is an additive "score centering" correction term that cancels this drift S¹. The term is derived mathematically, not learned. Because it is additive, it composes with existing importance sampling methods rather than replacing them. In the authors' staleness experiments, the combination of score centering and importance sampling outperforms pure importance-sampling baselines S¹.

What the numbers show

The authors report results across model sizes from 0.6B to 30B parameters. Score centering alone matches or outperforms importance sampling under quantization, and the gap between the two methods grows as the mismatch becomes more severe S¹. This is the key claim: the worse the mismatch, the more score centering helps relative to the standard approach.

The paper provides no error bars, and nobody outside the authors' lab has checked these results. The paper is a preprint and has not been peer-reviewed S¹. The model size range is specific to the authors' experimental setup, and the method has not been tested outside the 0.6B to 30B range. These are the same kinds of unverified generalisation gaps that plague few-shot learning approaches across reviewed studies: a method works in the lab, on the models the authors chose, and the leap to production is yours to make.

Who would use this first

The natural adopters are teams running large-scale RLHF or RL training pipelines where rollout efficiency matters. OpenRLHF, a framework with nearly 10,000 GitHub stars that supports PPO, DAPO, and REINFORCE++ with vLLM-based rollouts, is the kind of codebase where an additive correction term could be added with minimal disruption P⁵. A team running quantized inference for rollouts to save GPU cost, but losing training stability as a result, is the exact operator this targets.

The concept builds on earlier work in RL theory. Richard Sutton and colleagues at the Alberta Machine Intelligence Institute introduced "reward centering" as a related idea for stabilising value estimates P⁴. Marek and Ryabinin apply a similar centering principle to the score function in the context of LLM training under engine mismatch.

No code has been released for this specific paper. A GitHub repository under the username flxst exists for a related but separate paper on output embedding centering P³, not for the score centering method described here. Teams wanting to test the claim would need to implement the correction term from the paper's derivation.

The arXiv preprint is dated September 17, 2026 S¹. The next checkpoint for anyone tracking this work is whether the authors release code, or whether a framework like OpenRLHF integrates the method.

Sources: S1 — Score Centering Stabilizes Off-policy Reinforcement Learning · P2 — Score Centering Stabilizes Off-policy Reinforcement Learning · P3 — flxst/output-embedding-centering · P4 — Reward Centering · P5 — OpenRLHF/OpenRLHF

Written from 5 sourced items, 4 of them primary.

Model sizes tested with score centering