A new arXiv preprint from ETH Zurich and the Max Planck Institute proves that four leading discrete diffusion methods — MDM, UDM, SEDD, and GIDD — are all learning the same mathematical object, just written in different coordinates [S1]. But one of those coordinate systems, widely used in practice, makes the training objective explode to infinity before learning even begins. The question is which one — and why the choice matters more than anyone realised.

The quiet revolution in text generation

Discrete diffusion — where a model generates text by progressively un-corrupting tokens rather than predicting the next one — has been gaining ground as an alternative to autoregressive LLMs. SEDD, one of the methods this paper unifies, won ICML 2024's Best Paper award [P3]. Masked diffusion has its own growing literature [P2]. But despite the momentum, a basic question has gone unanswered: what are these models actually learning?

The preprint, authored by Rodrigo Casado Noguerales, Bernhard Schölkopf, Thomas Hofmann, and Aran Raoufi [P4], tackles this head-on. It has not been peer-reviewed [S1].

One object, three masks

The paper's central insight is that at the level of jump rates — the probabilities governing how a noisy token transitions to a cleaner one — three seemingly different predictors are the same thing in different coordinates: a denoiser, a score ratio, and a bridge plug-in [S1].

This matters because, as the authors show, reading a neural network in the wrong coordinate system silently changes both the process being trained and the process being sampled [S1]. You think you're training one model; you're actually training another.

For sequences with token-factorizing noise, the authors derive exact closed-form conversions among these three coordinates [S1]. They also show that the denoiser and cavity (bridge plug-in) coordinates coincide for masked diffusion — where tokens are blanked out — but do not coincide for uniform diffusion, where tokens are scrambled randomly [S1]. That distinction turns out to be more than academic.

The Oracle Distance theorem

The authors derive a continuous-time Markov chain evidence lower bound (ELBO — the standard training objective for diffusion models) for any noising process, including boundary terms that previous derivations have sometimes dropped [S1]. Then they prove what they call the Oracle Distance theorem: the negative ELBO is exactly equal to the data entropy plus the path KL divergence from the oracle reverse process to the learned one [S1].

Not a bound. An exact equality.

This means the unique optimizer — the best the model can possibly do — is the conditional expectation of the true reverse jump rate given the current noisy state [S1]. And every noising process, no matter how it corrupts the data, shares the same best achievable negative ELBO: the data entropy [S1]. The irreducible cost — the part no model can escape — is the rate at which the forward process destroys information about the clean data [S1].

The bombshell in the fine print

Here is where theory meets practice. The authors prove that a denoiser parameterization — one of the three coordinate systems — makes the uniform ELBO diverge at initialization [S1]. The bridge plug-in parameterization stays finite [S1].

If you initialise a uniform diffusion model with a denoiser setup, your training loss is infinite before you take a single gradient step. Switch to the bridge plug-in, and it's a number you can work with.

The framework also identifies which law each existing loss in the literature actually optimizes, recovering MDM, UDM, SEDD, and GIDD as special cases [S1]. Every identity was verified numerically, without approximation, on an exactly solvable model [S1].

What it means

For researchers and engineers working on discrete diffusion, this paper offers something rare: a map. Instead of choosing among MDM, SEDD, GIDD, and UDM as if they were competing products, you can now see them as points on the same mathematical surface, each optimizing the same object through a different lens [S1].

The practical takeaway is concrete: if you're working with uniform diffusion — where tokens are randomly replaced rather than masked — avoid the denoiser parameterization at initialization. Use the bridge plug-in instead. The denoiser coordinate works fine for masked diffusion, where the two coincide, but for uniform noise it produces an infinite loss before training begins [S1].

The Oracle Distance theorem also gives practitioners a precise diagnostic. Because the negative ELBO decomposes exactly into data entropy plus a KL divergence, you can separate the irreducible cost — information destroyed by the noising process — from the part your model can actually improve [S1]. That is a sharper tool than the loose bounds most diffusion papers work with.

What it means for business

For the small AI labs and two-person teams experimenting with discrete diffusion as an alternative to autoregressive models, this paper is a debugging guide disguised as theory. If your uniform diffusion model is producing NaN losses at initialization — a common and frustrating failure mode — the likely culprit is your parameterization, not your architecture or hyperparameters [S1].

For teams already running masked diffusion in production, the paper offers reassurance: the denoiser coordinate you're probably already using is mathematically sound for that setting [S1]. No change needed.

For larger organisations evaluating whether discrete diffusion is worth investing in over autoregressive LLMs, the unification matters strategically. It means the field's fragmentation — four methods, each with its own paper and codebase — is less of a barrier than it appeared. The underlying theory is one theory. Engineering effort spent on one approach transfers, with known conversions, to the others [S1].

What we don't know yet

This is a single preprint, not peer-reviewed, from one research group [S1]. The numerical verification was done on an exactly solvable model — a controlled, simplified setting — not on large-scale real-world data or production-scale discrete diffusion models [S1]. Whether the practical recommendations, particularly the divergence of the denoiser parameterization for uniform diffusion, hold at scale remains untested.

The paper is purely theoretical; it introduces no new architecture and reports no benchmark results [S1]. It tells you what the models are learning and which coordinate to use, but not how fast they converge or how well they generate text compared to autoregressive baselines.

Independent replication of the mathematical results has not been reported. The authors' affiliations — ETH Zurich and the Max Planck Institute for Intelligent Systems [P4] — lend credibility, but the claims await verification.

The next concrete signal to watch: whether any of the discrete diffusion codebases, such as the SEDD repository [P3], incorporate the bridge plug-in parameterization for uniform noise in response to this work.

If you want to understand where text generation is heading next, this is the kind of paper that separates the signal from the noise. Subscribe to keep reading the research that actually moves the field.

Sources

[S1] "What Does a Discrete Diffusion Model Learn?" arXiv preprint (cs.AI, cs.LG), 2026. https://arxiv.org/abs/2607.05381v1

[P2] "Simplified and Generalized Masked Diffusion for Discrete Data," arXiv, 2024. https://arxiv.org/pdf/2406.04329

[P3] louaaron/Score-Entropy-Discrete-Diffusion, GitHub. https://github.com/louaaron/score-entropy-discrete-diffusion

[P4] "What Does a Discrete Diffusion Model Learn?" arXiv HTML. https://arxiv.org/html/2607.05381

Sources

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.