A new arXiv preprint proposes FactorDiff, a framework that breaks discrete diffusion samples into pixel-level factors and routes each one to the most relevant pre-trained expert [S1]. Validated on the ARC-AGI reasoning benchmark, the approach consistently beats the global weighting schemes that have dominated expert composition so far [S1]. The gap between these two methods points to something deeper about how AI models combine knowledge, and why treating a sample as a single block has been capping performance on exactly the tasks that demand the most from composed models.

The problem with one-size-fits-all composition

Discrete diffusion models generate data by gradually turning noise into structured output, one discrete step at a time. This family of models has been gaining traction as an alternative to autoregressive generation for reasoning tasks.

One of the big promises is compositional generation: taking multiple pre-trained expert models and combining them so the combined system can handle tasks none of the individual experts saw during training [S1]. Think of it like assembling a team where each member brings a different specialism.

Until now, the standard approach has used time-dependent mixing weights to blend these experts together. Recent theoretical work refined this by adjusting those weights over the course of the diffusion process to better match the intended target distribution [S1]. But there is a structural flaw. These methods work on a per-sample basis. They treat each generated state as one monolithic block and apply a single weighting across the whole thing [S1]. A pixel in the corner gets the same expert assignment as a pixel in the centre, even if different experts specialise in different regions or functions.

How FactorDiff splits the difference

The authors' core insight is that samples can be broken down into smaller factors [S1]. Instead of asking which expert should generate this entire sample, FactorDiff asks which expert should generate this specific factor of the sample.

The framework instantiates this idea with spatial, pixel-level compositions [S1]. During sampling, each factor is dynamically routed to the expert best suited to handle it [S1]. An expert trained on grid patterns handles the grid-like region. An expert trained on shapes handles the shape region. The routing happens at every step of the diffusion process.

The researchers tested this on the ARC-AGI benchmark, a set of reasoning tasks that require logical consistency and spatial disentanglement [S1]. The results, described qualitatively in the preprint, show that simple factor-specific routing consistently outperforms complex global scalar weighting schemes on these tasks [S1].

What it means

The finding challenges a basic assumption in expert composition: that you need a single, sophisticated weighting scheme to blend experts well. FactorDiff's routing is simpler in mechanism, yet it performs better on tasks where different parts of the output demand different kinds of knowledge.

This matters because discrete diffusion is still an emerging field. The foundational score-entropy approach that made discrete diffusion practical won best paper at ICML 2024 [P5], and follow-up work has been tackling remaining problems like factorization errors in language models [P2] and latent augmentation for richer generation [P4]. Expert composition is one of the field's most ambitious goals: combining specialised models to achieve generalisation beyond any single model's training data. If the bottleneck was not the weighting maths but the granularity of the assignment, that changes the research direction.

For anyone following the broader push toward more capable AI reasoning, FactorDiff suggests that the way experts are combined deserves as much attention as the mathematics behind the weighting. Routing each piece of a sample to the right expert is a different mental model from blending all experts into one averaged opinion. Separate work on expert merging, which aligns and combines specialised model components without retraining [P3], points in a similar direction: the field is moving toward finer-grained control over which knowledge applies where.

What it means for business

No code has been released, and the paper is an unreviewed preprint, so nothing here is deployable today [S1]. But the direction matters for operators watching the cost and capability of AI reasoning systems.

A small AI firm building custom reasoning tools might find the idea of composing pre-trained experts appealing. Rather than training one giant model, you add a new capability by training or acquiring a narrow expert and slotting it into the composition pipeline. FactorDiff's factor-level routing suggests the value of those narrow experts increases when you can assign them to the parts of a task where they actually outperform.

Teams already working with mixture-of-experts architectures in production will see a direct parallel. The industry already routes tokens to different experts inside large language models. FactorDiff applies a similar principle at the output level of diffusion models, routing factors of the generated sample rather than tokens of the input. If this approach scales, it could lower the compute cost of compositional generation by avoiding the overhead of global weight optimisation.

A two-person consultancy experimenting with open-weight diffusion models for structured output should watch for factor-level routing implementations. When code appears, the first question to test is whether your specific use case has enough spatial or functional variation across the output for factor routing to matter. Tasks with uniform output structure may see little benefit.

What we don't know yet

The preprint describes results only qualitatively. No specific benchmark scores, percentages, or confidence intervals are reported [S1]. We know FactorDiff "consistently outperforms" global weighting on ARC-AGI tasks requiring logical consistency and spatial disentanglement, but we do not know the margin.

The framework has been validated only on ARC-AGI [S1]. Whether factor-level routing helps on other reasoning benchmarks, on language tasks, or on larger-scale generation problems remains untested.

No code, models, or data have been released [S1]. The paper has not been peer-reviewed [S1]. Reproducibility cannot be assessed until independent researchers can run the framework.

The spatial, pixel-level instantiation is the only one demonstrated. The authors' claim that samples "can be further decomposed into smaller factors" [S1] suggests other decompositions are possible, but whether non-spatial factorisations work as well is an open question.

The next thing to watch is whether the authors release code and benchmark numbers, and whether other research groups attempt to reproduce or extend the factor-routing approach on additional benchmarks. The gap between a promising preprint and a usable technique is often measured in months.

If you want to follow the next step in this story, subscribe to keep reading as the evidence arrives.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.