Chao Feng at Cornell and Adobe researchers posted a preprint on September 23 showing that diffusion models in high-dimensional latent spaces waste optimization on noise that carries no signal S¹P². The standard prediction target in flow matching, velocity prediction, forces the model to fit orthogonal noise outside the low-dimensional signal manifold where the actual data lives S¹. Their alternative, predicting clean data directly, consistently improved text-to-image generation across multiple encoders in their experiments S¹.
My read: This is the first paper I've seen that cleanly separates the geometry problem in representation autoencoders from the usual architecture debates. The claim that finetuning encoders for reconstruction reduces effective dimensionality is plausible but unverified outside the lab, and the "consistently improves" language in the abstract gives no numbers to judge by. I'd watch for whether independent teams reproduce the generation gains with other encoder families, and whether the dimensionality reduction holds at production scale.
Representation Autoencoders, introduced by Boyang Zheng and Saining Xie at NYU P⁴, let diffusion models work in the feature spaces of pretrained visual encoders instead of training a custom autoencoder from scratch. The official PyTorch implementation has gathered 1,973 stars on GitHub since September 2025 P⁵. The appeal is simple: reuse a powerful, already-trained encoder and skip the expensive autoencoder training step. But many off-the-shelf encoders were built for recognition, not reconstruction, and they discard fine-grained visual details that a generative model needs S¹.
Reconstruction comes at a price
Feng's team confirms that finetuning these encoders for image reconstruction recovers those details S¹. But the finetuning also reduces the effective dimensionality of the resulting representation, meaning the data occupies fewer meaningful dimensions than the space provides S¹.
The data collapses onto a low-dimensional manifold inside a much larger space.
Why high-dimensional latents break velocity prediction
That collapse is where velocity prediction runs into trouble. In flow matching, a standard training approach for diffusion models, velocity prediction asks the model to learn the direction and speed of movement from noise to data across the entire space S¹. When the data sits on a low-dimensional manifold inside a much larger space, most of those directions point into empty regions where there is no signal to learn.
The model spends capacity fitting noise that does not matter.
Clean-data parameterization, called x0-prediction, takes a different approach. Instead of predicting the velocity of the denoising trajectory, the model predicts the clean image directly S¹. This focuses learning on the signal manifold itself, where the data actually sits, and ignores the empty dimensions around it.
Across experiments with multiple strong-reconstruction encoders, the team reports that x0-prediction consistently improves text-to-image generation S¹.
The word "consistently" is doing a lot of work. The abstract provides no benchmark scores, no FID numbers, no CLIP scores, and the paper has not been peer-reviewed S¹.
No benchmark numbers, no peer review
The improvement is shown for text-to-image generation with strong-reconstruction encoders S¹. Whether x0-prediction helps for other tasks, or with weaker encoders that have not been finetuned, is not established by this work. The findings should not be read as a universal verdict against velocity prediction.
For a team building a text-to-image pipeline on top of a pretrained encoder, the practical signal is narrow but concrete. If you finetune your encoder for reconstruction and your latent space turns out to be effectively low-dimensional, switching from velocity prediction to x0-prediction is a parameterization change, not an architecture change. It costs nothing in model size. The open question is whether the generation gains survive at the scale and resolution that production systems demand.
The RAE codebase that this work builds on is publicly available under an MIT licence P⁵, with its most recent code push on February 25, 2026. The preprint appeared on arXiv on September 23 under cs.AI and cs.LG S¹.
Sources: S1 — On the Diffusibility of High-Dimensional Latents · P2 — On the Diffusibility of High-Dimensional Latents · P3 — latent-reasoning-works/manylatents · P4 — Diffusion Transformers with Representation Autoencoders · P5 — bytetriper/RAE
Written from 5 sourced items, 4 of them primary.
