NVIDIA ships eigendecomposition utilities for matrix-aware optimizers in its emerging-optimizers toolkit P⁴. A 29 October arXiv preprint called Dobb cuts optimizer-call time from 124.1 milliseconds to 9.2 milliseconds across 32 H100 GPUs by computing updates locally within each GPU shard S¹. That challenges the assumption that matrix-aware methods need global communication across devices S¹.
My read: This is the first optimizer I have seen that tries to keep matrix-aware update quality while removing the cross-shard communication that makes distributed Muon expensive. The 13-fold speedup on optimizer calls is striking, but I do not buy the generalisation claim yet, because the entire benchmark is one model size, self-reported, and carries no variance bars. The local-update trick only works when blocks align with shards, an assumption that could break on architectures with unusual sharding patterns.
How local updates work
When a language model's parameters and optimizer state outgrow one GPU's memory, training systems split those tensors across devices using tensor parallelism or fully sharded data parallelism S¹. Matrix-aware optimizers like Muon improve update quality by treating weight matrices as whole objects, but that means running global matrix operations across shards, which requires communication S¹.
Dobb, short for Distributed Optimizer with Block-diagonal Eigenbases, breaks each weight matrix's input-activation second moment, a statistical summary of recent training signals, into diagonal blocks S¹. Within each block it tracks a complete eigenbasis: the set of directions that matter most for that block. It projects the optimizer's momentum into those bases, applies sign normalisation (replacing each coefficient with its sign), and reconstructs the update S¹.
When each block fits within a single shard, the entire matrix update runs locally. The only communication left is the data-parallel synchronisation of activation second moments, which standard training already performs S¹. That is not zero communication, but it removes the global matrix operations that made distributed Muon costly.
What the benchmarks show
The authors tested Dobb against Dion, another distributed matrix-aware optimizer that tries to keep the benefits of orthonormalised updates while cutting communication S¹. On GPT-2 XL, a 1.56-billion-parameter model, Dobb with 16 blocks reached a validation loss of 2.476 S¹. Dion, at a rank fraction of 1/16, reached 2.511 S¹. Lower loss means the model predicts held-out text more accurately.
The speed gap is wider. Under fully sharded data parallelism on 32 H100 GPUs, Dobb reduced the amortised optimizer-call time from 124.1 milliseconds for Dion to 9.2 milliseconds S¹. That is a 13-fold drop in the time spent inside the optimizer per step.
Total step time is not reported, so the end-to-end training speedup is not verifiable from the preprint alone.
What is not yet verified
The preprint, posted on arXiv in the cs.AI and cs.LG categories, has not been peer-reviewed S¹. The benchmarks are self-reported on a single model size. The validation-loss gap of 2.476 versus 2.511 comes with no variance or statistical-significance data S¹. No independent group has replicated the results.
Dobb has not been compared against non-matrix-aware optimizers such as AdamW or standard Muon S¹. The speedup is specific to FSDP on 32 H100 GPUs and depends on the assumption that blocks align with shards S¹. The paper does not test models larger than 1.56 billion parameters, so whether the approach scales to 7-billion or 70-billion-parameter models is unknown.
The broader field is active. Google Research has published preconditioning code on GitHub P³, and the HomebrewML HeavyBall library released version 3.2.1 in August with improved FSDP and distributed-training support P². The mathematical foundation Dobb builds on, simultaneous block diagonalisation, has been available since 2020 in the SBD repository P⁵.
Who this reaches first
A team training a 1-to-3-billion-parameter model on a 32-GPU H100 cluster using FSDP is the most direct candidate. If Dobb's local-update approach holds at larger scale, the payoff is concrete: less time waiting for GPUs to exchange data means more training steps per hour, which means faster convergence or cheaper runs. A practitioner running distributed Muon today and hitting communication bottlenecks could swap in Dobb, provided their sharding scheme keeps each block within one shard.
The preprint does not indicate when code will be released.
The next checkpoint is either peer review or an open-source implementation that independent teams can test against their own training pipelines.
Sources: S1 — Dobb: Shard-Local Preconditioning in Block-Diagonal Eigenbases · P2 — Releases · HomebrewML/HeavyBall · GitHub · P3 — google-research/precondition · P4 — emerging_optimizers.utils.eig — Emerging-Optimizers · P5 — y-z-zhang/SBD
Written from 5 sourced items, 4 of them primary.
