> ## Content Index > Fetch the complete content index at: https://www.notatechguy.com/llms.txt > Use this file to discover other available public pages before exploring further. # Transformer attention gets momentum from Hamiltonian dynamics - URL: https://www.notatechguy.com/transformer-attention-gets-momentum-from-hamiltonian-dynamics/ - Published: 2026-10-23T23:40:41.000Z - Updated: 2026-10-23T23:40:41.000Z - Description: A TU Berlin preprint rewrites attention as damped Hamiltonian dynamics, lowering validation loss on small models but reporting no numbers. - Author: Marcello Babbili - Tags: Technology & AI, OpenAI, Nvidia Viktor Stein and Gabriele Steidl at TU Berlin, with Wuchen Li, posted a preprint on 22 October 2026 that adds physical momentum to transformer attention via damped Hamiltonian dynamics [S¹](https://arxiv.org/abs/2610.31600v1?ref=notatechguy.com) [P⁴](https://arxiv.org/html/2603.16535?ref=notatechguy.com). The paper reports lower validation loss on small language models but gives no specific numbers, and the work is not peer-reviewed [S¹](https://arxiv.org/abs/2610.31600v1?ref=notatechguy.com). **My read:** This is the first paper I've seen that treats attention as a Hamiltonian system, the mathematical framework behind planetary motion and pendulums, and then damps it to get something practical. The Nesterov connection is clever: just as Nesterov accelerated gradient adds momentum to speed up optimisation, these authors add inertia to the gradient-flow view of attention. But I don't buy the practical relevance yet. The experiments are nanoGPT-scale and the paper gives no numerical values for the improvement. The theoretical work on token-norm blow-up is the part I'd watch. That is a real problem in deep transformers, and a rigorous characterisation of when it happens could matter even if the momentum idea stays in the lab. ## When token norms run away Stack enough transformer layers and the vectors representing individual tokens can grow without bound. Practitioners fight this with layer normalisation, residual scaling, and careful initialisation, but the mathematical conditions that trigger blow-up have remained fuzzy. A line of recent work reframes attention as a gradient flow, a first-order optimisation process on a space of probability distributions equipped with Wasserstein-type metrics that measure the distance between those distributions [S¹](https://arxiv.org/abs/2610.31600v1?ref=notatechguy.com). Gradient flows converge, but they do so without momentum. They are the mathematical equivalent of a ball that stops at every instant rather than carrying inertia downhill. ## Adding momentum to the gradient flow Stein, Li, and Steidl extend that viewpoint by deriving an inertial, Nesterov-type damped Hamiltonian dynamics on density spaces [S¹](https://arxiv.org/abs/2610.31600v1?ref=notatechguy.com). In plain terms, they give the attention mechanism a memory of where it has been, so it moves faster toward its target. They build this framework for three settings: standard softmax attention, linear attention (a cheaper variant), and a quotient Stein geometry they construct to fix a degeneracy in linear attention's kernel that made its metric ill-defined [S¹](https://arxiv.org/abs/2610.31600v1?ref=notatechguy.com). For linear attention, they prove sharp conditions separating global existence from finite-time second-moment blow-up [S¹](https://arxiv.org/abs/2610.31600v1?ref=notatechguy.com). They then connect the continuous-time analysis to the discrete transformer stack, showing quantitatively at which layer depth token norms will cross a given threshold [S¹](https://arxiv.org/abs/2610.31600v1?ref=notatechguy.com). That link between a continuous mathematical model and a countable set of transformer layers is the kind of result that could give practitioners a diagnostic tool, if it holds up. ## Small models, no numbers In decoder-only nanoGPT-style experiments on TinyStories and OpenWebText, the authors propose geometric corrections to token and momentum updates, adjustments motivated by the fact that real transformers have finitely many tokens rather than a continuous distribution [S¹](https://arxiv.org/abs/2610.31600v1?ref=notatechguy.com). These corrections lower validation negative log-likelihood (a standard measure of how well a model predicts the next word) relative to momentum-only controls [S¹](https://arxiv.org/abs/2610.31600v1?ref=notatechguy.com). The paper does not report specific numerical values for the improvement, making its magnitude and statistical significance impossible to judge. The experiments cover only small-scale decoder-only models on two datasets. There is no evidence the method helps on larger architectures, encoder-decoder designs, or production-scale training, and the theoretical results have not been independently reproduced. An earlier version of the framework appeared on arXiv in March 2026 [P²](https://arxiv.org/abs/2603.16535?ref=notatechguy.com), giving the work roughly seven months of revision, but the current posting still lacks released code or benchmark numbers that a third party could verify. Researchers working on the mathematical foundations of attention are the immediate audience. A group exploring linear attention for long-context inference, for instance, could test whether the blow-up characterisation helps them predict and control token-norm growth as they stack more layers. The next checkpoint is peer review and, if the authors release code, independent reproduction on a model large enough to matter. --- *Sources: [S1 — SympFormer: Inertial Attention Blocks from Damped Hamiltonian Dynamics](https://arxiv.org/abs/2610.31600v1?ref=notatechguy.com) · [P2 — \[2603.16535\] SympFormer: Accelerated attention blocks via Inertial Dyn](https://arxiv.org/abs/2603.16535?ref=notatechguy.com) · [P3 — \[2603.16535v1\] SympFormer: Accelerated attention blocks via Inertial D](https://arxiv.org/abs/2603.16535v1?ref=notatechguy.com) · [P4 — SympFormer: Accelerated attention blocks via Inertial Dynamics on Dens](https://arxiv.org/html/2603.16535?ref=notatechguy.com) · [P5 — greydanus/dissipative\_hnns](https://github.com/greydanus/dissipative%5Fhnns?ref=notatechguy.com)* --- *Written from 5 sourced items, 4 of them primary.* ## More from Not A Tech Guy - [NVIDIA RTX Spark lands in Europe with 1-petaflop local AI](https://www.notatechguy.com/nvidia-rtx-spark-lands-in-europe-with-1-petaflop-local-ai/) - [OpenAI cuts ChatGPT storage reads 12× with Cosmos DB](https://www.notatechguy.com/openai-cuts-chatgpt-storage-reads-12-with-cosmos-db/) - [NVIDIA Agent Toolkit grounds AI agents in enterprise data](https://www.notatechguy.com/nvidia-agent-toolkit-grounds-ai-agents-in-enterprise-data/)