Reproduce “When Geometry Aligns: Dihedral Hidden-State Transformations in UNet, ViT, and DiT Architectures” on OpenResearch

About this paper

Mojtaba Faramarzi, Alex Lamb, Irina Rish

Researchers from the University of Montreal, Mila - Quebec AI Institute, and Tsinghua University introduced a unified framework for applying geometrically consistent dihedral transformations to the hidden states of U-Net, ViT, and DiT architectures. Their findings demonstrate that these interventions enhance internal feature stability and can improve model generalization, such as increasing ViT classification accuracy by 0.89% on CIFAR-100, without compromising output fidelity in generative models.

Key insights

• Maintaining geometric consistency during hidden-state interventions is crucial for preserving internal model stability and computational coherence across diverse architectures.
• Symmetry-aware hidden-state augmentation can serve as an effective regularization technique, leading to improved generalization in classification tasks and better internal stability in generative models.
• Inconsistent geometric transformations applied to hidden states lead to predictable architectural failures, such as disrupted attention fields and misaligned feature representations.

Summary by alphaXiv.