Machine Learning

Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers

Tim Tsz-Kit Lau, Weijie Su

Featured May 29, 2026

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

Instead of using one-size-fits-all optimizers, this paper shows that matching the optimizer's update rule to the unique symmetries of different neural network parts, like embeddings or expert routers, makes models train better.

In depth

The paper introduces a symmetry-compatible principle for designing optimizers, arguing that gradient update rules should respect the inherent geometric symmetries of different neural network parameter blocks. This leads to a layerwise optimizer stack where each matrix-valued parameter (e.g., embeddings, MoE routers) is assigned an update rule whose equivariance matches its specific symmetry group, moving beyond geometry-blind coordinate-wise methods like Adam. The authors demonstrate that these tailored optimizers consistently improve validation loss and training stability in large language models.

Key Takeaways

1
Traditional coordinate-wise optimizers like Adam are geometrically mismatched for matrix-valued neural network parameters, ignoring their rich symmetry and equivariance properties.
2
The authors propose a symmetry-compatible principle for optimizer design, where each layer's update rule is specifically tailored to its unique symmetry group (e.g., bi-orthogonal, left-permutation right-orthogonal, expert-permutation).
3
This leads to a family of equivariant optimizers (e.g., spectral, one-sided spectral, row-norm, hybrid) that consistently improve validation loss and training stability across various dense and sparse language models.

Conceptual Flow

HIGH LEVEL

Methodology: Tailored Optimizer Updates

Instead of one general update rule, the new method uses different, specialized update rules for each type of network part, based on its unique internal structure.

Network Part 1

Network Part 2

Network Part 3

Match Symmetry

Special Update 1

Special Update 2

Special Update 3

Results: Better Model Performance

By using these specialized update rules, the models learn more effectively, leading to better predictions and more stable training.

Old Update Method

New Update Method

Compare Learning

Higher Error

Lower Error

This breakdown was generated by SciGrove. Get the same analysis — intuition, storyboard, peer review, a runnable prototype and a glossary — on any paper you upload or paste a DOI for.