Blog
Posts
-
Denoise First, Orthogonalize Later
Understanding Momentum in Muon via Spectral Filtering
A theoretical and empirical study of why placing momentum before orthogonalization makes Muon work. Under a signal-plus-perturbation gradient model, momentum acts as a spectral filter that suppresses perturbations while preserving the dominant gradient signal — hence the rule: denoise first, then orthogonalize.