Reproduce “Align Your Tangent: Training Better Consistency Models via Manifold-Aligned Tangents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Beomsu Kim, Byunghee Cha, Jong Chul Ye
AYT trains consistency models with manifold feature distance, comparing outputs in features learned from controlled real-image perturbations to steer updates toward the data distribution. On unconditional CIFAR-10, it improves one-step FID from 3.60 to 2.61 versus Easy Consistency Training, while two-step FID is essentially unchanged: 2.13 versus 2.11. It reports up to 10× faster convergence by training iteration.
Key insights
- CM tangents exhibit significant oscillatory components during training, causing the model to move parallel to the data manifold rather than efficiently converging towards it.
- By designing a feature space where its Jacobian's rows (gradients) consistently point towards the data manifold, the effective CM tangent can be aligned, leading to more stable and faster training.
- Self-supervised learning of these manifold features, based on diverse data perturbations, provides a robust and interpretable approach that avoids the limitations and dataset dependency of pre-trained perceptual metrics like LPIPS.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.