Reproduce “Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lavanya Nigam, Ishaan Bansal, Aryan Sood, Vidit Aggarwal, Gaurav Kumar Nayak
Researchers from the Indian Institute of Technology Roorkee found that Masked Diffusion Language Model embeddings exhibit a hyperspherical geometry, causing linear interpolation in predictive feedback to fail. They propose Spherical Soft-Masking (S-SM), which uses spherical aggregation and interpolation, leading to MAUVE score improvements of 27.7–56.1% and a reduction in generative perplexity by 12.2–14.2% over existing linear soft-masking methods.
Key insights
- MDLM token embeddings are concentrated on a hyperspherical manifold, not a Euclidean space, with a consistent 73° geodesic angle between mask and predicted embedding directions.
- Linear interpolation is geometrically inconsistent in a hyperspherical space, whereas spherical interpolation (SLERP) respects the intrinsic geometry, maintaining semantic fidelity.
- The model's optimizer sustains a significantly larger feedback magnitude (higher learned λ) when using geometrically principled spherical feedback, indicating greater trust in the refined information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.