Reproduce “RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation” on OpenResearch

About this paper

Yuchuan Ding, Linfei Li, Lin Zhang, Ying Shen

RaysUp presents an ultra-lightweight, universal framework for upsampling low-resolution features from Vision Foundation Models by integrating a geometry-aware ray representation. This method achieves state-of-the-art efficiency, operating up to 7 times faster with 8 times fewer GFLOPs than prior methods, while maintaining competitive performance across diverse dense prediction tasks.

Key insights

• Integrating 3D geometric priors through a ray representation is crucial for robust feature upsampling, as it addresses the limitations of 2D Euclidean assumptions at depth discontinuities and preserves structural integrity.
• Decoupling spatial guidance into orthogonal directional components significantly improves feature learning efficiency and interpretability while reducing computational overhead.
• An attention mechanism that operates across different resolutions and is guided by ray geometry allows for flexible, universal upsampling without requiring task-specific fine-tuning or VFM-specific retraining.
• Localizing the attention mechanism within geometry-aware neighborhoods maintains computational efficiency while still benefiting from the global context provided by ray-based encoding.

Summary by alphaXiv.