Reproduce “Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners” on OpenResearch

About this paper

Nikita Araslanov, Martin Sundermeyer, Hidenobu Matsuki, David Joseph Tan, Federico Tombari

Researchers from Google, TU Munich, and Munich Center for Machine Learning developed LILA, a method that learns high-resolution, pixel-accurate feature descriptors for dynamic 3D scenes by fusing semantic and geometric information from uncurated videos. LILA's novel linear in-context learning approach significantly improves performance on video object segmentation by nearly 10% JF, surface normal estimation, and semantic segmentation tasks compared to baselines, while enabling single-image inference.

Key insights

• The linear in-context learning strategy effectively suppresses frame-specific, temporally unstable noise from automatically generated cues, resulting in more robust and consistent feature representations than direct cue prediction.
• A synergistic combination of self-distillation, depth, and optical flow cues enables the model to jointly embed semantic, geometric, and dynamic properties into high-resolution pixel descriptors.
• Training with video-based spatio-temporal cues allows the learned features to capture dynamic scene information, yet the model retains the ability to extract rich, dense features from a single static image during inference for practical application.

Summary by alphaXiv.