Reproduce “M2SVid: End-to-End Inpainting and Refinement for Monocular-to-Stereo Video Conversion” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nina Shvetsova, Goutam Bhat, Prune Truong, Hilde Kuehne, Federico Tombari

M2SVid, developed at Google, introduces an end-to-end system for converting monocular videos into high-quality stereoscopic 3D videos, focusing on refining depth-warped right-eye views. This approach improves perceptual quality with an LPIPS score of 0.180 and achieves substantial inference speed, running 6 times faster than StereoCrafter and 635 times faster than SVG for 512x512 videos.

Key insights

  • Conditioning the refinement process on the original left-eye view effectively guides the model to preserve fine details and correct subtle warping distortions, even in regions not explicitly marked as disoccluded.
  • The specific nature of monocular-to-stereo disocclusions (often small and temporally coherent) allows for an optimized spatio-temporal attention mechanism that enhances inpainting quality without prohibitive computational cost.
  • Training the generative model specifically for a single, direct prediction step, rather than multi-step denoising, significantly boosts inference speed while maintaining or improving visual quality for the constrained M2S task.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.