Reproduce “Consistent Direct Time-of-Flight Video Depth Super-Resolution” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhanghao Sun, Wei Ye, Jinhui Xiong, Gyeongmin Choe, Jialiang Wang, Shuochen Su, Rakesh Ranjan

Researchers from Stanford University and Meta developed deep learning frameworks, DVSR and HVSR, that super-resolve low-resolution direct Time-of-Flight (dToF) video depth data by fusing it with high-resolution RGB video, yielding improved depth accuracy and temporal consistency. They also introduced DyDToF, a synthetic RGB-Depth video dataset with dynamic objects and physics-based dToF simulation, to facilitate realistic training and evaluation.

Key insights

  • Leveraging multi-frame temporal information with flexible alignment significantly improves both depth accuracy and temporal consistency compared to single-frame methods.
  • Directly incorporating dToF sensor's unique histogram data provides valuable additional information, further alleviating spatial ambiguity and enhancing depth detail.
  • Realistic dToF sensor simulation within synthetic datasets, particularly considering radiance from material albedo and normals, is crucial for developing robust models for real-world scenarios.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.