Reproduce “TR2M: Transferring Monocular Relative Depth to Metric Depth with Language Descriptions and Dual-Level Scale-Oriented Contrast” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Beilei Cui, Yiming Huang, Long Bai, Hongliang Ren
TR2M, a framework from The Chinese University of Hong Kong, translates monocular relative depth maps into metric depth by predicting pixel-wise scale and shift maps using both an RGB image and its textual description. The model integrates dual-level scale-oriented contrastive learning, achieving competitive zero-shot metric depth estimation across diverse, unseen domains with a lightweight architecture of only 19M trainable parameters.
Key insights
- Predicting pixel-wise scale and shift maps with language guidance provides more granular and effective corrections for localized inaccuracies in relative depth compared to global scaling factors.
- Augmenting sparse ground truth with confidence-filtered pseudo metric depth maps significantly improves overall supervision, particularly in promoting cross-domain generalization.
- Enforcing feature consistency based on depth and scale distributions through dual-level contrastive learning is crucial for improving the model's intrinsic understanding of scale and enhancing its generalization capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.