Reproduce “RepSAM: Bridging Foundation Models to Robotic Vision via Representation-Guided Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenhui Chu
Researchers at Texas A&M University developed RepSAM, a parameter-efficient fine-tuning framework that adapts large foundation models like SAM for robotic vision, incorporating representation-guided layer-wise adaptation, depth-aware fusion, and edge-enhanced loss. It achieves 89.0% mIoU on robotic benchmarks, representing 97.9% of full fine-tuning performance with 158 times fewer trainable parameters, and demonstrates improved grasp success rates in robotic manipulation tasks.
Key insights
- Transformer layers in foundation models exhibit non-uniform domain shifts: shallow layers show significant domain gaps (low CKA), while deeper layers demonstrate strong transferability (high CKA) between natural and robotic images.
- Allocating adaptation capacity based on layer-wise representation similarity substantially improves efficiency and performance over uniform allocation strategies, especially for low-level feature adaptation.
- Combining this representation-guided adaptation with domain-specific enhancements (depth fusion and edge-enhanced loss) effectively addresses unique robotic vision challenges and translates directly to improved downstream robotic manipulation task performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.