Reproduce “Tactile-Augmented Radiance Fields” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiming Dou, Fengyu Yang, Yi Liu, Antonio Loquercio, Andrew Owens
Researchers from the University of Michigan, Yale University, and UC Berkeley introduce Tactile-Augmented Radiance Fields (TaRFs), a novel 3D scene representation that unifies visual appearance and tactile properties. The approach combines Neural Radiance Fields with vision-based touch sensors, enabling scalable collection of 19.3k spatially aligned visual-tactile pairs and training a diffusion model to generate accurate tactile signals with an FID score of 28.97, outperforming previous methods.
Key insights
- Vision-based tactile sensors, being camera-based, can be precisely registered with Neural Radiance Fields (NeRFs) using standard multi-view geometry, enabling accurate spatial alignment of visual and tactile data.
- Visual and structural similarities in a scene strongly correlate with tactile features, allowing generative models to effectively impute or "propagate" tactile information from sparse measurements to unprobed locations.
- Combining real collected data with estimated (synthetically generated) tactile samples significantly improves the performance of learned cross-modal embeddings for tasks like 3D localization and material classification.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.