Reproduce “Differentiable Robot Rendering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruoshi Liu, Alper Canberk, Shuran Song, Carl Vondrick
Differentiable Robot Rendering (Dr. Robot) introduces a self-model for robots that enables direct, gradient-based control from visual foundation models by creating a fully differentiable pipeline from a robot's visual appearance to its underlying joint parameters. It achieves this by combining 3D Gaussian Splatting with implicit linear blend skinning and pose-conditioned appearance deformation, demonstrating superior performance in pose reconstruction and diverse applications like semantic text-to-pose generation and visual motion retargeting.
Key insights
- Integrating Linear Blend Skinning as a prior for geometric deformation is crucial for accurately modeling robots with complex, high-degree-of-freedom kinematic chains.
- The pose-conditioned appearance deformation module significantly contributes to both the visual and geometric fidelity of the robot self-model.
- Full differentiability allows for direct, gradient-based optimization of robot actions using visual signals from foundation models, simplifying complex control tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.