Reproduce “EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Changjing Liu, Yiming Huang, Long Bai, Beilei Cui, Hongliang Ren
Researchers from The Chinese University of Hong Kong developed EndoGSim, a unified framework for physics-aware 4D dynamic endoscopic scene simulations using MLLM-guided Gaussian Splatting. The method generates visually realistic and physically consistent simulations of deformable surgical scenes directly from video inputs.
Key insights
- MLLMs can effectively provide robust, informed initial material parameter estimates from visual context, significantly improving the efficiency and accuracy of physics-aware simulation optimization.
- An object-wise material field, coupled with semantic segmentation, is crucial for accurately modeling multi-object surgical scenes with distinct deformable tissues and rigid instruments.
- The integration of a differentiable MPM with joint rendering and optical flow losses enables precise, physically consistent material parameter estimation and dynamic scene simulation from endoscopic video.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.