Reproduce “Foveated Instance Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongyi Zeng, Wenxuan Liu, Tianhua Xia, Jinhui Chen, Ziyun Li, Sai Qian Zhang
Researchers from New York University and Meta Reality Labs developed FovealSeg, a framework that leverages human gaze to perform instance segmentation only on regions of interest, significantly reducing computational load for AR/VR applications. The accompanying FSNet model and multi-frame algorithm reduce processing latency by over 20x compared to traditional methods, achieving real-time performance on edge GPUs.
Key insights
- Human users in AR/VR environments typically focus their gaze on small regions, allowing computational resources to be concentrated on "instances of interest."
- Leveraging saccadic suppression, where visual perception is reduced during rapid eye movements, enables temporary halting of processing without user awareness, saving compute.
- Saliency-guided downsampling, combined with a specialized loss function, can effectively magnify and segment IOIs from high-resolution inputs, maintaining accuracy where it matters most to the user.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.