Reproduce “RMem: Restricted Memory Banks Improve Video Object Segmentation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junbao Zhou, Ziqi Pang, Yu-Xiong Wang

A study by University of Illinois Urbana-Champaign researchers demonstrates that continuously expanding memory banks in video object segmentation can hinder performance due to redundant information. Their RMem approach, leveraging a restricted, intelligently updated memory bank and temporal positional embedding, establishes new state-of-the-art performance on benchmarks such as VOST and Long Videos, boosting AOT's J_tr on VOST by 3.4 points and J&F on Long Videos by 3.6 points.

Key insights

  • A "memory deciphering" study revealed that expanding memory banks actually degrade a VOS module's ability to decode information, as attention becomes scattered by redundant frames.
  • Restricting the memory bank to a small, relevant subset of frames significantly mitigates this performance degradation, maintaining high decoding accuracy.
  • Aligning memory lengths during training and inference by using restricted memory enables the effective application of temporal positional embeddings to enhance VOS performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.