Reproduce “ENCORE: Event-Assisted Complementary Motion Refinement for Learned Video Compression” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shuhan Ye, Hongbin Yu, Chenqi Kong, Pingchuan Ma, Chong Wang, Jun Wan, Qixin Zhang

Researchers from Wuhan University and collaborators developed ENCORE, an encoder-side framework that refines motion estimation in learned RGB video compression by integrating auxiliary event camera data. This system achieved up to 20.80% PSNR-RGB BD-rate savings and 22.14% MS-SSIM-RGB BD-rate savings across various datasets, outperforming RGB-only and naive multimodal baselines.

Key insights

  • Event camera data offers complementary, high-temporal-resolution motion information capable of resolving ambiguities in RGB-only motion estimation, particularly in challenging visual conditions.
  • Effective integration of auxiliary event data requires intelligently separating common motion from modality-specific cues and selectively calibrating event evidence based on its activity and novelty relative to RGB.
  • Spatially controlled application of event-derived motion corrections is essential to avoid degrading regions where initial RGB flow is reliable, ensuring targeted improvements in compression efficiency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.