Reproduce “Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Imad Eddine Marouf, Enzo Tartaglione, Stephane Lathuiliere, Joost van de Weijer

Researchers introduce QUestion-only replay with Attention Distillation (QUAD), a method for Continual Visual Question Answering (VQACL) that mitigates catastrophic forgetting by storing only questions from past tasks. This approach achieves 39.25% Average Performance with a 4.91% forgetting rate on VQAv2 and 31.70% AP with a 2.91% forgetting rate on NExT-QA, demonstrating competitive performance while significantly reducing memory overhead and enhancing data privacy compared to methods that store visual data.

Key insights

  • Storing only questions from previous tasks is sufficient for effective catastrophic forgetting mitigation in VQACL, offering a privacy-preserving and memory-efficient alternative to full-exemplar replay.
  • Aligning self-attention distributions (via cross-entropy on normalized attention maps) is more effective for maintaining internal representation consistency in multimodal continual learning than L1-based feature distillation, allowing flexibility while preserving critical relationships.
  • A targeted object-matched selection strategy for replayed questions significantly improves contextual relevance and reinforces cross-task associations, effectively preventing the 'out-of-answer-set' problem.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.