Reproduce “IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hamed Rahimi, Clemence Grislain, Adrien Jacquet Cretides, Olivier Sigaud, Mohamed Chetouani

Researchers from Sorbonne University developed IntentVLM, a video-language framework for open-vocabulary human intention recognition that uses a two-stage forward-inverse modeling approach. The framework achieved an average accuracy of approximately 80% on the IntentQA benchmark, performing comparably to human-level understanding and showing a consistent 5-point improvement in open-vocabulary performance metrics over direct generation methods.

Key insights

  • Decomposing complex cognitive tasks like intention understanding into explicit stages of candidate generation and structured selection significantly enhances reliability and reduces 'hallucinations' in open-vocabulary settings.
  • The framework demonstrates that explicitly modeling an intermediate goal inference step improves intention understanding by bridging robust classical probabilistic inference with the broad world knowledge of foundation models.
  • Task-specific adaptation through lightweight LoRA adapters can improve performance for intention recognition without causing catastrophic forgetting of the VLM's general visual understanding capabilities.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.