Reproduce “Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hao Zheng, Jinyi Huang, Tiantian Zheng, Xun Xu, Tuka Alhanai
A framework called Compositional Context Fine-Tuning (CCFT) coupled with Layer-Partitioned Alternating Training (LP-AT) adapts Vision-Language Models to understand complex assembly actions from videos, yielding interpretable, deterministic element-level predictions. The method demonstrated improved holistic action recognition, achieving 66.28% accuracy on IKEA-ASM-VQA, outperforming baseline models, and high accuracy for individual action elements like tool recognition (up to 97.62%).
Key insights
- Decomposing complex assembly actions into semantic elements (verb, object, tool) enhances VLM performance by reducing label ambiguity and focusing model attention.
- Layer-partitioned and alternating training strategies effectively prevent cross-task interference and catastrophic forgetting, enabling stable multi-task adaptation for specialized domains.
- Explicit compositional fine-tuning, rather than general VLM capabilities or prompting, is necessary for generating precise, deterministic, and interpretable outputs suitable for manufacturing.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.