Reproduce “Procedural Mistake Detection via Action Effect Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenliang Guo, Yujiang Pu, Yu Kong

Action Effect Modeling (AEM) introduces a framework for procedural mistake detection that analyzes both action execution and its visual outcome, or "action effect". This approach identifies errors evident in object states or spatial relationships, yielding improved AUC and Error Detection Accuracy on egocentric video benchmarks like EgoPER and CaptainCook4D under one-class classification settings.

Key insights

  • Detecting errors in procedural tasks requires analyzing not only *how* an action is performed but also *what* its effect or outcome is, as many mistakes are subtly reflected in resulting object states or spatial relations.
  • Multimodal supervision from advanced vision-language and grounding models can effectively distill rich, interpretable, and compact representations of action effects, capturing both visual and symbolic aspects.
  • A probabilistic formulation with informed effect frame sampling (considering semantic relevance and visual clarity) and cross-modal alignment of effect features enables robust integration of outcome information into mistake detection.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.