Reproduce “NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hyeonjeong Ha, Jinjin Ge, Bo Feng, Kaixin Ma, Gargi Chakraborty
NarrativeTrack evaluates whether multimodal language models can maintain entity identities and track their changing actions, outfits, and scenes throughout a video. Across 20 models, Gemini-2.5-Pro reaches 83.80% accuracy, while results reveal persistent weaknesses in identity ambiguity, backward reasoning, and robustness to distractors.
Key insights
- Models are substantially better at reasoning forward through a narrative than backward: the forward-to-backward gap reaches 20.65 percentage points for general-purpose open-source models and 17.55 points for proprietary models.
- Entity ambiguity and fine-grained outfit or action changes are harder than broad scene changes, indicating that temporal context alone does not guarantee reliable identity tracking.
- Adding frames or model parameters does not consistently improve performance; redundant visual input can hurt, and video-specialized models often trade stronger temporal handling for weaker perceptual precision.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.