Reproduce “TimeRefine: Temporal Grounding with Time Refining Video LLM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xizi Wang, Feng Cheng, Ziyang Wang, Huiyu Wang, Md Mohaiminul Islam, Lorenzo Torresani, Mohit Bansal, Gedas Bertasius and 1 more
TIMEREFINE enhances Video Large Language Models' (LLMs) performance in video temporal grounding by reformulating the task as an iterative refinement process and incorporating an auxiliary L1 loss for continuous timestamp prediction. This approach improves mean Intersection over Union (mIoU) by up to 5.0% on Charades-STA and 3.6% on ActivityNet Captions when integrated with existing Video LLMs.
Key insights
- Iterative offset prediction, where the model self-corrects by predicting deviations from an initial guess, is more effective for refining temporal boundaries than direct prediction or IoU prediction.
- An auxiliary L1 loss branch is crucial for enhancing temporal perception of continuous values, outperforming L2 and combined losses by providing error signals proportional to magnitude.
- Generating training refinement sequences with progressively decreasing Gaussian noise and a fixed number of steps (e.g., 4) optimizes learning for the iterative self-correction mechanism.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.