Reproduce “Teaching What to Look For: Privileged Perception Questions for Agentic On-Policy Distillation in Video Captioning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yifan Xu, Ruopeng Gao, Zhixiao Wang, Jiafu Tang, Yichun Yang, Zihao Wang, Yuanxing Zhang, Limin Wang

PQ-OPD trains a video-seeking agent to inspect relevant moments and regions using answer-free Perception Questions (PQs) during training, without PQs at inference. A frozen teacher supervises the student’s own tool-use history. On DREAM-1K, the 9B student/9B teacher setup raises event F1 from 31.3 to 33.3 under the paper’s evaluation protocol; inference takes longer than for the untrained agent.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.