Reproduce “TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuyang Liu, Chuan Wen, Yihang Hu, Dinesh Jayaraman, Yang Gao
TimeRewarder learns dense, task-instructive reward functions for robotic reinforcement learning by estimating frame-wise temporal distance from passive expert videos. The method achieves higher success rates and sample efficiency than hand-engineered rewards and prior reward learning techniques across 9 out of 10 Meta-World tasks.
Key insights
- Task progress in expert videos can be effectively modeled as a frame-wise temporal distance estimation problem, providing a dense reward signal without explicit goal conditioning.
- Explicitly accounting for suboptimality via implicit negative sampling and focusing on fine-grained temporal resolution through weighted sampling are crucial for robust reward learning.
- A theoretical justification exists for equating task progress with temporal distance in expert videos, guiding the design of effective progress models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.