Reproduce “Latent Actions from Factorized Transition Effects under Agent Ambiguity” on OpenResearch

About this paper

Heejeong Nam, Chandradithya S Jonnalagadda, Harshit Aggarwal, Eric Xu, Randall Balestriero

The authors introduce Observed Transition Factorization (OTF), which learns reusable local visual-transition codes before a state-aware model combines them into continuous latent actions. In a first-order DCS transfer test, OTF reconstruction MSE rises from 0.037 to 0.055, a 49.34% increase, versus 64.78% for a monolithic VQ-VAE. This supports transfer in that evaluation, not identification of true causes.

Key insights

• Factorizing observed visual transition effects into reusable primitives, largely independent of specific appearance or morphology, provides a more transferable and robust representation than monolithic approaches.
• "Action alignment" (how well a latent action maps to a true environment action) is a distinct concept from "latent predictability" (how easy a latent action is to predict from observation), and stronger alignment is crucial for sample-efficient downstream supervision.
• Predicting future states in a frozen, high-level feature space (e.g., DINOv2) for the latent action model significantly improves quality by suppressing pixel-level nuisance factors and retaining control-relevant information.

Summary by alphaXiv.