Reproduce “Generalized Decision Transformer for Offline Hindsight Information Matching” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hiroki Furuta, Yutaka Matsuo, Shixiang Shane Gu

Generalized Decision Transformer (GDT) reframes offline reinforcement learning as hindsight information matching: conditioning behavior on future trajectory statistics rather than only scalar returns. Its categorical variant improves Wasserstein distribution-matching distances over Decision Transformer across MuJoCo tasks, while the bi-directional variant improves one-shot full-trajectory imitation; these gains do not consistently translate into higher conventional task reward.

Key insights

  • Future trajectory information and the transformer architecture can be treated as separate design choices, allowing the same conditional sequence-modeling approach to cover return conditioning, feature-distribution matching, and demonstration imitation.
  • Histogram conditioning retains multimodal structure that scalar return or feature sums erase, enabling the policy to distinguish targets with similar averages but different distributions.
  • Offline behavioral cloning can support state-marginal and trajectory matching without online interaction, density-ratio estimation, or adversarial training, although it remains dependent on behaviors represented in the dataset.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.