Reproduce “TraDE: Transformers for Density Estimation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rasool Fakoor, Pratik Chaudhari, Jonas Mueller, Alexander J. Smola

TraDE, a novel auto-regressive density estimator, leverages Transformer encoders to model probability density functions for continuous and discrete data, addressing limitations of traditional RNNs and flow-based methods. It achieved higher log-likelihoods on several benchmark tabular and image datasets, and generated high-fidelity samples that yielded regression performance comparable to real data while being statistically indistinguishable from actual samples in two-sample tests.

Key insights

  • Transformer encoders, when equipped with effective positional encoding like the GRU-based embedding, can overcome the limitations of RNNs in auto-regressive density estimation for both continuous and discrete data.
  • Incorporating MMD regularization into the maximum likelihood objective significantly improves the fidelity of generated samples and can lead to better density estimates, particularly in scenarios with limited data.
  • Evaluating density estimators comprehensively through downstream tasks (e.g., regression with synthetic data, two-sample tests, OOD detection) provides a more robust and application-oriented assessment of model utility than log-likelihood alone.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.