Reproduce “On The Statistical Representation Properties Of The Perturb-Softmax And The Perturb-Argmax Probability Distributions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hedda Cohen Indelman, Tamir Hazan
The authors analyze how perturbation-based categorical models represent distributions and identify their parameters. Perturb-Softmax averages noisy softmax probabilities, while Perturb-Argmax averages noisy winners; convexity links these mappings to expected log-sum-exp and maximum functions. The models cover strictly positive distributions under stated conditions, with boundary distributions reached as limits. After 300 iterations, Normal-Softmax’s Poisson L1 error was 0.090 ± 0.001 versus 0.618 ± 0.007 for Gumbel-Softmax; this is not a runtime result.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.