Reproduce “Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gonçalo M. Correia, Vlad Niculae, Wilker Aziz, André F. T. Martins
The authors introduce sparse marginalization for training models with discrete hidden choices. They use sparsemax to assign some choices zero probability and evaluate the loss only over the remaining support. In an image-communication task, it matches dense marginalization’s 93.37% success with 3.13 decoder calls versus 256. In structured settings, top-k may omit choices, while SparseMAP has gradient limitations; decoder calls do not measure runtime.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.