Reproduce “An Identifiability Theory of Masked Prediction: Mode Blindness and Mask Schedules” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yichao Cai, Javen Qinfeng Shi
The paper analyzes mode blindness in masked prediction: conditional predictions can fit while global mode weights remain wrong. It compares distributions that preserve within-mode structure but change mode weights, showing that residual mode uncertainty controls the signal about the weights. Under assumptions, schedules retaining large contexts permit constant total-variation error; low-visibility interventions improve recovery only within the reweighting family.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.