Reproduce “Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi Liao, Xin Jiang, Qun Liu
Researchers at Huawei Noah’s Ark Lab introduce the Probabilistically Masked Language Model (PMLM), a generalized masked language model capable of autoregressive text generation in arbitrary word order. This model achieves strong performance across both natural language understanding (NLU) and generation (NLG) tasks, outperforming BERT on GLUE and SQUAD 2.0 benchmarks while demonstrating generation quality comparable to GPT for both sequential and randomly ordered text.
Key insights
- A probabilistic masking scheme, where the masking ratio is sampled from a uniform distribution, allows a masked language model to achieve autoregressive generation in arbitrary word order.
- u-PMLM is mathematically equivalent to an autoregressive permutated language model, providing a theoretical basis for its arbitrary-order generation capability.
- A single model built on a bidirectional Transformer architecture can effectively bridge the gap between NLU and NLG, performing strongly in both.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.