Reproduce “Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoyi Wu, Kewei Tu
Researchers from ShanghaiTech University developed a non-neural, probabilistic dependency model for contextual word representation. The model's computation closely mirrors Transformer architectures, suggesting that Transformers may implicitly learn syntactic dependencies, and it performs competitively on NLP tasks with fewer parameters.
Key insights
- The iterative update equations from the probabilistic model's Mean Field Variational Inference (MFVI) closely resemble the computational flow of Transformer architectures.
- Specifically, the probabilistic distribution over dependency heads corresponds to self-attention scores, and latent word representations correspond to intermediate word embeddings.
- This suggests that Transformer self-attention layers may implicitly perform a form of syntactic inference, capturing dependency-like relationships between words.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.