Reproduce “FlowNIB: An Information Bottleneck Analysis of Bidirectional vs. Unidirectional Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Md Kowsher, Nusrat Jahan Prottasha, Shiyun Xu, Shetu Mohanto, Ozlem Garibay, Niloofar Yousefi, Chen Chen
FlowNIB estimates input and target mutual information in frozen language-model layers, using neural lower-bound estimates to compare bidirectional and unidirectional representations. The authors report higher layerwise estimates for bidirectional models, but comparisons across model families do not isolate directionality. DeBERTa-v3-Large achieves 84.73% average accuracy with masked-token prediction versus 80.96% with pooling across nine classification benchmarks, with task-level exceptions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.