Reproduce “DEPTH: Discourse Education through Pre-Training Hierarchically” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zachary Bamberger, Ofek Glick, Chaim Baskin, Yonatan Belinkov
DEPTH trains a T5-style encoder–decoder to fill masked spans and restore shuffled sentence order, using sentence markers and restricted attention to represent sentence-level structure. In continuous pretraining from T5-Base, its 220M-parameter model scores 65.59 versus T5’s 59.48 on DiscoEval sentence permutation, but 82.49 versus 82.27 on coherence. The study lacks controlled component ablations and confidence intervals.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.