Reproduce “Language Models as Hierarchy Encoders” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuan He, Zhangdie Yuan, Jiaoyan Chen, Ian Horrocks

Researchers at the University of Oxford, Cambridge, and Manchester developed Hierarchy Transformer encoders (HITs), a method to explicitly encode hierarchical structures in transformer LMs by re-training them in hyperbolic space with specialized loss functions. HITs consistently surpassed pre-trained and fine-tuned LMs across hierarchical tasks, demonstrating improved transitive inference and cross-hierarchy knowledge transfer with F-score improvements of at least 0.150 in transfer scenarios.

Key insights

  • Explicitly re-training LMs in hyperbolic space with specialized loss functions enables them to effectively encode and interpret hierarchical structures.
  • The hyperbolic centripetal loss, by positioning parent entities closer to the origin of the Poincaré ball than their children, effectively maps conceptual generality to geometric proximity to the origin.
  • The re-trained HIT models demonstrate enhanced generalization capabilities in hierarchical tasks, including multi-hop inference and cross-hierarchy transfer, surpassing standard fine-tuning approaches.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.