Reproduce “HTLM: Hyper-Text Pre-Training and Prompting of Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, Luke Zettlemoyer
HTLM, a language model from Facebook AI and University of Washington, learns directly from a simplified version of HTML, leveraging web document structure beyond plain text. The approach enables more controlled language generation through structured prompting and size hints, achieving state-of-the-art zero-shot summarization and strong one-shot table-to-text generation while demonstrating improved data efficiency on various NLP tasks.
Key insights
- HTML markup contains rich structural and semantic information that, when retained during pre-training, can act as valuable supervision signals for language models.
- Pre-training on hyper-text enables "structured prompting," offering fine-grained control over generative model outputs (e.g., length via "size hints") and improving data efficiency in few-shot and zero-shot settings.
- A simplified "Minimal-HTML" (MHTML) representation, retaining essential structural and semantic elements while removing noise, is effective for large-scale language model pre-training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.