Reproduce “Sentence Length” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gábor Borbély, András Kornai
Researchers propose a random walk model that tracks valency—the number of unfilled syntactic dependencies—to describe sentence length distributions. This linguistically motivated model achieved a superior fit to empirical data across multiple corpora and offered high compression efficiency, outperforming existing statistical and neural language model approaches.
Key insights
- Sentence formation can be modeled as a random walk that tracks the number of open syntactic dependencies (valency), completing a sentence when all dependencies are satisfied.
- A mathematically rigorous framework, grounded in linguistic valency theory, allows for deriving analytical expressions for sentence length probabilities and provides interpretable model parameters.
- Sophisticated Bayesian model selection, accounting for corpus noise, identifies simple models (order 1, 1-2 mixture components) that provide superior fits to empirical sentence length distributions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.