Reproduce “CASTILLO: Characterizing Response Length Distributions of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daniel F. Perez-Ramirez, Dejan Kostic, Magnus Boman
The CASTILLO dataset provides an empirical characterization of Large Language Model response length distributions, covering 13 models and 7 datasets. This work from KTH Royal Institute of Technology, RISE Computer Science, and Karolinska Institutet reveals substantial inter-model and dataset-specific variability in response lengths, alongside rates of text degeneration across different model-dataset combinations.
Key insights
- Response length variability across LLMs is substantial and highly dependent on the specific model and the input dataset.
- Text degeneration, where models produce repetitive or incoherent outputs, is a measurable phenomenon, with higher rates observed in certain datasets (e.g., ShareGPT, Apps) and smaller models (e.g., Llama-1B).
- Larger LLMs do not consistently produce longer or more predictable responses than smaller models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.