Reproduce “Language Model Maps for Prompt-Response Distributions via Log-Likelihood Vectors” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Momose Oyama, Yusuke Takase, Hidetoshi Shimodaira
A shared behavioral space is constructed for language models by recording their conditional log-likelihoods on the same prompt–response pairs. Across hundreds of models, the resulting geometry reflects model families and capabilities, approximates conditional KL divergence, supports task-score prediction, and reveals approximately additive effects of prompt modifications.
Key insights
- Conditional likelihood vectors organize models into recognizable architecture and family clusters, while also reflecting broad differences in capability.
- The effects of repeating a prompt and appending a chain-of-thought instruction are approximately compositional: the combined displacement is close to the sum of the individual displacements, with mean-log-likelihood change correlation of 0.987.
- Subtracting unconditional response likelihood can expose training-data-related structure, although the PMI interpretation depends on an approximate marginal and varies across datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.