Reproduce “Estimating Rare Events in Language Models with Proper Evaluation” on OpenResearch

About this paper

Nikita Y. Parulekar, Anqi Liu

Authors introduce GA-AMLS to estimate rare single-token argmax events under specified input distributions. It samples activations through levels under a fitted prior instead of searching discrete inputs for triggers. In small-transformer benchmarks, average SPB loss is 17.30 versus QLD’s 67.30 for the one-layer model under symmetric penalties, with wins on seven of eight distributions. Realistic prompts and larger models remain untested.

Key insights

• Estimator rankings reflect the evaluation cost model: GA-AMLS performs well with roughly balanced penalties, whereas QLD's tendency to overestimate can be preferable when underestimation is penalized much more heavily.
• GA-AMLS estimates are more centered on ground truth in the reported comparisons, but show greater variation; input-space baselines often return zero for the rarest events, while QLD tends to overestimate.
• The activation-space approach estimates probabilities rather than merely finding triggering examples, but its results depend on how well the fitted activation prior and MCMC sampling represent the relevant distribution.

Summary by alphaXiv.