Reproduce “Reading Between the Dots: Decoding Hidden Computation across Filler Tokens” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kaley Brauer, Claudio Mayrink Verdun, Samuel Marks
The authors test whether residual-stream states at filler-token positions reveal intermediate computations in language models. Across four clean, decomposable task families, logit-lens readouts recover task-relevant values, and position-specific KV-cache transplants show some decoded values causally affect answers. On DeepSeek V3 one-fact addition, 100–500 filler tokens raise accuracy from 54% to about 72%; this is partial access, not general transparency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.