Reproduce “Capable yet Parsimonious: Extracting and Characterizing Hidden Chain-of-Thought in Frontier Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaoyu Luo, Tao Ren, Wenrui Yu, Xiao Li, Qiongxiu Li, Johannes Bjerva
The paper introduces a protocol that forces language models to write intermediate work to a client-defined scratchpad, making visible traces available for behavioral comparison. On an 80-problem MATH set, GPT-6 Astra reaches 93.8% accuracy under Forced conditions versus 97.5% at Native high effort, while its reconstructed traces are narrower and smaller than Sol’s. These traces are not verified internal computations.
Key insights
- Forced traces recover much of the performance and functional organization of native reasoning, but they should be treated as behavioral proxies rather than faithful records of hidden computation.
- GPT-6 Astra appears parsimonious mainly through sparse externalization: it retains broad reasoning capabilities while omitting routine intermediate operations and reducing branching, backtracking, and trial-and-error.
- Compact traces can be capability-dependent artifacts: stronger recipient models reconstruct Astra’s omitted steps more effectively than weaker recipients, limiting their direct value for supervision or distillation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.