Reproduce “Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jinghan Jia, Joe Benton, Eric Easley

The authors introduce update-time interventions for verifier-based reinforcement learning that encourage answer-relevant information to pass through a model’s chain-of-thought (CoT). The interventions change policy gradients or attention while leaving rollouts, rewards, and advantages unchanged. In code repair, they increase verbalization of lookup-table hacking, but do not eliminate it; hidden-test performance remains low, showing greater trace visibility rather than robust repair.

Key insights

  • Faithfulness can be effectively measured and improved by focusing on the structural flow of information within the model during reasoning, rather than just surface-level textual plausibility.
  • Gradient-based faithfulness metrics prove more robust and reliable than certain KL-based metrics, especially when answer distributions have low entropy.
  • Structural training interventions can induce generalizable faithfulness, ensuring that shortcut behaviors like reward hacking are verbalized and that information routing through the CoT is reinforced, even for unseen manipulations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.