Reproduce “A Stitch in Time Saves Nine: Proactive Self-Refinement for Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jinyi Han, Xinyi Wang, Haiquan Zhao, Tingyun li, Zishang Jiang, Sihang Jiang, Jiaqing Liang, Xin Lin and 4 more

A new reinforcement learning approach, ProActive Self-Refinement (PASR), enables large language models to dynamically refine their outputs during the generation process. This method autonomously improves output accuracy by up to 8.2% on challenging tasks while simultaneously reducing token consumption by over 40% compared to standard generation.

Key insights

  • Empowering LLMs with proactive, in-process self-refinement, akin to human cognition, enables dynamic error correction and avoids error propagation.
  • Reinforcement Learning, specifically GRPO, is vital for training LLMs in adaptive self-refinement, outperforming prompt-based or instruction-following fine-tuning approaches.
  • A sophisticated, comparison-based reward system that evaluates the effectiveness and necessity of refinement against unrefined baselines is critical for preventing superfluous modifications and 'reward hacking'.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.