Reproduce “Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rikuto Kotoge, Mai Nishimura, Jiaxin Ma

Researchers at The University of Osaka and OMRON SINIC X Corporation developed Distillation-Guided Policy Optimization (DGPO) to enable compact language models (0.5-1 billion parameters) to perform agentic Retrieval-Augmented Generation (RAG). This method yielded a 55x improvement in Exact Match score over initial compact models, achieving performance comparable to or surpassing larger 3B parameter teacher models on several datasets.

Key insights

  • A cold-start KD phase is crucial for stabilizing RL training in compact models starting with near-zero performance.
  • Selective teacher guidance, where the teacher intervenes actively upon student error, is more effective than uniform regularization or passive imitation for transferring complex agentic behaviors.
  • Compact models can learn to coordinate complex multi-hop reasoning and retrieval steps, compensating for limited capacity with potentially more extensive exploration.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.