Reproduce “Curriculum Design for Trajectory-Constrained Agent: Compressing Chain-of-Thought Tokens in LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Georgios Tzannetos, Parameswaran Kamalaruban, Adish Singla

Researchers developed CURLTRAC, a curriculum learning strategy that adaptively tightens training-time cost budgets for AI agents to meet strict deployment constraints. Applied to Large Language Models, this method compresses Chain-of-Thought tokens, achieving inference speedups (e.g., 7.5s to 0.6s on M1 Pro) with preserved accuracy on math reasoning benchmarks.

Key insights

  • Gradually tightening strict, trajectory-level constraints through an adaptive curriculum effectively mitigates the sparse reward problem in reinforcement learning.
  • The proposed adaptive curriculum strategy accelerates training convergence, achieving O(H^3) sample complexity versus O(2^H) for direct constraint application in a binary-tree MDP.
  • RL fine-tuning with curriculum learning can compel LLMs to compress Chain-of-Thought reasoning into fewer tokens, enhancing efficiency without notable accuracy loss.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.