Reproduce “Learning to Hand Off: Provably Convergent Workflow Learning under Interface Constraints” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiayu Li, Enpei Zhang, Dawei Zhou, Elynn Chen, Yujun Yan

This research introduces a formal model, the Interface-Constrained Semi-Markov Decision Process (IC-SMDP), and a decentralized Q-learning algorithm, IC-Q, for multi-agent workflows with interface-limited observations. The IC-Q algorithm is provably convergent with finite-sample guarantees, requires only a single scalar value communicated at handoffs, and empirically achieves performance comparable to optimal centralized workflows in multi-LLM mathematical reasoning tasks.

Key insights

  • Workflow learning in interface-constrained multi-agent systems can be rigorously modeled as an IC-SMDP, allowing for the extension of the Approximate Information State (AIS) framework to this decentralized, semi-Markov context.
  • Provably convergent decentralized learning is achievable in such restrictive settings by transmitting only minimal information (a single scalar representing the expected future value) between agents at handoffs.
  • The total learning error can be decomposed into interpretable components—interface representation gap, function approximation error, and statistical error—offering insights into performance limits and guiding system design choices.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.