Reproduce “RunAgent: Interpreting Natural-Language Plans with Constraint-Guided Execution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Arunabh Srivastava, Mohammad A., Khojastepour, Srimat Chakradhar, Sennur Ulukus
RunAgent, a multi-agent LLM system developed by researchers at the University of Maryland and NEC Laboratories America, enables more reliable and deterministic execution of natural language plans by integrating an agentic language with explicit control flow, autonomous constraint derivation and validation, dynamic execution modality selection, and robust error correction. It achieved an 81.1% exact match accuracy on the Natural-plan Calendar Scheduling dataset and improved performance across all SciBench Math subsets compared to direct GPT-4o execution and plan implementation baselines.
Key insights
- Explicitly defined agentic language constructs within natural language plans significantly improve LLM reliability in executing complex control flows (e.g., branching, iteration, jumps).
- Autonomous derivation and validation of grounded constraints at runtime are critical for ensuring correctness and guiding LLM behavior, contributing substantially to overall task accuracy.
- Dynamically choosing the appropriate execution modality (LLM, tool, or generated code) for each plan step, combined with iterative error correction, leads to more robust and deterministic multi-step task completion compared to unguided LLM execution.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.