Reproduce “TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shoufa Chen, Luyuan Wang, Xuan Yang, Zhiheng Liu, Yuren Cong, Yuanfeng Ji, Feiyan Zhou, Xiaohui Zhang and 2 more
Meta AI researchers introduced TUA-Bench, a benchmark comprising 120 real-world tasks designed to evaluate general-purpose terminal-use agents across diverse scenarios, from everyday digital activities to expert-level scientific workflows. Evaluations showed that the strongest frontier agent achieved a 65.8% success rate, indicating a significant remaining challenge for reliable general-purpose terminal interaction.
Key insights
- A substantial performance gap persists for reliable general-purpose terminal-use agents, with the strongest evaluated frontier agent achieving a 65.8% success rate.
- Task execution time budgets and reasoning effort significantly impact agent success, with diminishing returns observed at higher effort levels (e.g., 27.1 percentage point increase in success for Terminus-2 + GPT-5.5 by increasing time from 150s to 2400s).
- The relative performance of language models is not constant across different agent scaffolds, indicating that the choice of agent framework is as influential as the underlying model in determining overall efficiency and success.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.