Reproduce “SETA: Scaling Environments for Terminal Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qijia Shen, Zhiqi Huang, Vamsidhar Kamanuru, Aznaur Aliev, Jay Rainton, Ahmed Awelkair, Zhichen Zeng, Jiajun Li and 14 more
SETA turns human-validated sources into executable terminal tasks, using agents to build environments, reference solutions, and tests. It adapts task difficulty and context to a model’s pass rate, then revalidates evolved tasks. On Terminal-Bench 2.0, the best Qwen3-8B reinforcement-learning run scored 12% pass@1 versus 3.6% for the best base run; results do not isolate adaptive evolution as the cause.
Key insights
- The dataset combines 3,255 synthesized and 1,312 evolved environments across 14 categories, linking task generation and verification with downstream model training.
- Difficulty adaptation shifted pass rates in the intended direction for 77% of simplified task pairs and 60% of harder pairs, though results vary across tasks.
- Training on terminal environments can transfer to some code-focused evaluations, but the reported experiments cover a limited set of models and tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.