Reproduce “The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks” on OpenResearch

About this paper

Wenbo Pan, Zhichao Liu, Shujie Liu, Jingying Zeng, Chin-Yew Lin, Xianfeng Tang, Yan Lu, Qi He and 1 more

Taste-Bench evaluates whether language-model agents can choose promising directions before their long-term consequences are visible. Built from 502 outcome-labeled decision forks in software engineering and AI research trajectories, it shows that leading models reach only 59.7% accuracy, while distilling hindsight-informed judgments improves held-out decision accuracy by 17.9 percentage points and raises executor task success from 14.6% to 33.7%.

Key insights

• Long-horizon judgment deteriorates as supporting evidence moves farther beyond the decision point: average accuracy falls from 62.3% for immediately supported choices to 21.0% when substantial later work is required.
• Detour decisions are especially difficult, with engineering detour accuracy averaging 35.9%, suggesting that agents struggle to reject approaches before their failure becomes explicit.
• Taste is related to but not equivalent to end-to-end coding performance; the benchmark exposes differences among models that conventional task scores compress.

Summary by alphaXiv.