Reproduce “What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Daisuke Kikuta

NTT, Inc. introduces RevPropBench, a new benchmark for evaluating Large Language Models' ability to propagate revisions across implicitly dependent elements in JSON artifacts generated through multi-turn conversations. The study identifies cost-effective test-time compute methods, such as SELECT and MED, which improve revision propagation accuracy by up to 12.5% over baselines.

Key insights

  • Conversation history is critical for LLMs to identify implicit dependencies and correctly propagate revisions, consistently outperforming methods that rely solely on the final JSON artifact.
  • Test-time compute methods, particularly LLM-based selection (SELECT) and medoid-based parallel sampling (MED), substantially improve revision propagation accuracy over baselines.
  • LLM errors in revision propagation are predominantly 'misses' (failing to propagate necessary revisions), indicating a tendency to under-propagate changes rather than over-edit.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.