Reproduce “FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xueqing Wu, Zihan Xue, Da Yin, Shuyan Zhou, Kai-Wei Chang, Nanyun Peng, Yeming Wen
FRONTALK evaluates whether AI coding systems can build and maintain websites through ten rounds of conversational, text- or image-based feedback. Across 100 dialogues and 3,676 atomic requirements, the benchmark exposes visual-understanding, usability, and feature-forgetting failures; its ACECODER verification loop raises GPT-4o’s textual pass rate from 56.0% to 65.3% and reduces forgetting from 21.4% to 0.4%.
Key insights
- Models often interpret visual instructions as appearance-only changes, overlooking implied behavior such as making a drawn carousel arrow clickable; dense annotations and ambiguous visual cues create additional errors.
- Feature forgetting is widespread, with reported rates from 4.3% to 44.6%, and commonly occurs when a later request modifies the same component as an earlier one rather than because of simple context-length limits.
- Sparse feedback is especially challenging: for GPT-4o, pass rate drops from 56.0% under standard textual interaction to 23.9% for clarification-only feedback and 16.4% for preference-only feedback.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.