Reproduce “VisualArtifactSpec 1.0: Evaluating Visual Artifacts under Task Specifications” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shuyi Ouyang, Xiaojun Tang, Zijian Zou, Xintong Zhang, Mingli Song, Guannan Zhang, Hongwei Xue

The authors introduce VisualArtifactSpec 1.0 (VAS), a benchmark that separates Task Fulfillment from Artifact Performance across four visual artifact families. It evaluates required content and functionality separately from how content, organization, visuals, and delivery fit an artifact’s audience, purpose, and medium. Across nine model–harness configurations, Artifact Performance accounts for 20.85 points, or 73.4%, of a 28.40-point score gap; one AI judge was used.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.