Reproduce “Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Timur Galimzyanov, Sergey Titov, Yaroslav Golubev, Egor Bogomolov

JetBrains Research introduces PandasPlotBench, a new benchmark of 175 human-curated tasks with synthetically generated data to evaluate Large Language Models (LLMs) in generating Python plotting code from natural language instructions. The work demonstrates that top LLMs achieve up to 89 task-based scores for Matplotlib and Seaborn but struggle considerably with Plotly (22% incorrect code rate), and surprisingly maintain high performance even with concise user prompts when provided with rich data descriptions.

Key insights

  • Providing LLMs with concise, structured DataFrame descriptions (e.g., `head(5)` with column names and types) is crucial for optimal plotting code generation performance.
  • State-of-the-art LLMs are highly proficient with Matplotlib and Seaborn, achieving high task-based scores, but show significant weaknesses and higher error rates when generating code for Plotly.
  • LLMs demonstrate robust performance even with highly condensed, single-sentence user instructions, provided that comprehensive DataFrame descriptions and general context are included in the prompt.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.