Reproduce “NL2Formula: Generating Spreadsheet Formulas from Natural Language Queries” on OpenResearch

About this paper

Wei Zhao, Zhitao Hou, Siyuan Wu, Yan Gao, Haoyu Dong, Yao Wan, Hongyu Zhang, Yulei Sui and 1 more

Researchers from Huazhong University of Science and Technology and Microsoft introduced NL2FORMULA, a new task and a large-scale dataset for generating spreadsheet formulas from natural language queries. Their proposed fCODER model, based on T5, achieved 70.6% Exact Match and 77.1% Execution Result Assessment accuracy, outperforming existing baselines.

Key insights

• Fine-tuned, specialized sequence-to-sequence models, like the proposed fCODER, significantly outperform larger general-purpose language models (GPT-3.5) and adapted formula synthesis models (FORTAP) on the NL2FORMULA task.
• Evaluating formula generation requires both Exact Match (EM) and Execution Result Assessment (ERA) metrics, as models can produce semantically equivalent but syntactically different formulas that achieve high ERA but lower EM.
• Accurately inferring cell indices and handling variations in table placement within a spreadsheet are critical challenges, with model performance degrading sharply when tables are shifted from their training positions.

Summary by alphaXiv.