Reproduce “SODBench: A Large Language Model Approach to Documenting Spreadsheet Operations” on OpenResearch

About this paper

Amila Indika, Igor Molybog

Researchers introduced Spreadsheet Operations Documentation (SOD) as a novel AI task, generating natural language explanations directly from spreadsheet operations represented as xwAPI code. Their new SODBench dataset facilitated evaluation of large language models, demonstrating that top models like GPT-4o, GPT-4o-mini, and LLaMA-3.3-70B can produce semantically relevant descriptions with statistically comparable performance.

Key insights

• Large Language Models are capable of effectively translating spreadsheet operation code into semantically relevant and meaningful natural language summaries for documentation purposes.
• LLM performance on specialized tasks like SOD is not solely dictated by model size, but significantly influenced by architectural design, training data quality, and methodological choices such as few-shot learning and prompt engineering.
• The SODBench dataset provides a robust and non-saturated benchmark that is appropriately challenging for current state-of-the-art LLMs in the context of spreadsheet operation documentation.

Summary by alphaXiv.