Reproduce “A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets” on OpenResearch

About this paper

Tejas Agrawal, Vu Le, Sumit Gulwani, Gust Verbruggen

A Microsoft research team introduced the first benchmark and an online framework for next-action prediction in spreadsheets, demonstrating the learnability of such tasks. Their findings highlight the critical role of abstention and frequent prediction requests in achieving up to 32.7% User Actions Saved, thereby providing foundational insights for developing more effective spreadsheet assistants.

Key insights

• Predictive systems require robust abstention mechanisms, as accepting all non-empty predictions can lead to strongly negative user action savings due to numerous false positives.
• Frequent prediction requests (e.g., after every user action) yield the highest user action savings, suggesting that continuous, low-overhead triggers are more effective.
• The most predictive signals for next actions are primarily found within the recent history, with performance gains diminishing for context windows beyond approximately 128 prior operations.

Summary by alphaXiv.