用于评估电子表格中下一步行动预测的基准与框架
A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets
摘要
预测性代码补全大大加快了开发者的工作速度。在电子表格中，虽然这种自动补全功能更为常见，但实际上却几乎不存在。为了弥补这一不足，我们提出了一种评估标准，用于衡量那些能够观察电子表格中的用户操作序列并预测未来操作的系统。存在的两个挑战是：第一，公共电子表格数据集中缺乏编辑历史记录；第二，电子表格中的操作行为具有复杂的特性，包括空间、时间以及复合操作等。针对第一个问题，我们手动整理了52个包含12K种操作的序列，这些序列是从公共数据集中提取出来的，通过参数化启发式算法和大型语言模型进行优化得到。针对第二个问题，我们提出了一种在线评估方法：在每次用户操作后都进行预测，接受或拒绝该预测结果，然后根据接受的结果来更新未来的操作行为，如此循环直到获得目标电子表格为止。我们使用多种基准预测器（包括零样本大型语言模型、微调后的小语言模型以及传统模型），并分析该评估标准所揭示的各种特性，包括但不仅限于：保存的操作行为特性、误报情况、效率、用户档案的影响、触发条件的影响以及上下文的影响等。
English Abstract
Predictive code completion greatly accelerates how quickly developers work. In spreadsheets, despite being much more common, such auto-completion features are virtually non-existent. To address this gap, we introduce a benchmark for systems that observe a sequence of user actions in a spreadsheet and predict future actions. Two challenges are (1) the absence of edit histories in public spreadsheet corpora and (2) the complex space of spreadsheet actions (spatial, temporal, composite). To address (1), we manually curate 52 sequences of 12K actions that recreate spreadsheets from public corpora, seeded by parametrized heuristics and LLM refinement. To address (2), we propose an online evaluation that expects a prediction after each user action, accepts or rejects that prediction, updates the future actions upon acceptance, and repeats this until the target spreadsheet is obtained. We use multiple baseline predictors (including zero-shot LLMs, fine-tuned SLMs, and classical models) and analyze different properties that our benchmark teaches us, including but not limited to: properties of saved actions and false positives, efficiency, effect of user profiles, effect of triggers, and effect of context.