Reproduce “Rubric-Aligned Disentangled Evaluation of Human Simultaneous Interpreting” on OpenResearch

About this paper

Ziyu Zhang, Satoshi Nakamura

A rubric-aware evaluator for human simultaneous interpreting separates semantic fidelity (LQ) from delivery quality (EXP) instead of collapsing performance into one score. Using 1,101 annotated Chinese–English interpreting segments, its dual-head COMET-KIWI model reaches Pearson correlations of 0.388 for LQ and 0.301 for EXP on held-out talks, outperforming frozen COMET-KIWI and reducing the cross-dimension collapse seen in prompted LLM judges.

Key insights

• Explicit prompting does not reliably disentangle evaluation criteria: prompted LLM scores show roughly 0.90 correlation between LQ and EXP, compared with 0.56 in human ratings.
• Training separate heads on separate rubric labels preserves more dimension-specific information than predicting an averaged scalar score, whose development correlations were weak or negative.
• Perceived latency has almost no correlation with segment onset delay, suggesting that synchrony judgments incorporate discourse context and cognitive load rather than timing alone.

Summary by alphaXiv.