Reproduce “Do Cantonese-Adapted Language Models Better Predict Cantonese Reading? A Cross-Model Eye-Tracking Evaluation” on OpenResearch

About this paper

Ziqi Zhang, Emmanuele Chersoni, Mohammad Momenian

Across 10,011 Cantonese eye-tracking observations, language-model information measures add modest predictive value beyond standard lexical and syntactic variables. Extensive Cantonese adaptation in CantoneseLLM-7B produced the strongest overall alignment with human reading times, while light adaptation in JED351 did not consistently improve on its CKIP predecessor; the best joint models reduced mean absolute error by up to 1.251 ms.

Key insights

• Language-model measures provide incremental but small gains: the joint four-measure model improves every model–outcome combination, with the clearest relative gains for first fixation duration.
• Lexical surprisal favors CantoneseLLM-7B, but entropy reduction favors CKIP, showing that predicting a target word and modeling uncertainty change after that word are distinct capabilities.
• Adaptation scale matters: CantoneseLLM-7B consistently exceeds Qwen2.5-7B, whereas JED351 does not generally surpass CKIP despite vocabulary modification and limited Cantonese fine-tuning.

Summary by alphaXiv.