Reproduce “Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speech” on OpenResearch

About this paper

Rikuto Kotoge, Yuichi Sasaki

Researchers from SpiralAI Inc. and The University of Osaka introduce Token-level Kahneman-Tversky Optimization (TKTO), a novel framework for LLM-based Text-to-Speech that optimizes pronunciation preferences at the token level using unpaired data. The method achieved a 39% increase in Japanese pronunciation accuracy and a 54% reduction in Character Error Rate for ambiguous words, outperforming existing preference optimization techniques.

Key insights

• By training two contrastive LLMs and using their log-ratio, the method can quantify the informativeness and assign importance weights to individual tokens for preference learning, even without paired preference data.
• Adapting prospect theory's value function to the token level with dynamic rewards allows for a more precise and stable optimization of pronunciation at a granular scale, addressing specific mispronunciations.
• The token-level weighting mechanism automatically prioritizes critical tokens (e.g., ambiguous characters) during optimization, yielding more targeted and effective learning signals than utterance-level approaches.

Summary by alphaXiv.