No Verifiable Reward for Prosody: Toward Preference-Guided Prosody Learning in TTS

arXiv:2509.18531v2 Announce Type: replace-cross
Abstract: Recent work reports gains in neural text-to-speech (TTS) with Group Relative Policy Optimization (GRPO). However, in the absence of a verifiable reward for textitprosody, GRPO trained on transcription-oriented signals (CER/NLL) lowers error rates yet collapses prosody into monotone, unnatural speech; adding speaker-similarity further destabilizes training and degrades CER. We address this with an textititerative Direct Preference Optimization (DPO) scheme that uses only a few hundred human-labeled preference pairs per round to directly optimize prosodic naturalness while regularizing to the current model. On textbfKoCC-TTS, a curated dataset of authentic Korean call center interactions capturing task-oriented dialogues, our method attains the highest human preference (ELO) with competitive CER, outperforming GRPO and strong commercial baselines. These results suggest that when prosody cannot be rewarded automatically, textithuman preference optimization offers a practical and data-efficient path to natural and robust TTS. The demo page is available at hrefhttps://tts.ch.dev