Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis

Xiaobao Wang (Tianjin University) · Haoyu Wang (Fudan University) · Tianrui Wang (Tianjin University) · Meng Ge (National University of Singapore) · Cheng Gong (China Telecom) · Chunyu Qiang (Tianjin University) · Ziyang Ma (Shanghai Jiao Tong University) · Zikang Huang (Tianjin University) · Guanrou Yang (Shanghai Jiaotong University) · Eng-Siong Chng (Nanyang Technological University) · Xie Chen (Shanghai Jiaotong University) · Longbiao Wang (Tianjin University) · Jianwu Dang (Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Chinese Academy of Sciences)

annotated datasets data scarcitydynamic speed control mechanismemotional attention bias mechanismemotional text-to-speechend-to-end synthesisintra-sentence emotional variationmulti-emotion transitionsmulti-round inference processpretrained zero-shot tts modelprosodic variationself-training frameworkstate-of-the-art performancetransition-smoothing strategyword-level control

While emotional text-to-speech (TTS) has made significant progress, most existing research remains limited to utterance-level emotional expression and fails to support word-level control. Achieving word-level expressive control poses fundamental challenges, primarily due to the complexity of modeling multi-emotion transitions and the scarcity of annotated datasets that capture intra-sentence emotional and prosodic variation. In this paper, we propose WeSCon, the first self-training framework that enables word-level control of both emotion and speaking rate in a pretrained zero-shot TTS model, without relying on datasets containing intra-sentence emotion or speed transitions. Our method introduces a transition-smoothing strategy and a dynamic speed control mechanism to guide the pretrained TTS model in performing word-level expressive synthesis through a multi-round inference process. To further simplify the inference, we incorporate a dynamic emotional attention bias mechanism and fine-tune the model via self-training, thereby activating its ability for word-level expressive control in an end-to-end manner. Experimental results show that WeSCon effectively overcomes data scarcity, achieving state-of-the-art performance in word-level emotional expression control while preserving the strong zero-shot synthesis capabilities of the original TTS model.