Researching Deep Reinforcement Learning Algorithms for

Text-to-Speech
Vo Xuan Vuong
June 24, 2024

Abstract

1 Introduction
Text-to-speech (TTS) is a crucial technology that enables the conversion of written text into spoken
language, with applications in various domains such as assistive technology, e-learning, and virtual
assistants. Despite significant progress in TTS over the years, producing high-quality synthetic speech
remains a challenging task, especially in the context of more expressive and natural-sounding speech.
Traditional TTS approaches often rely on rule-based or statistical models that have limited ability
to capture the complex relationship between text and speech. The emergence of deep learning has
revolutionized the field of TTS, with neural network-based models demonstrating superior performance
in generating more intelligible and natural-sounding speech. These deep learning models are capable
of automatically learning powerful feature representations from large datasets, enabling them to better
capture the nuances and patterns in human speech.
However, even with the advancements in deep learning-based TTS, there are still opportunities to
further improve the quality and expressiveness of the generated speech. One promising approach is
the integration of deep reinforcement learning (DRL) techniques, which can enable the TTS model
to learn optimal policies through interactions with the environment and feedback signals. DRL-based
TTS models have the potential to learn more contextual and personalized speech patterns, leading to
more natural and engaging synthetic speech.
This research aims to explore the application of deep reinforcement learning to enhance the quality
of text-to-speech systems. By leveraging the representation power of deep learning and the adaptive
learning capabilities of reinforcement learning, we seek to develop improved TTS models that can
generate more expressive and human-like speech. The proposed approach will be evaluated on a
Vietnamese dataset, contributing to the advancement of TTS technology in the Vietnamese language
domain..

2 Related Work
The field of object detection has seen remarkable advancements over the past decade, primarily driven
by deep learning techniques. Traditional approaches relied on handcrafted features and simple clas-
sifiers, which have been largely outperformed by deep neural networks (DNNs). Seminal works such
as R-CNN and Faster R-CNN introduced CNN-based object detection frameworks that significantly
improved accuracy and speed compared to previous methods.

2.1 Evolution of Object Detection

Early TTS systems used rule-based methods and waveform concatenation to generate synthetic speech,
but the quality was often very artificial-sounding. The introduction of Hidden Markov Models (HMMs)
in TTS marked a key turning point, allowing for more natural-sounding speech synthesis by modeling
the complex relationships between linguistic features and speech parameters.

1
 Rapid advancements in deep learning have continued to revolutionize the field of TTS. Neural
network-based TTS models, such as Tacotron and Transformer-based architectures, have achieved
significant improvements in quality, expressiveness, and speaker adaptability. More recently, TTS
models based on Generative Adversarial Networks (GANs) have also shown the ability to generate
highly realistic synthetic speech that is difficult to distinguish from natural human voice.

2.2 Deep Reinforcement Learning for Text-to-Speech

The incorporation of Deep Reinforcement Learning (DRL) has opened up new opportunities for ad-
vancing Text-to-Speech (TTS) technology. Conventional neural TTS models often rely on supervised
learning, which requires large annotated datasets. In contrast, DRL-based TTS frameworks enable the
system to learn speech generation policies directly through interaction with the environment, without
the need for extensive labeling.
In this approach, the TTS agent interacts with a simulated environment to generate speech outputs,
and learns to map text inputs to speech parameters by maximizing a reward signal that represents the
quality and naturalness of the generated speech. This allows the agent to explore and discover optimal
TTS strategies, leading to more expressive and personalized synthetic speech.
Furthermore, DRL-based TTS can be combined with adversarial training, similar to GAN-based
methods, where the generator network acts as the TTS agent and the discriminator network provides
feedback on the realism of the generated speech, guiding the agent towards more natural-sounding
outputs.
By harnessing the power of DRL, TTS systems can adapt to user preferences, speaking styles,
and acoustic environments, enhancing the overall user experience with personalized and context-aware
synthetic speech. As the field of DRL continues to evolve, the integration of these techniques in TTS
promises to unlock new frontiers in speech synthesis and human-computer interaction.

3 Environment
The environment consists of a text-to-speech synthesis system that interacts with a speech emotion
recognition (SER) model. The ETTS system generates speech from given text inputs, and the SER
model evaluates the emotional content of the generated speech

3.1 Text-to-Speech System (ETTS)

• Input: Character sequences representing the text to be synthesized.
• Output: Mel-spectrograms that are converted to waveforms using the Griffin-Lim algorithm.

3.2 Speech Emotion Recognition (SER) Model

• Input: Mel-spectrograms of synthesized speech.
• Output: Probabilities for different emotion classes (e.g., happy, sad, angry, neutral, surprise).

4 Agent
The agent in this DRL framework is the ETTS model, which is built on the Global Style Tokens (GST)-
based Tacotron architecture. The agent’s task is to generate mel-spectrograms from text inputs that
are not only natural-sounding but also emotionally expressive.

4.1 Architecture Components

• Text Encoder: Converts input text to high-level embeddings.
• Reference Encoder: Encodes the emotional content of reference audio into a fixed-length
emotion embedding.

2
 • GST Module: Generates style tokens corresponding to different emotions.
• Decoder: Produces mel-spectrograms from text and emotion embeddings.

5 Reward
The reward function is based on the accuracy of the SER model in recognizing the intended emotion
from the synthesized speech. This ensures that the ETTS model not only produces natural-sounding
speech but also accurately conveys the specified emotion.

5.1 Reward Calculation

The reward R is defined as: PK
N 1(pi > λ)
R= = i=1 (1)
K K
where pi is the recognition probability of the target emotion, K is the sample size, and λ is a threshold
(set to 0.5).

6 Datasheet: RAVDESS Emotional Speech Audio Dataset

6.1 Dataset Overview
The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) is a dataset con-
taining 7356 files and includes both speech and song recordings. The recordings are performed by
24 professional actors (12 male, 12 female), and they include expressions of various emotions such as
calm, happy, sad, angry, fearful, surprise, and disgust.

6.2 Data Structure

• Format: The dataset includes both audio-only and audio-visual files.
• Audio Files: 24 actors, each performing 60 different vocalizations, resulting in 1440 speech files
and 1012 song files.
• Emotions: Eight emotional states, each vocalized at two levels of emotional intensity (normal,
strong), with an additional neutral expression.
• Metadata: Each file is named according to a structured convention, indicating modality, vocal
channel, emotion, emotional intensity, statement, repetition, and actor.

6.3 Usage for DRL

The RAVDESS dataset can be used in DRL for training agents in tasks involving emotional recognition,
speech processing, and human-computer interaction. For example:

• Emotion Recognition: Training agents to recognize and classify emotions from speech signals.
• Speech Synthesis: Enhancing the emotional expression in synthesized speech by DRL agents.
• Human-Robot Interaction: Improving the responsiveness of robots to human emotions.

6.4 Example Implementation

A DRL agent can be trained using Convolutional Neural Networks (CNNs) to process the audio signals
and recognize emotions. The agent’s reward can be based on the accuracy of emotion classification,
with higher rewards for correct classifications and penalties for incorrect ones.

3
7 Algorithms
Several DRL algorithms can be utilized for this project, each with unique advantages and suitable
applications. Below are some key algorithms:

7.1 Policy Gradient Methods

These methods optimize the policy directly by following the gradient of the expected reward.

7.1.1 REINFORCE (Monte Carlo Policy Gradient)

" T #
X
∇θ J(θ) = Eπθ ∇θ log πθ (at |st )Rt (2)
t=0

7.2 Actor-Critic Methods

These methods combine the benefits of policy gradients and value-based methods by having two models:
the actor (policy) and the critic (value function).

7.2.1 Asynchronous Advantage Actor-Critic (A3C)

" T #
X
∇θ J(θ) = Eπθ ∇θ log πθ (at |st )A(st , at ) (3)
t=0

7.2.2 Proximal Policy Optimization (PPO)

h  i
LCLIP (θ) = Êt min rt (θ)Ât , clip(rt (θ), 1 − ϵ, 1 + ϵ)Ât (4)

7.3 Deep Q-Learning Methods

These methods approximate the Q-value function using deep neural networks.

7.3.1 Deep Q-Networks (DQN)

δ = r + γ max Q(s′ , a′ ; θ− ) − Q(s, a; θ) (5)
a

7.4 Deterministic Policy Gradient Methods

These methods are designed for continuous action spaces and learn deterministic policies.

7.4.1 Deep Deterministic Policy Gradient (DDPG)

∇θµ J = Est ∼ρβ ∇a Q(s, a|θQ )|a=µ(s|θµ ) ∇θµ µ(s|θµ )
 
(6)

8 Conclusion
This report presents a detailed overview of using deep reinforcement learning for emotional text-to-
speech synthesis. The combination of ETTS with an SER model in a reinforcement learning framework
allows for improved emotion discriminability in synthesized speech. The use of the RAVDESS dataset
provides a rich source of emotional speech data for training and evaluation. The inclusion of various
DRL algorithms offers multiple approaches to optimizing the performance of the ETTS system.

4
9 References
• Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., ... & Hassabis,
D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529-
533.
• Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy
optimization algorithms. arXiv preprint arXiv:1707.06347.

• Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., ... & Wierstra, D. (2015).
Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971.