tacotron

Overview of the Tacotron Project

The Tacotron project is an end-to-end text-to-speech (TTS) synthesis model implemented using TensorFlow. The purpose of Tacotron is to convert written text into human-like speech automatically. This project is notable for being extensively documented, making it easier for users to understand and implement the TTS model.

Requirements

To use Tacotron, several software components must be installed, including:

• NumPy: A package for scientific computing with Python.
• TensorFlow: An open-source platform for machine learning.
• librosa: A Python package for music and audio analysis.
• tqdm: A library for creating progress bars in Python.
• matplotlib: A plotting library for Python.
• scipy: A Python library used for scientific and technical computing.

Data Used in Tacotron

Tacotron leverages different speech datasets to train its model. Three primary datasets are utilized:

1. LJ Speech Dataset: This is a public dataset composed of 24 hours of high-quality audio. It serves as a standard benchmark in TTS tasks.

2. Nick Offerman's Audiobooks: Comprising 18 hours of narration by Nick Offerman, these audiobooks provide varied speech samples, which help test the model's ability to work with smaller datasets.

3. The World English Bible: This dataset is a modern English version of the Bible, with 72 hours of audio. It's available in the public domain, and the audio clips are carefully aligned with the text.

Training Process

To train Tacotron, follow these steps:

1. Data Preparation: Download the LJ Speech Dataset or use your own data.

2. Parameter Adjustment: Modify the hyperparameters in the hyperparams.py file. If preprocessing is needed, set prepro to True.

3. Model Training: Execute python train.py for training. If preprocessing was configured, run python prepro.py before starting the training.

4. Model Evaluation: Regularly run python eval.py to monitor the training progress.

Synthesis of Speech Samples

The Tacotron model generates speech samples using Harvard Sentences. To synthesize speech, users can run python synthesize.py, and the output files will be available in the samples directory.

Monitoring Training Performance

The project includes a training curve plot and attention plots, which help in assessing the model’s performance. The attention plot should be linear; deviations might indicate the need for retuning or restarting the training process.

Sample Outputs and Pretrained Files

Pretrained models are available for use, but note that 200k training steps may not yield the best performance:

• LJ Dataset at 200k Steps
• Nick Offerman’s Audiobooks at 215k Steps
• World English Bible at 183k Steps

Pretrained files are accessible for download via provided links.

Important Notes

• Always monitor attention plots during training to ensure proper model alignment. Misalignment requires intervention, often reverting to a previous checkpoint.
• The learning rate is a crucial parameter; a rate of 0.001 has proven effective compared to a higher rate, preventing loss explosions.

Project Adjustments

Tacotron includes some modifications from the original paper, such as the use of Noam style learning rate adjustment, gradient clipping, and bucketed training batches. Moreover, the model applies an affine transformation to adjust for dimensional differences between output layers.

References

This implementation has influenced various papers and projects, such as storytelling neural networks and efficient TTS systems.