VALL-E

An unofficial PyTorch implementation of VALL-E, based on the EnCodec tokenizer.

Install

Note that the code is only tested under Python 3.10.7.

Install with pip (remote)

pip install git+https://github.com/enhuiz/vall-e

Install with pip (locally)

git clone --recurse-submodules https://github.com/enhuiz/vall-e.git
cd vall-e
pip install --editable .

Usage

1. Put your data into a folder, e.g. data/your_data. Audio files should be named with the suffix .wav and text files with .normalized.txt.

2. Quantize the data:

valle-quantize data/your_data

3. Generate phonemes based on the text:

valle-phonemes data/your_data

4. Customize your configuration by creating config/your_data/ar.yml and config/your_data/nar.yml. Refer to the example configs in config/test and vall_e/config.py for details. You may choose different model presets, check vall_e/vall_e/__init__.py.

5. Train the AR or NAR model using the following scripts:

valle-train yaml=config/your_data/ar_or_nar.yml

TODO

• AR model for the first quantizer
• Audio decoding from tokens
• NAR model for the rest quantizers
• Trainers for both models
• Implement AdaLN for NAR model.
• Sample-wise quantization level sampling for NAR training.
• Pre-trained checkpoint and demos on LibriTTS

Citations

@article{wang2023neural,
  title={Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers},
  author={Wang, Chengyi and Chen, Sanyuan and Wu, Yu and Zhang, Ziqiang and Zhou, Long and Liu, Shujie and Chen, Zhuo and Liu, Yanqing and Wang, Huaming and Li, Jinyu and others},
  journal={arXiv preprint arXiv:2301.02111},
  year={2023}
}

@article{defossez2022highfi,
  title={High Fidelity Neural Audio Compression},
  author={Défossez, Alexandre and Copet, Jade and Synnaeve, Gabriel and Adi, Yossi},
  journal={arXiv preprint arXiv:2210.13438},
  year={2022}
}