How to finetune a target voice（about 20 hours） without decay of robust #486

New Issue

liangmin · 2024-07-03T07:54:07Z

liangmin commented

2024-07-03 07:54:07 +00:00

I train tortoise with 10w hours of chinese audio, and then finetune it with 20 hours dataset of a female voice.
I tried:

lr = 0.00001, finetune all the parameters
lr = 0.00001, finetune using lora
lr = 0.00001, freeze part of the model and only finetune few layers

Thought the synthesized waves sounds good, but the robustness declained.
Repeat sometimes happens especially for long sentences.
For example, the input text is "I think .. blabla ......, I am happy",
the syntheszied wave is "I think .. blabla ......, I am happy, I am happy"

How to finetune to keep robustness?

I train tortoise with 10w hours of chinese audio, and then finetune it with 20 hours dataset of a female voice. I tried: 1. lr = 0.00001, finetune all the parameters 2. lr = 0.00001, finetune using lora 3. lr = 0.00001, freeze part of the model and only finetune few layers Thought the synthesized waves sounds good, but the robustness declained. Repeat sometimes happens especially for long sentences. For example, the input text is "I think .. blabla ......, I am happy", the syntheszied wave is "I think .. blabla ......, I am happy, I am happy" How to finetune to keep robustness?

Sign in to join this conversation.