-
https://git.ecker.tech/ aims to provide a place to share my efforts while maintaining true ownership of my code, as I do not trust GitHub.
XMR: 4B9TQdkAkBFYrbj5ztvTx89e5LpucPeTSPzemCihdDi9EBnx7btn8RDNZTBz2zihWsjMnDkzn5As1LU6gLv3KQy8BLsZ8SG
- Joined on
2022-10-10
In the Training > Prepare Dataset tab, you can click "(Re)Create Datasets" for it force regenerate the train.txt file again, although it's odd that it seemed to have done the prior steps…
into an issue where the model stops training after only 10 minutes
Oh, I suppose this is a bit of a mismatch between what is considered an "epoch" between my side creating the YAML, and the…
This is more of a cosmetic issue than a functional one.
The printout on the left shows your loss at 0.563, which corresponds to the bottom of the green line on the right. Your LR is 3.239e-06,…
Given the graph, loss curve, and LR curve, I think your LR scheduling might have been too lax and ended up frying the finetune from the LR decaying very slowly. The default scheduling should be…
The training script should be able to resume training from the last checkpoint without needing to update anything else, even if you modified the dataset.
The "Resume Training" or whatever it…
mmm...
Verify what your batch size is set to in settings. If it's something higher than 16, then the error message might just be a misnomer, and the true issue is that you're just running out…
Ah yeah. I'm not too sure where the issue lies, as even using the repo to transcribe large datasets will eventually hang after a while, from presumably a memory leak somewhere. I'm just not too…
7f4388e591
added total samples processed and tokens processed (len of text tokens + len of target response tokens)
87c4bfedba
added ability to mark models as disabled for training, and hotloading them for eval/validation (useful if training only one model, or training a model per GPU)
I think I've got everything I wanted to do done before the next training session, so I can just leave the GPUs (yes, plural) training and shutting up for a while (or at least not overworking…
You'll either need to:
- update the symmap table in
./vall_e/data.py:35with the first line printed out when the…
78378ed1ce
overhauled dataloading code to be marginally faster, mostly cleaned up, and can leverage a metadata json to help things out
7b3be3d7bf
added helper scripts to process LibriTTS/LibriLight, detect duplicate speaker+books between them, and script to directly phonemize and quantize LibriTTS
def unload_tts():
global tts
if tts:
del tts
tts = None
print("Unloaded TTS")
do_gc()
I'm pretty sure there's a magical Python issue where this isn't actually working…
Additionally, while trying to make recurrent_forward work, I managed to finally fix the issue with inferencing. It seems that chunkwise_recurrent does in fact work, and it was actually…
16e0020901
disabled chunkwise_recurrent for 2x speed gains (I suppose it has been working the entire time, but I have not been properly grabbing things, and this might explain why the output is bad)
690947ad36
Do not double phonemize if using VALL-E backend (I wonder how many hours I've wasted from this oversight)
Any model (autoregressive.pth) trained through the web UI (trained with DLAS) are all compatible with the base TorToiSe and the forks. I haven't checked base TorToiSe in a while to see if it…
Idle hands are truly the devil's workshop.
I'm getting tempted to make another poor purchase decision. My gut wants to go with a 7900XTX despite:
- already knowing there's an inherent design…