81b05dabb9accurate epoch metric is now reported (based on samples processed / length of dataset's paths, rather than naive assumptions)mrq2023-09-03 08:03:36 -0500
922404285cfixed segfault from tts-c task token exceeding being too big (inserted it in the hypothetical svc task token because in reality that is never ever going to be a feasible task to train against)mrq2023-09-02 19:25:43 -0500
4613781e23integrated plot script, added tts-c task token to help the model be able to mix between normal VALL-E and VALL-E continuousmrq2023-09-02 16:29:53 -0500
f7e942ec99modified plotting script to be more agnostic to Xmrq2023-09-02 13:59:43 -0500
e40c0d34a0somewhat got recurrent forward working (it's as accurate as chunkwise forward: it's not accurate at all), added option to use AMP instead of blanket setting the weight's dtypemrq2023-09-01 20:58:29 -0500
2bc2d08b09(need to verify) added modifying model size and config bool to align with VALL-E continuous' methodologymrq2023-09-01 17:19:34 -0500
5c8694db8enasty bandaid if there's no validation dataset specified during training (for example, during finetunes)mrq2023-08-30 18:23:05 -0500
7f4388e591added total samples processed and tokens processed (len of text tokens + len of target response tokens)mrq2023-08-28 11:02:45 -0500
87c4bfedbaadded ability to mark models as disabled for training, and hotloading them for eval/validation (useful if training only one model, or training a model per GPU)mrq2023-08-27 12:26:12 -0500
165a1154e0Undo naive=False test flag, this shouldn't have made its way inmrq2023-08-26 22:00:43 -0500
78378ed1ceoverhauled dataloading code to be marginally faster, mostly cleaned up, and can leverage a metadata json to help things outmrq2023-08-26 19:53:23 -0500
7b3be3d7bfadded helper scripts to process LibriTTS/LibriLight, detect duplicate speaker+books between them, and script to directly phonemize and quantize LibriTTSmrq2023-08-26 10:21:12 -0500
16e0020901disabled chunkwise_recurrent for 2x speed gains (I suppose it has been working the entire time, but I have not been properly grabbing things, and this might explain why the output is bad)mrq2023-08-25 19:50:19 -0500
6455a2f9d7I think I fixed a bug?mrq2023-08-24 23:33:36 -0500
f3fbed5ffdupdated notices tailored for windows / low VRAM cardsmrq2023-08-24 17:19:10 -0500
0517d620b8fixes with the local backendmrq2023-08-24 17:05:56 -0500
00ad4af651updated draconian requirement for espeak-ng to be installed and the env var set to the dll for Windowsmrq2023-08-24 14:57:01 -0500
b6c9686f7dDo not install DeepSpeed under Windows (to-do: default backend to use local if on Windows)mrq2023-08-24 14:27:36 -0500
22904a8639more oversights fixed because I've been using a cached dataloader forever now and didn't catch these problemsmrq2023-08-24 10:25:33 -0500
a47029065bI don't know if the lack of start/stop tokens being added was causing my inference tests to fail, but it seems better nowmrq2023-08-20 19:21:54 -0500
b105f6211eadded ability to export weights mid-training to avoid CBT to yank the weights while the training script is runningmrq2023-08-20 13:39:58 -0500
fc576010cewrapped saving the checkpoint in a try/catch so I can stop waking up to the damn trainer crashing because it ran out of disk space; I'd much rather it keep training to give me time to eventually clear up disk space rather than it silently restarting on its ownmrq2023-08-20 06:29:17 -0500
2d1a9f10c0nightmare of spaghetti that might break compat; mechanism to increase RVQ bins of an existing model without retraining, keeps sampled proms/resps at max RVQ level and trim off excess levels according to what model receives them, some other things I already forgot (I really hope no one else has weights being baked right now)mrq2023-08-19 15:06:33 -0500
f7f6d3bf6dvalidated that SpeechX tasks cse and nse works, added a method to test each task by invoking python3 -m vall_e.data --action=tasks --tasks='sr,se,cse,nse'mrq2023-08-19 09:50:07 -0500
6ca347e1e1literally had a urethra moment before going to bed with a way to implement cse/nse tasksmrq2023-08-19 01:16:46 -0500
8f42c578c9setting up for allowing training for a partial amount of the speechx tasks (do NOT try this at home yet without a proper model, as performance is predecated on having a solid base vall-e model for the tasksmrq2023-08-19 00:16:08 -0500
ae9d38aa31forgot to have it pull from specified noise to the hdf5 datasetmrq2023-08-18 23:57:07 -0500
77292c42f9tested the training preparation for tasks ns, sr, and tse (I don't expect it to go well with only 2 RVQ bins)mrq2023-08-18 23:55:40 -0500
bbb0563b3dpseudocode polyfill stub some other flavor of working on adding the tasksmrq2023-08-18 22:22:13 -0500
0b46c1e312god I am inexperienced with retaining compat from previous weights, I hope no one actually has weightsmrq2023-08-18 21:29:20 -0500
508677fcd5repaired auraloss loss calc during eval/valmrq2023-08-18 21:19:47 -0500
3ff7cf8341maybe fix evaluation dataset not being capped to cfg.evaluation.sizemrq2023-08-17 18:56:37 -0500
ee58db746factually make the evaluation dataset shuffled for sample_type=speakermrq2023-08-17 15:04:45 -0500
18403a3523maybe fixes eval dataloader not shuffling under distributedmrq2023-08-17 13:41:53 -0500
03872b823fwhy did I type rglob, another 10 bucks down the drain...mrq2023-08-17 00:11:29 -0500
b5f247aa11just nuked about 9 hours of progress because I didn't make sure it pruned only on the global leadermrq2023-08-16 23:37:52 -0500
d7152fc7b9added pruning of old checkpoints if specified (cfg.trainer.keep_last_checkpoints)mrq2023-08-16 20:12:12 -0500
44c08d828eadded sample_type that samples from speakers to truly balance an epoch by speakers rather than the entire dataset and a sampler that tries to balance by speakersmrq2023-08-16 19:39:21 -0500
599e47a813might fix user inputted saving/quitting breaking when distributedmrq2023-08-15 23:52:20 -0500
13571380bemade exporter make more sensemrq2023-08-13 22:56:28 -0500
d7deaf6defdistributed training works now (hopefully)mrq2023-08-13 22:07:45 -0500
2af09d0beffixed that mysterious discepancy between the reported losses (I am so freaking mad, my piss is boiling, I had to interrupt halfway through an epoch)mrq2023-08-05 15:25:41 -0500
d1b9770d41set model to eval when inferencing (very important)mrq2023-08-05 04:29:05 +0000
d89568a96esome fixes for the local frameworkmrq2023-08-05 03:22:15 +0000
5970f254e3some fixes for the local frameworkmrq2023-08-05 02:17:30 +0000
012f54b7f1another classic commit so i can copy it to another machine to gut out things and use the trainer bits for a side project that I should really get around to working on sooner than latermrq2023-08-04 14:21:30 -0500
mrq
Orjwan Zaafarani
Zhe Niu