Commit Graph

530 Commits

Author SHA1 Message Date
mrq
2a71486cb6 preparing for SpeechX extensions 2023-08-18 20:58:07 -05:00
mrq
ced31fd9b7 removed the sampler as it's very misleading 2023-08-18 14:47:48 -05:00
mrq
8e7f900210 forgot the = 2023-08-17 19:07:59 -05:00
mrq
3ff7cf8341 maybe fix evaluation dataset not being capped to cfg.evaluation.size 2023-08-17 18:56:37 -05:00
mrq
ee58db746f actually make the evaluation dataset shuffled for sample_type=speaker 2023-08-17 15:04:45 -05:00
mrq
18403a3523 maybe fixes eval dataloader not shuffling under distributed 2023-08-17 13:41:53 -05:00
mrq
03872b823f why did I type rglob, another 10 bucks down the drain... 2023-08-17 00:11:29 -05:00
mrq
b5f247aa11 just nuked about 9 hours of progress because I didn't make sure it pruned only on the global leader 2023-08-16 23:37:52 -05:00
mrq
d7152fc7b9 added pruning of old checkpoints if specified (cfg.trainer.keep_last_checkpoints) 2023-08-16 20:12:12 -05:00
mrq
44c08d828e added sample_type that samples from speakers to truly balance an epoch by speakers rather than the entire dataset and a sampler that tries to balance by speakers 2023-08-16 19:39:21 -05:00
mrq
599e47a813 might fix user inputted saving/quitting breaking when distributed 2023-08-15 23:52:20 -05:00
mrq
1e3e1d9315 tweaks 2023-08-15 21:58:16 -05:00
mrq
277c759ab1 fixed issue with non-distributed training, oops 2023-08-14 21:42:35 -05:00
mrq
5fa86182b5 oops 2023-08-14 10:50:40 -05:00
mrq
13571380be made exporter make more sense 2023-08-13 22:56:28 -05:00
mrq
d7deaf6def distributed training works now (hopefully) 2023-08-13 22:07:45 -05:00
mrq
2af09d0bef fixed that mysterious discepancy between the reported losses (I am so freaking mad, my piss is boiling, I had to interrupt halfway through an epoch) 2023-08-05 15:25:41 -05:00
mrq
d1b9770d41 set model to eval when inferencing (very important) 2023-08-05 04:29:05 +00:00
mrq
d89568a96e some fixes for the local framework 2023-08-05 03:22:15 +00:00
mrq
5970f254e3 some fixes for the local framework 2023-08-05 02:17:30 +00:00
mrq
012f54b7f1 another classic commit so i can copy it to another machine to gut out things and use the trainer bits for a side project that I should really get around to working on sooner than later 2023-08-04 14:21:30 -05:00
mrq
0a524f1d59 reticulating splines 2023-08-03 21:39:00 -05:00
mrq
608c1970eb ops 2023-08-03 20:36:19 -05:00
mrq
c85101403f big cleanup 2023-08-03 20:26:36 -05:00
mrq
2e03e5ac93 Fixed an issue with having fairseq installed at all will brick logging 2023-08-02 22:57:10 -05:00
mrq
f6597e2dfe adjustments 2023-08-02 18:36:26 -05:00
mrq
0f9b81de75 oops 2023-08-02 18:12:36 -05:00
mrq
7a06b27a9c Tweaks 2023-08-02 22:06:39 +00:00
mrq
d88e43800b adjustments 2023-08-02 22:01:49 +00:00
mrq
bf8cedc9dd Rewrite init 2023-08-02 21:53:35 +00:00