7617b6485finstead just compute a bunch of stuff on the transcriptions to store later in different names so I can just retrieve what I want, also added tongue twisters for nefarious reasonsmrq2024-12-18 23:43:11 -0600
4775edaa41added text cleaning/normalization for wer purposes but it amounts to nothing desumrq2024-12-18 19:58:53 -0600
9090c34f10cringe script to process seed-tts-eval's eval dataset into something i can easily usemrq2024-12-17 22:47:12 -0600
ed152f78dftweaks to prompt duration to allow me to divorce how i use it for training with how I'm using it for the demo page, and demo page tweaks to make my life easiermrq2024-12-17 19:33:04 -0600
7129582303actually do proper wer/cer calculation by un-normalizing the scoresmrq2024-12-17 14:22:30 -0600
c2c6d912acactually do speaker verificationmrq2024-12-17 10:11:14 -0600
cd4a5f427cKO/ZH model soonmrq2024-12-15 17:01:14 -0600
4800e7179aremove nan checks because it causes problems in distributed training because I'm not syncing between GPUs (and nan losses gets ignored anyways with loss scaling)mrq2024-12-15 09:42:54 -0600
0fbfb8bbe8actually save the optimizer for the local engine backend because safetensors doesn't save itmrq2024-12-12 17:12:59 -0600
f41251f648more fixes for local engine backendmrq2024-12-12 14:38:42 -0600
6b237ae5e3tweaks for the local engine orchestrator (that I never caught since I always used the deepspeed backend)mrq2024-12-12 13:37:38 -0600
9a62e3b824APOLLO cringe (doesn't want to work with deepspeed)mrq2024-12-12 00:31:58 -0600
cddf8ca814sort batches to try and reduce number of padded tokens in batched inference (also commented out F5 samples getting added to the demo page because I would have to regenerate them)mrq2024-12-11 22:45:38 -0600
20b87bfbd0store metrics and only recalculate them if the output file is newer than the metrics filemrq2024-12-11 20:55:43 -0600
6f1ee0c6faAdded CER, transcription/similarity model args in demomrq2024-12-10 21:00:51 -0600
8568a93dadadded WER/SIM-O metrics, added APOLLO but I need to test itmrq2024-12-10 20:13:21 -0600
fc5e6d8599fixes to process_emilia.py scriptmrq2024-12-09 14:38:09 -0600
a6c745bafbchinese (mandarin?) support added (I guess I don't need pinyin, but tone markers are handled), korean validated, vocab adjustedmrq2024-12-09 14:26:19 -0600
1d460b9fe3logic fixes, I feel like output is better? (also NAR can have a temperature, I imagine it couldn't because it was having a causal masked passed to it for the longest time before I caught it a month ago)mrq2024-12-08 14:52:47 -0600
0c5a458b00deduce language per line to allow for a cheap way to allow for cross-lingual switching, kindamrq2024-12-07 22:57:29 -0600
a032ff588fdoc update, added automatically deducing language from a given text, also checks if the input is already phonemized text to allow direct control without being cringe (procrastinating adding WER/SIM-O)mrq2024-12-07 22:34:25 -0600
5d80a2d0d4fixed NAR-len issues with non-english maybe (langs weren't being passed), added interface to inference in batches through tts.batched_inference (no support for rolling context/prefixes because there's no way to do that), demo page uses batched inferencing nowmrq2024-12-07 19:21:05 -0600
1f54bf5b40revert sageattn back to optional dependency because it's not on windows, force resize_modules on by default because I broke somethingmrq2024-12-07 17:09:39 -0600
218d0e29fdugh (batchmean actually expects batch=seq_len, and not the actual batch)mrq2024-12-07 12:39:01 -0600
61ed662856ACTUALLY actually fix KD-loss (the -inf in the logits was caused by cringecode)mrq2024-12-07 12:31:54 -0600
f97e8b0c7fACTUALLY do KD-loss because of an oversight with masked_select outputting 1D tensors that get softmax'd in totalmrq2024-12-07 09:52:51 -0600
42fafbaacaactually fixed knowledge distillation because of errant -inf logits causing problems and needed to be filtered (and splitting text language / output audio language because it helps)mrq2024-12-06 21:55:20 -0600
23d402bf01added knowledge distillation in the trainer (sadly it is not agnostic because of the grave mistake of further processing the batch within the forward pass, so subsequent calls do not match......)mrq2024-12-05 23:05:52 -0600
c66a53492cforgot to add NTLK as a dependency, promoted sageattn as a default dependency since it works fine enough and seems agnosticmrq2024-12-04 20:33:25 -0600
93d27be539rolling context finally (use last N utterances as the prefix for the next gen), option to split input text prompt by sentences instead of lines (or no splitting)mrq2024-12-04 20:31:44 -0600
9dff68c0c5NAR-len tweaks (remasks a small amount of tokens per step, it seems to help with reducing the number of steps needed some of the time?, disable CFG for the first half to speed things up)mrq2024-12-04 09:30:29 -0600
cf97560e70minimum CFG of 3 for NAR-len because it seems the model will auto-default to NAR-len nowmrq2024-12-03 19:40:05 -0600
ca31da0a95sageattn (forgot to bother with testing this the other day, seems ifne)mrq2024-12-03 15:14:57 -0600
31ab90d84acringe code to convert to LlamaForCausalLM-happy weights + tokenizer dict (still need to write logic to actually use these weights for proper inferencing)mrq2024-12-03 10:18:58 -0600
84a05acb6dtouch ups in docsmrq2024-12-02 19:10:42 -0600
dcaf38b359fixed training tqdm being stubbornmrq2024-11-23 09:45:23 -0600
41d7c30ea5added much cleaner non-causal mask generationmrq2024-11-22 19:43:32 -0600
c99a74e834actually generate a causal mask because it seems sometimes it does not actually generate one because it makes assumptionsmrq2024-11-22 18:30:24 -0600
ccee5fc11cthat was actually all pointless since sdpa always had an attention mask fed to it and does not need is_causal to implicitly generate onemrq2024-11-22 16:51:50 -0600
4aa685e749what has science donemrq2024-11-22 16:45:40 -0600
147219a5e0huge oversight in the attention masking......... (i realized I have not been providing a non-causal mask to non-causal tasks)mrq2024-11-22 13:44:43 -0600
24d888c47ctemporarily dropping support for xformers because it's breaking when using an attention mask (which i dont remember commenting it out when being passed), default to not use wandb because it's being a pain when doing tests and not actual sessionsS)mrq2024-11-22 11:29:12 -0600
8aafae91fddont use timeembeddingmrq2024-11-21 23:14:52 -0600
6845c447c9added more harvard sentences to load from a text filemrq2024-11-21 13:18:11 -0600
2a084544e8moved duration padding for NAR-len to be a scalar instead (since it seems longer utterances need it much more so than shorter utterances)mrq2024-11-21 13:04:07 -0600
6aee08f9c0moved stuff in the web UI around (un-experimented the max NAR-len steps because its kind of important to adjust this value for better sounding audio / quicker generated audio)mrq2024-11-20 20:37:33 -0600
1a73ac6a20I cannot believe it's not actually called Wand DB (added wandb logging support since I think it would have been a much better way to look at my metrics)mrq2024-11-20 16:10:47 -0600
67f7bad168added mixed modality AR+NAR-len to generate a short prefix through the AR, then inference with said prefix through the NAR-len (need to experiment with it more to ensure that the masked off tokens are the only tokens getting updated)mrq2024-11-20 14:22:12 -0600
db64e6cb59dependency updates (gradio 5.x now works on my machine)mrq2024-11-20 12:33:01 -0600
b1369e7824better modality selection (pick AR+NAR by default for the ar+nar model, pick NAR-len by default for the nar-len model), lowered default CFG because it makes the AR+NAR output sped up (but can't be too low since it's required for the NAR-len)mrq2024-11-19 18:51:17 -0600
4a71981456normalize sampler index by batch size (if not using batched sampler), add option to cap out utterances for a speaker, some other thingsmrq2024-11-18 12:46:50 -0600
6cfdf94bf9swap priority to use nar-len if available, added notesmrq2024-11-18 09:40:04 -0600
069b27570fset option to set training masking ratio (I don't think for tts a fixed masking ratio is beneficial since the magic of the AR+NAR is being able to still reference the prior sequence of tokens for predicting things)mrq2024-11-17 17:04:07 -0600
88d840218ddefault set cfg strength to 3.0 since the reference model is updatedmrq2024-11-17 10:23:40 -0600
39096f8ff3redid loss calculation to be cleaner, and position ID generation, and other things (I might need to train the NAR-len from scratch and not resume from an existing checkpoint.........)mrq2024-11-14 22:17:47 -0600
ef05c951ffadjust fp16 loss scaling since I fried a model overnight when it hit 8K scalemrq2024-11-14 09:23:52 -0600
ad7cfffc00NAR-len RVQ-0 was being trained causally.............mrq2024-11-13 09:43:50 -0600
976ee87f6fresume iteration step in tqdm trainer, warn to logger if the sampler state dict was invalidatedmrq2024-11-13 09:09:28 -0600
8286aa54c8do not pass timestep token/embedding since it doesn't seem to matter at all after all, fixed training masking rate to 80% because a paper said somrq2024-11-13 09:07:10 -0600
caf721c67bset it to zero because it'll make the stop token hide more often than notmrq2024-11-12 22:30:50 -0600
0f2584eba7new meme sampler PogChamp new meme sampler PogChamp (it sort of helps?)mrq2024-11-12 22:30:09 -0600
663f07038dhaha... (do not create a token dropout/noise mask when not training (this sadly didnt fix NAR-len output))mrq2024-11-12 16:41:58 -0600
mrq