52d13b321fI rather have it default to non-strict loading instead so I can clean up YAMLsmrq2024-07-30 22:24:38 -0500
d7c6be6f78fix weird regression in handling checkpoints when backend is local, but deepspeed checkpoints are in (it was handled with LoRA loading but not real loading...)mrq2024-07-30 22:15:56 -0500
07f8e2ad06added option to set the causal size (how many tokens to sample per AR step), but requires the model to be trained for this (which explains why recurrent chunk sampling just doesn't work for the retnet tests, obvious in hindsight)mrq2024-07-30 20:53:51 -0500
ebf848d249possible speedup for samplers that require a list of previous tokens (the DRY sampler made me realize that I should copy the tolist() thing from the rep pen sampler for everything else)mrq2024-07-29 20:23:26 -0500
55b0121b1atrying (and failing) to nail a weird regression in fancier attentionsmrq2024-07-29 19:53:37 -0500
c2f5b916fcadded what I think is DRY samplingmrq2024-07-29 19:15:07 -0500
ce8bb1e4f7sanity cleanups with weird off-by-one-ness, cleaned up and validated vall_e.models.experimental works againmrq2024-07-27 15:36:05 -0500
06e948aec1suppress warning on exit about distributed not being cleaned up (because I updated my system)mrq2024-07-25 16:50:47 -0500
682e4387dcoops (fixed proms being erased from a config oversight)mrq2024-07-25 12:39:57 -0500
1acb0e9c84added experimental training setting to perform token dropout to MAYBE compensate for errors from the preceding RVQ level (two types: token error offset, token dropout embedding replace)mrq2024-07-24 19:35:17 -0500
75b04686f8added prom-less training / inferencing, some other thingsmrq2024-07-22 19:36:07 -0500
491ae2a684some insanity for sanity checks (some phonemes from phonemizing japanese are not in my tokenizer...)mrq2024-07-22 00:30:40 -0500
ad024f400factually pass language into dataset process script, fix coercing japanese into hiragana because espeak does not like kanjimrq2024-07-21 23:21:37 -0500
e19aa643a6cleaned up demo page creation, added option to pass in RVQ level sampling distribution for trainingmrq2024-07-21 19:12:03 -0500
ba7ee8c0eeadded demo link to readmemrq2024-07-19 21:22:30 -0500
9ec88d9444validated passing URI path for assets instead of base64 encoding themmrq2024-07-19 21:07:17 -0500
d87b492295added rudimentary demo page creator (currently just embeds base64 wavs into the page, need to test not doing that)mrq2024-07-19 20:49:40 -0500
d53038a9e4actually have split classifiers workingmrq2024-07-19 15:33:31 -0500
692d09f9c1eval/validation fix for SpeechX tasksmrq2024-07-19 09:16:37 -0500
39f961abcdtest trainer (vall_e.models.ar_nar) tests some SpeechX featuresmrq2024-07-18 18:46:45 -0500
83a0954f85fixes for re-introducing SpeechX tasks (need to actually validate if these all do the right things)mrq2024-07-18 17:16:32 -0500
bccbb77a1aadded option to either naively concat codes to concat audio waveforms (prior behavior) or to decode => concat => encode instead (although this only currently happens for prom sampling if an utternace is too small)mrq2024-07-18 16:48:41 -0500
97e768601cre-introducing SpeechX tasks (need to validate them all, everything works with base tts anyways)mrq2024-07-18 16:16:14 -0500
c2b8035e74oops, kept forgetting to actually pass in lang/tone tokens (despite not really using these at the moment)mrq2024-07-18 14:18:34 -0500
22fe53508cadded experimental disjointed position IDs (because I *think* this might help because technically a sequence is made up of several parts, and the position embeddings shouldn't be unified)mrq2024-07-16 19:52:41 -0500
fe0f235335mechanism to store the model config inside the weights and load them, some other things to allow LoRA training on the RetNet (gradient checkpointing will gripe about inputs not having require_grad and nothing seems to remedy it)mrq2024-07-16 18:23:13 -0500
3acc54df22allow loading a different model within the web ui (apparently I did not have the web UI in the documentation)mrq2024-07-15 19:59:48 -0500
1ecf2793f4(commented-out) support for facebookresearch/AudioDec, but support really didn't wow me (so I commented it out until I figure out why my output audio is super crusty with AudioDec)mrq2024-07-04 15:40:51 -0500
db62e55a38oops, I forgot to use the new thing for audio_backendmrq2024-07-04 14:54:11 -0500
bc2a6fa756sanity cleanup: moved experimental features under its own thingmrq2024-06-30 10:37:33 -0500
b21f74a5c5added summing of external embeddings (at this point i dont think any amount of cope bandaids will get DAC to train nicely, I think the RVQ levels the NAR tends add too much noise if they're not accurate)mrq2024-06-29 23:42:30 -0500
2808f881c8cleaned up subjugated audio embedding into a flag, flag can also have it include the original, underlying embedding as well (it seems to do better when set to inclusive)mrq2024-06-29 21:46:35 -0500
ec5eaebcbcexperimental method of using DACs quantizer ""embeddings"" to see if it helps with model qualitymrq2024-06-29 19:46:11 -0500
a8718d35a4nasty bandaid because some of my DAC dataset only has 8 RVQ levels instead of the full 9mrq2024-06-29 10:16:37 -0500
c4dd523b6fchange from chunk-slicing paths for distributed dataloader to instead interleavemrq2024-06-29 10:10:35 -0500
dd40463803limit eval size because the training batch size seems to be used for the eval dataloader, somehow (bandaid)mrq2024-06-29 09:11:28 -0500
591d3ac848have eval dataloader use eval batch size for batchedordersamplermrq2024-06-28 22:44:00 -0500
1a392b69f6local training backend should be a bit more aware of variable batch sizes, maybemrq2024-06-28 22:39:05 -0500
83075c1505sort duration buckets to ensure that paths sorted-by-duration are actually sorted by duration (because i didnt know that python dicts can have non-strings as keys), added batching samples based on total duration to ensure best training throughputmrq2024-06-28 22:28:54 -0500
8fffb94964backport fix from tortoise_tts with local trainer + loading state when training loramrq2024-06-25 13:41:29 -0500
62a53eed64fixed deducing tokenizer path, added option to default to naive tokenizer (for old models, like ar+nar-retnet-8)mrq2024-06-18 22:11:14 -0500
8a986eb480load exported LoRA weights if exists (to-do: make a better LoRA loading mechanism)mrq2024-06-18 21:45:46 -0500
2bfe786ebdban stop token for NAR levels (because sometimes it gets sampled and causes problems)mrq2024-06-17 22:14:43 -0500
7cfb78fa64enable LoRA for targetted RVQ levels (to experiment with, seems to help)mrq2024-06-17 21:45:03 -0500
7047fcc6e2actually make deepspeed work with LoRAsmrq2024-06-17 13:55:37 -0500
1d159b1476updated export routine to split LoRA weights from the state dict (should work with deepspeed)mrq2024-06-17 13:28:18 -0500
726a4b613fnaive, rudimentary DeepSpeed support (just live with the LoRA weights living with the original weights, they can be split later)mrq2024-06-17 13:17:24 -0500
bd0bc10ec0added LoRA policy to decide what layer of the model gets adapted based on simple inclusion/exclusion termsmrq2024-06-17 13:05:06 -0500
be051d9544added other LoRA method using parametrization rather than linear injectionmrq2024-06-17 09:58:34 -0500
45a39fb79fvery rudimentary lora support (no deepspeed support, tested training and saving but not loading yet)mrq2024-06-17 00:09:16 -0500
d343bde09bresidual_in_fp32=False for mamba arch backends because it breaks the classifier (output projection / lm head / what-have-you) under AMPmrq2024-06-15 12:08:03 -0500
ccb14c06efmamba2-hf using vasqu/mamba2-torch because it lets me use mamba2 without triton ops (training with my 4xV100s are not happy with mamba2 because of triton)mrq2024-06-14 19:42:17 -0500
31f71fa134sampler update (some brainworm just never actually had a sampler for sample_type=path)mrq2024-06-14 16:55:40 -0500
b3b67f34acadded option to sort paths by durations to better group equally lengthed sequences together (and there was maybe a logic error from creating the samplers and then interleave-reordering paths, desyncing them, maybe)mrq2024-06-13 22:37:34 -0500
83eab4fa59actually going for the suggested "2x layers, no intermediate scaling" is wrong for VALL-E, directly copying the normal transformer structure fixes mamba2 performance in the test trainermrq2024-06-13 20:08:22 -0500
ff97e7480dfixed pip shitting itself on setupmrq2024-06-13 13:03:36 -0500
26da24fd8dmamba updated to fix that pesky NaN error during trainingmrq2024-06-13 12:38:33 -0500
bcf3910a17the NAR only dream is dead (it just won't work)mrq2024-06-12 19:49:47 -0500
65a8960305option to split classifier per-level instead of sharing one (at this point I'm just scrambling to try and cope with training a DAC model, the NAR is being a pain)mrq2024-06-11 22:28:59 -0500
a7a6e0ac76validated that inferencing works, changed some defaults (NAR benefits from greedy sampling)mrq2024-06-09 17:11:38 -0500
58fb0a84dbadded experimental NAR only model (inferences text length, need more experimenting), AudioEmbedding logic cleanup (I still think it's being done wrong)mrq2024-06-08 15:42:02 -0500
7d6fff24f9un-tensor'd quant_level marker since it doesn't need to be one (I forgot why I had it as one but nothing seems to need it as a tensor that didn't already make it one)mrq2024-06-07 20:46:22 -0500
b0158a61d5fixed some logic errors with training (grabbing wrong quant level...)mrq2024-06-07 20:34:36 -0500
eafa622be2I forgot the actual reason I was cleaning things up was to re-include prom loss calculation (I realized the reason I did this was because of an prom embedding oversight, it seems to work now)mrq2024-06-07 20:29:25 -0500
da8242d086finally got around to removing omegaconfmrq2024-06-07 20:23:53 -0500
b2194b859are-added loading multiple models because I'm now entertaining having split AR/NAR models again (and need a way to load both at once)mrq2024-06-06 09:48:43 -0500
mrq