b09328069eactually do CFG sampling for base AR+NAR tasksmrq2024-11-12 13:42:39 -0600
2495a7ef67Fixed STT in the web UImrq2024-11-12 12:49:53 -0600
8927bad7bcactually fixed rep pen (for ar and nar, it seems to help with nar unmasking)mrq2024-11-11 21:40:19 -0600
ec92613847actually pass input prompt length size to inferencemrq2024-11-11 20:39:48 -0600
b1df6a7bedreverted rep pen sampler due to a regressionmrq2024-11-11 20:35:08 -0600
b1f4db39c8threw in CFG sampling for normal model as well to experiment withmrq2024-11-11 20:27:38 -0600
2f56696506overhauled inference/sampler kwargs to stop being a bloated messmrq2024-11-11 20:21:16 -0600
354f8e059dstore dataset hash alongside state dict so it can be ignored if mismatchedmrq2024-11-11 18:16:56 -0600
f7b8b1e825dropped subtrain dataloader since its useless to duplicatemrq2024-11-11 17:00:49 -0600
cf9df71f2cuse homwbrewed caching system for dataloader paths / durations (I'm pretty sure I am now triggering OOM killers with my entire dataset used)mrq2024-11-11 16:32:08 -0600
9cb0b6901bunified nar.py into ar_nar.pymrq2024-11-10 12:19:48 -0600
a9d2faf2d7all I can do now until I wait for the model to (re)train for pure NARmrq2024-11-09 22:57:34 -0600
ad7e290a5eugh (ROCm seems to silently clamp any token value >= logits.shape[-1] for loss calculation, while cuda will throw an assert, making it hard to find this dumb fuckup)mrq2024-11-09 19:40:02 -0600
943fe70c10I don't know why this fixes an assert thrown but it doesmrq2024-11-09 19:04:13 -0600
f50d92ba6cAlmost made a mistakemrq2024-11-09 18:12:54 -0600
dcd5fecff3some cleanup while I wait for the NAR-len to train to an acceptable state (currently it performs okay, but only on audo after 3 seconds or so)mrq2024-11-09 12:12:46 -0600
69b0b3b854set timestep tensor to whatever the time embedding's dtype is because it'll gripe under ampmrq2024-11-09 00:11:16 -0600
5a09a5f6e9I forgot about the time embedding...mrq2024-11-08 22:46:26 -0600
811b15d280I suppose I just have a shit training method since the sampler is as solid as I can get it...............mrq2024-11-08 22:05:41 -0600
77ff23e319repeat extend the prom to fill the initial tokens for nar-len (it somewhat works, the model just needs to train more)mrq2024-11-06 23:29:53 -0600
d606a693ffeval fix for nar-lenmrq2024-11-06 23:14:16 -0600
105ed51159I guess I'll fall for the NAR-len meme again (I don't know where my previous weights are, so I need to train it again to test something)mrq2024-11-06 19:17:12 -0600
9e65e05e83more windows specific fixes, limit gradio to <5.0.0 on linux (it works on windows, but not on my linux machine tm)mrq2024-11-04 18:00:33 -0600
c83670c38cWindows specific fixes (to-do: find libespeak-ng.dll automatically because it cannot be trusted to do it by default)mrq2024-11-03 19:19:15 -0600
d229725c76more adjustments (adjustments of early-exit entropy/varentropy thresholds, default rep pen being 1.5, experimental refine-on-stop, etc.)mrq2024-11-03 18:31:28 -0600
aee08b7307changed layerskip float16 training warning (since it didnt seem to fry on my 4xV100 system)mrq2024-11-03 09:58:29 -0600
ec79230965shuffled web UI options hidden by cfg.experimental to its own tab, expose early exit selection to inferencing (it kinda works naively, still need to implement self-speculation)mrq2024-11-01 21:30:06 -0500
ef1c17430fskip step on nan loss (ironically I have not had a nan loss after adding this), throw exception with invalid cfg.dataset.sample_type and sample_order combination (because I was tricked by this in my yaml and had inconsistent vram usage)mrq2024-11-01 20:54:53 -0500
fb8faa295bactually float16(+AMP) and layerskip is bad and will kill the model......mrq2024-11-01 18:36:44 -0500
edf1e66bf9layerskip_r=6 fries the model so hard the loss is sub-1...mrq2024-11-01 17:06:07 -0500
9b6c57bc57third time's the charm (for some reason it escaped me that I should treat early exit loss as an aux_loss to be used with the normal loss, as if I was training a MoE's router)mrq2024-11-01 12:50:37 -0500
a22534e8f4layer skip training implemented (need to gut the inferencing from the repo, and to actually see if the model can benefit from this)mrq2024-10-30 20:05:45 -0500
4049f51ba9added option to load lora directly from the model file itself with --loramrq2024-10-26 00:13:10 -0500
023c3af331updated readme to reflect changesmrq2024-10-25 22:17:05 -0500
ccf71dc1b6added option to load from a model state dict directly instead of a yaml (to-do: do this for LoRAs too), automatically download the default model if none is providedmrq2024-10-25 22:15:15 -0500
a96f5aee32adjusted how i want to pass eval kwargsmrq2024-10-25 20:38:09 -0500
92e6bff6dcactually ar temp 0.5 with rep pen 1.125 seems to have the benefits of better outputs without it degrading some of the time but not all the timemrq2024-10-23 00:03:35 -0500
8920e5e86bactually have beam_width in the webUI workmrq2024-10-22 22:06:22 -0500
910571ad34too brainlet to diagnose why low temp / greedy sampling is randomly unstable some of the timemrq2024-10-22 20:13:54 -0500
8eb9a4056bmodified default arguments (ar temp = 0 and rep pen = 1.125 seems to be stable, at least given the few things i tested), do not pass top k/top p/min p to NAR even though technically none of those things should matter when greedy samplingmrq2024-10-22 18:12:39 -0500
1a02cd5bcemodify demo template to say F5 instead of YourTTS, swap LoRA comparison around to make the lora'd the base file, and the no-lora the suffix'd filemrq2024-10-21 19:52:02 -0500
71731ed785added prefixing with silence (was to test something, currently hidden under cfg.experimental=True)mrq2024-10-18 17:19:52 -0500
6b04c13c56print warning if audio promtpless inferencing with low AR temp (it really doesn't like low temps / greedy sampling)mrq2024-10-18 17:01:40 -0500
c8f31db1dedefault to greedy sample AR (i should probably test this more but it seems to pass my harvard sentences and tongue twisters)mrq2024-10-18 16:58:56 -0500
fc8dfd8617made greedy AR sampling viable (and preferable), with caveats (per comment in vall_e.models.ar_nar)mrq2024-10-18 16:55:00 -0500
75b90be325cleaned up unused config flags, allow less strict yaml by pruning missing keys, renamed some dataset configs to be more unifiedmrq2024-10-17 17:06:48 -0500
eea70f5698kludge fix for an oversight in the model when trying to train for longer input prompt durations......mrq2024-10-15 19:25:03 -0500
84005c5b00entropix apparently processes the entire sequence of logits but it falls apart when doing thatmrq2024-10-13 12:01:12 -0500
c800d28bb8respect attention defined in the yaml for web UI (which might explain why theres been a discrepancy in outputs for me)mrq2024-10-13 11:02:24 -0500
04e983b86bmodified demo page to be more modular with demoing comparisons, actually provide a path to use modified naive attention, entropix sampling is not tied to an experimental yaml flag nowmrq2024-10-12 11:27:55 -0500
75a4c866d6more demo page tweaks, added arg to force enable/disable LoRAs for inferencing (to-do: setup arg flags to handle this, and checkbox in web UI)mrq2024-10-10 19:04:12 -0500
2ea978f318added --eval-random-text-prompts to use random text prompts for eval pass, added --random-prompts for demo page and --lora to use a sample with the lora disabled, probably finally fixed validation dataloader breaking on evalmrq2024-10-10 13:40:25 -0500
acdce66d4ereadme tweaks, set the (unused) default model download URL back to the base ar+nar-llama-8 model, as ar+nar-tts+stt-llama-8 was renamed back to it since it performs wellmrq2024-10-05 22:53:53 -0500
mrq