211 Commits

Author	SHA1	Message	Date
mrq	14709ac67f	ughh	2024-05-12 07:30:59 -05:00
mrq	3774fcbdee	ugh	2024-05-11 22:58:38 -05:00
mrq	856545f8bb	nan loss detection (should have added it earlier), loss scaling for local backend + fp16	2024-05-11 22:23:29 -05:00
mrq	a755eb3c62	ugh	2024-05-11 17:34:45 -05:00
mrq	88e9b9caff	local ddp fix	2024-05-11 17:29:01 -05:00
mrq	3337c69e5a	leverage between xformers and torch.backends.cuda.sdp_kernel for attention	2024-05-11 17:14:05 -05:00
mrq	d33c7bb7cf	ugh	2024-05-11 16:47:19 -05:00
mrq	0b6499601b	sanitizing	2024-05-11 16:31:05 -05:00
mrq	71e373064f	remove redundant loss, tweak readme	2024-05-11 15:02:47 -05:00
mrq	04a80d6b55	maybe it's better to be more explicit in deepspeed configs	2024-05-11 13:57:43 -05:00
mrq	4d93a16ef7	might just be better to explicitly define prompt duration ranges, especially under a "train small contexts then increase it" training paradigm	2024-05-11 09:50:54 -05:00
mrq	bd0a36ba8d	I swear I keep seeing tqdm flicker back a number	2024-05-10 18:36:01 -05:00
mrq	2109712e5b	resolve deprecation warning that doesn't show on my old training rig but does on my new one	2024-05-09 23:25:44 -05:00
mrq	1547de5020	haha...	2024-05-09 23:15:52 -05:00
mrq	b7bd885651	some possible sanity with deepspeed config	2024-05-09 22:48:42 -05:00
mrq	c4b696ebeb	oops	2024-05-09 22:33:40 -05:00
mrq	c22a177cf8	forgot to pass warmup to schedule free	2024-05-09 22:18:49 -05:00
mrq	b6131565ad	autotune?	2024-05-09 21:25:40 -05:00
mrq	6ed6ab8c03	a bit more cleanup for deepspeed ds_cfg creation	2024-05-09 21:00:26 -05:00
mrq	0d5d545a40	crammed in DAdaptation (doesn't seem worth it) and ScheduleFree (forgot I wanted to weeks ago, seems promising), optimization wrapper cleanup, test trainer changes, etc.	2024-05-09 20:28:20 -05:00
mrq	c6e0f905b5	final tweaks (again) before training restarts	2024-05-08 02:11:38 -05:00
mrq	215800484d	correcting my wrong of assuming I could just use raw 24Khz audio in the 44Khz DAC without too much of an issue (there are issues)	2024-05-04 23:49:15 -05:00
mrq	9f738fbd5b	seems I actually don't need RVQ bins 9-32 with the 24Khz DAC model........ (time to requantize my audio...)	2024-05-04 23:09:18 -05:00
mrq	33b7f81b94	small cleanups	2024-05-04 22:37:22 -05:00
mrq	8aa1b2dabf	documentation update	2024-05-04 21:03:46 -05:00
mrq	253441b750	forgot to disable verbose flag	2024-05-04 13:13:52 -05:00
mrq	3dca1125f5	implemented xformers in HF's Llama (because theres no flash attention for Volta cards)	2024-05-04 13:07:45 -05:00
mrq	277dcec484	apparently I got an error for trying to serialize an errant tensor that made its way into the json, this could be remedied easily with recursively traversing the dict and coercing any objects to primitives, but I'm tired and I just want to start training and nap	2024-05-04 12:33:43 -05:00
mrq	ffa200eec7	added option to specify frames per second for the given audio representation (Encodec is 75Hz, DAC is 41Hz (at 24K sources))	2024-05-04 12:05:41 -05:00
mrq	c494894261	simple DDP wrapper (for my NVlink test)	2024-05-04 11:48:26 -05:00
mrq	a7b43b98b5	renamed cfg.bitsandbytes to cfg.optimizations (and having it serve as cfg.optimizations.bitsandbytes)	2024-05-02 20:08:59 -05:00
mrq	b5d1456a09	backwards compat for my shitty old weights (was testing if disabling AudioEmbedding summing magically made things better (it did not))	2024-04-29 22:14:01 -05:00
mrq	5120ffdda7	god it would be nice to know the best way to handle audio embeddings, because I genuinely don't know without skimming through papers or devoting X amount of GPU hours in training	2024-04-29 18:24:05 -05:00
mrq	6a11bc9cb6	update tokenizer because, for some reason, it had the wrong order for the special tokens to where eos = unk	2024-04-29 09:09:26 -05:00
mrq	57810e4ba4	metadata only path (might drop HDF5 since its giving file sizes twice as large as my actual unpacked dataset)	2024-04-28 23:03:09 -05:00
mrq	caad7ee3c9	final tweaks, hopefully	2024-04-28 22:28:29 -05:00
mrq	ffc334cf58	added dataset transcription helper script (now I don't ever have to touch ai-voice-cloning) (to-do: unify scripts into the module)	2024-04-21 17:43:20 -05:00
mrq	b251669536	forgot to fix up the test trainer	2024-04-21 14:58:04 -05:00
mrq	071fb97777	dataset preparation script updates, caved and am using HF tokenizer now	2024-04-21 14:49:18 -05:00
mrq	a8ffa88844	it slipped my mind that technically DAC can be used at any sample rate, since it models waveforms; make it a config YAML option to allow this behavior	2024-04-19 18:36:54 -05:00
mrq	8214aa23d7	converting over to a different intermediary dataset format	2024-04-18 21:24:06 -05:00
mrq	4f5c9e518a	actually use the passed-through sample rate from encode for DAC because it does its own resampling I guess	2024-04-18 13:32:41 -05:00
mrq	2e9e6e68f7	Forgot I need to use the DAC's 44K model because 24K model has 32 codebooks instead of 9.	2024-04-17 20:59:25 -05:00
mrq	5ff2b4aab5	finally swallowing the Descript-Audio-Codec pill (I guess I'm going to have to regenerate my entire dataset)	2024-04-17 20:39:35 -05:00
mrq	b0bd88833c	refractor cleanup, had a revelation on how I can handle a batch of varying tasks	2024-04-16 21:04:48 -05:00
mrq	467fa1c5ee	wrapper fixes	2024-04-16 10:19:02 -05:00
mrq	aa1e25fbf5	backwards compat for old YAMLs with models, option to set flash attention 2 for Llama (and derivatives), included syncdoth/RetNets torchscale retnet for shits and grins, etc.	2024-04-16 10:02:31 -05:00
mrq	545162195b	deprecate sole AR/NAR model by only keeping the AR+NAR (the beauty of no one using this is that I can break compat as much as I want), add tone token for when I classify my dataset with tone/emotion in the future, some other things	2024-04-15 19:54:32 -05:00
mrq	d69a00e389	Properly pass retention_mask for retnet-HF, attempt to fix recurrent forward for retnet (doesn't work still)	2024-04-14 13:12:50 -05:00
mrq	789bb5d11b	add an optional label override for model loading (used for easy testing between 12/16/20/24 layered model)	2024-04-13 12:43:35 -05:00