45 Commits

Author	SHA1	Message	Date
mrq	e859a7c01d	experimental multi-gpu training (Linux only, because I can't into batch files)	2023-03-03 04:37:18 +00:00
mrq	c956d81baf	added button to just load a training set's loss information, added installing broncotc/bitsandbytes-rocm when running setup-rocm.sh	2023-03-02 01:35:12 +00:00
mrq	b989123bd4	leverage tensorboard to parse tb_logger files when starting training (it seems to give a nicer resolution of training data, need to see about reading it directly while training)	2023-03-01 19:32:11 +00:00
mrq	c2726fa0d4	added new training tunable: loss_text_ce_loss weight, added option to specify source model in case you want to finetune a finetuned model (for example, train a Japanese finetune on a large dataset, then finetune for a specific voice, need to truly validate if it produces usable output), some bug fixes that came up for some reason now and not earlier	2023-03-01 01:17:38 +00:00
mrq	5037752059	oops	2023-02-28 22:13:21 +00:00
mrq	787b44807a	added to embedded metadata: datetime, model path, model hash	2023-02-28 15:36:06 +00:00
mrq	81eb58f0d6	show different losses, rewordings	2023-02-28 06:18:18 +00:00
mrq	bc0d9ab3ed	added graph to chart loss_gpt_total rate, added option to prune X number of previous models/states, something else	2023-02-28 01:01:50 +00:00
mrq	6925ec731b	I don't remember.	2023-02-27 19:20:06 +00:00
mrq	92553973be	Added option to disable bitsandbytesoptimizations for systems that do not support it (systems without a Turing-onward Nvidia card), saves use of float16 and bitsandbytes for training into the config json	2023-02-26 01:57:56 +00:00
mrq	d5d8821a9d	fixed some files not copying for bitsandbytes (I was wrong to assume it copied folders too), fixed stopping generating and training, some other thing that I forgot since it's been slowly worked on in my small free times	2023-02-24 23:13:13 +00:00
mrq	f31ea9d5bc	oops	2023-02-24 16:23:30 +00:00
mrq	f6d0b66e10	finally added model refresh button, also searches in the training folder for outputted models so you don't even need to copy them	2023-02-24 12:58:41 +00:00
mrq	1e0fec4358	god i finally found some time and focus: reworded print/save freq per epoch => print/save freq (in epochs), added import config button to reread the last used settings (will check for the output folder's configs first, then the generated ones) and auto-grab the last resume state (if available), some other cleanups i genuinely don't remember what I did when I spaced out for 20 minutes	2023-02-23 23:22:23 +00:00
mrq	1cbcf14cff	oops	2023-02-23 13:18:51 +00:00
mrq	225dee22d4	huge success	2023-02-23 06:24:54 +00:00
mrq	2aa70532e8	added '''suggested''' voice chunk size (it just updates it to how many files you have, not based on combined voice length, like it should	2023-02-22 03:31:46 +00:00
mrq	cc47ed7242	kmsing	2023-02-22 03:27:28 +00:00
mrq	93b061fb4d	oops	2023-02-22 03:21:03 +00:00
mrq	fefc7aba03	oops	2023-02-21 22:13:30 +00:00
mrq	9e64dad785	clamp batch size to sample count when generating for the sickos that want that, added setting to remove non-final output after a generation, something else I forgot already	2023-02-21 21:50:05 +00:00
mrq	8a1a48f31e	Added very experimental float16 training for cards with not enough VRAM (10GiB and below, maybe) \!NOTE\! this is VERY EXPERIMETNAL, I have zero free time to validate it right now, I'll do it later	2023-02-21 19:31:57 +00:00
mrq	bbc2d26289	I finally figured out how to fix gr.Dropdown.change, so a lot of dumb UI decisions are fixed and makes sense	2023-02-21 03:00:45 +00:00
mrq	ee95616dfd	optimize batch sizes to be as evenly divisible as possible (noticed the calculated epochs mismatched the inputted epochs)	2023-02-19 21:06:14 +00:00
mrq	6260594a1e	Forgot to base print/save frequencies in terms of epochs in the UI, will get converted when saving the YAML	2023-02-19 20:38:00 +00:00
mrq	4694d622f4	doing something completely unrelated had me realize it's 1000x easier to just base things in terms of epochs, and calculate iteratsions from there	2023-02-19 20:22:03 +00:00
mrq	ec76676b16	i hate gradio I hate having to specify step=1	2023-02-19 17:12:39 +00:00
mrq	092dd7b2d7	added more safeties and parameters to training yaml generator, I think I tested it extensively enough	2023-02-19 16:16:44 +00:00
mrq	e7d0cfaa82	added some output parsing during training (print current iteration step, and checkpoint save), added option for verbose output (for debugging), added buffer size for output, full console output gets dumped on terminating training	2023-02-19 05:05:30 +00:00
mrq	5fcdb19f8b	I forgot to make it update the whisper model at runtime	2023-02-19 01:47:06 +00:00
mrq	47058db67f	oops	2023-02-18 20:56:34 +00:00
mrq	fc5b303319	we do a little garbage collection	2023-02-18 20:37:37 +00:00
mrq	cf758f4732	oops	2023-02-18 15:50:51 +00:00
mrq	843bfbfb96	Simplified generating training YAML, cleaned it up, training output is cleaned up and will "autoscroll" (only show the last 8 lines, refer to console for a full trace if needed)	2023-02-18 14:51:00 +00:00
mrq	2615cafd75	added dropdown to select autoregressive model for TTS, fixed a bug where the settings saveer constantly fires I hate gradio so much why are dropdown.change broken to contiuously fire and send an empty array	2023-02-18 14:10:26 +00:00
mrq	d5c1433268	a bit of UI cleanup, import multiple audio files at once, actually shows progress when importing voices, hides audio metadata / latents if no generated settings are detected, preparing datasets shows its progress, saving a training YAML shows a message when done, training now works within the web UI, training output shows to web UI, provided notebook is cleaned up and uses a venv, etc.	2023-02-18 02:07:22 +00:00
mrq	c75d0bc5da	pulls DLAS for any updates since I might be actually updating it, added option to not load TTS on initialization to save VRAM when training	2023-02-17 20:43:12 +00:00
mrq	bcec64af0f	cleanup, "injected" dvae.pth to download through tortoise's model loader, so I don't need to keep copying it	2023-02-17 19:06:05 +00:00
mrq	8d268bc7a3	training added, seems to work, need to test it more	2023-02-17 16:29:27 +00:00
mrq	f87764e7d0	Slight fix, getting close to be able to train from the web UI directly	2023-02-17 13:57:03 +00:00
mrq	59d0f08244	https://arch.b4k.co/v/search/text/%22TAKE%20YOUR%20DAMN%20CLOTHES%20OFF%22/type/op/	2023-02-17 06:06:50 +00:00
mrq	12933cfd60	added dropdown to select which whisper model to use for transcription, added note that FFMPEG is required	2023-02-17 06:01:14 +00:00
mrq	96e9acdeec	added preparation of LJSpeech-esque dataset	2023-02-17 05:42:55 +00:00
mrq	f8249aa826	tab to generate the training YAML	2023-02-17 03:05:27 +00:00
mrq	3a078df95e	Initial refractor	2023-02-17 00:08:27 +00:00