Validate Training Configuration always sets settings that take more VRAM than is available. #168

New Issue

sazandora · 2023-03-24T01:39:15Z

sazandora commented

2023-03-24 01:39:15 +00:00

Like the title says, I think you may have fine tuned it TOO well on using the most VRAM possible while staying in limits. No matter what I try, it always hits 16-24 MiB over the VRAM limit and crashes if the settings are adjusted with validation.

mrq commented

2023-03-24 02:00:11 +00:00

I only can give rough estimates given:

cards I've tested (6GiB 2060, 16GiB A4000, 16+16GiB 6800XTs, 80GiB A100)
semi-sane batch sizes
using bitsandbytes
a semi-normally distributed (by length) of audio files for training
having absolutely nothing else using the target GPU
- my 2060 tests were when it was a slave card
- my 2x6800XTs already get an overhead from having the model + kernels copied twice, so its metric of 32GiB is already conservative
- the A4000 and A100 were on paperspace instances which have a baseline of zero

Like everything the web UI does, it's merely a suggestion.

I only can give rough estimates given: * cards I've tested (6GiB 2060, 16GiB A4000, 16+16GiB 6800XTs, 80GiB A100) * semi-sane batch sizes * using bitsandbytes * a semi-normally distributed (by length) of audio files for training * having absolutely nothing else using the target GPU - my 2060 tests were when it was a slave card - my 2x6800XTs already get an overhead from having the model + kernels copied twice, so its metric of 32GiB is already conservative - the A4000 and A100 were on paperspace instances which have a baseline of zero Like everything the web UI does, it's merely a suggestion.

mrq closed this issue

2023-03-24 02:00:11 +00:00

Sign in to join this conversation.