Are conditioning latents harder to generate for larger datasets? #267

New Issue

Fresh12 · 2023-06-15T00:06:26Z

Fresh12 commented

2023-06-15 00:06:26 +00:00

I am attempting to generate a clip from a model that hasn't finished training but its failing due to OOM. I am thinking of two possibilities

I have the the .pth and .state files from the checkpoints but I notice that there is an additional conditioning latents pth file that is generated in the voices folder that doesn't seem to be present in unfinished models. When you load a finished model for clip generation it loads the cond_latents file. But if you don't have one it tries to 'leverage the dataset to compute latents' and then eventually runs out of memory. If this is the problem is it possible to generate the conditioning latents file at some midpoint?
The model I'm trying to test midway has a much larger dataset than the models I've successfully trained (thousands of clips-is this overkill?). Is this whats causing the failure to compute the latents? And it would just fail when it tries to create the cond_latents pth file at the end?

I am attempting to generate a clip from a model that hasn't finished training but its failing due to OOM. I am thinking of two possibilities 1. I have the the .pth and .state files from the checkpoints but I notice that there is an additional conditioning latents pth file that is generated in the voices folder that doesn't seem to be present in unfinished models. When you load a finished model for clip generation it loads the cond_latents file. But if you don't have one it tries to 'leverage the dataset to compute latents' and then eventually runs out of memory. If this is the problem is it possible to generate the conditioning latents file at some midpoint? 2. The model I'm trying to test midway has a much larger dataset than the models I've successfully trained (thousands of clips-is this overkill?). Is this whats causing the failure to compute the latents? And it would just fail when it tries to create the cond_latents pth file at the end?

psammites commented

2023-06-15 17:37:12 +00:00

The latents are generated from the wav files in the subdirectory for that voice, the more you have in there the longer it will take to generate. You can adjust the Sample Batch Size in Settings if you're OOM'ing. See discussion in #160 for more information on the effect of latents on synthesis.

The latents are generated from the wav files in the subdirectory for that voice, the more you have in there the longer it will take to generate. You can adjust the `Sample Batch Size` in Settings if you're OOM'ing. See discussion in #160 for more information on the effect of latents on synthesis.

Fresh12 commented

2023-06-23 04:50:15 +00:00

The latents are generated from the wav files in the subdirectory for that voice, the more you have in there the longer it will take to generate. You can adjust the Sample Batch Size in Settings if you're OOM'ing. See discussion in #160 for more information on the effect of latents on synthesis.

Can you tell me how to generate the cond_latents files? For some reason they aren't being generated anymore even when I go through an entire processing and training with a dataset.

> The latents are generated from the wav files in the subdirectory for that voice, the more you have in there the longer it will take to generate. You can adjust the `Sample Batch Size` in Settings if you're OOM'ing. See discussion in #160 for more information on the effect of latents on synthesis. Can you tell me how to generate the cond_latents files? For some reason they aren't being generated anymore even when I go through an entire processing and training with a dataset.

psammites commented

2023-06-23 05:16:01 +00:00