mrq
/
ai-voice-cloning
31
73
Fork 42
Code Issues 221 Pull Requests 5 Packages Projects Releases Wiki Activity

Labels Milestones

master #369

Merged
mrq merged 4 commits from ken11o2/ai-voice-cloning:master into master 2023-09-04 22:04:04 +07:00
Conversation 3 Commits 4 Files Changed 2
ken11o2 commented 2023-09-04 19:05:48 +07:00

Add DeepSpeed feature for tortoise

Add DeepSpeed feature for tortoise
ken11o2 added 3 commits 2023-09-04 19:05:58 +07:00
5d87418d3a add DeepSpeed feature 
add DeepSpeed feature
57f8da7802 add args.use_deepspeed 
add DeepSpeed feature
mrq commented 2023-09-04 20:04:17 +07:00
Owner

ai_voice_cloning_edited_to_use_deepspeed.zip

Resubmit the PR without this file, please.

> ai_voice_cloning_edited_to_use_deepspeed.zip Resubmit the PR without this file, please.
ken11o2 added 1 commit 2023-09-04 20:52:42 +07:00
ken11o2 commented 2023-09-04 20:55:00 +07:00
Poster

yes, i deleted zip file. Pls check also the PR in tortoise-tts. Thanks u

yes, i deleted zip file. Pls check also the PR in tortoise-tts. Thanks u
ken11o2 closed this pull request 2023-09-04 21:43:25 +07:00
ken11o2 reopened this pull request 2023-09-04 21:43:40 +07:00
mrq merged commit 29c270d1cc into master 2023-09-04 22:04:04 +07:00
mrq referenced this issue from a commit 2023-09-04 22:04:06 +07:00
master (#369)
ken11o2 commented 2023-09-05 08:45:39 +07:00
Poster

Can you take a look inside src/cli.py , webui.py and utils.py. I'm trying to make cli work with deepspeed but something went wrong,

I did some test with webui and cli.
If i run in webui with same setting, it took ~20s to inference with deepspeed and ~40s without deepspeed.
Here is output i run with webui

Whisper detected
[2023-09-05 10:21:56,713] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
VALL-E detected
Bark detected
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Loading TorToiSe... (AR: /mnt/e/ai-voice-cloning/models/tortoise/autoregressive.pth, diffusion: ./models/tortoise/diffusion_decoder.pth, vocoder: bigvgan_24khz_100band)
Hardware acceleration found: cuda
Loading tokenizer JSON: ./modules/tortoise-tts/tortoise/data/tokenizer.json
Loaded tokenizer
Loading autoregressive model: /mnt/e/ai-voice-cloning/models/tortoise/autoregressive.pth
[2023-09-05 10:23:13,696] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.10.2, git-hash=unknown, git-branch=unknown
[2023-09-05 10:23:13,698] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-09-05 10:23:13,698] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
WARNING! Setting BLOOMLayerPolicy._orig_layer_class to None due to Exception: module 'transformers.models' has no attribute 'bloom'
Using /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118/transformer_inference/build.ninja...
Building extension module transformer_inference...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.5863604545593262 seconds
[2023-09-05 10:23:15,232] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed-Inference config: {'layer_id': 0, 'hidden_size': 1024, 'intermediate_size': 4096, 'heads': 16, 'num_hidden_layers': -1, 'dtype': torch.float32, 'pre_layer_norm': True, 'norm_type': <NormType.LayerNorm: 1>, 'local_rank': -1, 'stochastic_mode': False, 'epsilon': 1e-05, 'mp_size': 1, 'scale_attention': True, 'triangular_masking': True, 'local_attention': False, 'window_size': 1, 'rotary_dim': -1, 'rotate_half': False, 'rotate_every_two': True, 'return_tuple': True, 'mlp_after_attn': True, 'mlp_act_func_type': <ActivationFuncType.GELU: 1>, 'specialized_mode': False, 'training_mp_size': 1, 'bigscience_bloom': False, 'max_out_tokens': 1024, 'min_out_tokens': 1, 'scale_attn_by_inverse_layer_idx': False, 'enable_qkv_quantization': False, 'use_mup': False, 'return_single_tuple': False, 'set_empty_params': False, 'transposed_mode': False, 'use_triton': False, 'triton_autotune': False}
Using /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.023633956909179688 seconds
Loaded autoregressive model
Loaded diffusion model
Loading vocoder model: bigvgan_24khz_100band
Loading vocoder model: bigvgan_24khz_100band.pth
Removing weight norm...
Loaded vocoder model
Loaded TTS, ready for generation.
[1/1] Generating line: [I am really happy,] When use DeepSpeed is True, the methods that load models will use DeepSpeed for loading, and when it's False, the models will be loaded without DeepSpeed.
Loading voice: gun with model d1f79232
Loading voice: gun
Reading from latent: ./voices/gun//cond_latents_d1f79232.pth
------------------------------------------------------
Free memory : 17.803711 (GigaBytes)
Total memory: 23.999390 (GigaBytes)
Requested memory: 5.375000 (GigaBytes)
Setting maximum total tokens (input + output) to 1024
WorkSpace: 0x1b4d800000
------------------------------------------------------
Generating line took 21.546339511871338 seconds
/home/ubuntu/miniconda3/envs/tortoise/lib/python3.10/site-packages/torchaudio/functional/functional.py:1458: UserWarning: "kaiser_window" resampling method name is being deprecated and replaced by "sinc_interp_kaiser" in the next release. The default behavior remains unchanged.
  warnings.warn(
Generation took 22.051681518554688 seconds, saved to './results//gun//gun_00039.wav'

If i run in cli with same setting it always take same ~40s with deepspeed and without deepspeed
Here is output i run with cli

python src/cli.py --use-deepspeed True --prune-nonfinal-outputs True \
> --autoregressive-model ./models/tortoise/autoregressive.pth \
> --diffusion-model ./models/tortoise/diffusion_decoder.pth \
> --vocoder-model bigvgan_24khz_100band \
> --tokenizer-json ./modules/tortoise-tts/tortoise/data/tokenizer.json \
> --voice gun --preset ultra_fast --candidates 1 \
> --text "When use DeepSpeed is True, the methods that load models will use DeepSpeed for loading, and when it's False, the models will be loaded without DeepSpeed."  \
> --num_autoregressive_samples 16 --diffusion_iterations 30 --emotion "Happy" --experimentals "Conditioning-Free" \
> --voice_latents_chunks 4 --candidates 1 --diffusion_sampler "DDIM" --output-sample-rate 44100 \
> --voice_latents_original_ar True --voice_latents_original_diffusion True --no-embed-output-metadata True
Whisper detected
[2023-09-05 10:29:38,770] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
VALL-E detected
Bark detected
Loading TorToiSe... (AR: ./models/tortoise/autoregressive.pth, diffusion: ./models/tortoise/diffusion_decoder.pth, vocoder: bigvgan_24khz_100band)
Hardware acceleration found: cuda
Loading tokenizer JSON: ./modules/tortoise-tts/tortoise/data/tokenizer.json
Loaded tokenizer
Loading autoregressive model: ./models/tortoise/autoregressive.pth
[2023-09-05 10:30:52,417] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.10.2, git-hash=unknown, git-branch=unknown
[2023-09-05 10:30:52,418] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-09-05 10:30:52,418] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
WARNING! Setting BLOOMLayerPolicy._orig_layer_class to None due to Exception: module 'transformers.models' has no attribute 'bloom'
Using /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118/transformer_inference/build.ninja...
Building extension module transformer_inference...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.5363190174102783 seconds
[2023-09-05 10:30:53,854] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed-Inference config: {'layer_id': 0, 'hidden_size': 1024, 'intermediate_size': 4096, 'heads': 16, 'num_hidden_layers': -1, 'dtype': torch.float32, 'pre_layer_norm': True, 'norm_type': <NormType.LayerNorm: 1>, 'local_rank': -1, 'stochastic_mode': False, 'epsilon': 1e-05, 'mp_size': 1, 'scale_attention': True, 'triangular_masking': True, 'local_attention': False, 'window_size': 1, 'rotary_dim': -1, 'rotate_half': False, 'rotate_every_two': True, 'return_tuple': True, 'mlp_after_attn': True, 'mlp_act_func_type': <ActivationFuncType.GELU: 1>, 'specialized_mode': False, 'training_mp_size': 1, 'bigscience_bloom': False, 'max_out_tokens': 1024, 'min_out_tokens': 1, 'scale_attn_by_inverse_layer_idx': False, 'enable_qkv_quantization': False, 'use_mup': False, 'return_single_tuple': False, 'set_empty_params': False, 'transposed_mode': False, 'use_triton': False, 'triton_autotune': False}
Using /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.025423049926757812 seconds
Loaded autoregressive model
Loaded diffusion model
Loading vocoder model: bigvgan_24khz_100band
Loading vocoder model: bigvgan_24khz_100band.pth
Removing weight norm...
Loaded vocoder model
Loaded TTS, ready for generation.
[1/1] Generating line: [I am really happy,] When use DeepSpeed is True, the methods that load models will use DeepSpeed for loading, and when it's False, the models will be loaded without DeepSpeed.
Loading voice: gun with model d1f79232
Reading from latent: ./voices/gun//cond_latents_d1f79232.pth
Generating autoregressive samples:   0%|                                                          | 0/1 [00:00<?, ?it/s]------------------------------------------------------
Free memory : 17.803711 (GigaBytes)
Total memory: 23.999390 (GigaBytes)
Requested memory: 5.375000 (GigaBytes)
Setting maximum total tokens (input + output) to 1024
WorkSpace: 0x1b4d800000
------------------------------------------------------
Generating autoregressive samples: 100%|██████████████████████████████████████████████████| 1/1 [00:08<00:00,  8.86s/it]Computing best candidates using CLVP: 100%|███████████████████████████████████████████████| 1/1 [00:20<00:00, 20.38s/it]Generating line took 40.81342697143555 seconds
/home/ubuntu/miniconda3/envs/tortoise/lib/python3.10/site-packages/torchaudio/functional/functional.py:1458: UserWarning: "kaiser_window" resampling method name is being deprecated and replaced by "sinc_interp_kaiser" in the next release. The default behavior remains unchanged.
  warnings.warn(
Embedding metadata...: 100%|██████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 27.25it/s]Generation took 41.372543811798096 seconds, saved to './results//gun//gun_00040.wav'

(tortoise) ubuntu@HP:/mnt/e/ai-voice-cloning$ python src/cli.py --use-deepspeed False --prune-nonfinal-outputs True --autoregressive-model ./models/tortoise/autoregressive.pth --diffusion-model ./models/tortoise/diffusion_decoder.pth --vocoder-model bigvgan_24khz_100band --tokenizer-json ./modules/tortoise-tts/tortoise/data/tokenizer.json --voice gun --preset ultra_fast --candidates 1 --text "When use DeepSpeed is True, the methods that load models will use DeepSpeed for loading, and when it's False, the models will be loaded without DeepSpeed."  --num_autoregressive_samples 16 --diffusion_iterations 30 --emotion "Happy" --experimentals "Conditioning-Free" --voice_latents_chunks 4 --candidates 1 --diffusion_sampler "DDIM" --output-sample-rate 44100 --voice_latents_original_ar True --voice_latents_original_diffusion True --no-embed-output-metadata True
Whisper detected
[2023-09-05 10:35:36,272] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
VALL-E detected
Bark detected
Loading TorToiSe... (AR: ./models/tortoise/autoregressive.pth, diffusion: ./models/tortoise/diffusion_decoder.pth, vocoder: bigvgan_24khz_100band)
Hardware acceleration found: cuda
Loading tokenizer JSON: ./modules/tortoise-tts/tortoise/data/tokenizer.json
Loaded tokenizer
Loading autoregressive model: ./models/tortoise/autoregressive.pth
[2023-09-05 10:36:49,144] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.10.2, git-hash=unknown, git-branch=unknown
[2023-09-05 10:36:49,145] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-09-05 10:36:49,146] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
WARNING! Setting BLOOMLayerPolicy._orig_layer_class to None due to Exception: module 'transformers.models' has no attribute 'bloom'
Using /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118/transformer_inference/build.ninja...
Building extension module transformer_inference...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.586035966873169 seconds
[2023-09-05 10:36:50,648] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed-Inference config: {'layer_id': 0, 'hidden_size': 1024, 'intermediate_size': 4096, 'heads': 16, 'num_hidden_layers': -1, 'dtype': torch.float32, 'pre_layer_norm': True, 'norm_type': <NormType.LayerNorm: 1>, 'local_rank': -1, 'stochastic_mode': False, 'epsilon': 1e-05, 'mp_size': 1, 'scale_attention': True, 'triangular_masking': True, 'local_attention': False, 'window_size': 1, 'rotary_dim': -1, 'rotate_half': False, 'rotate_every_two': True, 'return_tuple': True, 'mlp_after_attn': True, 'mlp_act_func_type': <ActivationFuncType.GELU: 1>, 'specialized_mode': False, 'training_mp_size': 1, 'bigscience_bloom': False, 'max_out_tokens': 1024, 'min_out_tokens': 1, 'scale_attn_by_inverse_layer_idx': False, 'enable_qkv_quantization': False, 'use_mup': False, 'return_single_tuple': False, 'set_empty_params': False, 'transposed_mode': False, 'use_triton': False, 'triton_autotune': False}
Using /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.028942584991455078 seconds
Loaded autoregressive model
Loaded diffusion model
Loading vocoder model: bigvgan_24khz_100band
Loading vocoder model: bigvgan_24khz_100band.pth
Removing weight norm...
Loaded vocoder model
Loaded TTS, ready for generation.
[1/1] Generating line: [I am really happy,] When use DeepSpeed is True, the methods that load models will use DeepSpeed for loading, and when it's False, the models will be loaded without DeepSpeed.
Loading voice: gun with model d1f79232
Reading from latent: ./voices/gun//cond_latents_d1f79232.pth
Generating autoregressive samples:   0%|                                                          | 0/1 [00:00<?, ?it/s]------------------------------------------------------
Free memory : 17.803711 (GigaBytes)
Total memory: 23.999390 (GigaBytes)
Requested memory: 5.375000 (GigaBytes)
Setting maximum total tokens (input + output) to 1024
WorkSpace: 0x1b4d800000
------------------------------------------------------
Generating autoregressive samples: 100%|██████████████████████████████████████████████████| 1/1 [00:07<00:00,  7.92s/it]Computing best candidates using CLVP: 100%|███████████████████████████████████████████████| 1/1 [00:20<00:00, 20.09s/it]Generating line took 39.286839962005615 seconds
/home/ubuntu/miniconda3/envs/tortoise/lib/python3.10/site-packages/torchaudio/functional/functional.py:1458: UserWarning: "kaiser_window" resampling method name is being deprecated and replaced by "sinc_interp_kaiser" in the next release. The default behavior remains unchanged.
  warnings.warn(
Embedding metadata...: 100%|██████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 22.81it/s]Generation took 39.848819732666016 seconds, saved to './results//gun//gun_00041.wav'
Can you take a look inside src/cli.py , webui.py and utils.py. I'm trying to make cli work with deepspeed but something went wrong, I did some test with webui and cli. If i run in webui with same setting, it took ~20s to inference with deepspeed and ~40s without deepspeed. Here is output i run with webui ``` Whisper detected [2023-09-05 10:21:56,713] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) VALL-E detected Bark detected Running on local URL: http://127.0.0.1:7860 To create a public link, set `share=True` in `launch()`. Loading TorToiSe... (AR: /mnt/e/ai-voice-cloning/models/tortoise/autoregressive.pth, diffusion: ./models/tortoise/diffusion_decoder.pth, vocoder: bigvgan_24khz_100band) Hardware acceleration found: cuda Loading tokenizer JSON: ./modules/tortoise-tts/tortoise/data/tokenizer.json Loaded tokenizer Loading autoregressive model: /mnt/e/ai-voice-cloning/models/tortoise/autoregressive.pth [2023-09-05 10:23:13,696] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.10.2, git-hash=unknown, git-branch=unknown [2023-09-05 10:23:13,698] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead [2023-09-05 10:23:13,698] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1 WARNING! Setting BLOOMLayerPolicy._orig_layer_class to None due to Exception: module 'transformers.models' has no attribute 'bloom' Using /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118/transformer_inference/build.ninja... Building extension module transformer_inference... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module transformer_inference... Time to load transformer_inference op: 0.5863604545593262 seconds [2023-09-05 10:23:15,232] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed-Inference config: {'layer_id': 0, 'hidden_size': 1024, 'intermediate_size': 4096, 'heads': 16, 'num_hidden_layers': -1, 'dtype': torch.float32, 'pre_layer_norm': True, 'norm_type': <NormType.LayerNorm: 1>, 'local_rank': -1, 'stochastic_mode': False, 'epsilon': 1e-05, 'mp_size': 1, 'scale_attention': True, 'triangular_masking': True, 'local_attention': False, 'window_size': 1, 'rotary_dim': -1, 'rotate_half': False, 'rotate_every_two': True, 'return_tuple': True, 'mlp_after_attn': True, 'mlp_act_func_type': <ActivationFuncType.GELU: 1>, 'specialized_mode': False, 'training_mp_size': 1, 'bigscience_bloom': False, 'max_out_tokens': 1024, 'min_out_tokens': 1, 'scale_attn_by_inverse_layer_idx': False, 'enable_qkv_quantization': False, 'use_mup': False, 'return_single_tuple': False, 'set_empty_params': False, 'transposed_mode': False, 'use_triton': False, 'triton_autotune': False} Using /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118 as PyTorch extensions root... No modifications detected for re-loaded extension module transformer_inference, skipping build step... Loading extension module transformer_inference... Time to load transformer_inference op: 0.023633956909179688 seconds Loaded autoregressive model Loaded diffusion model Loading vocoder model: bigvgan_24khz_100band Loading vocoder model: bigvgan_24khz_100band.pth Removing weight norm... Loaded vocoder model Loaded TTS, ready for generation. [1/1] Generating line: [I am really happy,] When use DeepSpeed is True, the methods that load models will use DeepSpeed for loading, and when it's False, the models will be loaded without DeepSpeed. Loading voice: gun with model d1f79232 Loading voice: gun Reading from latent: ./voices/gun//cond_latents_d1f79232.pth ------------------------------------------------------ Free memory : 17.803711 (GigaBytes) Total memory: 23.999390 (GigaBytes) Requested memory: 5.375000 (GigaBytes) Setting maximum total tokens (input + output) to 1024 WorkSpace: 0x1b4d800000 ------------------------------------------------------ Generating line took 21.546339511871338 seconds /home/ubuntu/miniconda3/envs/tortoise/lib/python3.10/site-packages/torchaudio/functional/functional.py:1458: UserWarning: "kaiser_window" resampling method name is being deprecated and replaced by "sinc_interp_kaiser" in the next release. The default behavior remains unchanged. warnings.warn( Generation took 22.051681518554688 seconds, saved to './results//gun//gun_00039.wav' ``` If i run in cli with same setting it always take same ~40s with deepspeed and without deepspeed Here is output i run with cli ``` python src/cli.py --use-deepspeed True --prune-nonfinal-outputs True \ > --autoregressive-model ./models/tortoise/autoregressive.pth \ > --diffusion-model ./models/tortoise/diffusion_decoder.pth \ > --vocoder-model bigvgan_24khz_100band \ > --tokenizer-json ./modules/tortoise-tts/tortoise/data/tokenizer.json \ > --voice gun --preset ultra_fast --candidates 1 \ > --text "When use DeepSpeed is True, the methods that load models will use DeepSpeed for loading, and when it's False, the models will be loaded without DeepSpeed." \ > --num_autoregressive_samples 16 --diffusion_iterations 30 --emotion "Happy" --experimentals "Conditioning-Free" \ > --voice_latents_chunks 4 --candidates 1 --diffusion_sampler "DDIM" --output-sample-rate 44100 \ > --voice_latents_original_ar True --voice_latents_original_diffusion True --no-embed-output-metadata True Whisper detected [2023-09-05 10:29:38,770] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) VALL-E detected Bark detected Loading TorToiSe... (AR: ./models/tortoise/autoregressive.pth, diffusion: ./models/tortoise/diffusion_decoder.pth, vocoder: bigvgan_24khz_100band) Hardware acceleration found: cuda Loading tokenizer JSON: ./modules/tortoise-tts/tortoise/data/tokenizer.json Loaded tokenizer Loading autoregressive model: ./models/tortoise/autoregressive.pth [2023-09-05 10:30:52,417] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.10.2, git-hash=unknown, git-branch=unknown [2023-09-05 10:30:52,418] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead [2023-09-05 10:30:52,418] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1 WARNING! Setting BLOOMLayerPolicy._orig_layer_class to None due to Exception: module 'transformers.models' has no attribute 'bloom' Using /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118/transformer_inference/build.ninja... Building extension module transformer_inference... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module transformer_inference... Time to load transformer_inference op: 0.5363190174102783 seconds [2023-09-05 10:30:53,854] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed-Inference config: {'layer_id': 0, 'hidden_size': 1024, 'intermediate_size': 4096, 'heads': 16, 'num_hidden_layers': -1, 'dtype': torch.float32, 'pre_layer_norm': True, 'norm_type': <NormType.LayerNorm: 1>, 'local_rank': -1, 'stochastic_mode': False, 'epsilon': 1e-05, 'mp_size': 1, 'scale_attention': True, 'triangular_masking': True, 'local_attention': False, 'window_size': 1, 'rotary_dim': -1, 'rotate_half': False, 'rotate_every_two': True, 'return_tuple': True, 'mlp_after_attn': True, 'mlp_act_func_type': <ActivationFuncType.GELU: 1>, 'specialized_mode': False, 'training_mp_size': 1, 'bigscience_bloom': False, 'max_out_tokens': 1024, 'min_out_tokens': 1, 'scale_attn_by_inverse_layer_idx': False, 'enable_qkv_quantization': False, 'use_mup': False, 'return_single_tuple': False, 'set_empty_params': False, 'transposed_mode': False, 'use_triton': False, 'triton_autotune': False} Using /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118 as PyTorch extensions root... No modifications detected for re-loaded extension module transformer_inference, skipping build step... Loading extension module transformer_inference... Time to load transformer_inference op: 0.025423049926757812 seconds Loaded autoregressive model Loaded diffusion model Loading vocoder model: bigvgan_24khz_100band Loading vocoder model: bigvgan_24khz_100band.pth Removing weight norm... Loaded vocoder model Loaded TTS, ready for generation. [1/1] Generating line: [I am really happy,] When use DeepSpeed is True, the methods that load models will use DeepSpeed for loading, and when it's False, the models will be loaded without DeepSpeed. Loading voice: gun with model d1f79232 Reading from latent: ./voices/gun//cond_latents_d1f79232.pth Generating autoregressive samples: 0%| | 0/1 [00:00<?, ?it/s]------------------------------------------------------ Free memory : 17.803711 (GigaBytes) Total memory: 23.999390 (GigaBytes) Requested memory: 5.375000 (GigaBytes) Setting maximum total tokens (input + output) to 1024 WorkSpace: 0x1b4d800000 ------------------------------------------------------ Generating autoregressive samples: 100%|██████████████████████████████████████████████████| 1/1 [00:08<00:00, 8.86s/it]Computing best candidates using CLVP: 100%|███████████████████████████████████████████████| 1/1 [00:20<00:00, 20.38s/it]Generating line took 40.81342697143555 seconds /home/ubuntu/miniconda3/envs/tortoise/lib/python3.10/site-packages/torchaudio/functional/functional.py:1458: UserWarning: "kaiser_window" resampling method name is being deprecated and replaced by "sinc_interp_kaiser" in the next release. The default behavior remains unchanged. warnings.warn( Embedding metadata...: 100%|██████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 27.25it/s]Generation took 41.372543811798096 seconds, saved to './results//gun//gun_00040.wav' (tortoise) ubuntu@HP:/mnt/e/ai-voice-cloning$ python src/cli.py --use-deepspeed False --prune-nonfinal-outputs True --autoregressive-model ./models/tortoise/autoregressive.pth --diffusion-model ./models/tortoise/diffusion_decoder.pth --vocoder-model bigvgan_24khz_100band --tokenizer-json ./modules/tortoise-tts/tortoise/data/tokenizer.json --voice gun --preset ultra_fast --candidates 1 --text "When use DeepSpeed is True, the methods that load models will use DeepSpeed for loading, and when it's False, the models will be loaded without DeepSpeed." --num_autoregressive_samples 16 --diffusion_iterations 30 --emotion "Happy" --experimentals "Conditioning-Free" --voice_latents_chunks 4 --candidates 1 --diffusion_sampler "DDIM" --output-sample-rate 44100 --voice_latents_original_ar True --voice_latents_original_diffusion True --no-embed-output-metadata True Whisper detected [2023-09-05 10:35:36,272] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) VALL-E detected Bark detected Loading TorToiSe... (AR: ./models/tortoise/autoregressive.pth, diffusion: ./models/tortoise/diffusion_decoder.pth, vocoder: bigvgan_24khz_100band) Hardware acceleration found: cuda Loading tokenizer JSON: ./modules/tortoise-tts/tortoise/data/tokenizer.json Loaded tokenizer Loading autoregressive model: ./models/tortoise/autoregressive.pth [2023-09-05 10:36:49,144] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.10.2, git-hash=unknown, git-branch=unknown [2023-09-05 10:36:49,145] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead [2023-09-05 10:36:49,146] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1 WARNING! Setting BLOOMLayerPolicy._orig_layer_class to None due to Exception: module 'transformers.models' has no attribute 'bloom' Using /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118/transformer_inference/build.ninja... Building extension module transformer_inference... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module transformer_inference... Time to load transformer_inference op: 0.586035966873169 seconds [2023-09-05 10:36:50,648] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed-Inference config: {'layer_id': 0, 'hidden_size': 1024, 'intermediate_size': 4096, 'heads': 16, 'num_hidden_layers': -1, 'dtype': torch.float32, 'pre_layer_norm': True, 'norm_type': <NormType.LayerNorm: 1>, 'local_rank': -1, 'stochastic_mode': False, 'epsilon': 1e-05, 'mp_size': 1, 'scale_attention': True, 'triangular_masking': True, 'local_attention': False, 'window_size': 1, 'rotary_dim': -1, 'rotate_half': False, 'rotate_every_two': True, 'return_tuple': True, 'mlp_after_attn': True, 'mlp_act_func_type': <ActivationFuncType.GELU: 1>, 'specialized_mode': False, 'training_mp_size': 1, 'bigscience_bloom': False, 'max_out_tokens': 1024, 'min_out_tokens': 1, 'scale_attn_by_inverse_layer_idx': False, 'enable_qkv_quantization': False, 'use_mup': False, 'return_single_tuple': False, 'set_empty_params': False, 'transposed_mode': False, 'use_triton': False, 'triton_autotune': False} Using /mnt/e/ai-voice-cloning/models/torch_extensions/py310_cu118 as PyTorch extensions root... No modifications detected for re-loaded extension module transformer_inference, skipping build step... Loading extension module transformer_inference... Time to load transformer_inference op: 0.028942584991455078 seconds Loaded autoregressive model Loaded diffusion model Loading vocoder model: bigvgan_24khz_100band Loading vocoder model: bigvgan_24khz_100band.pth Removing weight norm... Loaded vocoder model Loaded TTS, ready for generation. [1/1] Generating line: [I am really happy,] When use DeepSpeed is True, the methods that load models will use DeepSpeed for loading, and when it's False, the models will be loaded without DeepSpeed. Loading voice: gun with model d1f79232 Reading from latent: ./voices/gun//cond_latents_d1f79232.pth Generating autoregressive samples: 0%| | 0/1 [00:00<?, ?it/s]------------------------------------------------------ Free memory : 17.803711 (GigaBytes) Total memory: 23.999390 (GigaBytes) Requested memory: 5.375000 (GigaBytes) Setting maximum total tokens (input + output) to 1024 WorkSpace: 0x1b4d800000 ------------------------------------------------------ Generating autoregressive samples: 100%|██████████████████████████████████████████████████| 1/1 [00:07<00:00, 7.92s/it]Computing best candidates using CLVP: 100%|███████████████████████████████████████████████| 1/1 [00:20<00:00, 20.09s/it]Generating line took 39.286839962005615 seconds /home/ubuntu/miniconda3/envs/tortoise/lib/python3.10/site-packages/torchaudio/functional/functional.py:1458: UserWarning: "kaiser_window" resampling method name is being deprecated and replaced by "sinc_interp_kaiser" in the next release. The default behavior remains unchanged. warnings.warn( Embedding metadata...: 100%|██████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 22.81it/s]Generation took 39.848819732666016 seconds, saved to './results//gun//gun_00041.wav' ```

Pull request successfully merged and closed

The branch ken11o2/ai-voice-cloning:master can now be deleted.
Sign in to join this conversation.
Reviewers
Request review
No reviewers
Labels
Apply labels
Clear labels   
bug

Something is not working   
duplicate

This issue or pull request already exists   
enhancement

New feature   
help wanted

Need some help   
insufficient info

Need more information   
invalid

Something is wrong   
news

To make up for a lack of a way to provide any news-like   
not a bug

For non-bug things   
question

More information is needed   
wontfix

This won't be fixed
No Label
bug
duplicate
enhancement
help wanted
insufficient info
invalid
news
not a bug
question
wontfix
Milestone
Set milestone
Clear milestone
No items
No Milestone
Projects
Set Project
Clear projects
No project
Assignees
Assign users
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: mrq/ai-voice-cloning#369
There is no content yet.
Delete Branch "ken11o2/ai-voice-cloning:master"

Deleting a branch is permanent. It CANNOT be undone. Continue?