vall-e/data/config.yaml

dataset:
  training: []
  validation: []
  noise: []
  
  speaker_name_getter: "lambda p: f'{p.parts[-3]}_{p.parts[-2]}'"
  
  use_hdf5: True
  use_metadata: True
  hdf5_flag: r
  validate: True

  workers: 2
  cache: True

  phones_range: [4, 256]
  duration_range: [1.0, 16.0]
  min_utterances: 32

  random_utterance: 1.0
  max_prompts: 6
  prompt_duration: 6.0

  sample_type: speaker

  tasks_list: [ "tts" ] # , [ "tts", "tts-c", "ns", "sr", "tse", "cse", "nse", "tts"]

models:
  _prom_levels: 8
  _max_levels: 8

  _models:
  - name: "ar+nar"
    size: "double"
    resp_levels: 8
    prom_levels: 8
    tasks: 8
    arch_type: "retnet"
    training: True
    version: 2


hyperparameters:
  batch_size: 8
  gradient_accumulation_steps: 16
  gradient_clipping: 100
  
  # prodigyopt is nicer, but requires even more VRAM
  #optimizer: Prodigy
  #learning_rate: 1.0 # e-4

  optimizer: AdamW
  learning_rate: 1.0e-4
  torch_optimizer: True
  
  scheduler_type: ""
  #scheduler_type: OneCycle
  #scheduler_params:
  #  cycle_first_step_size: 10_000
  #  cycle_first_stair_count: 10_000

  #  cycle_second_step_size: 15_000
  #  cycle_second_stair_count: 15_000

  #  decay_step_size: 5_000

  #  cycle_min_lr: 2.5e-4 # 1.0e-5
  #  cycle_max_lr: 2.5e-4 # 1.0e-4
  #  decay_lr_rate: 0.0

  #  cycle_min_mom: 0.90
  #  cycle_max_mom: 0.99
  #  decay_mom_rate: 0.0

evaluation:
  batch_size: 16
  frequency: 250
  size: 16
  
  steps: 450
  ar_temperature: 0.95
  nar_temperature: 0.25
  load_disabled_engines: True

trainer:
  iterations: 1_000_000
  
  save_tag: step
  save_on_oom: True
  save_on_quit: True
  save_frequency: 100
  export_on_save: True

  keep_last_checkpoints: 4

  aggressive_optimizations: False
  load_disabled_engines: False

  #load_state_dict: True
  #strict_loading: False
  #load_tag: "9500"
  #load_states: False
  #restart_step_count: True
  
  gc_mode: None # "global_step"

  weight_dtype: bfloat16
  amp: False

  backend: deepspeed
  deepspeed:
    zero_optimization_level: 0
    use_compression_training: True

  activation_checkpointing: True

inference:
  use_vocos: True
  normalize: False

bitsandbytes:
  enabled: False
  injects: False
  linear: False
  embedding: False
Rewrite init 2023-08-02 21:53:35 +00:00			`dataset:`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`training: []`
			`validation: []`
			`noise: []`
Rewrite init 2023-08-02 21:53:35 +00:00
			`speaker_name_getter: "lambda p: f'{p.parts[-3]}_{p.parts[-2]}'"`

			`use_hdf5: True`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`use_metadata: True`
repaired auraloss loss calc during eval/val 2023-08-19 02:19:47 +00:00			`hdf5_flag: r`
Rewrite init 2023-08-02 21:53:35 +00:00			`validate: True`

fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`workers: 2`
Rewrite init 2023-08-02 21:53:35 +00:00			`cache: True`

fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`phones_range: [4, 256]`
repaired auraloss loss calc during eval/val 2023-08-19 02:19:47 +00:00			`duration_range: [1.0, 16.0]`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`min_utterances: 32`
Rewrite init 2023-08-02 21:53:35 +00:00
			`random_utterance: 1.0`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`max_prompts: 6`
			`prompt_duration: 6.0`
Rewrite init 2023-08-02 21:53:35 +00:00
repaired auraloss loss calc during eval/val 2023-08-19 02:19:47 +00:00			`sample_type: speaker`
nightmare of spaghetti that might break compat; mechanism to increase RVQ bins of an existing model without retraining, keeps sampled proms/resps at max RVQ level and trim off excess levels according to what model receives them, some other things I already forgot (I really hope no one else has weights being baked right now) 2023-08-19 20:06:33 +00:00
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`tasks_list: [ "tts" ] # , [ "tts", "tts-c", "ns", "sr", "tse", "cse", "nse", "tts"]`
repaired auraloss loss calc during eval/val 2023-08-19 02:19:47 +00:00
Rewrite init 2023-08-02 21:53:35 +00:00			`models:`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`_prom_levels: 8`
nightmare of spaghetti that might break compat; mechanism to increase RVQ bins of an existing model without retraining, keeps sampled proms/resps at max RVQ level and trim off excess levels according to what model receives them, some other things I already forgot (I really hope no one else has weights being baked right now) 2023-08-19 20:06:33 +00:00			`_max_levels: 8`
Rewrite init 2023-08-02 21:53:35 +00:00
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`_models:`
			`- name: "ar+nar"`
			`size: "double"`
			`resp_levels: 8`
			`prom_levels: 8`
oops 2023-08-19 02:11:19 +00:00			`tasks: 8`
repaired auraloss loss calc during eval/val 2023-08-19 02:19:47 +00:00			`arch_type: "retnet"`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`training: True`
			`version: 2`
Rewrite init 2023-08-02 21:53:35 +00:00
nightmare of spaghetti that might break compat; mechanism to increase RVQ bins of an existing model without retraining, keeps sampled proms/resps at max RVQ level and trim off excess levels according to what model receives them, some other things I already forgot (I really hope no one else has weights being baked right now) 2023-08-19 20:06:33 +00:00
Rewrite init 2023-08-02 21:53:35 +00:00			`hyperparameters:`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`batch_size: 8`
			`gradient_accumulation_steps: 16`
Rewrite init 2023-08-02 21:53:35 +00:00			`gradient_clipping: 100`

fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`# prodigyopt is nicer, but requires even more VRAM`
			`#optimizer: Prodigy`
			`#learning_rate: 1.0 # e-4`

nightmare of spaghetti that might break compat; mechanism to increase RVQ bins of an existing model without retraining, keeps sampled proms/resps at max RVQ level and trim off excess levels according to what model receives them, some other things I already forgot (I really hope no one else has weights being baked right now) 2023-08-19 20:06:33 +00:00			`optimizer: AdamW`
			`learning_rate: 1.0e-4`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`torch_optimizer: True`
Rewrite init 2023-08-02 21:53:35 +00:00
			`scheduler_type: ""`
			`#scheduler_type: OneCycle`
			`#scheduler_params:`
			`# cycle_first_step_size: 10_000`
			`# cycle_first_stair_count: 10_000`

			`# cycle_second_step_size: 15_000`
			`# cycle_second_stair_count: 15_000`

			`# decay_step_size: 5_000`

			`# cycle_min_lr: 2.5e-4 # 1.0e-5`
			`# cycle_max_lr: 2.5e-4 # 1.0e-4`
			`# decay_lr_rate: 0.0`

			`# cycle_min_mom: 0.90`
			`# cycle_max_mom: 0.99`
			`# decay_mom_rate: 0.0`

			`evaluation:`
nightmare of spaghetti that might break compat; mechanism to increase RVQ bins of an existing model without retraining, keeps sampled proms/resps at max RVQ level and trim off excess levels according to what model receives them, some other things I already forgot (I really hope no one else has weights being baked right now) 2023-08-19 20:06:33 +00:00			`batch_size: 16`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`frequency: 250`
nightmare of spaghetti that might break compat; mechanism to increase RVQ bins of an existing model without retraining, keeps sampled proms/resps at max RVQ level and trim off excess levels according to what model receives them, some other things I already forgot (I really hope no one else has weights being baked right now) 2023-08-19 20:06:33 +00:00			`size: 16`
Rewrite init 2023-08-02 21:53:35 +00:00
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`steps: 450`
nightmare of spaghetti that might break compat; mechanism to increase RVQ bins of an existing model without retraining, keeps sampled proms/resps at max RVQ level and trim off excess levels according to what model receives them, some other things I already forgot (I really hope no one else has weights being baked right now) 2023-08-19 20:06:33 +00:00			`ar_temperature: 0.95`
			`nar_temperature: 0.25`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`load_disabled_engines: True`
Rewrite init 2023-08-02 21:53:35 +00:00
			`trainer:`
repaired auraloss loss calc during eval/val 2023-08-19 02:19:47 +00:00			`iterations: 1_000_000`
Rewrite init 2023-08-02 21:53:35 +00:00
			`save_tag: step`
			`save_on_oom: True`
			`save_on_quit: True`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`save_frequency: 100`
			`export_on_save: True`
repaired auraloss loss calc during eval/val 2023-08-19 02:19:47 +00:00
			`keep_last_checkpoints: 4`
Rewrite init 2023-08-02 21:53:35 +00:00
			`aggressive_optimizations: False`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`load_disabled_engines: False`
Rewrite init 2023-08-02 21:53:35 +00:00
			`#load_state_dict: True`
			`#strict_loading: False`
repaired auraloss loss calc during eval/val 2023-08-19 02:19:47 +00:00			`#load_tag: "9500"`
			`#load_states: False`
Rewrite init 2023-08-02 21:53:35 +00:00			`#restart_step_count: True`

			`gc_mode: None # "global_step"`

repaired auraloss loss calc during eval/val 2023-08-19 02:19:47 +00:00			`weight_dtype: bfloat16`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`amp: False`
Rewrite init 2023-08-02 21:53:35 +00:00
big cleanup 2023-08-04 01:26:36 +00:00			`backend: deepspeed`
			`deepspeed:`
repaired auraloss loss calc during eval/val 2023-08-19 02:19:47 +00:00			`zero_optimization_level: 0`
big cleanup 2023-08-04 01:26:36 +00:00			`use_compression_training: True`
adjustments 2023-08-02 22:01:49 +00:00
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`activation_checkpointing: True`

big cleanup 2023-08-04 01:26:36 +00:00			`inference:`
			`use_vocos: True`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`normalize: False`
big cleanup 2023-08-04 01:26:36 +00:00
			`bitsandbytes:`
nightmare of spaghetti that might break compat; mechanism to increase RVQ bins of an existing model without retraining, keeps sampled proms/resps at max RVQ level and trim off excess levels according to what model receives them, some other things I already forgot (I really hope no one else has weights being baked right now) 2023-08-19 20:06:33 +00:00			`enabled: False`
fixed training stats not loading from exported weights, a bit of a readme cleanup, updated example training yaml 2023-09-24 00:59:00 +00:00			`injects: False`
			`linear: False`
			`embedding: False`