mrq/ai-voice-cloning
31
74
Fork 43
Code Issues 223 Pull Requests 5 Packages Projects Releases Wiki Activity

today I learned adamw_zero actually negates ANY LR schemes

Browse Source
This commit is contained in:
mrq 2023-03-09 19:42:31 +00:00
parent cb273b8428
commit c3b43d2429
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
src/utils.py
View File

@ -1394,7 +1394,8 @@ def save_training_settings( **kwargs ):
if settings['gpus'] > get_device_count():
settings['gpus'] = get_device_count()
settings['optimizer'] = 'adamw' if settings['gpus'] == 1 else 'adamw_zero'
# what an utter mistake this was
# settings['optimizer'] = 'adamw' if settings['gpus'] == 1 else 'adamw_zero'
if 'learning_rate_scheme' not in settings or settings['learning_rate_scheme'] not in LEARNING_RATE_SCHEMES:
settings['learning_rate_scheme'] = "Multistep"