Resuming Training keyError: 'iter' #429

I am trying to resume a training and getting the following:

[Training] [2023-10-25T12:41:49.117575] Disabled distributed training.                                                                                                                [Training] [2023-10-25T12:41:49.119210] Traceback (most recent call last):                                                                                                            [Training] [2023-10-25T12:41:49.120834]   File "/home/toast/AI/ai-voice-cloning/./src/train.py", line 64, in <module>                                                                 [Training] [2023-10-25T12:41:49.122467]     train(config_path, args.launcher)                                                                                                         [Training] [2023-10-25T12:41:49.124102]   File "/home/toast/AI/ai-voice-cloning/./src/train.py", line 30, in train
[Training] [2023-10-25T12:41:49.125734]     trainer.init(config_path, opt, launcher, '')                                                                                              [Training] [2023-10-25T12:41:49.127349]   File "/home/toast/AI/ai-voice-cloning/modules/dlas/dlas/train.py", line 107, in init                                                        [Training] [2023-10-25T12:41:49.128991]     option.check_resume(opt, resume_state['iter'])
[Training] [2023-10-25T12:41:49.130837] KeyError: 'iter'

in my generate config tab i have a link to ./training/paisley/finetune/models/10514_gpt.pth (which does exist, i did check) in the Resume State Path

Any ideas?

I am trying to resume a training and getting the following: ```trace [Training] [2023-10-25T12:41:49.117575] Disabled distributed training. [Training] [2023-10-25T12:41:49.119210] Traceback (most recent call last): [Training] [2023-10-25T12:41:49.120834] File "/home/toast/AI/ai-voice-cloning/./src/train.py", line 64, in <module> [Training] [2023-10-25T12:41:49.122467] train(config_path, args.launcher) [Training] [2023-10-25T12:41:49.124102] File "/home/toast/AI/ai-voice-cloning/./src/train.py", line 30, in train [Training] [2023-10-25T12:41:49.125734] trainer.init(config_path, opt, launcher, '') [Training] [2023-10-25T12:41:49.127349] File "/home/toast/AI/ai-voice-cloning/modules/dlas/dlas/train.py", line 107, in init [Training] [2023-10-25T12:41:49.128991] option.check_resume(opt, resume_state['iter']) [Training] [2023-10-25T12:41:49.130837] KeyError: 'iter' ``` in my generate config tab i have a link to `./training/paisley/finetune/models/10514_gpt.pth` (which does exist, i did check) in the Resume State Path Any ideas?

Screenshot 2023-10-25 at 12.45.12.png

113 KiB

Yup. I spotted it (eventually)

my resume state path was pointing to the model, not the state.

Yup. I spotted it (eventually) my resume state path was pointing to the model, not the state.

Labels Milestones

Resuming Training keyError: 'iter' #429