[Training] Detected call of lr_scheduler.step()
before optimizer.step()
#276
Closed
opened
Loading…
Reference in New Issue
There is no content yet.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. It CANNOT be undone. Continue?
I'm having issues running my own training on a local machine.
Everything else seems to work fine, but after validating training configuration and using it to train my own model, the process starts up until i get this console output:
After which the process hangs and I have to kill it. I'm pretty sure I followed instructions to the letter. I understand it has to do with an older version of torch but I would assume the app already deals with that since torch 2.x is a requirement? Any help would be appreciated. Thanks!
For reference here's my setup:
OS: Win10
GPU: Nvidia GeForce GTX 1070
torch: 2.0.1+cu118
Maybe because of
[Training] [2023-06-21T14:37:27.465151] NOTE: Redirects are currently not supported in Windows or MacOs.
On linux I get that error message and the training runs fine. With 2 gpu it sometimes stops and has to be restarted though. One GPU will keep going and one will be 0 but no OOM.
I actually realized this is just a warning. Even though my setup is slow by training standards, after a while the training does kick in. So it's not a blocker as I first thought.