Unable to complete training. #284

New Issue

Atoli · 2023-06-26T21:43:22Z

Atoli commented

2023-06-26 21:43:22 +00:00

I have been trying to train/finetune a model for a few hours, but everytime it gets stuck at step 13 and i get the following error:

[Training] RuntimeError: [enforce fail at ..\caffe2\serialize\inline_container.cc:337] . unexpected pos 58478784 vs 58478736

It doesn't seem to be an OOM error, what could cause the training to get stuck always at epoch 13?

I have been trying to train/finetune a model for a few hours, but everytime it gets stuck at step 13 and i get the following error: `[Training] RuntimeError: [enforce fail at ..\caffe2\serialize\inline_container.cc:337] . unexpected pos 58478784 vs 58478736` It doesn't seem to be an OOM error, what could cause the training to get stuck always at epoch 13?

tr.PNG

26 KiB

tr1.PNG

70 KiB

psammites commented

2023-06-27 01:19:46 +00:00

That's a pytorch error, if you google it you will find some suspected causes and possible workarounds.

Atoli commented

2023-06-27 14:07:00 +00:00

That's a pytorch error, if you google it you will find some suspected causes and possible workarounds.

Yeah, it seem to be a pytorch issue.

For anyone wondering, the issue seemed to be that pytorch needs more disk space.

After moving my folder to a hdd with lots of free space i could train normally.

> That's a pytorch error, if you google it you will find some suspected causes and possible workarounds. Yeah, it seem to be a pytorch issue. For anyone wondering, the issue seemed to be that pytorch needs more disk space. After moving my folder to a hdd with lots of free space i could train normally.

Atoli closed this issue

2023-06-27 14:07:06 +00:00

Sign in to join this conversation.