OverflowError: Python int too large to convert to C int #35

New Issue

sakharam_gatne · 2023-02-25T16:08:01Z

sakharam_gatne commented

2023-02-25 16:08:01 +00:00

I get the following error while trying to start training. My last pull (before the current one) was 7d1220e83e, and it works fine on that commit, so I'm pretty certain the cause of this error is in the commits that follow it.


[Training] [2023-02-25T21:25:39.168729] Using BitsAndBytes ADAMW optimizations
[Training] [2023-02-25T21:25:39.170089] Disabled distributed training.
[Training] [2023-02-25T21:25:39.170730] Loading from ./models/tortoise/dvae.pth
[Training] [2023-02-25T21:25:39.177765]
Traceback (most recent call last):
  File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\gradio\routes.py", line 374, in run_predict
    output = await app.get_blocks().process_api(
  File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\gradio\blocks.py", line 1017, in process_api
    result = await self.call_function(
  File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\gradio\blocks.py", line 849, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\gradio\utils.py", line 453, in async_iteration
    return next(iterator)
  File "C:\Users\Username\Desktop\ai-voice-cloning\src\utils.py", line 612, in run_training
    res = training_state.parse( line=line, verbose=verbose, buffer_size=buffer_size, progress=progress, owner=True )
  File "C:\Users\Username\Desktop\ai-voice-cloning\src\utils.py", line 533, in parse
    self.eta_hhmmss = str(timedelta(seconds=int(self.eta)))
OverflowError: Python int too large to convert to C int

I get the following error while trying to start training. My last pull (before the current one) was 7d1220e83e751cebe644b59018babf49e247c965, and it works fine on that commit, so I'm pretty certain the cause of this error is in the commits that follow it. ``` [Training] [2023-02-25T21:25:39.168729] Using BitsAndBytes ADAMW optimizations [Training] [2023-02-25T21:25:39.170089] Disabled distributed training. [Training] [2023-02-25T21:25:39.170730] Loading from ./models/tortoise/dvae.pth [Training] [2023-02-25T21:25:39.177765] Traceback (most recent call last): File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\gradio\routes.py", line 374, in run_predict output = await app.get_blocks().process_api( File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\gradio\blocks.py", line 1017, in process_api result = await self.call_function( File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\gradio\blocks.py", line 849, in call_function prediction = await anyio.to_thread.run_sync( File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run result = context.run(func, *args) File "C:\Users\Username\Desktop\ai-voice-cloning\venv\lib\site-packages\gradio\utils.py", line 453, in async_iteration return next(iterator) File "C:\Users\Username\Desktop\ai-voice-cloning\src\utils.py", line 612, in run_training res = training_state.parse( line=line, verbose=verbose, buffer_size=buffer_size, progress=progress, owner=True ) File "C:\Users\Username\Desktop\ai-voice-cloning\src\utils.py", line 533, in parse self.eta_hhmmss = str(timedelta(seconds=int(self.eta))) OverflowError: Python int too large to convert to C int ```

mrq commented

2023-02-25 16:13:46 +00:00

I'll slap it under a try/except block.

It's odd that it throws an error, since I have two different training sessions (one locally, one on a paperspace instance) and they haven't had issues.

Speaking of odd, I just tabbed into the local one and it's already wigging out due to some discrepancies:

I'll slap it under a try/except block. It's odd that it throws an error, since I have two different training sessions (one locally, one on a paperspace instance) and they haven't had issues. --- Speaking of odd, I just tabbed into the local one and it's already wigging out due to some discrepancies: ![image](/attachments/fef0fb18-bfec-4619-bf5c-c5ec0842760d)

image.png

6.4 KiB

mrq commented

2023-02-25 16:45:52 +00:00

Should be fixed in commit aafeb9f96a. I threw it and a possible division-by-zero in try/except blocks.

There were some other logic errors, probably due to [some cope about being up early with little sleep] and not actually babysitting the output.

Should be fixed in commit aafeb9f96a3fe39cc3ff74a02c509cd78c892d70. I threw it and a possible division-by-zero in try/except blocks. There were some other logic errors, probably due to `[some cope about being up early with little sleep]` and not actually babysitting the output.

mrq closed this issue

2023-02-26 02:11:42 +00:00

Sign in to join this conversation.