AttributeError: 'AsyncRequest' object has no attribute '_json_response_data' #182

I'm getting the error AttributeError: 'AsyncRequest' object has no attribute '_json_response_data' after some time in to the training

[Training] [2023-03-28T08:10:39.974381] 23-03-28 08:10:39.974 - INFO: Training Metrics: {"loss_text_ce": 3.0363988876342773, "loss_mel_ce": 1.6027159690856934, "loss_gpt_total": 1.6330801248550415, "lr": 7.8125e-08, "it": 865, "step": 1, "steps": 4, "epoch": 216, "iteration_rate": 4.793625831604004}
Task exception was never retrieved
future: <Task finished name='iujkw6g09l8_53' coro=<Queue.process_events() done, defined at C:\Users\Dominik\ai-voice-cloning\venv\lib\site-packages\gradio\queueing.py:343> exception=AttributeError("'AsyncRequest' object has no attribute '_json_response_data'")>
Traceback (most recent call last):
  File "C:\Users\Dominik\ai-voice-cloning\venv\lib\site-packages\gradio\queueing.py", line 370, in process_events
    while response.json.get("is_generating", False):
  File "C:\Users\Dominik\ai-voice-cloning\venv\lib\site-packages\gradio\utils.py", line 759, in json
    return self._json_response_data
AttributeError: 'AsyncRequest' object has no attribute '_json_response_data'

I'm getting the error `AttributeError: 'AsyncRequest' object has no attribute '_json_response_data'` after some time in to the training ``` [Training] [2023-03-28T08:10:39.974381] 23-03-28 08:10:39.974 - INFO: Training Metrics: {"loss_text_ce": 3.0363988876342773, "loss_mel_ce": 1.6027159690856934, "loss_gpt_total": 1.6330801248550415, "lr": 7.8125e-08, "it": 865, "step": 1, "steps": 4, "epoch": 216, "iteration_rate": 4.793625831604004} Task exception was never retrieved future: <Task finished name='iujkw6g09l8_53' coro=<Queue.process_events() done, defined at C:\Users\Dominik\ai-voice-cloning\venv\lib\site-packages\gradio\queueing.py:343> exception=AttributeError("'AsyncRequest' object has no attribute '_json_response_data'")> Traceback (most recent call last): File "C:\Users\Dominik\ai-voice-cloning\venv\lib\site-packages\gradio\queueing.py", line 370, in process_events while response.json.get("is_generating", False): File "C:\Users\Dominik\ai-voice-cloning\venv\lib\site-packages\gradio\utils.py", line 759, in json return self._json_response_data AttributeError: 'AsyncRequest' object has no attribute '_json_response_data' ```

Unfortunately it's a deep seated Gradio regression, there's not much I can do about it: #116.

I couldn't consistently replicate it, but I would advise if it's too intrusive, just manually invoke the training process (the command used is printed before training starts), and if you want to peep at metrics, you can click View Losses in the training tab; it's pretty much what I've been doing for VALL-E model training.

Unfortunately it's a deep seated Gradio regression, there's not much I can do about it: https://git.ecker.tech/mrq/ai-voice-cloning/issues/116. I couldn't consistently replicate it, but I would advise if it's too intrusive, just manually invoke the training process (the command used is printed before training starts), and if you want to peep at metrics, you can click `View Losses` in the training tab; it's pretty much what I've been doing for VALL-E model training.

Mhm when this error appears the training pauses, how can I resume it. I've refreshed Gradio and clicked on reconnect, which then seems to continue the training but I don't know if it's the right way to do it.

or does the training continue automatically? Because in gradio there's no progress als in the cmd window nothing happens.

Labels Milestones

AttributeError: 'AsyncRequest' object has no attribute '_json_response_data' #182