cli.py loads model everytime it's ran #263

New Issue

FrioGlakka · 2023-06-12T17:07:16Z

FrioGlakka commented

2023-06-12 17:07:16 +00:00

I guess this is intented behaviour but I'm not sure maybe I'm just doing something wrong or forgetting an obvious way to work around this limitation.

I'm using the cli script located at ./src/cli.py to generate an audio file.

It's my goal to build an API around this repo that I can access from my other non-python related environments. I've succesfully done this using the (I think automatically generated) gradio client API and it wasn't pretty (had to edit a lot of stuff to get it to be somewhat stable).

Now that the cli script was added I hoped I would be able to leave my hacky gradio client implementation behind.

If I run the cli script, it all works fine but it's doing the whole "Loading TorToiSe..." step before generating the audio file. And then it unloads it all again.

Like I said, I guess that's intented behaviour.

I'm not a python developer so excuse me for assuming/hoping:
Is there a way to get this to work so it keeps the model loaded in between generation calls?

I can manage my way to implement something but only if someone here can give me some direction as to what has to happen to get that to work.

Obviously I still have the gradio client implementation right now, but I much prefer something that isn't as destructive or hacky, preferably something that still works after updating the repository.

I guess this is intented behaviour but I'm not sure maybe I'm just doing something wrong or forgetting an obvious way to work around this limitation. I'm using the cli script located at ./src/cli.py to generate an audio file. It's my goal to build an API around this repo that I can access from my other non-python related environments. I've succesfully done this using the (I think automatically generated) gradio client API and it wasn't pretty (had to edit a lot of stuff to get it to be somewhat stable). Now that the cli script was added I hoped I would be able to leave my hacky gradio client implementation behind. If I run the cli script, it all works fine but it's doing the whole "Loading TorToiSe..." step before generating the audio file. And then it unloads it all again. Like I said, I guess that's intented behaviour. I'm not a python developer so excuse me for assuming/hoping: Is there a way to get this to work so it keeps the model loaded in between generation calls? I can manage my way to implement something but only if someone here can give me some direction as to what has to happen to get that to work. Obviously I still have the gradio client implementation right now, but I much prefer something that isn't as destructive or hacky, preferably something that still works after updating the repository.

mrq commented

2023-06-12 18:28:27 +00:00

I guess this is intented behaviour

"Intentional" in the sense there's no way to keep it loaded without it being a server such as with the main script, yes, it's intentional. There's no way to keep the model persisting between multiple Python processes.

In theory, there can be a way to make it persist, but it would be over-engineered spaghetti with a daemon service running as a server and subsequent actions are simply clients phoning it through IPC, which at that point, you're just better off either:

using the web UI's gradio API to do generations
manually opening a Python REPL prompt and manually instantiating everything to keep it "hotloaded".

> I guess this is intented behaviour "Intentional" in the sense there's no way to keep it loaded without it being a server such as with the main script, yes, it's intentional. There's no way to keep the model persisting between multiple Python processes. In *theory*, there can be a way to make it persist, but it would be over-engineered spaghetti with a daemon service running as a server and subsequent actions are simply clients phoning it through IPC, which at that point, you're just better off either: * using the web UI's gradio API to do generations * manually opening a Python REPL prompt and manually instantiating everything to keep it "hotloaded".

👍 1

FrioGlakka commented

2023-06-14 15:12:13 +00:00

I don't know why I expected to be able to use the cli.py script for my use case. I followed your advice and had really rapid success.

It was as easy as just having

args = setup_args(cli=True)
tts = load_tts()

at the beginning of the script and then inside a method routed to a POST endpoint:
building a kwargs object using the parameters in the posted .json and then just

result = generate(**kwargs)
output_wav_path = result[1][0]

This made it so the model is kept "hotloaded" as you call it. And then every POST call just starts inference on the model and returns the path.

I don't know why I expected to be able to use the cli.py script for my use case. I followed your advice and had really rapid success. It was as easy as just having ``` args = setup_args(cli=True) tts = load_tts() ``` at the beginning of the script and then inside a method routed to a POST endpoint: building a kwargs object using the parameters in the posted .json and then just ``` result = generate(**kwargs) output_wav_path = result[1][0] ``` This made it so the model is kept "hotloaded" as you call it. And then every POST call just starts inference on the model and returns the path.

FrioGlakka closed this issue

2023-06-14 15:12:15 +00:00

Sign in to join this conversation.