RAM memory leak #355

New Issue

voiceai_user · 2023-08-28T14:15:23Z

voiceai_user commented

2023-08-28 14:15:23 +00:00

Hello, I've been using this project for a couple of weeks, and been having issues after prolonged usage.

I am using the project to try and make audio books so my inputs are generally long. The inputs are usually fine but over time, the ram usage continues to climb higher until the program crashes or the system crashes. I'm not sure if there is a better way, but at the moment after generating for a period, I will have to close and reopen the app to reclaim the memory in ram.

I tried doing various things, it is similar to #61 sort of, but instead of gpu vram it is my ram.

I am using linux mint so linux flavor of install. It works fine until I run out of ram and either the application crashes or my system crashes because of no more ram.

Hello, I've been using this project for a couple of weeks, and been having issues after prolonged usage. I am using the project to try and make audio books so my inputs are generally long. The inputs are usually fine but over time, the ram usage continues to climb higher until the program crashes or the system crashes. I'm not sure if there is a better way, but at the moment after generating for a period, I will have to close and reopen the app to reclaim the memory in ram. I tried doing various things, it is similar to #61 sort of, but instead of gpu vram it is my ram. I am using linux mint so linux flavor of install. It works fine until I run out of ram and either the application crashes or my system crashes because of no more ram.

mrq commented

2023-08-28 16:11:00 +00:00

Ah yeah. I'm not too sure where the issue lies, as even using the repo to transcribe large datasets will eventually hang after a while, from presumably a memory leak somewhere. I'm just not too sure where if it's independent from continuous transcriptions. For my problem, I just modified the cli.py, and the only problem was having to constantly reload models.

For normal people, the existing cli.py should be fine enough, a shell script like this should be fine:

#!/bin/bash
python3 ./src/cli.py --voice="Voice Name" --text="The birch canoe slid on the smooth planks.\nGlue the sheet to the dark blue background." --delimiter="\n"
python3 ./src/cli.py --voice="Voice Name" --text="It's easy to tell the depth of a well.\nThese days a chicken leg is a rare dish." --delimiter="\n"
python3 ./src/cli.py --voice="Voice Name" --text="Rice is often served in round bowls.\nThe juice of lemons makes fine punch." --delimiter="\n"

The downside is that, each call to cli.py will require the models to reload again. Using the websocket server added in might be a better approach to work around constantly reloading the model, but you'll most likely still run into a memory leak problem (assuming the issue is not gradio).

Ah yeah. I'm not too sure where the issue lies, as even using the repo to transcribe large datasets will eventually hang after a while, from presumably a memory leak somewhere. I'm just not too sure where if it's independent from continuous transcriptions. For my problem, I just modified the `cli.py`, and the only problem was having to constantly reload models. For normal people, the existing [`cli.py`](https://git.ecker.tech/mrq/ai-voice-cloning/src/branch/master/src/cli.py) should be fine enough, a shell script like this should be fine: ``` #!/bin/bash python3 ./src/cli.py --voice="Voice Name" --text="The birch canoe slid on the smooth planks.\nGlue the sheet to the dark blue background." --delimiter="\n" python3 ./src/cli.py --voice="Voice Name" --text="It's easy to tell the depth of a well.\nThese days a chicken leg is a rare dish." --delimiter="\n" python3 ./src/cli.py --voice="Voice Name" --text="Rice is often served in round bowls.\nThe juice of lemons makes fine punch." --delimiter="\n" ``` The downside is that, each call to `cli.py` will require the models to reload again. Using the websocket server added in might be a better approach to work around constantly reloading the model, but you'll most likely still run into a memory leak problem (assuming the issue is not gradio).

voiceai_user commented

2023-08-28 16:43:43 +00:00

I see, thanks for the insight, yeah I'm not sure either. I tried doing a single long generation with a delimiter and multiple smaller ones but the memory never clears unless I just kill the python script.

Loading the models with massive amount of input sounds like it will take significantly longer so I'm unsure if the cli.py method is desirable sadly.

I may just try monitoring memory usage and then kill and restart it programmatically since I was sort of doing a process where I generate text, send it somewhere to clean up the output, and then send it somewhere else to match the voice more closely. I was going to automate it down the line anyways so another cog might not be too bad...

I will leave up the issue just in case

I see, thanks for the insight, yeah I'm not sure either. I tried doing a single long generation with a delimiter and multiple smaller ones but the memory never clears unless I just kill the python script. Loading the models with massive amount of input sounds like it will take significantly longer so I'm unsure if the cli.py method is desirable sadly. I may just try monitoring memory usage and then kill and restart it programmatically since I was sort of doing a process where I generate text, send it somewhere to clean up the output, and then send it somewhere else to match the voice more closely. I was going to automate it down the line anyways so another cog might not be too bad... I will leave up the issue just in case

voiceai_user commented

2023-09-13 04:23:54 +00:00

I finally got around to messing with this again, and it seems that the issue does not occur if using the api, I will have to test if providing longer inputs changes anything, but currently using roughly 100 lines of input does not seem to have the memory leak compared to using the webui. I made roughly 5 different api calls and the memory stayed roughly the same.

voiceai_user commented

2023-09-13 10:13:26 +00:00

Nevermind, after trying to do it overnight, it died from the memory leak.

DoctorPopi commented

2023-09-29 06:54:46 +00:00

Hey, after prolonged usage, I’m having the same problem I think. I cannot generate anything else through Mrq without having the OOM Cuda allocation error. What’s even weirder, I just shut down my computer yesterday thinking that it would clear the GPU ram, but this morning after launching a fresh instance of gradio, I immediately got the OOM message

EDIT : after experimenting with this, I found out that you can shut down the Gradio interface, and then while still in the dedicated environment (conda in my case), you write

Python
import torch
torch.cuda_empty_cache()

And then when I restart the UI I don't have any problem anymore. Will it remain so? Time will tell

Hey, after prolonged usage, I’m having the same problem I think. I cannot generate anything else through Mrq without having the OOM Cuda allocation error. What’s even weirder, I just shut down my computer yesterday thinking that it would clear the GPU ram, but this morning after launching a fresh instance of gradio, I immediately got the OOM message -------- EDIT : after experimenting with this, I found out that you can shut down the Gradio interface, and then while still in the dedicated environment (conda in my case), you write >> Python >> import torch >> torch.cuda_empty_cache() And then when I restart the UI I don't have any problem anymore. Will it remain so? Time will tell

Sign in to join this conversation.