-
https://git.ecker.tech/ aims to provide a place to share my efforts while maintaining true ownership of my code, as I do not trust GitHub.
XMR: 4B9TQdkAkBFYrbj5ztvTx89e5LpucPeTSPzemCihdDi9EBnx7btn8RDNZTBz2zihWsjMnDkzn5As1LU6gLv3KQy8BLsZ8SG
- Joined on
2022-10-10
11ea660700
added (fixed) LR not updating when changing it in the config, added ability to set LR from stdin (lr value)
c41e567950
implemnted microsoft/torchscale's RetNet implementation (needs improvement and testing, but the mini test trainers are happy so far)
Ugh. I suppose my current training needs to be scrapped, since it turns out my "just replace the Attention with Retention, it'll be fine, it did fix inferencing" approach to integrate the RetNet…
I feel some slight uneasiness.
I feel like there's something I'm missing with the implementation.
- I probably shouldn't be using the sinusoidal position embedding before the inputs get…
c117c060e1
tweaks, testing to see if just using Retention instead of Attention offers any benefits (so far no changes I can notice)
Ugh, inferencing is wonky again with RetNet. I managed to get it working when using the web UI (manual inferencing), but the evaluation / validation step just doesn't want to work, so I'll have to…
Mary? \n {"voice":
I think the issue is that you have a space after the \n.
I should be able to fix this by stripping for whitespace before checking if needed.
I genuinely had to do a double take when I woke back up and saw the AR already this far: 6 hours to get to what I imagine would be an astronomical time: , but I figured this should be it's own block as this is more about integrating RetNet.
I bit…
628dadff71
Added RetNet support (credit to https://github.com/Jamie-Stirling/RetNet/ for the implementation base)
mmm... maybe I was a little too hasty to get training back up again. Not only did I have the partial LibriLight-6K, I also forgot I wanted to:
- re-enable Vocos.
- I'll cry about it, it's…
c9249eb612
I genuinely don't remember, I just need to commit this so I can copy it to my main PC. It looks like some tweaks to inferencing with Vocos and also being able to filter the dataset by audio duration