Learning rate finder #49

New Issue

epp · 2023-10-19T11:14:50Z

epp commented

2023-10-19 11:14:50 +00:00

Hello,

I am trying to implement learning rate finder from https://github.com/davidtvs/pytorch-lr-finder so I would be able to find optimal learning rate when training new language. However I don't have much experience in ML and Pytorch.

I have been able to implement the model loading like this, but I am not sure how to load the dataset into the training

from torch_lr_finder import LRFinder
from torch.distributed.run import main as torchrun
from tortoise.models.autoregressive import UnifiedVoice
import torch
from torch import nn, optim

model = UnifiedVoice(
    max_mel_tokens=604,
    max_text_tokens=402,
    max_conditioning_inputs=2,
    layers=30,
    model_dim=1024,
    heads=16,
    number_text_tokens=255,  # updated from 257 to 256
    start_text_token=255,
    checkpointing=True,
    train_solo_embeddings=False,  
    mel_length_compression=1024,  
    number_mel_codes=8194,  
    start_mel_token=8192,  
    stop_mel_token=8193,  
    use_mel_codes_as_input=True,
    # tortoise_compat=True,
    # mel_solo_embedding=True, 
    # text_solo_embedding=True
).cpu().eval()

model.load_state_dict(torch.load('/models/tortoise/autoregressive.pth'))

train_dataset = CustomDataset(path='./training/MegaCzech/train.txt') # I don't know how to load dataset through the pytorch dataset loader
trainloader = DataLoader(train_dataset, batch_size=128, shuffle=True, num_workers=2)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-7, weight_decay=1e-2)
lr_finder = LRFinder(model, optimizer, criterion, device="cuda")
lr_finder.range_test(trainloader, end_lr=100, num_iter=100)
lr_finder.plot() # to inspect the loss-learning rate graph
lr_finder.reset() # to reset the model and optimizer to their initial state

Hello, I am trying to implement learning rate finder from https://github.com/davidtvs/pytorch-lr-finder so I would be able to find optimal learning rate when training new language. However I don't have much experience in ML and Pytorch. I have been able to implement the model loading like this, but I am not sure how to load the dataset into the training ``` from torch_lr_finder import LRFinder from torch.distributed.run import main as torchrun from tortoise.models.autoregressive import UnifiedVoice import torch from torch import nn, optim model = UnifiedVoice( max_mel_tokens=604, max_text_tokens=402, max_conditioning_inputs=2, layers=30, model_dim=1024, heads=16, number_text_tokens=255, # updated from 257 to 256 start_text_token=255, checkpointing=True, train_solo_embeddings=False, mel_length_compression=1024, number_mel_codes=8194, start_mel_token=8192, stop_mel_token=8193, use_mel_codes_as_input=True, # tortoise_compat=True, # mel_solo_embedding=True, # text_solo_embedding=True ).cpu().eval() model.load_state_dict(torch.load('/models/tortoise/autoregressive.pth')) train_dataset = CustomDataset(path='./training/MegaCzech/train.txt') # I don't know how to load dataset through the pytorch dataset loader trainloader = DataLoader(train_dataset, batch_size=128, shuffle=True, num_workers=2) criterion = nn.CrossEntropyLoss() optimizer = optim.Adam(model.parameters(), lr=1e-7, weight_decay=1e-2) lr_finder = LRFinder(model, optimizer, criterion, device="cuda") lr_finder.range_test(trainloader, end_lr=100, num_iter=100) lr_finder.plot() # to inspect the loss-learning rate graph lr_finder.reset() # to reset the model and optimizer to their initial state ```

mrq commented

2023-10-19 14:29:27 +00:00

It's a bit of a pickle, since all the relevant code to actually train the model is within DLAS. I believe here is the relevant code for preparing the dataloader.

You might have a better time instead trying to use it within DLAS, since you can probably just inject it here, rather than re-implement the dataloader preparation routines within TorToiSe.

It's a bit of a pickle, since all the relevant code to actually train the model is within [DLAS](https://git.ecker.tech/mrq/DL-Art-School). I believe [here](https://git.ecker.tech/mrq/DL-Art-School/src/branch/master/dlas/data/audio/paired_voice_audio_dataset.py) is the relevant code for preparing the dataloader. You *might* have a better time instead trying to use it within DLAS, since you can probably just inject it [here](https://git.ecker.tech/mrq/DL-Art-School/src/branch/master/dlas/train.py#L390), rather than re-implement the dataloader preparation routines within TorToiSe.

epp commented

2023-10-20 08:07:53 +00:00

Thank you, I didn't realize that could be the case. I have tried to insert/modify the code, but the complexity is beyond my capabilities. Thank you for this library though!

Sign in to join this conversation.

No Label

No Milestone

No project

No Assignees

2 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: mrq/tortoise-tts#49