DL-Art-School/README.md

110 lines
5.7 KiB
Markdown
Raw Normal View History

# (QoL improvements for) Deep Learning Art School
This fork of [neonbjb/DL-Art-School](https://github.com/neonbjb/DL-Art-School/) contains a few fixes and QoL improvements, including but not limited to:
* sanity tidying, like:
- not outputing to `./DL-Art-School/experiments/`
- the custom module loader for networks/injectors getting fixed
- BitsAndBytes integration:
+ working but output untested: Adam/AdamW
+ toggles available in `./codes/torch_indermediary/__init__.py`
---
2020-10-23 22:38:23 +00:00
# Deep Learning Art School
Send your Pytorch model to art class!
2020-12-20 23:52:57 +00:00
This repository is both a framework and a set of tools for training deep neural networks that create images. It started
as a branch of the [open-mmlab](https://github.com/open-mmlab) project developed by [Multimedia Laboratory, CUHK](http://mmlab.ie.cuhk.edu.hk)
but has been almost completely re-written at every level.
2020-10-23 22:38:23 +00:00
## Why do we need another training framework
2020-12-20 23:52:57 +00:00
These are a dime a dozen, no doubt. DL Art School (*DLAS*) differentiates itself by being configuration driven. You write
the model code (specifically, a torch.nn.Module) and (possibly) some losses, then you cobble together a config file written
in yaml that tells DLAS how to train it. Swapping model architectures and tuning hyper-parameters is simple and often
requires no changes to actual code. You also don't need to remember complex command line incantations. This effectively
enables you to run multiple concurrent experiments that use the same codebase, as well as retain backwards compatibility
for past experiments.
2020-10-23 22:38:23 +00:00
2020-12-20 23:52:57 +00:00
Training effective generators often means juggling multiple loss functions. As a result, DLAS' configuration language is
specifically designed to make it easy to support large number of losses and networks that interact with each other. As an
example: some GANs I have trained in this framework consist of more than 15 losses and use 2 separate discriminators and
require no bespoke code.
2020-10-23 22:38:23 +00:00
2020-12-20 23:52:57 +00:00
Generators are also notorious GPU memory hogs. I have spent substantial time streamlining the training framework to support
gradient checkpointing and FP16. DLAS also supports "mega batching", where multiple forward passes contribute to a single
backward pass. Most models can be trained on midrange GPUs with 8-11GB of memory.
2020-10-23 22:38:23 +00:00
2020-12-20 23:52:57 +00:00
The final value-added feature is interpretability. Tensorboard logging operates out of the box with no custom code.
Intermediate images from within the training pipeline can be intermittently surfaced as normal PNG files so you can
see what your network is up to. Validation passes are also cached as images so you can view how your network improves
over time.
2020-10-23 22:38:23 +00:00
## Modeling Capabilities
2020-12-20 23:52:57 +00:00
DLAS was built with extensibility in mind. One of the reasons I'm putting in the effort to better document this code is the
incredible ease with which I have been able to train entirely new model types with no changes to the core training code.
2020-10-23 22:38:23 +00:00
2020-12-20 23:52:57 +00:00
I intend to fill out the sections below with sample configurations which can be used to train different architectures.
You will need to bring your own data.
2020-10-23 22:38:23 +00:00
### Super-resolution
2020-12-20 23:52:57 +00:00
- [GAN-based SR (ESRGAN)](https://github.com/neonbjb/DL-Art-School/tree/gan_lab/recipes/esrgan)
- [SRFlow](https://github.com/neonbjb/DL-Art-School/tree/gan_lab/recipes/srflow)
- [GLEAN](https://github.com/neonbjb/DL-Art-School/tree/gan_lab/recipes/glean)
- Video SR (TecoGAN) (*documentation TBC*)
2020-10-23 22:38:23 +00:00
### Style Transfer
2020-12-20 23:52:57 +00:00
* Stylegan2 (*documentation TBC*)
### Latent development
* [BYOL](https://github.com/neonbjb/DL-Art-School/tree/gan_lab/recipes/byol)
* iGPT (*documentation TBC*)
2019-08-23 13:42:47 +00:00
## Dependencies and Installation
2020-10-23 22:38:23 +00:00
- Python 3
- [PyTorch >= 1.6](https://pytorch.org)
2019-08-23 13:42:47 +00:00
- NVIDIA GPU + [CUDA](https://developer.nvidia.com/cuda-downloads)
2019-11-24 07:47:57 +00:00
- Python packages: `pip install -r requirements.txt`
2020-10-23 22:38:23 +00:00
- Some video utilities require [FFMPEG](https://ffmpeg.org/)
2019-09-06 13:32:28 +00:00
2020-10-23 22:38:23 +00:00
## User Guide
TBC
2019-08-23 13:42:47 +00:00
### Development Environment
If you aren't already using [Pycharm](https://www.jetbrains.com/pycharm/) - now is the time to try it out. This project was built in Pycharm and comes with
an IDEA project for you to get started with. I've done all of my development on this repo in this IDE and lean heavily
on its incredible debugger. It's free. Try it out. You won't be sorry.
2020-10-23 22:38:23 +00:00
### Dataset Preparation
DLAS comes with some Dataset instances that I have created for my own use. Unless you want to use one of the recipes above, you'll need to provide your own. Here is how to add your own Dataset:
2019-08-23 13:42:47 +00:00
2020-10-23 22:38:23 +00:00
1. Create a Dataset in codes/data/ which takes a single Python dict as a constructor and extracts options from that dict.
2. Register your Dataset in codes/data/__init__.py
3. Your Dataset should return a dict of tensors. The keys of the dict are injected directly into the training state, which you can interact within your configuration file.
2019-08-23 13:42:47 +00:00
2020-10-23 22:38:23 +00:00
### Training and Testing
There are currently 3 base scripts for interacting with models. They all take a single parameter, `-opt` which specifies the configuration file which controls how they work. Configs (will be) documented above in the user guide.
2019-08-23 13:42:47 +00:00
2020-10-23 22:38:23 +00:00
#### train.py
2021-06-06 22:56:40 +00:00
Start (or continue) a training session:
2020-10-23 22:38:23 +00:00
`python train.py -opt <your_config.yml>`
2019-08-23 13:42:47 +00:00
2021-06-06 22:56:40 +00:00
Start a distributed training session:
`python -m torch.distributed.launch --nproc_per_node=<gpus> --master_port=1234 train.py -o <opt> --launcher=pytorch`
2020-10-23 22:38:23 +00:00
#### test.py
Runs a model against a validation or test set of data and reports metrics (for now, just PSNR and a custom perceptual metric)
`python test.py -opt <your_config.yml>`
2019-08-23 13:42:47 +00:00
2020-10-23 22:38:23 +00:00
#### process_video.py
Breaks a video into individual frames and uses a network to do processing on it, then reassembles the output back into video form.
`python process_video -opt <your_config.yml>`
## Contributing
At this time I am not taking feature requests or bug reports, but I appreciate all contributions.
2019-08-23 13:42:47 +00:00
## License
This project is released under the Apache 2.0 license.