James Betker
|
da53090ce6
|
More adjustments to support distributed training with teco & on multi_modal_train
|
2020-10-27 20:58:03 -06:00 |
|
James Betker
|
2a3eec8fd7
|
Fix some distributed training snafus
|
2020-10-27 15:24:05 -06:00 |
|
James Betker
|
54accfa693
|
Merge remote-tracking branch 'origin/gan_lab' into gan_lab
|
2020-10-26 11:12:37 -06:00 |
|
James Betker
|
ff58c6484a
|
Fixes to unified chunk datasets to support stereoscopic training
|
2020-10-26 11:12:22 -06:00 |
|
James Betker
|
f857eb00a8
|
Allow tecogan losses to compute at 32px
|
2020-10-26 11:09:55 -06:00 |
|
James Betker
|
9c3d059ef0
|
Updates to be able to train flownet2 in ExtensibleTrainer
Only supports basic losses for now, though.
|
2020-10-24 11:56:39 -06:00 |
|
James Betker
|
e9c0b9f0fd
|
More adjustments to support multi-modal training
Specifically - looks like at least MSE loss cannot handle autocasted tensors
|
2020-10-22 16:49:34 -06:00 |
|
James Betker
|
76789a456f
|
Class-ify train.py and workon multi-modal trainer
|
2020-10-22 16:15:31 -06:00 |
|
James Betker
|
15e00e9014
|
Finish integration with autocast
Note: autocast is broken when also using checkpoint(). Overcome this by modifying
torch's checkpoint() function in place to also use autocast.
|
2020-10-22 14:39:19 -06:00 |
|
James Betker
|
d7ee14f721
|
Move to torch.cuda.amp (not working)
Running into OOM errors, needs diagnosing. Checkpointing here.
|
2020-10-22 13:58:05 -06:00 |
|
James Betker
|
3e3d2af1f3
|
Add multi-modal trainer
|
2020-10-22 13:27:32 -06:00 |
|
James Betker
|
43c4f92123
|
Collapse progressive zoom candidates into the batch dimension
This contributes a significant speedup to training this type of network
since losses can operate on the entire prediction spectrum at once.
|
2020-10-21 22:37:23 -06:00 |
|
James Betker
|
680d635420
|
Enable ExtensibleTrainer to skip steps when state keys are missing
|
2020-10-21 22:22:28 -06:00 |
|
James Betker
|
d1175f0de1
|
Add FFT injector
|
2020-10-21 22:22:00 -06:00 |
|
James Betker
|
931aa65dd0
|
Allow recurrent losses to be weighted
|
2020-10-21 16:59:44 -06:00 |
|
James Betker
|
b28e4d9cc7
|
Add spread loss
Experimental loss that peaks around 0.
|
2020-10-19 11:31:19 -06:00 |
|
James Betker
|
668cafa798
|
Push correct patch of recurrent embedding to upstream image, rather than whole thing
|
2020-10-18 22:39:52 -06:00 |
|
James Betker
|
7df378a944
|
Remove separated vgg discriminator
Checkpointing happens inline instead. Was a dumb idea..
Also fixes some loss reporting issues.
|
2020-10-18 12:10:24 -06:00 |
|
James Betker
|
c709d38cd5
|
Fix memory leak with recurrent loss
|
2020-10-18 10:22:10 -06:00 |
|
James Betker
|
552e70a032
|
Get rid of excessive checkpointed disc params
|
2020-10-18 10:09:37 -06:00 |
|
James Betker
|
6a0d5f4813
|
Add a checkpointable discriminator
|
2020-10-18 09:57:47 -06:00 |
|
James Betker
|
9ead2c0a08
|
Multiscale training in!
|
2020-10-17 22:54:12 -06:00 |
|
James Betker
|
eda75c9779
|
Cleanup fixes
|
2020-10-15 10:13:17 -06:00 |
|
James Betker
|
24792bdb4f
|
Codebase cleanup
Removed a lot of legacy stuff I have no intent on using again.
Plan is to shape this repo into something more extensible (get it? hah!)
|
2020-10-13 20:56:39 -06:00 |
|
James Betker
|
e620fc05ba
|
Mods to support video processing with teco networks
|
2020-10-13 20:47:05 -06:00 |
|
James Betker
|
17d78195ee
|
Mods to SRG to support returning switch logits
|
2020-10-13 20:46:37 -06:00 |
|
James Betker
|
8014f050ac
|
Clear metrics properly
Holy cow, what a PITA bug.
|
2020-10-13 10:07:49 -06:00 |
|
James Betker
|
05377973bf
|
Allow initial recurrent input to be specified (optionally)
|
2020-10-12 17:36:43 -06:00 |
|
James Betker
|
d7d7590f3e
|
Fix constant injector - wasn't working in test
|
2020-10-12 10:36:30 -06:00 |
|
James Betker
|
a9c2e97391
|
Constant injector and teco fixes
|
2020-10-11 08:20:07 -06:00 |
|
James Betker
|
e785029936
|
Mods needed to support SPSR archs with teco gan
|
2020-10-10 22:39:55 -06:00 |
|
James Betker
|
120072d464
|
Add constant injector
|
2020-10-10 21:50:23 -06:00 |
|
James Betker
|
f99812e14d
|
Fix tecogan_losses errors
|
2020-10-10 20:30:14 -06:00 |
|
James Betker
|
3a5b23b9f7
|
Alter teco_losses to feed a recurrent input in as separate
|
2020-10-10 20:21:09 -06:00 |
|
James Betker
|
0d30d18a3d
|
Add MarginRemoval injector
|
2020-10-09 20:35:56 -06:00 |
|
James Betker
|
0011d445c8
|
Fix loss indexing
|
2020-10-09 20:20:51 -06:00 |
|
James Betker
|
202eb11fdc
|
For element loss added
|
2020-10-09 19:51:44 -06:00 |
|
James Betker
|
7e777ea34c
|
Allow tecogan to be used in process_video
|
2020-10-09 19:21:43 -06:00 |
|
James Betker
|
1eb516d686
|
Fix more distributed bugs
|
2020-10-08 14:32:45 -06:00 |
|
James Betker
|
c174ac0fd5
|
Allow tecogan to support generators that only output a tensor (instead of a list)
|
2020-10-08 09:26:25 -06:00 |
|
James Betker
|
c93dd623d7
|
Tecogan losses work
|
2020-10-07 23:11:58 -06:00 |
|
James Betker
|
c352c8bce4
|
More tecogan fixes
|
2020-10-07 12:41:17 -06:00 |
|
James Betker
|
1c44d395af
|
Tecogan work
Its training! There's still probably plenty of bugs though..
|
2020-10-07 09:03:30 -06:00 |
|
James Betker
|
e9d7371a61
|
Add concatenate injector
|
2020-10-07 09:02:42 -06:00 |
|
James Betker
|
cffc596141
|
Integrate flownet2 into codebase, add teco visual debugs
|
2020-10-06 20:35:39 -06:00 |
|
James Betker
|
840927063a
|
Work on tecogan losses
|
2020-10-05 19:35:28 -06:00 |
|
James Betker
|
51044929af
|
Don't compute attention statistics on multiple generator invocations of the same data
|
2020-10-05 00:34:29 -06:00 |
|
James Betker
|
e760658fdb
|
Another fix..
|
2020-10-04 21:08:00 -06:00 |
|
James Betker
|
a890e3a9c0
|
Fix geometric loss not handling 0 index
|
2020-10-04 21:05:01 -06:00 |
|
James Betker
|
13f97e1e97
|
Add recursive loss
|
2020-10-04 20:48:15 -06:00 |
|