James Betker
5c6c8f6904
back to convs?
2022-06-11 15:26:10 -06:00
James Betker
b684622b06
tfd10
2022-06-11 14:06:19 -06:00
James Betker
2a787ec910
more mods
2022-06-11 11:44:33 -06:00
James Betker
999a140f9f
whoops
2022-06-11 11:22:34 -06:00
James Betker
00b9f332ee
rework arch
2022-06-11 11:17:16 -06:00
James Betker
36b5e89a69
Rework the diet blocks a bit
2022-06-11 08:28:10 -06:00
James Betker
0dd3883662
one last update..
2022-06-11 08:06:29 -06:00
James Betker
41170f97e9
one more adjustment
2022-06-11 08:01:46 -06:00
James Betker
df0cdf1a4f
tfd9 returns with some optimizations
2022-06-11 08:00:09 -06:00
James Betker
acfe9cf880
fp16
2022-06-10 22:39:15 -06:00
James Betker
aca9024d9b
qcodes
2022-06-10 16:23:08 -06:00
James Betker
38a00f29c0
now theres deprecation warnings, fml
2022-06-10 15:41:39 -06:00
James Betker
561a6b8ff7
damn this sucks
2022-06-10 15:38:59 -06:00
James Betker
0316063e2d
.
2022-06-10 15:37:02 -06:00
James Betker
dca16e6447
.
2022-06-10 15:35:36 -06:00
James Betker
ee2827dee9
Debug warmup state
2022-06-10 15:23:31 -06:00
James Betker
6d85fe05f6
:/ oh well.
2022-06-10 15:17:41 -06:00
James Betker
33178e89c4
harharhack
2022-06-10 15:13:24 -06:00
James Betker
7198bd8bd0
forgot other customizations I want to keep
2022-06-10 15:09:05 -06:00
James Betker
8f40108f5b
lets try a different tact
2022-06-10 14:51:59 -06:00
James Betker
2158383fa4
Revert previous changes
2022-06-10 14:34:05 -06:00
James Betker
89bd40d39f
eval bug fix
2022-06-10 13:51:06 -06:00
James Betker
84469f3538
get rid of encoder checkpointing
2022-06-10 10:50:34 -06:00
James Betker
97b32dd39d
try to make tfd8 be able to be trained e2e in quantizer mode
2022-06-10 10:40:56 -06:00
James Betker
e78c4b422c
tfd8
2022-06-10 09:24:41 -06:00
James Betker
d98b895307
loss aware fix and report gumbel temperature
2022-06-09 21:56:47 -06:00
James Betker
6e57eaa186
fix bug
2022-06-09 21:52:57 -06:00
James Betker
07bdd865dc
some checks
2022-06-09 21:46:32 -06:00
James Betker
34005367fd
setup for partial channel diffusion
2022-06-09 21:41:20 -06:00
James Betker
47b34f5cb9
mup work checkin
2022-06-09 21:15:09 -06:00
James Betker
e67e82be2d
misc
2022-06-09 21:14:48 -06:00
James Betker
16936881e5
allow freezing the upper quantizer
2022-06-08 18:30:22 -06:00
James Betker
43f225c35c
debug gumbel temperature
2022-06-08 12:12:08 -06:00
James Betker
91be38cba3
.
2022-06-08 11:54:46 -06:00
James Betker
dee2b72786
checkpointing bugs, smh
2022-06-08 11:53:10 -06:00
James Betker
c61cd64bc9
network updates
2022-06-08 09:26:59 -06:00
James Betker
5a54d7db11
unet with ar prior
2022-06-07 17:52:36 -06:00
James Betker
5028703b3d
ci not required
2022-06-06 09:26:25 -06:00
James Betker
08597bfaf5
fix
2022-06-06 09:21:58 -06:00
James Betker
49568ee16f
some updates
2022-06-06 09:13:47 -06:00
James Betker
602df0abbc
revert changes to dietattentionblock
2022-06-05 10:06:17 -06:00
James Betker
51d1908e94
update
2022-06-05 09:35:43 -06:00
James Betker
f9ebcf11d8
fix2
2022-06-05 01:31:37 -06:00
James Betker
aac92b01b3
fix
2022-06-05 01:27:28 -06:00
James Betker
38d8b17d18
tfd8 gets real verbose grad norm metrics
2022-06-04 23:09:54 -06:00
James Betker
0a9d4d4afc
bunch of new stuff
2022-06-04 22:23:08 -06:00
James Betker
8f8b189025
Support legacy vqvae quantizer in music_quantizer
2022-06-04 10:16:24 -06:00
James Betker
4819f15521
undo quantile sampler in music_diffusion_fid
2022-06-04 10:15:31 -06:00
James Betker
a9387179db
add channel loss balancing
2022-06-03 15:19:23 -06:00
James Betker
40ba802104
padding
2022-06-03 12:09:59 -06:00
James Betker
581bc7ac5c
udmc update
2022-06-03 12:02:22 -06:00
James Betker
a14274c845
fix memory issue
2022-06-03 11:20:09 -06:00
James Betker
9d8c2bddb1
classical unet for music
2022-06-03 11:03:14 -06:00
James Betker
d3a60633a3
codes generation script
2022-06-03 11:02:28 -06:00
James Betker
2f4d990ad1
tfd7
2022-06-02 09:27:40 -06:00
James Betker
b2a83efe50
a few fixes
2022-06-01 16:35:15 -06:00
James Betker
712e0e82f7
fix bug
2022-06-01 14:21:44 -06:00
James Betker
de54be5570
propagate diversity loss
2022-06-01 14:18:50 -06:00
James Betker
4c6ef42b38
freeze quantizer until step
2022-06-01 08:06:05 -06:00
James Betker
64b6ae2f4a
fix
2022-06-01 01:01:32 -06:00
James Betker
1ac02acdc3
tfd7
2022-06-01 00:50:40 -06:00
James Betker
e8cb93a4e9
fix size issues
2022-05-31 21:23:26 -06:00
James Betker
8a1b8e3e62
add checkpointing
2022-05-31 21:09:05 -06:00
James Betker
c0db85bf4f
music quantizer
2022-05-31 21:06:54 -06:00
James Betker
96da10415e
facepalm
2022-05-30 16:48:18 -06:00
James Betker
479c3195f3
make wandb histograms work again
2022-05-30 16:44:39 -06:00
James Betker
29b55d42a5
one more
2022-05-30 16:33:49 -06:00
James Betker
71cf654957
fix unused parameters
2022-05-30 16:31:40 -06:00
James Betker
f7d237a50a
train quantizer with diffusion
2022-05-30 16:25:33 -06:00
James Betker
136021bf8d
tfd6
2022-05-30 09:09:42 -06:00
James Betker
eab1162d2b
hmm..
2022-05-29 22:32:25 -06:00
James Betker
2e72fddaeb
td_tts_2
2022-05-29 22:22:14 -06:00
James Betker
27f347cdd3
gotta project that shit!
2022-05-28 23:19:36 -06:00
James Betker
536c8558ae
fix
2022-05-28 22:32:38 -06:00
James Betker
da367da411
df5
2022-05-28 22:30:23 -06:00
James Betker
6b43915eb8
support projecting to vectors
2022-05-28 22:27:45 -06:00
James Betker
86694aef4e
tfd5
2022-05-28 22:27:04 -06:00
James Betker
b6b4f10e1b
...
2022-05-28 10:59:03 -06:00
James Betker
0d3b831cf9
big fatty
2022-05-28 10:55:43 -06:00
James Betker
76aeba7843
fail gracefully from mfd
2022-05-27 20:24:16 -06:00
James Betker
f691f5faa1
f
2022-05-27 13:47:05 -06:00
James Betker
031769150d
clip adf test dataset
2022-05-27 13:42:52 -06:00
James Betker
31dec016e0
adf
2022-05-27 12:28:04 -06:00
James Betker
b4269af61b
fix circular deps
2022-05-27 11:44:27 -06:00
James Betker
34ee1d0bc3
mdf
2022-05-27 11:40:47 -06:00
James Betker
490d39b967
some stuff
2022-05-27 11:40:31 -06:00
James Betker
5efeee6b97
fix type bug
2022-05-27 11:19:30 -06:00
James Betker
0659fe3d1e
tfd3 mods
2022-05-27 11:16:26 -06:00
James Betker
bed3df4888
propagate type
2022-05-27 11:12:03 -06:00
James Betker
c46da0285c
Move stuff around
2022-05-27 11:06:58 -06:00
James Betker
9852599b34
tfd5 - with clvp!
2022-05-27 09:49:10 -06:00
James Betker
3db862dd32
adf update
2022-05-27 09:25:53 -06:00
James Betker
8587a18717
fd fix
2022-05-26 20:19:09 -06:00
James Betker
dd13b883ac
td4
2022-05-26 14:56:03 -06:00
James Betker
1dbe0b6b2e
a
2022-05-26 10:13:27 -06:00
James Betker
aa653115f1
tfd3
2022-05-26 10:09:11 -06:00
James Betker
36c68692a6
forgot to add rotary embeddings
2022-05-26 09:25:42 -06:00
James Betker
8ce48f04ff
transformer diffusion 2
2022-05-26 09:08:35 -06:00
James Betker
56f19a23cd
fix nh
2022-05-25 12:31:56 -06:00
James Betker
52a20f3aa3
und10
2022-05-25 12:19:21 -06:00
James Betker
00e133afa9
support not_ew too
2022-05-25 08:58:23 -06:00
James Betker
5188866bd5
facepalm
2022-05-24 14:55:51 -06:00
James Betker
8b4b5ffa72
slight rework
2022-05-24 14:38:37 -06:00
James Betker
48aab2babe
ressurect ctc code gen with some cool new ideas
2022-05-24 14:02:33 -06:00
James Betker
65b441d74e
transformer diffusion
2022-05-24 14:02:05 -06:00
James Betker
1e1bbe1a27
whoops
2022-05-23 12:28:36 -06:00
James Betker
560b83e770
default to residual encoder
2022-05-23 12:24:00 -06:00
James Betker
f432bdf7ae
deeper resblock encoder
2022-05-23 11:46:40 -06:00
James Betker
bb36bb4db5
ugh it never ends
2022-05-23 10:58:28 -06:00
James Betker
5d13d38119
allow opt states to be reset
2022-05-23 10:54:37 -06:00
James Betker
f4a97ca0a7
and this
2022-05-23 10:38:28 -06:00
James Betker
874de1775d
Update mdf spectral
2022-05-23 10:37:15 -06:00
James Betker
36dd4eb61f
no grads for mel injectors
2022-05-23 10:34:53 -06:00
James Betker
dc471f5c6d
residual features
2022-05-23 09:58:30 -06:00
James Betker
1f521d6a1d
add reconstruction loss to m2v
2022-05-23 09:28:41 -06:00
James Betker
1d758c3bc8
try and make repeated failures recover better
2022-05-23 08:50:27 -06:00
James Betker
2270c89fdc
.
2022-05-23 08:47:15 -06:00
James Betker
40f844657b
tolong
2022-05-23 08:27:54 -06:00
James Betker
9f16b25ce5
introduce prepadlength
2022-05-23 08:21:27 -06:00
James Betker
10f4a742bd
reintroduce attention masks
2022-05-23 08:16:04 -06:00
James Betker
2da8f8a666
fmp
2022-05-23 07:06:25 -06:00
James Betker
68c0afcbcc
m2v frequency masking
2022-05-23 07:04:12 -06:00
James Betker
4093e38717
revert flat diffusion back...
2022-05-22 23:10:58 -06:00
James Betker
8f28404645
another fix
2022-05-22 21:32:43 -06:00
James Betker
41809a6330
Add 8x dim reductor
2022-05-22 20:23:16 -06:00
James Betker
07f7be24ce
:/
2022-05-22 20:00:55 -06:00
James Betker
c0bc466aad
mdf for shortened mel test
2022-05-22 19:29:20 -06:00
James Betker
1095248caf
Revert "retest"
...
This reverts commit ed7768c73b
2022-05-22 19:23:01 -06:00
James Betker
2798bfab8c
Revert "update mdf for legacy try"
...
This reverts commit 693b19ea3b
2022-05-22 19:22:44 -06:00
James Betker
693b19ea3b
update mdf for legacy try
2022-05-22 16:37:02 -06:00
James Betker
ed7768c73b
retest
2022-05-22 16:30:09 -06:00
James Betker
2dd0b9e6e9
mel_head should be optional
2022-05-22 12:25:45 -06:00
James Betker
37640e9759
Squelch annoying warning
2022-05-22 12:12:40 -06:00
James Betker
0c60f22197
fix unused parameters
2022-05-22 08:16:31 -06:00
James Betker
57d6f6d366
Big rework of flat_diffusion
...
Back to the drawing board, boys. Time to waste some resources catching bugs....
2022-05-22 08:09:33 -06:00
James Betker
db38672dae
precompute diffusion embeddings for from_codes
2022-05-22 06:45:57 -06:00
James Betker
ea21a8b107
Update music_diffusion_fid to support waveform diffusion from codes
2022-05-22 05:23:54 -06:00
James Betker
e0bf3a0ddc
Save myself some time in the future
2022-05-20 17:18:35 -06:00
James Betker
be937d202e
new attempt
2022-05-20 17:04:22 -06:00
James Betker
968660c248
another update
2022-05-20 11:25:00 -06:00
James Betker
28f950b7d3
fix
2022-05-20 11:18:52 -06:00
James Betker
b317c68ac9
fix
2022-05-20 11:12:53 -06:00
James Betker
3121bc4e43
flat diffusion
2022-05-20 11:01:48 -06:00
James Betker
e9fb2ead9a
m2v stuff
2022-05-20 11:01:17 -06:00
James Betker
c9c16e3b01
misc updates
2022-05-19 13:39:32 -06:00
James Betker
10378fc37f
make codebooks specifiable
2022-05-18 11:07:12 -06:00
James Betker
efc2657b48
fiddle with init
2022-05-18 10:56:01 -06:00
James Betker
208a703080
use gelu act
2022-05-18 09:34:01 -06:00
James Betker
b2b37453df
make the codebook bigger
2022-05-17 20:58:56 -06:00
James Betker
9a9c3cafba
Make feature encoder a bit more descriptive
2022-05-17 18:14:52 -06:00
James Betker
ee364f4eeb
just take the mean...
2022-05-17 18:09:23 -06:00
James Betker
6130391a85
fix div
2022-05-17 18:04:20 -06:00
James Betker
7213ad2b89
Do grad reduction
2022-05-17 17:59:40 -06:00
James Betker
7c82e18c6c
darn mpi
2022-05-17 17:16:09 -06:00
James Betker
88ec0512f7
Scale losses
2022-05-17 17:12:20 -06:00
James Betker
a6397ce84a
Fix incorrect projections
2022-05-17 16:53:52 -06:00
James Betker
c37fc3b4ed
m2v grad norm groups
2022-05-17 16:29:36 -06:00
James Betker
c1bdb4f9a1
degrade gumbel softmax over time
2022-05-17 16:23:04 -06:00
James Betker
3853f37257
stable layernorm
2022-05-17 16:07:03 -06:00
James Betker
6a2c29f596
Fix inverted logic
2022-05-17 15:39:07 -06:00
James Betker
519151d83f
m2v
2022-05-17 15:37:59 -06:00
James Betker
d1de94d75c
Stash mel2vec work (gonna throw it all away..)
2022-05-17 12:35:01 -06:00
James Betker
8202b9f39c
some stuff
2022-05-15 21:50:54 -06:00
James Betker
ab5acead0e
add exp loss for diffusion models
2022-05-15 21:50:38 -06:00
James Betker
ee218ab9b7
uv3
2022-05-13 17:57:47 -06:00
James Betker
eb64d18075
Fix phoneme tokenizer
2022-05-13 17:56:26 -06:00
James Betker
51f8c1bced
phonetic dataset
2022-05-12 11:57:28 -06:00
James Betker
3d7e2a2846
fix collection
2022-05-11 21:50:05 -06:00
James Betker
ba2b71796a
k
2022-05-11 21:20:06 -06:00
James Betker
efa737b685
re-add distributed collect to clvp
2022-05-11 21:14:18 -06:00
James Betker
545453077e
uv3
2022-05-09 15:36:22 -06:00
James Betker
96a5cc66ee
uv3
2022-05-09 15:35:51 -06:00
James Betker
b42b4e18de
clean up unified voice
...
- remove unused code
- fix inference model to use the terms "prior" and "posterior" to properly define the modeling order (they were inverted before)
- default some settings I never intend to change in the future
2022-05-09 14:45:49 -06:00
James Betker
9118f58849
uncomment music projector..
2022-05-09 09:19:26 -06:00
James Betker
74dd095326
a
2022-05-08 18:54:09 -06:00
James Betker
1177c35dec
music fid updates
2022-05-08 18:49:39 -06:00
James Betker
7812c23c7a
revert fill_gaps back to old masking behavior
2022-05-08 00:10:19 -06:00
James Betker
58ed27d7a8
new gap_filler
2022-05-07 12:44:23 -06:00
James Betker
6c8032b4be
more work
2022-05-06 21:56:49 -06:00
James Betker
f541610256
contrastive_audio
2022-05-06 16:37:22 -06:00
James Betker
79543e5488
Simpler form of the wavegen model
2022-05-06 16:37:04 -06:00
James Betker
d8925ccde5
few things with gap filling
2022-05-06 14:33:44 -06:00
James Betker
b83b53cf84
norm mel
2022-05-06 00:49:54 -06:00
James Betker
b13d983c24
and mel_head
2022-05-06 00:25:27 -06:00
James Betker
d5fb79564a
remove mel_pred
2022-05-06 00:24:05 -06:00
James Betker
e9bb692490
fixed aligned_latent
2022-05-06 00:20:21 -06:00
James Betker
1609101a42
musical gap filler
2022-05-05 16:47:08 -06:00
James Betker
d66ab2d28c
Remove unused waveform_gens
2022-05-04 21:06:54 -06:00
James Betker
47662b9ec5
some random crap
2022-05-04 20:29:23 -06:00
James Betker
6655f7845a
add pixel shuffling for 1d cases
2022-05-04 08:03:09 -06:00
James Betker
c42c53e75a
Add a trainable network for converting a normal distribution into a latent space
2022-05-02 09:47:30 -06:00
James Betker
e402089556
abstractify
2022-05-02 00:11:26 -06:00
James Betker
ab219fbefb
output variance
2022-05-02 00:10:33 -06:00
James Betker
3b074aac34
add checkpointing
2022-05-02 00:07:42 -06:00
James Betker
ae5f934ea1
diffwave
2022-05-02 00:05:04 -06:00
James Betker
f4254609c1
MDF
...
around and around in circles........
2022-05-01 23:04:56 -06:00
James Betker
b712d3b72b
break out get_conditioning_latent from unified_voice
2022-05-01 23:04:44 -06:00
James Betker
afa2df57c9
gen3
2022-04-30 10:41:38 -06:00
James Betker
64c7582bf5
full pipeline
2022-04-28 22:47:26 -06:00
James Betker
8aa6651fc7
fix surrogate loss return in waveform_gen2
2022-04-28 10:10:11 -06:00
James Betker
e208d9fb80
gate augmentations with a flag
2022-04-28 10:09:22 -06:00
James Betker
3f67cb2023
music diffusion fid adjustments
2022-04-28 10:08:55 -06:00
James Betker
ab8176b217
audio prep misc
2022-04-28 10:08:38 -06:00
James Betker
f02b01bd9d
reverse univnet classifier
2022-04-20 21:37:55 -06:00
James Betker
9df85c902e
New gen2
...
Which is basically a autoencoder with a giant diffusion appendage attached
2022-04-20 21:37:34 -06:00
James Betker
b1c2c48720
music diffusion fid
2022-04-20 00:28:03 -06:00
James Betker
084b1c1527
file splitter
2022-04-20 00:27:49 -06:00
James Betker
b4549eed9f
uv2 fix
2022-04-20 00:27:38 -06:00
James Betker
24fdafd855
fix2
2022-04-20 00:03:29 -06:00
James Betker
0af0051399
fix
2022-04-20 00:01:57 -06:00
James Betker
419f4d37bd
gen2 music
2022-04-19 23:38:37 -06:00
James Betker
c85ab738c5
paired fix
2022-04-16 23:41:57 -06:00
James Betker
8fe0dff33c
support tts typing
2022-04-16 23:36:57 -06:00
James Betker
48cb6a5abd
misc
2022-04-16 20:28:04 -06:00
James Betker
147478a148
cvvp
2022-04-16 20:27:46 -06:00
James Betker
546ecd5aeb
music!
2022-04-15 21:21:37 -06:00
James Betker
254357724d
gradprop
2022-04-15 09:37:20 -06:00
James Betker
fbf1f4f637
update
2022-04-15 09:34:44 -06:00
James Betker
82aad335ba
add distributued logic for loss
2022-04-15 09:31:48 -06:00
James Betker
efe12cb816
Update clvp to add masking probabilities in conditioning and to support code inputs
2022-04-15 09:11:23 -06:00
James Betker
3cad1b8114
more fixes
2022-04-11 15:18:44 -06:00
James Betker
6dea7da7a8
another fix
2022-04-11 12:29:43 -06:00
James Betker
f2c172291f
fix audio_diffusion_fid for autoregressive latent inputs
2022-04-11 12:08:15 -06:00
James Betker
8ea5c307fb
Fixes for training the diffusion model on autoregressive inputs
2022-04-11 11:02:44 -06:00
James Betker
a3622462c1
Change latent_conditioner back
2022-04-11 09:00:13 -06:00
James Betker
03d0b90bda
fixes
2022-04-10 21:02:12 -06:00
James Betker
19ca5b26c1
Remove flat0 and move it into flat
2022-04-10 21:01:59 -06:00
James Betker
81c952a00a
undo relative
2022-04-08 16:32:52 -06:00
James Betker
944b4c3335
more undos
2022-04-08 16:31:08 -06:00
James Betker
032983e2ed
fix bug and allow position encodings to be trained separately from the rest of the model
2022-04-08 16:26:01 -06:00
James Betker
09ab1aa9bc
revert rotary embeddings work
...
I'm not really sure that this is going to work. I'd rather explore re-using what I've already trained
2022-04-08 16:18:35 -06:00
James Betker
2fb9ffb0aa
Align autoregressive text using start and stop tokens
2022-04-08 09:41:59 -06:00
James Betker
628569af7b
Another fix
2022-04-08 09:41:18 -06:00
James Betker
423293e518
fix xtransformers bug
2022-04-08 09:12:46 -06:00
James Betker
048f6f729a
remove lightweight_gan
2022-04-07 23:12:08 -07:00
James Betker
e634996a9c
autoregressive_codegen: support key_value caching for faster inference
2022-04-07 23:08:46 -07:00
James Betker
d05e162f95
reformat x_transformers
2022-04-07 23:08:03 -07:00
James Betker
7c578eb59b
Fix inference in new autoregressive_codegen
2022-04-07 21:22:46 -06:00
James Betker
3f8d7955ef
unified_voice with rotary embeddings
2022-04-07 20:11:14 -06:00
James Betker
573e5552b9
CLVP v1
2022-04-07 20:10:57 -06:00
James Betker
71b73db044
clean up
2022-04-07 11:34:10 -06:00
James Betker
6fc4f49e86
some dumb stuff
2022-04-07 11:32:34 -06:00
James Betker
e6387c7613
Fix eval logic to not run immediately
2022-04-07 11:29:57 -06:00
James Betker
305dc95e4b
cg2
2022-04-06 21:24:36 -06:00
James Betker
e011166dd6
autoregressive_codegen r3
2022-04-06 21:04:23 -06:00
James Betker
33ef17e9e5
fix context
2022-04-06 00:45:42 -06:00
James Betker
37bdfe82b2
Modify x_transformers to do checkpointing and use relative positional biases
2022-04-06 00:35:29 -06:00
James Betker
09879b434d
bring in x_transformers
2022-04-06 00:21:58 -06:00
James Betker
3d916e7687
Fix evaluation when using multiple batch sizes
2022-04-05 07:51:09 -06:00
James Betker
572d137589
track iteration rate
2022-04-04 12:33:25 -06:00
