mrq/ai-voice-cloning
31
74
Fork 43
Code Issues 223 Pull Requests 5 Packages Projects Releases Wiki Activity

Error compile with TORCH_USE_CUDA_DSA #452

New Issue
Closed
opened 2023-11-16 07:33:33 +00:00 by Aeternus · 2 comments
Aeternus commented 2023-11-16 07:33:33 +00:00
Copy Link

[Training] [2023-11-16T07:32:02.763633] warnings.warn("Detected call of lr_scheduler.step() before optimizer.step(). "
[Training] [2023-11-16T07:32:03.729833] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [32,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.729883] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [33,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.729894] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [34,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.729925] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [35,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.729943] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [36,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.729959] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [37,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.729975] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [38,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.729998] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [39,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730025] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [40,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730041] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [41,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730066] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [42,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730093] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [43,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730116] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [44,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730142] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [45,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730158] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [46,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730194] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [47,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730217] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [48,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730237] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [49,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730248] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [50,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730267] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [51,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730287] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [52,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730304] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [53,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730315] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [54,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730331] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [55,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730341] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [56,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730367] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [57,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730385] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [58,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730411] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [59,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730430] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [60,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730456] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [61,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730474] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [62,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730500] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [63,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730518] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [0,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730544] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [1,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730563] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [2,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730584] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [3,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730603] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [4,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730623] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [5,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730649] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [6,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730675] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [7,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730693] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [8,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730710] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [9,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730725] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [10,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730745] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [11,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730773] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [12,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730810] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [13,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730832] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [14,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730857] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [15,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730881] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [16,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730899] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [17,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730922] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [18,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730939] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [19,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730953] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [20,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730971] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [21,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.730986] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [22,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731004] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [23,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731026] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [24,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731044] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [25,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731065] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [26,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731082] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [27,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731096] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [28,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731112] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [29,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731128] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [30,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731147] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [31,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731162] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [0,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731181] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [1,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731203] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [2,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731220] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [3,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731240] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [4,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731256] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [5,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731270] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [6,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731286] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [7,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731301] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [8,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731317] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [9,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731331] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [10,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731347] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [11,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731361] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [12,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731377] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [13,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731391] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [14,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731406] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [15,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731420] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [16,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731436] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [17,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731461] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [18,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731478] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [19,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731495] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [20,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731510] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [21,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731524] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [22,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731540] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [23,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731555] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [24,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731571] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [25,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731585] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [26,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731601] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [27,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731615] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [28,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731631] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [29,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731644] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [30,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.731660] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [31,0,0] Assertion srcIndex < srcSelectDimSize failed.
[Training] [2023-11-16T07:32:03.738275] Traceback (most recent call last):
[Training] [2023-11-16T07:32:03.738308] File "/mnt/efs/tortoise/train/ai-voice-cloning/./src/train.py", line 64, in
[Training] [2023-11-16T07:32:03.739601] train(config_path, args.launcher)
[Training] [2023-11-16T07:32:03.739631] File "/mnt/efs/tortoise/train/ai-voice-cloning/./src/train.py", line 31, in train
[Training] [2023-11-16T07:32:03.740531] trainer.do_training()
[Training] [2023-11-16T07:32:03.740560] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/train.py", line 408, in do_training
[Training] [2023-11-16T07:32:03.741489] metric = self.do_step(train_data)
[Training] [2023-11-16T07:32:03.741518] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/train.py", line 271, in do_step
[Training] [2023-11-16T07:32:03.742392] gradient_norms_dict = self.model.optimize_parameters(
[Training] [2023-11-16T07:32:03.742425] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/trainer/ExtensibleTrainer.py", line 321, in optimize_parameters
[Training] [2023-11-16T07:32:03.743379] ns = step.do_forward_backward(
[Training] [2023-11-16T07:32:03.743412] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/trainer/steps.py", line 274, in do_forward_backward
[Training] [2023-11-16T07:32:03.744471] injected = inj(local_state)
[Training] [2023-11-16T07:32:03.744505] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
[Training] [2023-11-16T07:32:03.745668] return self._call_impl(*args, **kwargs)
[Training] [2023-11-16T07:32:03.745687] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
[Training] [2023-11-16T07:32:03.746775] return forward_call(*args, **kwargs)
[Training] [2023-11-16T07:32:03.746796] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/trainer/injectors/base_injectors.py", line 94, in forward
[Training] [2023-11-16T07:32:03.747851] results = method(*params, **self.args)
[Training] [2023-11-16T07:32:03.747869] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
[Training] [2023-11-16T07:32:03.749248] return self._call_impl(*args, **kwargs)
[Training] [2023-11-16T07:32:03.749264] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
[Training] [2023-11-16T07:32:03.750420] return forward_call(*args, **kwargs)
[Training] [2023-11-16T07:32:03.750437] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 183, in forward
[Training] [2023-11-16T07:32:03.751483] return self.module(*inputs[0], **module_kwargs[0])
[Training] [2023-11-16T07:32:03.751499] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
[Training] [2023-11-16T07:32:03.752499] return self._call_impl(*args, **kwargs)
[Training] [2023-11-16T07:32:03.752516] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
[Training] [2023-11-16T07:32:03.753809] return forward_call(*args, **kwargs)
[Training] [2023-11-16T07:32:03.753826] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/models/audio/tts/unified_voice2.py", line 441, in forward
[Training] [2023-11-16T07:32:03.754792] text_inputs) + self.text_pos_embedding(text_inputs)
[Training] [2023-11-16T07:32:03.754810] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
[Training] [2023-11-16T07:32:03.755936] return self._call_impl(*args, **kwargs)
[Training] [2023-11-16T07:32:03.755953] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
[Training] [2023-11-16T07:32:03.757159] return forward_call(*args, **kwargs)
[Training] [2023-11-16T07:32:03.757175] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/models/audio/tts/transformer_builders.py", line 54, in forward
[Training] [2023-11-16T07:32:03.758136] return self.emb(torch.arange(0, sl, device=x.device))
[Training] [2023-11-16T07:32:03.758164] RuntimeError: CUDA error: device-side assert triggered
[Training] [2023-11-16T07:32:03.758172] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[Training] [2023-11-16T07:32:03.758191] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[Training] [2023-11-16T07:32:03.758210] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
[Training] [2023-11-16T07:32:03.758220]

[Training] [2023-11-16T07:32:02.763633] warnings.warn("Detected call of `lr_scheduler.step()` before `optimizer.step()`. " [Training] [2023-11-16T07:32:03.729833] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.729883] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.729894] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.729925] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.729943] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.729959] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.729975] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.729998] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730025] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730041] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730066] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730093] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730116] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730142] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730158] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730194] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730217] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730237] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730248] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730267] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730287] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730304] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730315] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730331] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730341] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730367] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730385] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730411] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730430] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730456] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730474] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730500] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730518] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730544] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730563] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730584] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730603] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730623] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730649] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730675] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730693] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730710] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730725] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730745] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730773] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730810] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730832] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730857] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730881] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730899] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730922] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730939] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730953] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730971] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.730986] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731004] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731026] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731044] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731065] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731082] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731096] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731112] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731128] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731147] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [260,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731162] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731181] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731203] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731220] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731240] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731256] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731270] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731286] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731301] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731317] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731331] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731347] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731361] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731377] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731391] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731406] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731420] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731436] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731461] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731478] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731495] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731510] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731524] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731540] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731555] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731571] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731585] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731601] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731615] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731631] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731644] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.731660] ../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [258,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed. [Training] [2023-11-16T07:32:03.738275] Traceback (most recent call last): [Training] [2023-11-16T07:32:03.738308] File "/mnt/efs/tortoise/train/ai-voice-cloning/./src/train.py", line 64, in <module> [Training] [2023-11-16T07:32:03.739601] train(config_path, args.launcher) [Training] [2023-11-16T07:32:03.739631] File "/mnt/efs/tortoise/train/ai-voice-cloning/./src/train.py", line 31, in train [Training] [2023-11-16T07:32:03.740531] trainer.do_training() [Training] [2023-11-16T07:32:03.740560] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/train.py", line 408, in do_training [Training] [2023-11-16T07:32:03.741489] metric = self.do_step(train_data) [Training] [2023-11-16T07:32:03.741518] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/train.py", line 271, in do_step [Training] [2023-11-16T07:32:03.742392] gradient_norms_dict = self.model.optimize_parameters( [Training] [2023-11-16T07:32:03.742425] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/trainer/ExtensibleTrainer.py", line 321, in optimize_parameters [Training] [2023-11-16T07:32:03.743379] ns = step.do_forward_backward( [Training] [2023-11-16T07:32:03.743412] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/trainer/steps.py", line 274, in do_forward_backward [Training] [2023-11-16T07:32:03.744471] injected = inj(local_state) [Training] [2023-11-16T07:32:03.744505] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl [Training] [2023-11-16T07:32:03.745668] return self._call_impl(*args, **kwargs) [Training] [2023-11-16T07:32:03.745687] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl [Training] [2023-11-16T07:32:03.746775] return forward_call(*args, **kwargs) [Training] [2023-11-16T07:32:03.746796] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/trainer/injectors/base_injectors.py", line 94, in forward [Training] [2023-11-16T07:32:03.747851] results = method(*params, **self.args) [Training] [2023-11-16T07:32:03.747869] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl [Training] [2023-11-16T07:32:03.749248] return self._call_impl(*args, **kwargs) [Training] [2023-11-16T07:32:03.749264] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl [Training] [2023-11-16T07:32:03.750420] return forward_call(*args, **kwargs) [Training] [2023-11-16T07:32:03.750437] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 183, in forward [Training] [2023-11-16T07:32:03.751483] return self.module(*inputs[0], **module_kwargs[0]) [Training] [2023-11-16T07:32:03.751499] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl [Training] [2023-11-16T07:32:03.752499] return self._call_impl(*args, **kwargs) [Training] [2023-11-16T07:32:03.752516] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl [Training] [2023-11-16T07:32:03.753809] return forward_call(*args, **kwargs) [Training] [2023-11-16T07:32:03.753826] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/models/audio/tts/unified_voice2.py", line 441, in forward [Training] [2023-11-16T07:32:03.754792] text_inputs) + self.text_pos_embedding(text_inputs) [Training] [2023-11-16T07:32:03.754810] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl [Training] [2023-11-16T07:32:03.755936] return self._call_impl(*args, **kwargs) [Training] [2023-11-16T07:32:03.755953] File "/mnt/efs/tortoise/train/ai-voice-cloning/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl [Training] [2023-11-16T07:32:03.757159] return forward_call(*args, **kwargs) [Training] [2023-11-16T07:32:03.757175] File "/mnt/efs/tortoise/train/ai-voice-cloning/modules/dlas/dlas/models/audio/tts/transformer_builders.py", line 54, in forward [Training] [2023-11-16T07:32:03.758136] return self.emb(torch.arange(0, sl, device=x.device)) [Training] [2023-11-16T07:32:03.758164] RuntimeError: CUDA error: device-side assert triggered [Training] [2023-11-16T07:32:03.758172] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. [Training] [2023-11-16T07:32:03.758191] For debugging consider passing CUDA_LAUNCH_BLOCKING=1. [Training] [2023-11-16T07:32:03.758210] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. [Training] [2023-11-16T07:32:03.758220]
Aeternus commented 2023-11-16 11:43:18 +00:00
Author
Copy Link

Torch and Cuda are 12.1

Torch and Cuda are 12.1
Aeternus commented 2023-11-16 13:04:36 +00:00
Author
Copy Link

Problem was tokenizer

Problem was tokenizer
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
insufficient info

Need more information

  
invalid

Something is wrong

  
news

To make up for a lack of a way to provide any news-like

  
not a bug

For non-bug things

  
question

More information is needed

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
insufficient info
invalid
news
not a bug
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: mrq/ai-voice-cloning#452
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?