CUDA error: device-side assert triggered

Error Message:

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

The above information does not make it possible to determine exactly where the error is.

Debugging Tip:
If there is a CUDA error, try using the cpu, debugging code.