Assertion srcIndex < srcSelectDimSize failed in the inference.py #141
Description
When I run the inference.py on the hundreds of audio inputs with the A1A2 task, I meet the following question:
... (correct audio output)
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [16,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [16,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [16,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [16,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [16,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [16,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [16,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [16,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[error] failed to process /data/21.mp3
[error] failed to process /data/22.mp3
[error] failed to process /data/23.mp3
[error] failed to process /data/24.mp3
[error] failed to process /data/25.mp3
[error] failed to process /data/26.mp3
...
I meet the situation on two different machines; the machines' information is:
System 1:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
GPU 1:
NVIDIA H20
System 2:
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"
NAME_ORIG="CentOS Linux"
GPU 2:
Tesla T4
Has anyone encountered this problem?