Test Failure Investigation - Run #1302

Summary

Daily Tests run 34293967852 failed in two independent long-test jobs. The MI200 gfx90a GPU full-system smoke test failed after a cold boot, while the MI355X gfx950 test in the same job passed. Separately, the x86 boot-test job could not download a required build artifact after five retries.

Failure Details

• Run: 34293967852
• Commit: d670200
• Trigger: workflow_dispatch

Failed Jobs and Errors

• long-tests (gem5/x86_boot_tests): actions/download-artifact v8 failed with: Unable to download and extract artifact: Artifact download failed after 5 retries. No test-results files were produced.
• long-tests (gem5/gpu): gpu-fs-mi200-stdlib-driver-ALL-x86_64-opt and its MatchFileRegex verifier failed; the paired MI355X test and verifier passed. The GPU job reported 2 passed and 2 failed tests after approximately 43000 seconds.

Root Cause Analysis

• long-tests (gem5/x86_boot_tests): Infrastructure/artifact-service failure before tests started. Multiple artifacts were discovered and several downloads completed with matching SHA-256 digests, but a required artifact exhausted the action retry budget. This is unrelated to x86 boot-test source behavior.
• long-tests (gem5/gpu): The new long MI200 standard-library driver smoke test did not emit the required GPU full-system test passed: gfx90a serial marker. The test starts configs/example/gem5_library/x86-mi200-gpu.py with --cpu-type atomic --num-cus 4, then runs tests/gem5/gpu/gpu_fs_check.sh gfx90a. That script fails when rocminfo fails or its output does not contain gfx90a. The MI355X control test passed, confirming the shared TestLib registration, ALL build, and basic GPUFS execution path work.

Reproduction Steps

long-tests (gem5/x86_boot_tests)

This failure is in artifact retrieval, not the suite. Re-run the affected workflow job after confirming GitHub Actions artifact availability.

long-tests (gem5/gpu)

1. Build an ALL optimized binary with GPU_VIPER support.
2. From tests, select the long GPU suite with ./main.py list --isa=ALL --length=long -q --suites.
3. Run its gpu-fs-mi200-stdlib-driver-ALL-x86_64-opt suite, using --skip-build only when the matching ALL binary already exists.
4. Inspect the test simerr.txt, simout.txt, and serial output. The expected marker is produced by tests/gem5/gpu/gpu_fs_check.sh only when rocminfo reports gfx90a.

Investigation Findings

The failed-job log reports only TestLib status for the MI200 simulation. The uploaded GPU result artifact could not be retrieved in this investigation environment, so the guest-level rocminfo or simulator diagnostic from simerr.txt is not available here. The next reproduction should retain those files to distinguish driver load failure from ROCr device enumeration failure.

Recommended Actions

• Re-run the x86 boot-test job to clear the artifact-service failure; investigate workflow artifact handling only if it recurs.
• Reproduce the MI200 suite and inspect its simerr.txt and simout.txt to identify why rocminfo does not report gfx90a.
• Review the MI200 initialization in src/python/gem5/components/devices/gpus/amdgpu.py and the common board setup in configs/example/gem5_library/x86_gpu.py against the working MI355X path, then make the smallest configuration or driver fix supported by the guest diagnostics.

Generated by Test Failure Doctor · auto · 35.6 AIC · ⌖ 10.8 AIC · ⊞ 20.3K · ◷