Skip to content

fix(export): pass CUDA device to ONNX Runtime benchmark - #1346

Merged
Borda merged 1 commit into
roboflow:developfrom
voropaevv:fix/1345-onnx-benchmark-cuda-device
Aug 15, 2026
Merged

Borda merged 1 commit into
roboflow:developfrom
voropaevv:fix/1345-onnx-benchmark-cuda-device

Conversation

@voropaevv

Copy link
Copy Markdown
Contributor

Why

The ONNX benchmark documents device as the CUDA device index and routes PyTorch post-processing to cuda:{device}, but the ONNX Runtime session did not receive that device index.

Because the CUDA Execution Provider defaults device_id to 0, requesting a non-default logical device could run ONNX model execution on logical GPU 0 and PyTorch post-processing on another GPU.

A two-GPU Tesla T4 reproduction with device=1 confirmed that the current implementation selected ORT device 0 and allocated memory on both GPUs.

Fixes #1345

What changed

  • pass the requested logical CUDA index to the ONNX Runtime CUDA Execution Provider
  • add regression coverage for both the default device and a non-default logical device
  • verify that ONNX Runtime and PyTorch post-processing receive the same requested device index

Verification

  • regression test against the unmodified parent:
    • 2 expected failures because the provider omitted device_id
  • hard-coded device_id=0 mutant:
    • rejected by the non-default-device regression case
  • focused benchmark tests:
    • 13 passed
  • full CPU suite:
    • 3910 passed, 76 skipped
  • git diff --check
  • applicable formatting, Ruff, docformatter, codespell, license, and security pre-commit hooks passed

Real CUDA validation:

  • RTX 4090:
    • CUDA Execution Provider active
    • six profiled CUDA nodes
    • identical parent/candidate raw and post-processing outputs
  • 2 × Tesla T4, requested device=1, two repetitions:
    • parent reported device_id=0
    • this branch reported device_id=1
    • parent memory delta: [126, 140] MiB across GPUs 0 and 1
    • this branch memory delta: [0, 162] MiB
    • 18 profiled CUDA node events per run
    • identical raw and post-processing output hashes across both implementations and repetitions

Known local baseline:

  • project-wide mypy reports four errors in unchanged files:
    src/rfdetr/export/_topk.py:31-32 and
    src/rfdetr/evaluation/coco_eval.py:178,190
  • the same four errors reproduce on the clean parent

Not verified

  • the full upstream CUDA/TensorRT CI matrix
  • TensorRT engine execution
  • Jetson
  • a full RF-DETR checkpoint/COCO benchmark

The hardware tests used the exact production session-construction and post-processing path with deterministic ONNX graphs to isolate CUDA device routing.

Risks / rollback

The change is limited to one existing ONNX Runtime provider option. The default device=0 behavior is preserved. The TensorRT path is unchanged.

Reverting the commit restores the previous provider configuration.

Migrations / external effects

None.

Screenshots / preview

Not applicable; this changes CUDA device routing in the command-line benchmark.

Pass the requested logical CUDA device to the ONNX Runtime execution provider so model execution stays aligned with PyTorch post-processing. Add CPU-only regression coverage for default and non-default device indices.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes ONNX benchmark CUDA routing so ONNX Runtime and PyTorch post-processing use the same requested GPU.

Changes:

  • Passes device_id to the CUDA Execution Provider.
  • Adds regression tests for default and non-default devices.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/rfdetr/export/benchmark.py Configures ONNX Runtime with the requested CUDA device.
tests/inference/test_trt_inference.py Verifies consistent device routing.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@codecov

codecov Bot commented Aug 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85%. Comparing base (a981c6d) to head (eef3fb2).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files
@@           Coverage Diff           @@
##           develop   #1346   +/-   ##
=======================================
  Coverage       85%     85%           
=======================================
  Files          111     111           
  Lines        14058   14058           
=======================================
+ Hits         11983   11998   +15     
+ Misses        2075    2060   -15     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Borda
Borda merged commit 65bd85b into roboflow:develop Aug 15, 2026
39 checks passed
@Borda Borda added the bug Something isn't working label Aug 15, 2026
Borda pushed a commit that referenced this pull request Aug 17, 2026
Pass the requested logical CUDA device to the ONNX Runtime execution provider so model execution stays aligned with PyTorch post-processing. Add CPU-only regression coverage for default and non-default device indices.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ONNX benchmark ignores the requested non-default CUDA device

3 participants