feat(cookbooks): add inference latency benchmark notebook by Borda · Pull Request #1152 · roboflow/rf-detr

Borda · 2026-06-22T20:53:40Z

What does this PR do?

New inference-latency-benchmark.ipynb benchmarks FP32 / FP16+JIT / ONNX across detection (RFDETRMedium), segmentation (RFDETRSegSmall), and keypoint (RFDETRKeypointPreview); downloads COCO-128 from Universe for sample images; CUDA event timing
cards.yaml: add benchmark card with INFERENCE / ONNX / GPU labels
export/_onnx/inference.py: fix silent CPU fallback in _create_onnx_session — now explicitly selects CUDAExecutionProvider when available; warns when falling back to CPU
export/main.py: add weights_only=False to torch.load() for PyTorch ≥2.6 compatibility; relax strict=True → strict=False to match checkpoint key handling in r-flow
detr.py: switch predict() decorator to @torch.inference_mode(); batch GPU transfer, resize, and normalize outside per-image loop; update warning to mention optimize_for_inference(dtype=torch.float16)

Additional Context

                    RFDETRMedium  RFDETRSegSmall  RFDETRKeypointPreview
Config \ Model                                                         
predict() FP32              33.2            29.5                   21.3
predict() FP16+JIT          52.4            47.8                   38.0
ONNX (CPU)                   4.4             4.2                    2.0
ONNX (CUDA)                 87.4            99.8                   36.1

FPS — 100 timed + 20 warmup runs, batch 1, GPU: NVIDIA L4.

- New `inference-latency-benchmark.ipynb` benchmarks FP32 / FP16+JIT / ONNX across detection (RFDETRMedium), segmentation (RFDETRSegSmall), and keypoint (RFDETRKeypointPreview); downloads COCO-128 from Universe for sample images; CUDA event timing - `cards.yaml`: add benchmark card with INFERENCE / ONNX / GPU labels - `export/_onnx/inference.py`: fix silent CPU fallback in `_create_onnx_session` — now explicitly selects CUDAExecutionProvider when available; warns when falling back to CPU - `export/main.py`: add `weights_only=False` to `torch.load()` for PyTorch ≥2.6 compatibility; relax `strict=True` → `strict=False` to match checkpoint key handling in r-flow - `detr.py`: switch `predict()` decorator to `@torch.inference_mode()`; batch GPU transfer, resize, and normalize outside per-image loop; update warning to mention `optimize_for_inference(dtype=torch.float16)` --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

`folder` is invalid for object-detection projects; switch to `coco`. --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Copilot

Pull request overview

This PR adds a new cookbooks notebook to benchmark RF-DETR inference latency (FP32 vs FP16+JIT vs ONNX Runtime) and includes supporting runtime/export tweaks to improve inference throughput and make ONNX execution provider selection explicit.

Changes:

Add a new cookbook notebook for GPU latency/FPS benchmarking across detection, segmentation, and keypoint models, and surface it via a new cookbooks card.
Update ONNX Runtime session creation to prefer CUDAExecutionProvider and warn when falling back to CPU.
Optimize RFDETR.predict() for inference (switch to torch.inference_mode() and batch preprocessing) and adjust export CLI checkpoint loading behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`src/rfdetr/export/main.py`	Adjusts checkpoint loading (`torch.load`) and relaxes `load_state_dict` strictness during export resume.
`src/rfdetr/export/_onnx/inference.py`	Explicitly selects ONNX Runtime providers (CUDA preferred) and logs providers in use.
`src/rfdetr/detr.py`	Switches `predict()` to inference mode and refactors preprocessing to batch operations.
`docs/cookbooks/inference-latency-benchmark.ipynb`	New benchmark notebook measuring latency/FPS across FP32, FP16+JIT, and ONNX.
`docs/cookbooks/cards.yaml`	Adds a new cookbook card linking to the benchmark notebook.

Roboflow export is async — zip not ready on first download call. For a latency benchmark image content is irrelevant; switch to numpy random 640×640 RGB images. Removes roboflow + API-key dependency from notebook. --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

- Extracted ONNX export logic to `_export_onnx`. - Generalized ONNX runtime benchmarking to support multiple providers. - Replaced plain output formatting with a pandas DataFrame for cleaner summary display. - Added `pandas` as a dependency.

…sing - Add `_onnx_runtime(path, image, providers, warmup, runs)` to `export/_onnx/inference.py` — returns `(mean_ms, std_ms, provider_label)`; raises RuntimeError if requested provider fell back silently (fixes double "ONNX (CPU)" rows in benchmark output) - Extract `_preprocess_pil_to_nchw` — eliminates duplicated ImageNet normalize+NCHW block shared between `_run_inference` and `_onnx_runtime` - Extend `_create_onnx_session` with optional `providers` param; explicit list bypasses auto-selection - Cookbook: remove `_onnx_rt` wrapper; import `_onnx_runtime` directly from package; ONNX providers loop replaces two separate calls --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

CUDA RuntimeError was aborting the entire ONNX loop — CPU result survived but CUDA row was silently dropped. Now each provider has its own try/except so CPU and CUDA failures are independent and clearly labelled. --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

… provider torch.cuda.is_available() can be True while onnxruntime-gpu is absent — ORT then silently uses CPU for a CUDA-requested session, producing duplicate ONNX (CPU) rows. Raise immediately on provider mismatch so the caller (cookbook torch.cuda check skips no-GPU, this raises missing-package). --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

…pu on Colab Colab pre-installs onnxruntime (CPU-only); pip install onnxruntime-gpu leaves both installed and ORT picks CPU. Uninstall both first to ensure GPU build wins. --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

…ark starts Fail fast in section 2 if onnxruntime-gpu is absent or CPU-only build is active, with a clear fix command — avoids silent CPU fallback mid-run. --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

… notebook Removed an outdated RF-DETR copyright header block from the notebook to streamline its content.

--- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

…check Switch install cell to onnxruntime-cuda-12 index (PyPI default is CUDA 11.8 build which silently falls back on Colab CUDA 12.8). Remove broken session probe — _onnx_runtime already raises clearly on provider init failure. --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

…ncy notebook

Add explanatory paragraphs to all 8 markdown sections in inference-latency-benchmark — covering CUDA-12 index rationale, warmup/measurement runs, synthetic image equivalence, CUDA event timing accuracy, FP32/FP16+JIT/ONNX path tradeoffs, provider loop design, model family task-complexity breakdown, and how to read the FPS summary table. --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

… fix/export

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

…puts - detr.py: move per-image .to(device) inside loop and resize before torch.stack; previous batch-first approach raised RuntimeError when images had different sizes - export/main.py: add missing-key warning for strict=False load; add explanatory comment for weights_only=False matching pattern at detr.py:354 - export/_onnx/inference.py: fix _onnx_rt -> _onnx_runtime in docstring example; pass channels from ONNX metadata to _preprocess_pil_to_nchw; remove unverified CUDA sync claim from _onnx_runtime docstring; fix broken session example; add section comment marking _onnx_runtime as benchmarking-only --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

codecov · 2026-06-23T18:25:36Z

Codecov Report

❌ Patch coverage is 15.68627% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 81%. Comparing base (dc79e78) to head (b2e2499).

❌ Your patch check has failed because the patch coverage (16%) is below the target coverage (95%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@           Coverage Diff           @@
##           develop   #1152   +/-   ##
=======================================
- Coverage       82%     81%   -0%     
=======================================
  Files          110     110           
  Lines        11459   11488   +29     
=======================================
  Hits          9350    9350           
- Misses        2109    2138   +29

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…no-any-return `np.ndarray.__getitem__(np.newaxis)` returns `Any` in numpy stubs; `np.expand_dims` is fully typed, resolving the `no-any-return` error at `_preprocess_pil_to_nchw` line 122. --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Borda requested review from SkalskiP, isaacrob and probicheaux as code owners June 22, 2026 20:53

Borda requested a review from Copilot June 22, 2026 20:53

Copilot started reviewing on behalf of Borda June 22, 2026 20:54 View session

fix(cookbooks): use coco format for Universe dataset download

c5ea065

`folder` is invalid for object-detection projects; switch to `coco`. --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Copilot AI reviewed Jun 22, 2026

View reviewed changes

Comment thread src/rfdetr/detr.py

Comment thread src/rfdetr/export/main.py

Comment thread docs/cookbooks/inference-latency-benchmark.ipynb Outdated

Comment thread docs/cookbooks/inference-latency-benchmark.ipynb

Borda and others added 15 commits June 22, 2026 23:04

fix(cookbooks): remove unused copyright header from inference latency…

5c34393

… notebook Removed an outdated RF-DETR copyright header block from the notebook to streamline its content.

docs(cookbooks): add Colab restart note after install cell

09ef0da

--- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

docs(cookbooks): remove outdated copyright header from inference late…

4c9f568

…ncy notebook

Merge branch 'develop' into fix/export

1942fbb

Merge branch 'fix/export' of https://github.com/roboflow/rf-detr into…

eb50193

… fix/export

Borda requested a review from Copilot June 23, 2026 15:44

Copilot started reviewing on behalf of Borda June 23, 2026 15:45 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread src/rfdetr/export/main.py

Comment thread src/rfdetr/export/_onnx/inference.py

Comment thread src/rfdetr/export/_onnx/inference.py

Comment thread docs/cookbooks/inference-latency-benchmark.ipynb

Borda merged commit 70d968e into develop Jun 23, 2026
26 checks passed

Borda deleted the fix/export branch June 23, 2026 19:51

Borda added the enhancement New feature or request label Jun 23, 2026

Borda mentioned this pull request Jun 25, 2026

releasing 1.8.2 #1162

Merged

Navigation Menu

feat(cookbooks): add inference latency benchmark notebook - #1152

feat(cookbooks): add inference latency benchmark notebook#1152
Borda merged 19 commits into
developfrom
fix/export

Borda commented Jun 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jun 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Borda commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Additional Context

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Borda commented Jun 22, 2026 •

edited

Loading

codecov Bot commented Jun 23, 2026 •

edited

Loading