Add in-place inference optimization - #1089
Merged
Borda merged 7 commits intoJun 27, 2026
Merged
Conversation
Codecov Report❌ Patch coverage is ❌ Your project check has failed because the head coverage (82%) is below the target coverage (95%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## develop #1089 +/- ##
========================================
+ Coverage 80% 82% +2%
========================================
Files 110 110
Lines 11879 11901 +22
========================================
+ Hits 9468 9748 +280
+ Misses 2411 2153 -258 🚀 New features to boost your workflow:
|
pirnerjonas
marked this pull request as ready for review
June 5, 2026 12:18
pirnerjonas
requested review from
Borda,
SkalskiP,
isaacrob and
probicheaux
as code owners
June 5, 2026 12:18
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a supported low-memory inference-only workflow by adding an inplace option to RFDETR.optimize_for_inference(), allowing optimization to reuse the already-loaded module (instead of deep-copying it) and freeing the base model reference to reduce peak/steady-state memory.
Changes:
- Added
inplace: bool = Falsetooptimize_for_inference()with validation (inplace=Truerequirescompile=False) and a new internal flag_optimized_inplace. - Implemented the destructive in-place inference optimization path that exports/casts the loaded module and clears
self.model.modelon success. - Added tests and documentation snippets demonstrating
optimize_for_inference(compile=False, inplace=True)for memory-constrained inference.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tests/models/test_optimize_for_inference.py | Adds test coverage for new inplace behavior, error cases, and state flags. |
| src/rfdetr/detr.py | Implements inplace optimization path, new _optimized_inplace state, and updated docs/errors. |
| docs/learn/run/segmentation.md | Documents in-place optimization for memory-constrained segmentation inference. |
| docs/learn/run/detection.md | Documents in-place optimization for memory-constrained detection inference. |
- Fix flag ordering: _optimized_inplace set before model.model=None so
exception recovery sees correct state if failure occurs between the two
- Add RuntimeError guard for second optimize_for_inference() call after
inplace (M6) — model.model is None, create a new instance
- Add export() guard: raises RuntimeError immediately after inplace instead
of crashing on model.model.to("cpu")
- Change remove_optimized_model() after inplace from RuntimeError to
UserWarning + no-op (M5), preserving optimized state
- Add is_optimized_inplace property (M4)
- Improve compile=True+inplace=True error message explaining why jit.trace
retains weight storage refs
- Update predict() resolution/batch mismatch messages to hint "create new
instance" after inplace instead of "call remove_optimized_model()"
- Docstring: note 1.5× transient peak for dtype conversion, export()
mutation caveat, restructure Examples to show non-inplace path first
- Docs: add dtype="float16" to inplace snippet and irreversibility note
- Tests: update remove_optimized_model inplace test to expect UserWarning,
add float32 no-op test, second-optimize guard test, mutation-not-undone
test, and _optimized_has_been_compiled/_optimized_batch_size assertions
---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Borda
approved these changes
Jun 27, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a supported low-memory inference-only path via
optimize_for_inference(..., inplace=True).inplace=Falsebehavior unchanged: the loaded model is deep-copied before export, soremove_optimized_model()can still return the instance to the unoptimized state.inplace=Trueexports and casts the loaded module itself, stores it asmodel.inference_model, and clearsmodel.modelafter optimization succeeds.inplace=Truewithcompile=True, since tracing would create another module and undermine the low-memory path.compile=False, inplace=Trueusage for detection and segmentation inference.Issue
Closes #1080
Validation
RFDETRNano(device="cpu"),compile=False, and isolated subprocesses: