Restore latest and interval training checkpoints in PTL path by Copilot · Pull Request #847 · roboflow/rf-detr

Copilot · 2026-03-22T09:51:39Z

In v1.6.0, training stopped producing checkpoint.pth (latest) and checkpoint_<N>.pth (interval), leaving only best-model artifacts (checkpoint_best_*). This broke expected resume/retention behavior for epoch snapshots and checkpoint_interval.

Checkpoint behavior restoration
- Uses native PyTorch Lightning ModelCheckpoint callbacks to save:
  - latest rolling checkpoint each epoch (filename="last", every_n_epochs=1, save_top_k=1)
  - interval archive checkpoints (filename="checkpoint_{epoch}", every_n_epochs=checkpoint_interval, save_top_k=-1)
- These are full PTL resume checkpoints.
Trainer wiring
- build_trainer(...) now always registers the two PTL ModelCheckpoint callbacks with:
  - dirpath=train_config.output_dir
  - every_n_epochs=train_config.checkpoint_interval for interval snapshots
- Preserves existing best-checkpoint logic (checkpoint_best_regular.pth, checkpoint_best_ema.pth, checkpoint_best_total.pth) unchanged.
Targeted regression coverage
- Updated tests to validate:
  - latest ModelCheckpoint callback is present (every_n_epochs == 1)
  - interval ModelCheckpoint callback is present and uses checkpoint_interval
  - TrainConfig validation rejects invalid checkpoint_interval values

# trainer.py (callback stack)
callbacks.append(
    ModelCheckpoint(
        dirpath=tc.output_dir,
        filename="last",
        every_n_epochs=1,
        save_top_k=1,
        enable_version_counter=False,
        auto_insert_metric_name=False,
        verbose=False,
    )
)

callbacks.append(
    ModelCheckpoint(
        dirpath=tc.output_dir,
        filename="checkpoint_{epoch}",
        every_n_epochs=tc.checkpoint_interval,
        save_top_k=-1,
        enable_version_counter=False,
        auto_insert_metric_name=False,
        verbose=False,
    )
)

Original prompt

This section details on the original issue you should resolve

<issue_title>v1.6.0: lack of latest epoch checkpoint and interval checkpoint file</issue_title>
<issue_description>### Search before asking

I have searched the RF-DETR issues and found no similar bug report.

Bug

I found bug. The new version has not saved the latest epoch checkpoint and interval checkpoint (defined by 'checkpoint_interval' ). The system just make checkpoint_best_total.pth, checkpoint_best_regular.pth, checkpoint_best_ema.pth

We espcially need the interval checkpoints. Please check

Environment

rfdetr: 1.6.0

OS: windows11

RTX 3090

Python 3.10

Minimal Reproducible Example

parameters= {'dataset_dir': cfg.dataset_dir, 'epochs': cfg.epochs, 'batch_size': cfg.batch_size, 'grad_accum_steps': cfg.grad_accum_steps, 'lr': cfg.lr, 'progress_bar': cfg.progress_bar,
'checkpoint_interval': cfg.checkpoint_interval, 'output_dir': cfg.output_dir, 'early_stopping': False, 'run_test': False, 'resume': cfg.checkpoint}

model.train(**parameters)

Additional

No response

Are you willing to submit a PR?

Yes, I'd like to help by submitting a PR!</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes v1.6.0: lack of latest epoch checkpoint and interval checkpoint file #843

📱 Kick off Copilot coding agent tasks wherever you are with GitHub Mobile, available on iOS and Android.

Co-authored-by: Borda <6035284+Borda@users.noreply.github.com> Agent-Logs-Url: https://github.com/roboflow/rf-detr/sessions/cb6dfc2c-b6e2-4937-ad66-28c1ef138f30

Copilot

Pull request overview

Restores legacy-style “latest” and “interval” training checkpoint artifacts in the PyTorch Lightning training path so users can resume from epoch snapshots (checkpoint.pth, checkpoint_<N>.pth) in addition to best-model checkpoints.

Changes:

Added RFDETRTrainCheckpointCallback to write a full PTL checkpoint to checkpoint.pth every epoch and to checkpoint_<N>.pth at checkpoint_interval boundaries.
Wired the new callback into build_trainer(...) unconditionally and exported it from rfdetr.training.callbacks.
Added regression tests covering latest/interval checkpoint behavior and trainer wiring.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`src/rfdetr/training/callbacks/train_checkpoint.py`	Implements new callback that saves latest + interval PTL checkpoints.
`src/rfdetr/training/trainer.py`	Always registers the new checkpoint callback in the standard trainer callback stack.
`src/rfdetr/training/callbacks/__init__.py`	Re-exports the new callback for a consistent callbacks module surface.
`tests/training/test_train_checkpoint_callback.py`	Unit tests for latest/interval checkpoint save behavior and global-rank gating.
`tests/training/test_build_trainer.py`	Ensures the trainer includes the callback and threads `checkpoint_interval` correctly.

codecov · 2026-03-22T14:43:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76%. Comparing base (6c06c38) to head (6f7c6e1).
⚠️ Report is 2 commits behind head on develop.

Additional details and impacted files

@@          Coverage Diff           @@
##           develop   #847   +/-   ##
======================================
  Coverage       76%    76%           
======================================
  Files           94     94           
  Lines         7269   7271    +2     
======================================
+ Hits          5502   5508    +6     
+ Misses        1767   1763    -4

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…point callback trainer.save_checkpoint() calls strategy.barrier(), a collective that requires all ranks to participate. The early return on non-global-zero ranks caused rank 0 to hang waiting for peers that had already exited. Removes the guard (PTL's DDP strategy already ensures the file write happens only on rank 0), replaces the test that validated the broken behaviour, and adds a test covering output_dir auto-creation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Removes ~60 lines of custom RFDETRTrainCheckpointCallback in favour of two standard ModelCheckpoint instances — one for latest (last.ckpt, every epoch) and one for interval archives (checkpoint_{epoch}.ckpt). Using .ckpt extension eliminates the naming collision with weight-only .pth artifacts from BestModelCallback. Also adds ge=1 validation to TrainConfig.checkpoint_interval (previously only validated inside the deleted callback's __init__). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Borda

@copilot I revaamped you implementation and the pivodted how to address the issue, update PR description according to the implementation

Copilot · 2026-03-23T10:11:46Z

@copilot I revaamped you implementation and the pivodted how to address the issue, update PR description according to the implementation

Updated the PR description to match the current implementation pivot (native PTL ModelCheckpoint for latest + interval checkpoints, with best-checkpoint flow unchanged). Addressed in 1400d71 (with description clarification in f2754fa).

* fix: restore latest and interval training checkpoints * refactor: replace custom checkpoint callback with PTL ModelCheckpoint --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Borda <6035284+Borda@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Initial plan

e16de85

Copilot AI assigned Copilot and Borda Mar 22, 2026

Copilot started work on behalf of Borda March 22, 2026 09:51 View session

fix: restore latest and interval training checkpoints

72c2af3

Co-authored-by: Borda <6035284+Borda@users.noreply.github.com> Agent-Logs-Url: https://github.com/roboflow/rf-detr/sessions/cb6dfc2c-b6e2-4937-ad66-28c1ef138f30

Copilot AI changed the title ~~[WIP] Fix missing latest epoch and interval checkpoint files~~ Restore latest and interval training checkpoints in PTL path Mar 22, 2026

Copilot AI requested a review from Borda March 22, 2026 10:01

Copilot finished work on behalf of Borda March 22, 2026 10:01

Borda marked this pull request as ready for review March 22, 2026 12:15

Borda requested review from SkalskiP, isaacrob and probicheaux as code owners March 22, 2026 12:15

Copilot AI review requested due to automatic review settings March 22, 2026 12:15

Copilot started reviewing on behalf of Borda March 22, 2026 12:15 View session

Copilot AI reviewed Mar 22, 2026

View reviewed changes

Comment thread src/rfdetr/training/callbacks/train_checkpoint.py Outdated

Borda requested a review from Copilot March 22, 2026 18:26

Copilot started reviewing on behalf of Borda March 22, 2026 18:27 View session

Copilot AI reviewed Mar 22, 2026

View reviewed changes

Comment thread tests/training/test_train_checkpoint_callback.py Outdated

Comment thread src/rfdetr/training/callbacks/train_checkpoint.py Outdated

Comment thread src/rfdetr/training/callbacks/train_checkpoint.py Outdated

Comment thread tests/training/test_train_checkpoint_callback.py Outdated

Borda requested a review from Copilot March 22, 2026 19:27

Copilot started reviewing on behalf of Borda March 22, 2026 19:27 View session

Borda and others added 3 commits March 22, 2026 20:28

Update src/rfdetr/training/callbacks/train_checkpoint.py

c33dcf1

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/rfdetr/training/callbacks/train_checkpoint.py

fe6aacd

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

fix(pre-commit): 🎨 auto format pre-commit hooks

502d2cd

Copilot AI reviewed Mar 22, 2026

View reviewed changes

Borda and others added 2 commits March 22, 2026 20:42

Clarify PTL checkpoint semantics

f2754fa

Borda reviewed Mar 23, 2026

View reviewed changes

Copilot started work on behalf of Borda March 23, 2026 10:09 View session

Merge branch 'develop' into copilot/fix-latest-epoch-checkpoint

6f7c6e1

Copilot AI requested a review from Borda March 23, 2026 10:12

Copilot finished work on behalf of Borda March 23, 2026 10:12

Borda approved these changes Mar 23, 2026

View reviewed changes

Borda merged commit 3b377ef into develop Mar 23, 2026
23 checks passed

Borda deleted the copilot/fix-latest-epoch-checkpoint branch March 23, 2026 10:43

Copilot AI mentioned this pull request Mar 24, 2026

Avoid duplicate ModelCheckpoint state keys when checkpoint_interval=1 #859

Merged

Borda mentioned this pull request Mar 25, 2026

releasing 1.6.1 [rebase & merge] #869

Merged

omkar-334 mentioned this pull request Mar 27, 2026

checkpoint.pth not generated in v1.6.1 despite checkpointing working per epoch #881

Closed

2 tasks

Navigation Menu

Restore latest and interval training checkpoints in PTL path - #847

Restore latest and interval training checkpoints in PTL path#847
Borda merged 9 commits into
developfrom
copilot/fix-latest-epoch-checkpoint

Copilot AI commented Mar 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

codecov Bot commented Mar 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Borda left a comment

Uh oh!

Copilot AI commented Mar 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug

Environment

Minimal Reproducible Example

Additional

Are you willing to submit a PR?

Comments on the Issue (you are @copilot in this section)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

codecov Bot commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Borda left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Mar 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Mar 22, 2026 •

edited

Loading

codecov Bot commented Mar 22, 2026 •

edited

Loading