Skip to content

Report exact transfer files to Transmission clients - #52

Merged
elsbrock merged 5 commits into
elsbrock:mainfrom
alexg0:alexg0/transmission-file-manifests
Sep 12, 2026
Merged

elsbrock merged 5 commits into
elsbrock:mainfrom
alexg0:alexg0/transmission-file-manifests

Conversation

@alexg0

@alexg0 alexg0 commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Return exact persisted transfer-file manifests to Transmission clients, including
numeric-ID/hash selection and restart-safe local ownership. Supports Bindery's
exact-file integration (vavallee/bindery#903 and vavallee/bindery#933).

The independent Put.io root-deletion guard and race CI
and stalled-download cancellation fix
merged on September 9. This branch now merges current main at ef8cc9e,
so the review diff excludes those landed prerequisites.

Review changes

  • Bound remote deletion to three attempts per request. Persist removal intent
    before mutation, release active in-memory tracking, and keep disk ownership
    evidence until confirmed remote absence and worker drain or successful retry.
  • Capture category ownership atomically, reject corrupt/null state, prevent old
    retries from reviving removals, and avoid holding removal locks across blocked
    queue sends or remote listing calls.
  • Document whole-transfer rather than per-file bytesCompleted granularity.
  • Reserve .plundrio-files/; coordinated feat: reconcile unmanaged downloads #51 excludes internal state and protects
    pending payloads even when their remote transfer moves or disappears.
  • Give removal regression tests a persistence-aware 10-second deadlock watchdog,
    preserving real fsync and blocked queues/listings, and join removal goroutines
    during failure cleanup so they cannot race temporary-directory removal.

Upgrade behavior / release notes

A Put.io COMPLETED/SEEDING transfer without local context now reports downloading
at 50%, not complete/seeding, until local restoration or download finishes.
Remote completion is not proof of local completion. Truly absent legacy manifests
with either FileID zero or a confirmed source-root404 enter durable NeedsReview
instead of assumed completion or a failed download. They report stopped/50%, no
files, zero rates, unknown ETA, and an explanatory warning with automatic
removal disabled. Restart, source reappearance and remote ERROR cannot release
the hold. Corrupt/unreadable/dangling manifests and missing child-folder404s
remain errors, not legacy absence.

Review markers retain original identity/category across storage repair. Ordinary
removal cannot race initial or retry-time classification. Explicit record-only
retirement requires one exact positive numeric ID, plundrio-retire-reviewed,
plundrio-copy-verified, delete-local-data=false, and fresh identity/source
validation. It never calls source/local-file deletion; failed retirement keeps
its warning and durable hold. This is documented as an operator action, not
automatic migration or proof of historical imports.

Dependencies and validation

Remaining merge order: this PR → #49 → #50 → #51, with #51 still
pending agreement on product scope. The dependent branches include the same
upstream merge and batch-removal fix through ordinary merges, without rewriting
published history.

Independent Astra implementation/review completed in separate worktrees. Local
full unit/race tests, vet and build pass at this exact PR boundary. The combined
reviewed tree passes native and NAS Linux/Alpine full unit/race tests, vet/build,
100 focused legacy/removal race repetitions and Linux ARM64 cross-compilation.
The root/removal/storage/race regressions were reproduced before their fixes.

The combined integration at 9c1a8cf was built and rolled out locally to Radarr,
Sonarr and Bindery. Repeated read-only checks retain all 38 records, verify all
25 legacy review markers, and show zero numeric RPC errors or matching startup
errors. Sonarr exposes the checked legacy queue entry as warning/importPending.
This live result includes #49/#50/#51, including #50's integer-error correction;
it is not a standalone deployment test of this PR. No legacy retirement,
manifest backfill, consumer scan, or new download/import cycle was exercised.
Nix builds and historical retained-copy/import verification remain untested.

Follow-up verification (2026-09-09)

Ordinary torrent-remove now attempts every selector independently when a
transfer fails preparation or exhausts its three remote deletion attempts.
It returns the combined preparation/deletion errors after the batch. Failed
IDs retain local payload and ownership evidence; successful later IDs complete
cleanup. The explicit reviewed-retirement guard remains a separate operation.

The new regression reproduces classification failure, exhausted remote deletion,
and mixed failures on the previous implementation; those cases and a successful
control now pass, including absent selectors and retained manifests/categories.

Full go test -race ./..., go vet ./..., go build ./..., and diff checks pass
at this updated branch boundary on darwin/arm64. Upstream main is included,
including its OAuth2 dependency update. Hosted CI must validate the new published
head separately. Earlier deployment and platform evidence above is historical;
this follow-up has not been deployed or exercised against live Put.io/Arr services.
Nix builds remain untested.

@alexg0
alexg0 marked this pull request as draft September 4, 2026 05:17
@elsbrock

elsbrock commented Sep 4, 2026

Copy link
Copy Markdown
Owner

The bindery side (#903 / #933) lines up.

Two changes should be separate from the manifest feature because they're independent bug fixes:

  • The FileID == 0 guards (manager.go:174, transfers.go:430). Put.io uses 0 for the root folder, so the old code could hand the account root to DeleteFile. That's a serious bug on its own, and TestCleanupNeverDeletesPutioRoot is exactly the right test for it.
  • Switching to context.WithCancelCause so a stall-cancelled download is classified transient instead of as a shutdown cancellation.

Can you pull those into their own commits or a separate PR? I'd merge them ahead of the manifest work.

On the rest:

  1. handleTorrentRemove now only calls RemoveCategory/RemoveTransfer when DeleteTransfer succeeded (torrent.go:553). I understand why, but a transfer whose remote deletion keeps failing now keeps its category and manifest for the process lifetime. torrent-remove was the thing that bounded tracking growth. Can we cap this — retry N times, or drop local tracking after a deadline and log it?

  2. localTorrentFiles sets bytesCompleted = length for every file once percentDone >= 1. That's fine as a simplification, but the field promises per-file progress. Add a comment saying it's whole-transfer granularity, or source it from the per-file state we already track.

  3. The progress change (no-context COMPLETED/SEEDING now 0.5 / downloading instead of 1.0 / seeding) is correct — put.io done is not plundrio done — but it changes what every existing *arr sees on restart. Please call it out in the PR body as a behaviour change so it lands in the release notes.

  4. .plundrio-files is a new reserved directory in the download root. It's excluded from deleteLocalData and from manifest names, but it'll be visible to anything else that walks TargetDir. See my note on feat: reconcile unmanaged downloads #51 — the two PRs collide there.

@elsbrock

elsbrock commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Cross-PR note, since all four of these are open at once.

All four touch internal/server/torrent.go in the same region, and #49 and #50 conflict textually. Proposed order:

  1. Report exact transfer files to Transmission clients #52 (fixes the root-folder delete bug and the premature-completion state)
  2. fix(rpc): mark completed transfers removable #49 (only safe after Report exact transfer files to Transmission clients #52)
  3. feat: report stalled Put.io transfers #50 (rebase; keep the int error field, Report exact transfer files to Transmission clients #52 still emits a bool)
  4. feat: reconcile unmanaged downloads #51 (pending the scope discussion)

Separately, CI only runs go build and go test. Given how much of this repo is concurrency, -race should be in the pipeline before we take any of these. I've run go vet and go test -race against all four locally and they're clean.

Retain the exact commits published in elsbrock#53 and elsbrock#54 before manifest ownership work.
…ship

Persist exact Transmission file ownership and local completion, bound remote removal retries, and retain durable removal intent without unbounded active tracking. Include reviewed ownership, queue-locking, and Linux test-cleanup fixes. Reserve the manifest directory and document whole-transfer progress.
Persist NeedsReview for missing legacy source roots without ownership manifests. Keep holds non-removable across restarts and source changes; permit only verified, exact-ID record retirement. Include persistence recovery and classification/removal race fixes with regression coverage.
@alexg0
alexg0 force-pushed the alexg0/transmission-file-manifests branch from f11045a to 84d77a9 Compare September 7, 2026 04:36
@alexg0

alexg0 commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for both the detailed review and the cross-PR sequencing note. I've updated the original PR description, including the upgrade/restart behavior for release notes, and incorporated your suggestions.

  • Root-deletion protection plus race CI are separate in fix: prevent Put.io root deletion and enable race CI #53; stalled cancellation/retry classification is separate in fix: preserve stalled download cancellation for retries #54. Both can land first.
  • Remote removal is bounded to three attempts per request. Durable removal intent preserves ownership evidence while active in-memory tracking is released; regression coverage includes blocked queues, storage, and restart behavior.
  • Whole-transfer bytesCompleted granularity is documented, and feat: reconcile unmanaged downloads #51 excludes .plundrio-files/ and protects pending-removal payloads.
  • Remote completion without proven local completion no longer reports complete. Unverifiable legacy records enter persistent warning-only review holds, with explicit record-only retirement after retained-copy verification.

The history is now two feature commits after the shared safety prerequisites, with the exact previously published source tree preserved. The new boundary passes go test -race ./.... The stack retains #53/#54 → #52 → #49 → #50 → #51, including #50's separate integer-error fix; #51 remains subject to scope agreement.

@elsbrock

elsbrock commented Sep 9, 2026

Copy link
Copy Markdown
Owner

Thanks for the extremely thorough writeup and testing notes on this one — the durability/generation-reservation discipline throughout removal.go/review.go/transfers.go is great, and it all builds, vets, and passes go test -race ./... cleanly for me on a rebase onto current main.

One thing I want to confirm intent on before merging.

handleTorrentRemove used to process each ID in a batch independently — a failure (missing transfer, failed DeleteTransfer) was logged and the loop moved on to the next ID. With this PR, a PrepareRemoval failure (e.g. a completed transfer that hasn't hit its first monitor poll yet) or an exhausted 3-attempt DeleteTransfer now returns an error immediately, which aborts the RPC call for the rest of the batch — any IDs after the failing one in the same torrent-remove request are never attempted.

Is that an intentional fail-fast tradeoff, or should a per-ID failure still let the rest of the batch proceed (matching the old best-effort behavior, just with the new safety bookkeeping)? Sonarr/Radarr can send multi-ID torrent-remove calls, so this changes what a bulk-delete-from-queue actually does on partial failure. No test currently covers a multi-ID batch either way, so happy to help pin down whichever behavior you intend.

Everything else looks solid to me — nothing further blocking on my end once this is settled.

@alexg0

alexg0 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for catching this. The fail-fast behavior was unintended; ordinary multi-ID torrent-remove should continue attempting later IDs after a per-ID failure.

Commit 6ae264a restores that behavior. Preparation failures and exhausted three-attempt remote deletions are collected, processing continues through the batch, and the combined errors are returned afterward. Failed IDs retain local payload and ownership evidence; successful later IDs complete cleanup. The explicit reviewed-retirement guard remains separate.

The added regression covers preparation failure, exhausted remote deletion, mixed failures, and a successful control, including absent selectors and retained manifests/categories. The PR description records the before/after regression and local race-test, vet, and build results.

The updated head also passes hosted CI. This follow-up has not been deployed or exercised against live Put.io/Arr services.

Could you take another look at the batch-removal fix? With #53 and #54 merged, this remains the next PR in the agreed sequence.

@elsbrock

Copy link
Copy Markdown
Owner

The batch-removal fix looks right — I reverted torrent.go to the pre-fix version locally and confirmed TestTorrentRemoveContinuesBatchAfterFailure genuinely fails 3/4 subtests without it, so this isn't a vacuous test. errors.Join + continue is the correct shape, and it matches the old best-effort semantics exactly. go test -race ./..., vet, and build all clean.

LGTM, merging.

@elsbrock
elsbrock merged commit 0663511 into elsbrock:main Sep 12, 2026
1 check passed
elsbrock added a commit that referenced this pull request Sep 12, 2026
…al docs

PR #52's README additions were authored before de5c48a landed on main and
silently reverted its correction when rebased: the 'stateless architecture'
claim and the FAQ answer both understated the local state plundrio keeps,
which now includes the removal/review markers and file manifests under
.plundrio-files/. Restore the accurate wording and extend the FAQ answer to
cover the new state alongside .plundrio-state.json.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants