Skip to content

Fix dedupe scripts leaving duplicate data behind through hardlink siblings - #24

Draft
Joloxx9 wants to merge 1 commit into
thrill-burn:mainfrom
Joloxx9:fix/dedupe-hardlink-siblings
Draft

Joloxx9 wants to merge 1 commit into
thrill-burn:mainfrom
Joloxx9:fix/dedupe-hardlink-siblings

Conversation

@Joloxx9

@Joloxx9 Joloxx9 commented Sep 17, 2026

Copy link
Copy Markdown

The duplicate map kept only one representative path per inode. If duplicate B had multiple hardlinks (for example, Luminarr and Darkpeers), replacing only Luminarr with a link to A left B's data alive through Darkpeers while the script still reported reclaimed space.

This change retains all known hardlink paths for each physical copy, merges overlapping groups, and groups copies by device and inode. The generated script compares content once per separate copy, creates a replacement hardlink before atomically replacing each included target, and counts each old copy's logical size only when its last hardlink is gone. Failed operations are reported, and partial or completed runs can be safely repeated.

Excluded paths remain untouched, and excluded or unscanned hardlinks prevent a reclamation claim. Filesystem boundaries are respected. The script runs directly on the Unraid host using Bash and standard system tools, with no Python dependency on the host.

Validation:

  • 342 backend tests passed, including 18 deduplication cases.
  • Tests execute generated scripts on real small files and hardlinks with a PATH containing no Python, covering exclusions, overlapping groups, content differences, link/rename/comparison errors, interruptions, and reruns.
  • git diff --check passed.

Limitations:

  • Tests were run locally on macOS; the Unraid/GNU tools execution path has not been tested on an Unraid host.
  • The Docker build was not run because Docker is unavailable in the local environment.
  • SIGKILL may leave .auditorr-dedupe-* temporary directories, although target paths remain present and the script can be rerun.
  • Writers must be stopped during deduplication. Reported reclaimed bytes are logical file size, not measured free disk space; open files or snapshots may retain data.

thrill-burn added a commit that referenced this pull request Sep 18, 2026
CONTRIBUTING.md told people to branch from main. main only moves at
releases, so it sits well behind experimental, and a contributor
following that instruction can rebuild work that already exists — which
is what happened in #24.

Adds a "Which branch to work from" section, switches the pull-request
steps to experimental, says to change the PR base branch GitHub defaults
to main, asks for an issue before anything substantial, and documents the
backend test suite that now gates the CI build.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
thrill-burn added a commit that referenced this pull request Sep 18, 2026
CONTRIBUTING.md told people to branch from main. main only moves at
releases, so it sits well behind experimental, and a contributor
following that instruction can rebuild work that already exists — which
is what happened in #24.

Adds a "Which branch to work from" section, switches the pull-request
steps to experimental, says to change the PR base branch GitHub defaults
to main, asks for an issue before anything substantial, and documents the
backend test suite that now gates the CI build.

Identical to the experimental copy, so the eventual merge is clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thrill-burn

thrill-burn commented Sep 18, 2026

Copy link
Copy Markdown
Owner

Thank you for this, the testing especially (running the script with no Python on PATH, injecting ln/mv/cmp failures, SIGKILL mid-rename) is better than I'd have asked for.

Apologies but I do development work on the experimental branch which is currently 19 commits ahead of main, with a Dedupe rework that landed on 15 September a day before your commit. It independently reaches five of the six things you describe: union-find group merging, device+inode bucketing, a staged plain ln instead of ln -f, bytes counted only when the last link to a copy is gone, and idempotent reruns. I've updated CONTRIBUTING.md so the next person doesn't hit this.

The bug you found is real - experimental currently refuses to act when an inode has more links than it knows about. Safe, but it leaves space on the table. Yours is a better approach and I'd like to incorporate it into the new dedupe script and credit you by name.

Because we both rewrote _build_dup_groups and the script generator from different premises where mine has no canonical copy at all, the script picks what to keep at runtime there's no clean merge path. I'll leave this open rather than close it.

Please feel free to work from experimental, just note it's a working branch and can be mid-refactor on any given commit. auditorr itself never touches your files, but the scripts it generates do, so read one before you run it, especially while dedupe is in flux. Let me know if the dedupe script is missing anything and I'll incorporate your work as soon as I can.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants