CLI utility for finding duplicated files.

Weekend project to scratch an itch. I think it works, but not tested very well.

NB: Use at your own risk.

dedupe FOLDER1 FOLDER2

This speeds up the program greatly.

dedupe --db-path cache.db FOLDER1 FOLDER2

dedupe FOLDER1 FOLDER2 --report-dir FOLDER1/test/sub/folder

It will then report how many files are unique to FOLDER1/test/sub/folder and how many are duplicates from other places in FOLDER1 and FOLDER2.

Add scoring of files based on the file path. To add a score:

dedupe FOLDER1 FOLDER2 --score=50=MY_REGEX --score=100=MY_OTHER_REGEX

Each file will get the score of the first regex that matches. Files from a duplication group will only be deleted if all files in the group has a score and not all files have the same score.

To perform deletion:

dedupe FOLDER1 FOLDER2 --score=50=MY_REGEX --score=100=MY_OTHER_REGEX --delete

If a file is duplicated four times with the following scores:

• path1: score=3
• path2: score=2
• path3: score=1
• path4: score=1

Then path3 and path4 will be deleted. If the program is run again, it will then delete path2.

1. Iterate through the directories recursively and retrieves file paths and file sizes.
2. Remove files that are empty. All empty files are duplicate anyway.
3. Find and retain the files that are duplicated by size.
4. Hash the first 4096 bytes of the files.
5. Retain the files that are duplicated by the hash of the first 4096 bytes.
6. Hash the full content of the files.
7. Group by the full hash and report duplicates.

The file paths, sizes, and hash values are optionally stored in a sqlite3 database if --db-path is set.