Using Krokiet on a very large number of files - Hash db questions. #2090

I missed the cause in my earlier reply. In 12.0.2 the cache writer opens the destination with create and truncate, then bincode serializes the prehash entries. Each DuplicateEntry stores the scanned path as a PathBuf. Serde 1.0.229 returns exactly path contains invalid UTF-8 characters when that path cannot be represented as UTF-8. The cache file can therefore open successfully and then be left empty or incomplete when serialization fails. The full-hash cache uses a separate save pass, so its successful write does not rule this out.

The likely source is a scanned filename or directory component with non-UTF-8 bytes. If you rename or exclude the offending path, then remove the empty prehash cache and rescan, the prehash cache should save.

12.0.2 cache writer, duplicate entry path field, separate prehash and full-hash saves, Serde's path serializer.

Okay, so the issue is UTF-8 names or paths of the files. That is an issue and makes sense since I do have some UTF-16 and UTF-32 filenames and possibly paths. It has not caused me any issues for years so never worried about them. I do know that they are throughout the full archives I am checking.

Guess it is time to fix that issue. Looks like convmv may be the tool to use. Will search for non-UTF-8 directories/files before I start the next level of scans. I am almost done this set and will find and convert them, where I can, before I proceed, as the file count is only going to get larger. Move all the ones that I cannot convert to a separate directory for last scanning.

Thank you. At least it isn't a bug.

The UTF-8 warning in your log is explained by cache serialization rejecting a path. A timing race is not needed to explain it. In 12.0.2, the prehash cache stores each path as a PathBuf, and Serde reports path contains invalid UTF-8 characters when one cannot be converted to UTF-8. The cache file is opened with truncation before serialization, so a failed save can leave it empty or incomplete. Adding directories can change whether a bad path is included in the cache, so a group passing once does not rule this out. Cache save code, cache file open code, cached path field, Serde path handling.

On Linux, a filename is a byte sequence. A name written in Japanese or another language is not a problem if those bytes are valid UTF-8. To find names that are not valid UTF-8, run this read-only scan on each archive root. It prints the raw filename bytes in hex so the result stays readable:

python3 - /path/to/archive <<'PY'
import os, sys

for base, dirs, files in os.walk(sys.argv[1]):
    for name in dirs + files:
        path = os.path.join(base, name)
        if any("\udc80" <= char <= "\udcff" for char in path):
            print(os.fsencode(path).hex())
PY

I tested the check with both a valid Japanese directory name and a filename containing an invalid byte. It reported only the invalid-byte name. If the cache error keeps appearing, this should help identify the exact path. The error in your log is already a specific match for this cause.

Thank you for the program. I spent hours looking for something that would work to check but everything I tried failed. I just ordered a book on learning python. :) I can program an Arduino, some Pi and C(++) stuff enough to be dangerous. One of the projects I want to work with, requires python. My field is electronics design and repair.

I am assuming that the python3 - /path/to/archive <<'PY' is the command or does that go in the *.py script or do I have to change the path for each test?

Much of what you said about Linux file names, I read about last night, while I was trying to look at this issue an what I am dealing with. All the the convert programs, like convmv, require knowing the character encoding in of the original name. That is even harder.

What I saw last night was a fail and pass and fail on the same directories for the prehash. Nothing added or changed between scans. I want to test more to see what happened. I didn't look at the log files at the time so I don't have any idea what was happening. Not sure if that will be today or tomorrow.

I also found out that you don't have to delete the prehash file when you do a scan. If the scan works, the prehash is written over with the new file. It is also much larger than the hash file.

Again, thank you.

Got the script working. Thank you.

Made a slight change for ease of use.
UTF8_test.py

#!/usr/bin/python
import os, sys

for base, dirs, files in os.walk(sys.argv[1]):
    for name in dirs + files:
        path = os.path.join(base, name)
        if any("\udc80" <= char <= "\udcff" for char in path):
            print(os.fsencode(path).hex())

Now I can just run it as
UTF8_test.py {directory}

Found 8 files from the quick test. Biggest issue is the names are all printed as hex so need to search in the directories but that is now much easier.

I do know that in my tests the other night, one of the directories with a file found here did generate a cache_duplicates_Blake3_prehash_120.bin file. It could be an indication that the file is smaller than the scanned size.

Made a slight change to the script to show the directory that the bad file is in.

#!/usr/bin/python
import os, sys

for base, dirs, files in os.walk(sys.argv[1]):
    for name in dirs + files:
        path = os.path.join(base, name)
        if any("\udc80" <= char <= "\udcff" for char in path):
            print(os.fsencode(path).hex())
            print("Path to file:",base)

I did this to find out what directory the "bad" file is in. Saves time to find the file in a directory tree.

This is a very useful script.