You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Scan: high peak memory on trees with many hard-linked files (pnpm) #5
measure in internal/sweep/scan.go keeps a shared map[inode]int64 per target with one entry for every hard-linked file, and Scan holds all of those maps until the scan ends so that a file linked into several targets is counted once.
pnpm hard-links nearly every installed file. A map entry costs about 55 bytes (measured with a 1M-entry map[inode]int64), so a scan that sees 5 million hard-linked files holds roughly 275 MB.
This is not a leak: the memory is released when Scan returns. It is a peak-memory cost that grows with the size of the tree.
Idea
Store each inode once for the whole scan instead of once per target, for example a single shared map from inode to the first target (in path order) that contains it. A file linked into many targets then costs one entry instead of one per target. Sizes must stay deterministic regardless of which worker gets there first, as they are today.
That only helps when the same file is linked into several targets. If most links point outside the scanned tree (pnpm's global store), a different approach is needed, such as not remembering inodes whose link count can't be reached inside the tree.
Notes
Progress.OnFound reports a provisional size that counts shared files in full; that behaviour can stay.
TestScan… cases around hard links in scan_test.go cover the current attribution rule.
Activity
Ask git once per repo, cut hard-link memory, stop the scan on quit, c…