Skip to content

Scan: high peak memory on trees with many hard-linked files (pnpm) #5

Closed

Description

@devloop01

Problem

measure in internal/sweep/scan.go keeps a shared map[inode]int64 per target with one entry for every hard-linked file, and Scan holds all of those maps until the scan ends so that a file linked into several targets is counted once.

pnpm hard-links nearly every installed file. A map entry costs about 55 bytes (measured with a 1M-entry map[inode]int64), so a scan that sees 5 million hard-linked files holds roughly 275 MB.

This is not a leak: the memory is released when Scan returns. It is a peak-memory cost that grows with the size of the tree.

Idea

Store each inode once for the whole scan instead of once per target, for example a single shared map from inode to the first target (in path order) that contains it. A file linked into many targets then costs one entry instead of one per target. Sizes must stay deterministic regardless of which worker gets there first, as they are today.

That only helps when the same file is linked into several targets. If most links point outside the scanned tree (pnpm's global store), a different approach is needed, such as not remembering inodes whose link count can't be reached inside the tree.

Notes

  • Progress.OnFound reports a provisional size that counts shared files in full; that behaviour can stay.
  • TestScan… cases around hard links in scan_test.go cover the current attribution rule.

Activity

  1. added a commit that references this issue on Nov 13, 2026
    5abe807
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions