Make recursive iterdir linear on read-only archives - #164

Summary

Path.iterdir() builds a Path for every name in the archive and then filters down to direct children. If you walk the whole tree, you rescan the archive once per directory, which makes the walk O(dirs × entries). importlib.resources.as_file() on a zipped package does exactly this walk.

Read-only archives already go through FastLookup, which caches the name list. This adds a cached parent → children index next to it:

 class FastLookup(CompleteDirs):
     namelist()      # cached
     _name_set()     # cached
+    _children       # cached: {parent_dir: [names]}

 Path.iterdir()
   if not is_dir(): raise
+  if root is FastLookup:
+    return map(_next, root._children[at])
   return filter(_is_child, map(_next, root.namelist()))

Same names, same order, same parent rule (dirname(name.rstrip('/'))) as _is_child. Write-mode archives don't use FastLookup, so they keep the old path.

Evidence

Full tree walk with zipp on Python 3.14, median of 7, macOS arm64. n is the number of directories, each holding one file:

Same patch applied to CPython's zipfile._path (3.16 dev build):

New test test_recursive_iterdir_work counts Path objects built while walking a 32-directory archive:

• Before: AssertionError: assert 2113 <= (4 * 32)
• After: 65 built, passes

tox -e py314: 115 passed, ruff clean. In CPython, test_zipfile + test_importlib with the patch: 952 run, 42 skipped, all pass.

Merge Danger

Door: two-way

The index is private, cached per FastLookup instance, and built on first iterdir().

Blast Radius: memory

Each read-only archive that calls iterdir() now keeps one extra dict. On the 1,024-directory walk (CPython build), retained traversal memory went from 79 KB to 262 KB and peak from 178 KB to 364 KB. Archives that never call iterdir() don't pay anything.

AI disclosure: the profiling and the patch were done with AI tools (Codex, Claude).

🤖 Generated with Claude Code