Skip to content

feat(dropbox): grep/rg search push-down via files/search_v2 - #568

Merged
zechengz merged 2 commits into
strukto-ai:mainfrom
bytecii:feat/dropbox-search-pushdown
Jul 19, 2026
Merged

zechengz merged 2 commits into
strukto-ai:mainfrom
bytecii:feat/dropbox-search-pushdown

Conversation

@bytecii

@bytecii bytecii commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

What

Recursive grep/rg on a Dropbox mount previously walked the tree and downloaded every file. This adds an opt-in search push-down: with content_search=True (TS contentSearch: true), both commands first ask /2/files/search_v2 which files contain the pattern's literal, then download and scan only those candidates. Output stays exactly GNU/ripgrep because the local scan still decides every match — search only narrows the candidate set.

Follows the GitHub backend's narrow-then-scan blueprint and the Slack backend's availability-gate pattern, with three deliberate hardenings the GitHub narrow doesn't have:

  • -v / -c (grep) and -v / --type / --glob (rg) force the full walk — a narrowed superset would hide files those output modes must see.
  • No -l short-circuit: Dropbox search is case-insensitive, so narrowed candidates are always re-scanned locally.
  • Binary-extension candidates are dropped (the walk skips them), rg prunes hidden candidates segment-wise, and narrowed rg runs force walk-style filename labels.

Why the knob is off by default

Full-text content search is plan-gated (Professional/Essentials/Business+); on other plans search_v2 silently matches file names only, which would make a narrowed scan miss content matches. Dropbox's search index also lags recent writes. Docs spell out both caveats.

Details

  • Core (both languages): search_files/searchFiles pages search_v2 + search/continue_v2, dedups across pages (API may repeat results), and flags the 10,000-match ceiling; narrow_paths/narrowPaths maps path_lower/path_display back to mount paths under root_path (case-insensitive scope filter, display-cased output), sorts candidates into sorted-readdir walk order (component-wise, not lexicographic), and rebases raw_path onto the scope's as-typed spelling so labels match a walk. Failed/empty/truncated searches fall back to the full walk.
  • Gating (narrow_scope/narrowScope): knob on + recursive + single-line literal (regexes narrow on an extracted required literal) + all scope operands stat as directories. Unlike GitHub there is no scope-size gate: one search call plus targeted downloads beats a full walk at every scope size.
  • Provision: bespoke wrappers re-attach the factory's default_provision, so cost estimates are byte-identical to before (prov_* battery cases).
  • GNU fix (python generic): grep -Rl with a file operand now stats first and scans the file instead of readdir-walking it — GNU behavior and TS parity; narrowed candidates exercised this latent gap.
  • Fakes: integ/server/dropbox.ts + dropbox_server.py implement search_v2/search/continue_v2 with cursor paging; matching is case-insensitive substring over names and content — a superset of real token matching, which is exactly the contract narrowing needs.
  • Battery: adapters enable the knob, so every dropbox/dropbox-root grep/rg case now exercises the push-down live.

Verification

  • Battery: dropbox + dropbox-root 988/988 on both hosts (push-down active); ram/disk/s3 regression targets clean on both hosts (python 3271/0 incl. s3; TS 2170/0, s3 skipped locally).
  • Python: full suite green (only known env failures: native tac, fuse, redis); mypy 1435 files clean.
  • TS: core 4304 / browser 165 / node 1540 (+4 known native_tac env); typecheck, eslint, knip, pre-commit clean.
  • spec/ regenerated — no drift (registered command names unchanged).
  • Docs: search push-down sections in docs/python/resource/dropbox.mdx + docs/typescript/dropbox.mdx, resource-matrix note; python example extended with a write roundtrip + search demo.

🤖 Generated with Claude Code

@bytecii
bytecii requested a review from zechengz as a code owner July 19, 2026 07:44
bytecii and others added 2 commits July 19, 2026 04:04
Recursive grep/rg on a Dropbox mount previously downloaded every file.
With the new content_search / contentSearch config knob (off by default:
full-text search is plan-gated and its index lags recent writes), both
commands now ask /2/files/search_v2 which files contain the pattern's
literal and download only those candidates. Output stays exactly
GNU/ripgrep because the local scan still decides every match:

- Core searchFiles pages search_v2 + search/continue_v2, dedups across
  pages, and reports the 10,000-match ceiling; narrowPaths maps
  path_lower/path_display back to mount paths under root_path, sorts
  narrowed candidates into sorted-readdir walk order, and rebases
  raw_path onto the scope spelling so labels match a walk's.
- narrow_scope gates the push-down: literal (or regex-required-literal)
  single patterns only, recursive scans only, directory operands only,
  and never for output modes that must see every file (grep -v/-c,
  rg -v/--type/--glob). Empty/failed/truncated searches fall back to
  the full walk; binary-extension candidates are dropped to mirror the
  walk's skip; rg prunes hidden candidates segment-wise and forces
  walk-style filename labels.
- Both wrappers keep the factory's default_provision so cost estimates
  are unchanged.
- fix(grep, python): grep -Rl with a file operand now stats first and
  scans the file instead of readdir-walking it (GNU + TS parity);
  narrowed candidates exercised this path.
- Fakes gain search_v2 + search/continue_v2 (case-insensitive substring
  over names and content — a superset of real token matching, which is
  what narrowing needs) with cursor paging; battery adapters enable the
  knob, so all dropbox/dropbox-root grep/rg cases now exercise the
  push-down live: 988/988 per target on both hosts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…matting

The local TS battery ran against a stale mirage-node dist (only core was
rebuilt), so the node DropboxResource never forwarded contentSearch and
search narrowing was silently inactive on the TS host; CI's fresh build
activated it and exposed two latent TS bugs python had already fixed:

- grepFilesOnly walked file operands under -r (readdir on a narrowed
  file candidate -> ENOENT warnings, empty output). It now stats first
  and takes the single-file scan for file operands (GNU + python
  grep_files_only parity); regression tests in both languages.
- rg's plain-line path delegates to grepGeneric, whose single-file body
  honors -H over -h, so the wrapper's forced label defeated -I
  suppression. Both wrappers now skip forcing H when -I is set;
  regression tests in both languages.

Also formats the new files pre-commit never saw locally (they were
untracked when it ran; --all-files only covers git ls-files) and settles
two formatter fights: the provision calls are hoisted onto a shared
dropboxResolveGlob const so Prettier/ESLint agree, and the
test_grep_helper import gets grep_helper via a module import so
yapf/isort converge.

Verified with fresh core+node dists: dropbox/dropbox-root 988/988 on
both hosts (narrowing live), ram/disk 2170/0, core vitest green,
pre-commit converges.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bytecii
bytecii force-pushed the feat/dropbox-search-pushdown branch from 3a8a39e to ccc43d0 Compare July 19, 2026 11:06
@zechengz
zechengz merged commit db35b48 into strukto-ai:main Jul 19, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants