Skip to content

improve(web-fetch): speed up provider fallback loading - #98559

Merged
vincentkoc merged 9 commits into
mainfrom
perf/web-scraping-loading
Jul 4, 2026
Merged

improve(web-fetch): speed up provider fallback loading#98559
vincentkoc merged 9 commits into
mainfrom
perf/web-scraping-loading

Conversation

@vincentkoc

@vincentkoc vincentkoc commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What Problem This Solves

Resolves a performance problem where web_fetch shell-like HTML fallback paths loaded the complete bundled Firecrawl plugin graph after direct readability extraction returned no content. In a fresh process, that path spent roughly 6.4-6.9 seconds in plugin discovery/loading before doing useful fetch work.

Why This Change Was Made

This adds a deterministic perf:web-fetch benchmark, caches provider fallback discovery with bounded invalidation, and gives bundled web-fetch providers a narrow runtime-artifact path. The fast path uses the canonical bundled-plugin activation policy, so plugin deny lists, explicit disables, allow lists, and global plugin disablement still apply. Explicit activation and mixed/external provider origins continue through the full loader.

The bundled Firecrawl provider now exposes a narrow runtime artifact and lazily imports its client only when the tool executes. The branch also removes duplicate Firecrawl scrape payload wrapping and trims HTML/readability hot-path work.

User Impact

Fresh shell-like HTML fallback now resolves in roughly 0.92-1.11 seconds instead of 6.4-6.9 seconds on the same machine, an approximately 83-86% reduction. The first old-path sample reached 13.9 seconds and 811 MB RSS; the new path remained near one second across repeated fresh-process runs.

Provider behavior still updates when plugins load or unload, config changes, credentials appear, or callers explicitly request activation. External and non-bundled providers retain the existing full-loader behavior.

Evidence

  • Fresh-process baseline from the rebased parent: 6933.901ms, 6658.163ms, 6417.874ms, 6476.341ms after a first-run outlier of 13945.523ms.
  • Fresh-process result on this branch: 1222.835ms, 1102.780ms, 1113.427ms, 949.066ms, 969.586ms, 916.912ms, 988.535ms.
  • Focused validation passed with 178 tests across core, agent, plugin, extension, readability, Firecrawl, and web-fetch runtime shards.
  • pnpm build passed through Testbox id tbx_01kwn1evj9cjt8xz43hmj5yahg in 75.5s.
  • Path-scoped pnpm check:changed passed through Crabbox/Testbox id tbx_01kwn1r744fs714103fns86pr0, Actions run 28686129791, exit 0, total 10m05.984s. This covered all typecheck lanes, core/extensions/tooling lint, dependency and shrinkwrap guards, runtime import cycles, and repository policy guards.
  • Post-rebase branch autoreview passed with no accepted/actionable findings; overall verdict patch is correct, confidence 0.90.

@vincentkoc vincentkoc self-assigned this Jul 1, 2026
@openclaw-barnacle openclaw-barnacle Bot added scripts Repository scripts extensions: firecrawl size: L maintainer Maintainer-authored PR labels Jul 1, 2026
@clawsweeper

clawsweeper Bot commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: stale review; fresh review needed.

Summary
The latest durable ClawSweeper review was for head 53d8df7c738e02cd64a25ce849103aa290f7c015, but the PR head is now bc48ec568a203a7b389301eca471c3b6912c58d2. Its old verdict and PR readiness labels are no longer current.

Next step
Run or wait for a fresh ClawSweeper review on the current PR head.

Review history (5 earlier review cycles)
  • reviewed 2026-07-03T22:50:07.676Z sha 4721b4b :: needs maintainer review before merge. :: none
  • reviewed 2026-07-04T18:34:37.764Z sha 06d0392626691160670f76c983e0ff268642281d :: needs maintainer review before merge. :: none
  • reviewed 2026-07-04T18:40:13.592Z sha 06d0392626691160670f76c983e0ff268642281d :: needs maintainer review before merge. :: none
  • reviewed 2026-07-04T18:59:38.996Z sha 50d130d8c8e654fe9da10fbbc7456390a83579dd :: needs maintainer review before merge. :: none
  • reviewed 2026-07-04T19:19:18.350Z sha 7e2b03af137d542838f5d1b3d036575534d53a81 :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. labels Jul 1, 2026
@vincentkoc
vincentkoc force-pushed the perf/web-scraping-loading branch from 988776d to 1b67241 Compare July 2, 2026 05:55
@openclaw-barnacle openclaw-barnacle Bot added the agents Agent runtime and tooling label Jul 2, 2026
@vincentkoc
vincentkoc force-pushed the perf/web-scraping-loading branch from 1b67241 to 4721b4b Compare July 3, 2026 22:42
@vincentkoc
vincentkoc force-pushed the perf/web-scraping-loading branch from 4721b4b to 06d0392 Compare July 4, 2026 18:30
@vincentkoc
vincentkoc marked this pull request as ready for review July 4, 2026 18:49
@vincentkoc

vincentkoc commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Land-ready maintainer proof for exact head 06d0392626691160670f76c983e0ff268642281d:

  • Focused web-fetch, provider runtime, Firecrawl, readability, and agent fallback suites: 178 tests passed on Testbox tbx_01kwq6bs8n7vvt0pr1mqmzq1m0 in 45.452s.
  • pnpm build: passed on the same Testbox in 74.748s.
  • Path-scoped pnpm check:changed: passed on Testbox tbx_01kwq6j61gy9rzrb9m3889esrr in 10m05.089s.
  • GitHub exact-head checks: no failing or pending checks.
  • Fresh branch autoreview against current origin/main: no actionable findings, confidence 0.88.
  • scripts/pr review-validate-artifacts 98559 and OPENCLAW_TESTBOX=1 scripts/pr prepare-run 98559: passed.

Best-fix verdict: yes. Provider discovery caching lives at the shared runtime owner, bundled Firecrawl resolution stays behind canonical activation policy, external providers retain the full-loader fallback, and credentials are still read per selection. No known proof gaps block landing.

@vincentkoc
vincentkoc force-pushed the perf/web-scraping-loading branch from 06d0392 to 50d130d Compare July 4, 2026 18:50
@vincentkoc

vincentkoc commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Updated land-ready maintainer proof after required mainline sync, for exact head 50d130d8c8e654fe9da10fbbc7456390a83579dd:

  • Focused web-fetch, provider runtime, Firecrawl, readability, and agent fallback suites: 178 tests passed on Testbox tbx_01kwq7gt44ephayjx86czwc3c9 in 49.748s.
  • pnpm build: passed on the same Testbox in 75.586s.
  • Path-scoped pnpm check:changed: passed on the same Testbox in 9m29.703s.
  • GitHub exact-head checks: no failing or pending checks.
  • Fresh branch autoreview against current origin/main: no actionable findings, confidence 0.90.
  • scripts/pr review-validate-artifacts 98559 and OPENCLAW_TESTBOX=1 scripts/pr prepare-run 98559: passed.

Best-fix verdict: yes. Provider discovery caching lives at the shared runtime owner, bundled Firecrawl resolution stays behind canonical activation policy, external providers retain the full-loader fallback, and credentials are still read per selection. No known proof gaps block landing.

@vincentkoc
vincentkoc force-pushed the perf/web-scraping-loading branch from 50d130d to 7e2b03a Compare July 4, 2026 19:11
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 4, 2026
@vincentkoc

vincentkoc commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Updated land-ready maintainer proof after the required package-overlap mainline sync, for exact head 7e2b03af137d542838f5d1b3d036575534d53a81:

  • Focused web-fetch, provider runtime, Firecrawl, readability, and agent fallback suites: 178 tests passed on Testbox tbx_01kwq7gt44ephayjx86czwc3c9 in 31.438s after reconciling the updated mainline workspace dependencies.
  • pnpm build: passed on the same Testbox in 76.086s.
  • Path-scoped pnpm check:changed: passed on the same Testbox in 9m44.434s.
  • GitHub exact-head checks: no failing or pending checks.
  • Fresh branch autoreview against current origin/main: no actionable findings, confidence 0.91.
  • scripts/pr review-validate-artifacts 98559 and OPENCLAW_TESTBOX=1 scripts/pr prepare-run 98559: passed.

Best-fix verdict: yes. Provider discovery caching lives at the shared runtime owner, bundled Firecrawl resolution stays behind canonical activation policy, external providers retain the full-loader fallback, and credentials are still read per selection. No known proof gaps block landing.

@vincentkoc
vincentkoc force-pushed the perf/web-scraping-loading branch from 7e2b03a to 53d8df7 Compare July 4, 2026 19:30
@vincentkoc
vincentkoc force-pushed the perf/web-scraping-loading branch from 53d8df7 to bc48ec5 Compare July 4, 2026 19:38
@clawsweeper clawsweeper Bot removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 4, 2026
@vincentkoc
vincentkoc merged commit 2f64590 into main Jul 4, 2026
105 of 109 checks passed
@vincentkoc
vincentkoc deleted the perf/web-scraping-loading branch July 4, 2026 19:47
@vincentkoc

vincentkoc commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Merged via squash.

@vincentkoc

vincentkoc commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Final landing proof:

  • Prepared and merged exact PR head: bc48ec568a203a7b389301eca471c3b6912c58d2.
  • Exact-head CI run 28717445539: completed successfully with no failed jobs.
  • Exact-head focused web-fetch, provider runtime, Firecrawl, readability, and agent fallback suites: 178 tests passed on Testbox tbx_01kwq7gt44ephayjx86czwc3c9 in 25.683s.
  • The immediately preceding synced head also passed pnpm build, path-scoped pnpm check:changed, and fresh autoreview; the final sync only incorporated unrelated mainline merge infrastructure, while exact-head CI revalidated the final tree.
  • Squash commit on main: 2f6459093ad3abe4369f686b517b5d78568af35a.

The Testbox lease is stopped and the remote topic branch is removed.

@github-actions github-actions Bot mentioned this pull request Jul 4, 2026
Open
@clawsweeper clawsweeper Bot mentioned this pull request Jul 5, 2026
Merged
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 5, 2026
@vincentkoc
improve(web-fetch): speed up provider fallback loading (openclaw#98559)
0d37a52 
* perf(web-fetch): add web fetch benchmark

* perf(web-fetch): cache provider fallback resolution

* perf(firecrawl): avoid duplicate scrape payload wrapping

* perf(web-readability): reuse void tag lookup

* fix(web-fetch): keep provider cache live

* fix(web-fetch): invalidate provider cache on registry swap

* perf(web-fetch): reduce html extraction overhead

* fix(web-fetch): bound provider discovery cache

* perf(web-fetch): avoid full plugin load on fallback
@vincentkoc vincentkoc mentioned this pull request Aug 5, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@vincentkoc vincentkoc

Labels

agents Agent runtime and tooling extensions: firecrawl maintainer Maintainer-authored PR P2 Normal backlog priority with limited blast radius. scripts Repository scripts size: XL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@vincentkoc