Skip to content

perf(readability): load the public bundled DOM entry - #139851

Merged
steipete merged 2 commits into
mainfrom
codex/perf51-readability
Sep 6, 2026
Merged

steipete merged 2 commits into
mainfrom
codex/perf51-readability

Conversation

@steipete

@steipete steipete commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

The first HTML web_fetch in a Gateway process spends time loading Readability's DOM dependencies. A live Gateway CPU profile showed substantial module-read, compilation, and resolution activity beneath that first extraction.

Why This Change Was Made

Use LinkeDOM's documented linkedom/worker entry, which bundles its DOM dependencies, through the existing lazy Readability loader. The dependency version, plugin boundary, sanitization, extraction modes, limits, and fallback behavior remain unchanged. The worker entry omits native Canvas support; OpenClaw removes canvas elements before Readability sees the HTML.

The measured tradeoff is approximately 30 ms less first-extraction work with some slower early Markdown calls. The complete fixed workload still finishes faster. The real Gateway proof below exercises both extraction modes; no universal or steady-state speedup is claimed.

User Impact

In the fixed comparison, median first extraction fell from 64.795 to 32.944 ms and from 62.392 to 32.595 ms in the reverse order, reductions of 49.16% and 47.76%. Including extractor module loading, cold latency improved 32.83% and 31.91%.

The same processes showed early-call regressions: article Markdown increased 0.782/0.728 ms, and formatting Markdown increased 1.506/1.644 ms. These were the second and fourth extractor calls respectively, not sustained throughput measurements. A roughly 20-fetch break-even assumes the latter penalty persists and is not established by this experiment.

Evidence

  • One fixed B0/C0/C1/B1 cohort, five fresh processes for each dependency/extractor mode per phase: 40 children, no reruns or discarded samples. Complete results match across all variants: 20 dependency parses and 380 extractor results, including text/Markdown, relative links/media, malformed HTML, hidden content, canvas removal, and null results at empty/size/depth boundaries.
  • All 40 native receipts show exit 0, process-family closure, and no owner problems; stdout/stderr hashes and byte counts were independently checked. Whole extractor process median wall time improved 14.03%/15.52%; kernel-child peak RSS medians decreased 3.37%/2.77%. Sampled tree RSS is too sparse for these short processes and is not used for a memory claim.
  • Every measured field is retained: 3/30 cold timing comparisons, 68/108 fixture timing comparisons, 1/204 process-reported resource comparisons, and 4/60 native timing/resource comparisons were adverse. These counters overlap and are not independent experiments. Of the fixture medians, 22/36 were slower; observed extrema are not population tails.
  • Existing owner/sibling tests passed 54/54 across three complete files: the Readability extractor, HTML visibility/unicode handling, and runtime extractor dispatch/cache behavior.
  • Normal pnpm build passed in 273.07 seconds, including declarations and plugin outputs. The built extractor uses the lazy public worker entry. Source identity stayed unchanged; 122 observed processes and 45 groups were closed. The extractor has no sourcemap.
  • Managed P2 source review was scoped-clean. Production delta is +1 line; no tests, dependency versions, configuration, or persistent-store contracts changed.

Timing, owner tests, source review, and the completed build above belong to frozen base 5b099995413589f41904cef1413c62239542aa7c plus this exact patch. The checkout has since fast-forwarded to fd22d08e2dd4d5fd9134d9a627262b665115a532; the candidate patch, affected extractor/caller/sibling sources, package file, and lockfile are unchanged. The normal 24-command changed gate passed against the new base in 238.52 seconds, without skips. A fresh normal build at clean candidate commit a30b73b470514c426fa4d6bb5bc4a054495d7b9e passed in 218.54 seconds, with unchanged source and complete closure of 118 observed processes / 44 groups. The built extractor bytes match the earlier measured entry. Live Gateway proof ran on that clean commit; no old build receipt has been relabeled.

The real OpenAI Gateway run completed all 10 turns. The original nine campaign turns passed all answer/tool checks. Both actual web-fetch results explicitly selected Readability, returned HTTP 200, and matched their persisted replies: Markdown took 228 ms; text took 188 ms. The separate text follow-up failed its frozen exact-answer oracle because it returned purpose: "documentation examples" rather than "documentation"; its tool result was successful and supported that description. The aggregate remains failed (9/10 exact-answer passes, 10/10 tool-oracle passes), with no rerun or oracle change.

Readiness was observed at 4.033 seconds; sampled process-tree RSS peaked at 1.639 GB. These are contextual observations, not attributed readiness or memory gains. Both CPU profiles have valid sample/delta arithmetic and node references, but hit-count inconsistencies are retained and no precise per-function or per-turn timing claim is made. Runtime and audit processes are closed. Required exact-head GitHub review/CI checks remain a separate landing gate.

Before publication, the unchanged source patch was merged with main d1dce5857a3accfa791e13a1673d8def5d6031ce, including upstream #139794, which repairs an unrelated history-fixture type error. Earlier measurements, local checks, and any local build/live proof retain their explicitly stated source identities. No numerical or live run was repeated or relabeled for this mechanical refresh.

@clawsweeper

clawsweeper Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 6, 2026
@clawsweeper

clawsweeper Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed September 6, 2026, 2:47 AM ET / 06:47 UTC.

ClawSweeper review

What this changes

The Readability plugin loads LinkeDOM’s public bundled DOM entry through its existing lazy loader to reduce first-use HTML extraction overhead.

Merge readiness

✅ Ready for maintainer review

The optimization remains useful and absent from current main. No actionable correctness or security defect was found, and the recorded real Gateway evidence supports the changed extraction path. The prior throughput caveat does not establish a blocker against the explicitly limited cold-start claim.

Priority: P2
Reviewed head: 49d3f8ca33ab1b3225a62d9ba53189732a07e30f

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused, maintainable optimization with substantial recorded behavior evidence and appropriately bounded performance claims.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (live_output): The captured body records the production Readability extractor running through a real Gateway on candidate a30b73b, with successful HTTP 200 text and Markdown results matching persisted replies; the affected source remains unchanged at the reviewed head. The separate answer-wording failure does not invalidate those extraction observations.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (live_output): The captured body records the production Readability extractor running through a real Gateway on candidate a30b73b, with successful HTTP 200 text and Markdown results matching persisted replies; the affected source remains unchanged at the reviewed head. The separate answer-wording failure does not invalidate those extraction observations.
Evidence reviewed 11 items Verified introduced change: The pinned merge-base-to-head diff changes only the LinkeDOM runtime/type entry and adds its explanatory comment. The host-verified test merge likewise changes only this extractor, so unrelated base drift is not attributed to this PR.
Optimization remains absent from main: Fetched main still dynamically imports linkedom through the same lazy loader; this PR's bundled-entry optimization is not already implemented there.
Latest release comparison: The supplied latest release, v2026.9.2, also uses the original linkedom entry.
Findings None None.
Security None None.

How this fits together

OpenClaw’s web-fetch tool sends downloaded HTML to the Readability plugin for main-content extraction. The plugin sanitizes the HTML, parses it into a document, and returns text or Markdown; unsuccessful extraction leaves the existing fallback path available.

flowchart LR
  A[Downloaded HTML] --> B[Remove hidden content and canvas]
  B --> C{Within size and depth limits}
  C -->|Yes| D[Lazy bundled DOM and Readability]
  D --> E[Text or Markdown]
  C -->|No| F[Existing extraction fallback]
  D -->|No readable content| F
Loading

Before merge

None.

Agent review details

Security

None.

PR surface

Source +1. Total +1 across 1 file.

View PR surface stats
Area Files Added Removed Net
Source 1 3 2 +1
Tests 0 0 0 0
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 1 3 2 +1

Review metrics

None.

Technical review

Best possible solution:

Use the existing lazy loader with the supported bundled entry while preserving extraction behavior and limiting performance claims to the measured workload.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a functional bug reproduction: this is a performance optimization supported by a fixed before/after workload and real Gateway observations; no benchmark was executed during review.

Is this the best way to solve the issue?

Yes. Switching the existing lazy import to the dependency's public entry is a narrow solution that preserves plugin APIs, configuration, and fallback behavior.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 9718729bbcd0.

Labels

Label justifications:

  • P2: This is a bounded first-use performance improvement with recorded benefit and no demonstrated urgent regression.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The captured body records the production Readability extractor running through a real Gateway on candidate a30b73b, with successful HTTP 200 text and Markdown results matching persisted replies; the affected source remains unchanged at the reviewed head. The separate answer-wording failure does not invalidate those extraction observations.
  • proof: sufficient: Contributor real behavior proof is sufficient. The captured body records the production Readability extractor running through a real Gateway on candidate a30b73b, with successful HTTP 200 text and Markdown results matching persisted replies; the affected source remains unchanged at the reviewed head. The separate answer-wording failure does not invalidate those extraction observations.

Evidence

What I checked:

  • Verified introduced change: The pinned merge-base-to-head diff changes only the LinkeDOM runtime/type entry and adds its explanatory comment. The host-verified test merge likewise changes only this extractor, so unrelated base drift is not attributed to this PR. (extensions/web-readability/web-content-extractor.ts:32, 49d3f8ca33ab)
  • Optimization remains absent from main: Fetched main still dynamically imports linkedom through the same lazy loader; this PR's bundled-entry optimization is not already implemented there. (extensions/web-readability/web-content-extractor.ts:32, 9718729bbcd0)
  • Latest release comparison: The supplied latest release, v2026.9.2, also uses the original linkedom entry. (extensions/web-readability/web-content-extractor.ts:32, 3928bad9badf)
  • Dependency ownership and public export contract: The versioned package manifest identifies WebReflection/linkedom as its repository and exports ./worker as ./worker.js with the same declaration entry as the normal ESM import. This is the dependency contract directly invoked by the patch. (package.json:57)
  • Worker entry limitations: The upstream documentation documents the public worker export, its omission of native canvas, and its use of globalThis.performance. (README.md)
  • Canvas removal precedes document parsing: The sanitizer removes canvas elements, and the extractor awaits sanitization before invoking parseHTML. Existing size/depth limits, extraction modes, and null-on-failure behavior are unchanged. (src/agents/tools/web-fetch-visibility.ts:142, 49d3f8ca33ab)

Likely related people:

  • Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • Vincent Koc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (2 earlier review cycles)
  • reviewed 2026-09-06T06:34:29.165Z sha 49d3f8c :: blocked before merge. :: none
  • reviewed 2026-09-06T06:39:31.137Z sha 49d3f8c :: blocked before merge. :: none

@steipete
steipete marked this pull request as ready for review September 6, 2026 06:43
@steipete
steipete merged commit 7e55839 into main Sep 6, 2026
164 checks passed
@steipete
steipete deleted the codex/perf51-readability branch September 6, 2026 07:06
@steipete

steipete commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: web-readability maintainer Maintainer-authored PR P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: XS status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant