Repository navigation
perf(readability): load the public bundled DOM entry - #139851
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
|
Codex review: needs maintainer review before merge. Reviewed September 6, 2026, 2:47 AM ET / 06:47 UTC. ClawSweeper reviewWhat this changesThe Readability plugin loads LinkeDOM’s public bundled DOM entry through its existing lazy loader to reduce first-use HTML extraction overhead. Merge readiness✅ Ready for maintainer review The optimization remains useful and absent from current main. No actionable correctness or security defect was found, and the recorded real Gateway evidence supports the changed extraction path. The prior throughput caveat does not establish a blocker against the explicitly limited cold-start claim. Priority: P2 Review scores
Verification
How this fits togetherOpenClaw’s web-fetch tool sends downloaded HTML to the Readability plugin for main-content extraction. The plugin sanitizes the HTML, parses it into a document, and returns text or Markdown; unsuccessful extraction leaves the existing fallback path available. flowchart LR
A[Downloaded HTML] --> B[Remove hidden content and canvas]
B --> C{Within size and depth limits}
C -->|Yes| D[Lazy bundled DOM and Readability]
D --> E[Text or Markdown]
C -->|No| F[Existing extraction fallback]
D -->|No readable content| F
Before mergeNone. Agent review detailsSecurityNone. PR surfaceSource +1. Total +1 across 1 file. View PR surface stats
Review metricsNone. Technical reviewBest possible solution: Use the existing lazy loader with the supported bundled entry while preserving extraction behavior and limiting performance claims to the measured workload. Do we have a high-confidence way to reproduce the issue? Not applicable as a functional bug reproduction: this is a performance optimization supported by a fixed before/after workload and real Gateway observations; no benchmark was executed during review. Is this the best way to solve the issue? Yes. Switching the existing lazy import to the dependency's public entry is a narrow solution that preserves plugin APIs, configuration, and fallback behavior. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 9718729bbcd0. LabelsLabel justifications:
EvidenceWhat I checked:
Likely related people:
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
History |
|
Merged via squash.
|
What Problem This Solves
The first HTML
web_fetchin a Gateway process spends time loading Readability's DOM dependencies. A live Gateway CPU profile showed substantial module-read, compilation, and resolution activity beneath that first extraction.Why This Change Was Made
Use LinkeDOM's documented
linkedom/workerentry, which bundles its DOM dependencies, through the existing lazy Readability loader. The dependency version, plugin boundary, sanitization, extraction modes, limits, and fallback behavior remain unchanged. The worker entry omits native Canvas support; OpenClaw removes canvas elements before Readability sees the HTML.The measured tradeoff is approximately 30 ms less first-extraction work with some slower early Markdown calls. The complete fixed workload still finishes faster. The real Gateway proof below exercises both extraction modes; no universal or steady-state speedup is claimed.
User Impact
In the fixed comparison, median first extraction fell from 64.795 to 32.944 ms and from 62.392 to 32.595 ms in the reverse order, reductions of 49.16% and 47.76%. Including extractor module loading, cold latency improved 32.83% and 31.91%.
The same processes showed early-call regressions: article Markdown increased 0.782/0.728 ms, and formatting Markdown increased 1.506/1.644 ms. These were the second and fourth extractor calls respectively, not sustained throughput measurements. A roughly 20-fetch break-even assumes the latter penalty persists and is not established by this experiment.
Evidence
pnpm buildpassed in 273.07 seconds, including declarations and plugin outputs. The built extractor uses the lazy public worker entry. Source identity stayed unchanged; 122 observed processes and 45 groups were closed. The extractor has no sourcemap.Timing, owner tests, source review, and the completed build above belong to frozen base
5b099995413589f41904cef1413c62239542aa7cplus this exact patch. The checkout has since fast-forwarded tofd22d08e2dd4d5fd9134d9a627262b665115a532; the candidate patch, affected extractor/caller/sibling sources, package file, and lockfile are unchanged. The normal 24-command changed gate passed against the new base in 238.52 seconds, without skips. A fresh normal build at clean candidate commita30b73b470514c426fa4d6bb5bc4a054495d7b9epassed in 218.54 seconds, with unchanged source and complete closure of 118 observed processes / 44 groups. The built extractor bytes match the earlier measured entry. Live Gateway proof ran on that clean commit; no old build receipt has been relabeled.The real OpenAI Gateway run completed all 10 turns. The original nine campaign turns passed all answer/tool checks. Both actual web-fetch results explicitly selected Readability, returned HTTP 200, and matched their persisted replies: Markdown took 228 ms; text took 188 ms. The separate text follow-up failed its frozen exact-answer oracle because it returned
purpose: "documentation examples"rather than"documentation"; its tool result was successful and supported that description. The aggregate remains failed (9/10 exact-answer passes, 10/10 tool-oracle passes), with no rerun or oracle change.Readiness was observed at 4.033 seconds; sampled process-tree RSS peaked at 1.639 GB. These are contextual observations, not attributed readiness or memory gains. Both CPU profiles have valid sample/delta arithmetic and node references, but hit-count inconsistencies are retained and no precise per-function or per-turn timing claim is made. Runtime and audit processes are closed. Required exact-head GitHub review/CI checks remain a separate landing gate.
Before publication, the unchanged source patch was merged with main
d1dce5857a3accfa791e13a1673d8def5d6031ce, including upstream #139794, which repairs an unrelated history-fixture type error. Earlier measurements, local checks, and any local build/live proof retain their explicitly stated source identities. No numerical or live run was repeated or relabeled for this mechanical refresh.