feat: expose garbage collection duration metrics by steipete · Pull Request #139592 · openclaw/openclaw

steipete · 2026-09-06T01:19:14Z

What Problem This Solves

Operators investigating intermittent event-loop stalls can see memory and delay metrics, but have no GC-duration distribution covering the gaps between short CPU profiles. Sampled GC frames cannot establish actual GC-entry counts or elapsed durations.

Why This Change Was Made

The existing diagnostics heartbeat now owns one demand-driven Node GC observer. It reconciles demand at startup and on its existing 30-second tick, stops observation on diagnostics teardown, and fences stale callbacks and entries when observation restarts. A duration-only internal event uses the existing bounded diagnostic queue and the existing Prometheus and OpenTelemetry histogram implementations.

There is no new configuration, dependency, persistent store, public RPC, timer, or registry. Readiness/recovery policy and exporter counter-reset behavior are unchanged. The change adds about 101 production lines for the missing observation and both exporter adapters.

User Impact

Metrics consumers gain openclaw_gc_duration_seconds and openclaw.gc.duration_ms. These measure elapsed GC duration for the hosting JavaScript isolate, not CPU time, allocation ownership, or guaranteed stop-the-world pauses. Bucket counts provide co-evidence when compared with event-loop windows; they do not establish request-level causality.

Collection can begin at the next heartbeat after a consumer registers, with no backfill before observer activation. Brief subscriber gaps, queue drops and series limits constrain coverage. Runtimes that do not advertise GC performance entries leave this observer inactive.

Evidence

Thirteen focused tests passed across diagnostic lifecycle, bounded queue/public-stream policy, Prometheus HTTP/cap behavior, and real OpenTelemetry SDK signal modes and trace isolation. The corrected lifecycle test also passed independently.
Full local pnpm build passed. All four type lanes and Knip scans passed; two test-only lint findings were corrected and both lint scopes plus the affected canonical services test graph passed. Remaining runtime guards and the 753-page MDX check passed.
Independent review through P2 is clean on the final change.
Actual Node 26.7 component integration passed through real plugin registration, capability leases, diagnostic dispatch, Prometheus HTTP rendering, and OpenTelemetry SDK aggregation: matching counts/sums, cumulative disable/re-enable, stale-entry rejection and cleanup. This is component integration, not a compiled Gateway claim.
Actual compiled Linux Gateway on Node 26.7 passed authenticated metrics and the real config-watcher hot-toggle flow. GC counts progressed 0 → 8 → 12, stayed at 12 through disable, and resumed to 17 after enable; the existing three-call RPC counter stayed unchanged. Source/build/runtime seals and exact process/listener identity stayed stable; the Gateway closed gracefully and the fixture was removed. This proves the Prometheus Gateway flow; the separate component run proves both-exporter parity.
Controlled warmed Linux comparison completed: six alternating OFF/ON pairs, four CPU affinity, 3,276,800 allocations per run, both real exporters and four HTTP/flush cycles. ON exported 22 GC entries per run; OFF exported zero; both histograms agreed. Median paired CPU difference was +0.145 ms (+0.104% ratio), with individual differences −2.167 to +8.611 ms (ratios 0.986–1.055). Wall-time differences were −148.6 to +54.1 ms and RSS-growth differences −5.4 to +1.9 MiB. These are observed component-workload results, not a precise isolated observer cost or a production bound. All children joined without cleanup signals, and the test lease was stopped. The earlier noisy macOS result remains inconclusive.
Native evidence is bound to f00f29cb6374e1b274bc144133ebb6c5ca0f410c. The current-main rebase preserves the identical GC patch; the only changed GC-owner file inherits main's already-landed Prometheus rendering allocation improvement (perf(prometheus): reduce temporary scrape allocations #139535). All thirteen combined lifecycle/exporter tests passed again on the refreshed head. Main also supplies the unrelated native-report test mock repair (refactor(test): simplify catalog guards and fix native report fixture #139496) for fresh CI.
The new GC documentation anchor resolves through the publishing parser. A full optional anchor audit reports unrelated stale anchors in mirrored ClawHub validation docs; ordinary link checking found no broken routes.

clawsweeper · 2026-09-06T01:19:18Z

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

clawsweeper · 2026-09-06T01:28:16Z

Codex review: needs maintainer review before merge. Reviewed September 5, 2026, 10:09 PM ET / September 6, 2026, 02:09 UTC.

ClawSweeper review

What this changes

Adds garbage-collection duration histograms to the existing Prometheus and OpenTelemetry exporters, with demand-driven collection, lifecycle coverage, and operator documentation.

Merge readiness

✅ Ready for maintainer review

The PR remains useful and has no actionable correctness findings. The updated evidence addresses the previous Gateway hot-toggle and overhead qualification request; maintainer-authored, protected work also remains outside automatic cleanup.

Priority: P2
Reviewed head: e3805714e7f87e1a8fe2e1bef5990dc9d0bc9d69

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused, maintainable implementation with useful runtime evidence and no actionable findings.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (live_output): Captured Linux Node 26.7 results exercise the heartbeat-owned observer through authenticated compiled Gateway scrapes and real hot toggles, while component results cover both exporters and controlled overhead. They resolve the prior proof request; runtime evidence remains explicitly bound to the earlier head, whose reported patch-identical rebase could not be independently compared locally.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (live_output): Captured Linux Node 26.7 results exercise the heartbeat-owned observer through authenticated compiled Gateway scrapes and real hot toggles, while component results cover both exporters and controlled overhead. They resolve the prior proof request; runtime evidence remains explicitly bound to the earlier head, whose reported patch-identical rebase could not be independently compared locally.
Evidence reviewed	7 items	Pinned patch and lifecycle owner: Inspected the complete introduced patch from the pinned base to head. The observer checks diagnostics enablement and consumer interest, fences replaced callbacks, rejects entries preceding activation, and disconnects through heartbeat teardown. Hot-disable reaches cleanup: Gateway configuration changes update the process diagnostics flag and immediately stop the heartbeat when disabled. This reaches the newly added observer disconnect without restarting exporters or resetting their counters. Exporter isolation and bounded coverage: Reviewed lifecycle, queue-pressure, public-stream exclusion, Prometheus HTTP/cap, and OpenTelemetry signal-mode tests. The new measurements retain existing internal-provenance filtering, carry no attributes, and use ROOT_CONTEXT in OpenTelemetry. Tests were inspected, not executed during this read-only review.
Findings	None	None.
Security	None	None.

How this fits together

OpenClaw diagnostics collects runtime measurements and sends them through a bounded internal event queue. The existing metrics plugins turn these observations into Prometheus scrapes and OpenTelemetry metrics for operators.

flowchart LR
  A[Diagnostics heartbeat] --> B{Enabled and consumer interested?}
  B -->|Yes| C[Node GC observer]
  B -->|No| D[Disconnect observer]
  C --> E[Bounded internal event queue]
  E --> F[Prometheus histogram]
  E --> G[OpenTelemetry histogram]

Before merge

None.

Agent review details

Security

None.

PR surface

Source +101, Tests +134, Docs +37. Total +272 across 15 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	9	104	3	+101
Tests	4	181	47	+134
Docs	2	37	0	+37
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	15	322	50	+272

Review metrics

Metric	Value	Why it matters
Production versus test growth	production +101 net lines; tests +134 net lines	Production growth supplies the missing observer and exporter adapters; tests cover lifecycle, bounded delivery, and exporter isolation.
Operator surface	2 histograms; 0 new configuration options	Both existing exporters gain GC duration distributions through their current enablement paths.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #139547
Summary: This PR implements the linked request for cumulative GC duration histograms.

Members:

canonical: Expose garbage collection duration in diagnostics metrics #139547 - The open request describes the same heartbeat-owned observation and both exporter histograms.
adjacent_distinct: perf(prometheus): reduce temporary scrape allocations #139535 - The merged change optimizes Prometheus rendering allocations without adding GC observations.
adjacent_distinct: refactor(test): simplify catalog guards and fix native report fixture #139496 - The merged test-fixture repair supplies rebase context and does not implement this feature.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Technical review

Best possible solution:

Keep GC collection owned by the existing diagnostics lifecycle, with cumulative exporter histograms and explicit documentation of observation gaps and attribution limits.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this adds an observability capability rather than repairing an established behavior; the inspected main exporter paths lack the requested GC histograms.

Is this the best way to solve the issue?

Yes. Reusing the heartbeat, bounded dispatcher, and existing exporters avoids a competing telemetry lifecycle while preserving current settings and counter behavior.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 4da5d447f695.

Labels

Label justifications:

P2: This is a bounded operational diagnostics improvement for investigating intermittent stalls.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): Captured Linux Node 26.7 results exercise the heartbeat-owned observer through authenticated compiled Gateway scrapes and real hot toggles, while component results cover both exporters and controlled overhead. They resolve the prior proof request; runtime evidence remains explicitly bound to the earlier head, whose reported patch-identical rebase could not be independently compared locally.
proof: sufficient: Contributor real behavior proof is sufficient. Captured Linux Node 26.7 results exercise the heartbeat-owned observer through authenticated compiled Gateway scrapes and real hot toggles, while component results cover both exporters and controlled overhead. They resolve the prior proof request; runtime evidence remains explicitly bound to the earlier head, whose reported patch-identical rebase could not be independently compared locally.

Evidence

What I checked:

Pinned patch and lifecycle owner: Inspected the complete introduced patch from the pinned base to head. The observer checks diagnostics enablement and consumer interest, fences replaced callbacks, rejects entries preceding activation, and disconnects through heartbeat teardown. (src/logging/diagnostic-gc.ts:17, e3805714e7f8)
Hot-disable reaches cleanup: Gateway configuration changes update the process diagnostics flag and immediately stop the heartbeat when disabled. This reaches the newly added observer disconnect without restarting exporters or resetting their counters. (src/gateway/server-lifecycle.ts:632, e3805714e7f8)
Exporter isolation and bounded coverage: Reviewed lifecycle, queue-pressure, public-stream exclusion, Prometheus HTTP/cap, and OpenTelemetry signal-mode tests. The new measurements retain existing internal-provenance filtering, carry no attributes, and use ROOT_CONTEXT in OpenTelemetry. Tests were inspected, not executed during this read-only review. (extensions/diagnostics-otel/src/service-recorders-tools.ts:276, e3805714e7f8)
Previous proof request addressed: The captured PR body at context sourceRevision 462fe2d87b1aa6d5e56a3feff0c8a8c898283b9f55c61fca9a7ea3b03733dc23 records compiled Linux Gateway counts of 0→8→12, retention at 12 while disabled, and resumption to 17 after enable. It also records both-exporter component parity and six alternating OFF/ON overhead pairs, with 22 GC entries per enabled run and zero per disabled run. These directly address the rank-up move retained from feat: expose garbage collection duration metrics #139592 (comment). (f00f29cb6374)
Proof identity limitation: The body explicitly binds runtime evidence to the earlier reviewed head and reports an unchanged GC patch after rebase. That earlier commit is unavailable locally, so the attempted comparison could not independently establish patch identity. This is a reviewer limitation, not missing contributor evidence.
Still necessary on current main: The inspected main diagnostics owners retain event-loop measurements but lack these GC metric names; the GC observer file is absent from both fetched main and v2026.9.2. The provided related context identifies this PR as the implementation candidate for Expose garbage collection duration in diagnostics metrics #139547, while the merged scrape-allocation and test-fixture PRs address different work. (extensions/diagnostics-prometheus/src/service.ts:543, 4da5d447f695)

Likely related people:

steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
Vincent Koc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (1 earlier review cycle)

reviewed 2026-09-06T01:27:12.169Z sha f00f29cb6374e1b274bc144133ebb6c5ca0f410c :: blocked before merge. :: none

steipete · 2026-09-06T02:18:27Z

Merged via squash.

Prepared head SHA: e3805714e7f87e1a8fe2e1bef5990dc9d0bc9d69
Landed commit: 41fba39c40aa2840d7a3a8bb24eeac525c871ac9

* commit '6c55a99226ed0611ca088004c81b00fb296f1f71': (182 commits) test(ui): scope dashboard settings setup to the top frame (openclaw#139618) refactor(skills): share pending proposal creation (openclaw#139617) fix(release): accept frozen targets before worker bundles (openclaw#136699) fix(gateway): preserve percent sequences in media filenames (openclaw#139508) fix: preserve current update availability alongside update history (openclaw#139501) feat(diagnostics): expose garbage collection duration metrics (openclaw#139592) fix(ui): keep Logbook timestamps clear of the activity stripe (openclaw#139539) fix(ui): keep Swarm outcomes visible after completion (openclaw#139490) test: check file accessibility without runtime-specific return values (openclaw#139590) refactor(sdk): keep helper types private (openclaw#139601) fix(cron): preserve current-session delivery intent (openclaw#139491) fix: prevent browser profile permission hangs in Doctor (openclaw#139612) fix(agents): repair storage-failure test fixture after failover owner move (openclaw#139531) fix(cli): keep note text with Windows and Unicode line endings (openclaw#139512) fix(agents): keep isolated completion context on its admitted generation (openclaw#139607) fix: avoid declaration builds during source-server updates (openclaw#139392) fix(models): keep discovered models after catalog refresh failures (openclaw#139357) fix(gateway): record a transcript notice when a run fails before replying (openclaw#139439) refactor(ios): reuse shared integer conversion (openclaw#139581) fix(sessions): prevent database lock failures during reclamation (openclaw#139469) ...

* commit 'refs/openclaw/tasks/pr-135167-main-latest': test(ui): scope dashboard settings setup to the top frame (#139618) refactor(skills): share pending proposal creation (#139617) fix(release): accept frozen targets before worker bundles (#136699) fix(gateway): preserve percent sequences in media filenames (#139508) fix: preserve current update availability alongside update history (#139501) feat(diagnostics): expose garbage collection duration metrics (#139592) fix(ui): keep Logbook timestamps clear of the activity stripe (#139539) fix(ui): keep Swarm outcomes visible after completion (#139490) test: check file accessibility without runtime-specific return values (#139590) refactor(sdk): keep helper types private (#139601) fix(cron): preserve current-session delivery intent (#139491) fix: prevent browser profile permission hangs in Doctor (#139612) fix(agents): repair storage-failure test fixture after failover owner move (#139531) fix(cli): keep note text with Windows and Unicode line endings (#139512) fix(agents): keep isolated completion context on its admitted generation (#139607) fix: avoid declaration builds during source-server updates (#139392) fix(models): keep discovered models after catalog refresh failures (#139357) fix(gateway): record a transcript notice when a run fails before replying (#139439)

…aw#139592)

openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime extensions: diagnostics-otel Extension: diagnostics-otel extensions: diagnostics-prometheus size: M maintainer Maintainer-authored PR labels Sep 6, 2026

steipete marked this pull request as ready for review September 6, 2026 01:21

feat(diagnostics): expose garbage collection duration metrics

e380571

steipete force-pushed the codex/gc-duration-telemetry-20260905 branch from f00f29c to e380571 Compare September 6, 2026 01:58

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Sep 6, 2026

steipete merged commit 41fba39 into main Sep 6, 2026
182 checks passed

steipete deleted the codex/gc-duration-telemetry-20260905 branch September 6, 2026 02:17

ayaangazali mentioned this pull request Sep 6, 2026

fix(irc): config validation rejects the documented healthMonitor override #117302

Merged

steipete mentioned this pull request Sep 6, 2026

fix(sdk): preserve replay order for late run subscribers #139563

Merged

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Sep 6, 2026

feat(diagnostics): expose garbage collection duration metrics (opencl…

d91308e

…aw#139592)

clawsweeper Bot mentioned this pull request Oct 9, 2026

fix(gateway): prevent workspace icon failures after startup #167697

Merged

Navigation Menu

Repository navigation

feat: expose garbage collection duration metrics - #139592

feat: expose garbage collection duration metrics#139592
steipete merged 1 commit into
mainfrom
codex/gc-duration-telemetry-20260905

steipete commented Sep 6, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Sep 6, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Sep 6, 2026 •

edited

Loading

Security

PR surface

Review metrics

Root-cause cluster

Technical review

Labels

Evidence

Rating scale

Workflow

History

Uh oh!

Uh oh!

steipete commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

steipete commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ClawSweeper review complete

Uh oh!

clawsweeper Bot commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ClawSweeper review

What this changes

Merge readiness

Review scores

Verification

How this fits together

Before merge

Security

PR surface

Review metrics

Root-cause cluster

Technical review

Labels

Evidence

Rating scale

Workflow

History

Uh oh!

Uh oh!

steipete commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

steipete commented Sep 6, 2026 •

edited

Loading

clawsweeper Bot commented Sep 6, 2026 •

edited

Loading

clawsweeper Bot commented Sep 6, 2026 •

edited

Loading