Skip to content

fix(heartbeat): empty scheduled heartbeats time out behind busy queues and enter retry storms - #137517

Merged
steipete merged 4 commits into
openclaw:mainfrom
LiuwqGit:feat/issue-137492-heartbeat-empty-preflight
Sep 5, 2026
Merged

fix(heartbeat): empty scheduled heartbeats time out behind busy queues and enter retry storms#137517
steipete merged 4 commits into
openclaw:mainfrom
LiuwqGit:feat/issue-137492-heartbeat-empty-preflight

Conversation

@LiuwqGit

@LiuwqGit LiuwqGit commented Sep 3, 2026 •
edited by steipete
Loading

Copy link
Copy Markdown
Contributor

Closes #137492

What Problem This Solves

Why This Change Was Made

  • Intended outcome: Scheduled interval heartbeats without heartbeat tasks resolve their existing preflight before the shared busy guards, so an effectively empty scratch terminates immediately with empty-heartbeat-file (non-retryable; the wake layer settles it at once) instead of returning a retryable requests-in-flight/cron-in-progress skip first.
  • Out of scope: Queue deferral is fully preserved for heartbeats that actually have work — actionable or missing scratch, event-triggered wakes, tagged cron events, and scheduled heartbeat tasks still wait behind busy queues as before. No changes to cron timeouts, watchdog contract, or the wake-layer retry policy.
  • Highest-risk area: The pre-busy preflight predicate in src/infra/heartbeat-runner-prompt.ts now also covers interval wakes; the resolved preflight is reused after the guards, matching the existing exec-event pattern. Per review feedback, the interval rule is folded into the existing predicate as a single expression with one concise invariant comment (no PR-specific commentary growth).
  • How is that risk mitigated? Two regression tests pin both halves of the ordering: empty scheduled scratch + busy main queue skips terminally, while actionable scratch + the same busy queue still defers with requests-in-flight. All existing busy-guard and scratch-gating suites pass unchanged.

User Impact

  • Success criteria: With agents.defaults.heartbeat.every configured and an empty/comments-only monitor scratch, openclaw automations runs --id <heartbeat-job-id> shows scheduled heartbeat runs finishing quickly with status=skipped, reason=empty-heartbeat-file (hundreds of ms, as before the regression) even while the Gateway is busy — no more errorReason=timeout at durationMs≈600000, no more "automations have failed 3+ runs in a row" retry storms in openclaw doctor.
  • What should reviewers focus on? src/infra/heartbeat-runner-prompt.ts (shouldPreflightWakeBeforeBusy), src/infra/heartbeat-runner-execution.ts (call site rename), and the two new cases in src/infra/heartbeat-runner.skips-busy-session-lane.test.ts.

Evidence

  • Real environment tested: Two real Gateway processes (isolated state dirs), one on unmodified origin/main and one on this PR branch, each with two agents: b-empty (heartbeat every: "20s", timeoutSeconds: 600, comments-only monitor scratch seeded through the production writeCronJobScratch path) and a-busy (an every-20s isolated automation whose real agent turn holds a model HTTP call on a local OpenAI-compatible endpoint for 15 minutes — real cron scheduler, real SQLite state, real wake settlement; no stubbed queue internals).
  • Exact steps or commands run after this patch: 
    # 1. Regression test (RED first on unmodified main, then GREEN after fix)
node scripts/run-vitest.mjs run src/infra/heartbeat-runner.skips-busy-session-lane.test.ts
# 2. Full targeted validation (oxfmt + oxlint + same-dir tests + tsgo:core)
skills/fix-pr/scripts/validate.sh src/infra/heartbeat-runner-execution.ts src/infra/heartbeat-runner-prompt.ts src/infra/heartbeat-runner.skips-busy-session-lane.test.ts
# 3. Live Gateway trace, identical scenario on both builds:
openclaw gateway run --port <port>            # main build vs PR build
openclaw automations runs --id <heartbeat-b-empty-job-id>
openclaw automations show  <busy-loop-job-id> # busy-window source
  • Evidence after fix:
    Contributor Gateway trace on the original production patch (head fbba43ffb4aabbde33f9006979fc414ee7c4fb73; same scenario as the main-build trace below): 
    # b-empty heartbeat monitor while a-busy's agent turn is in flight (busy window
# 06:26:11→06:41:11, then a second busy turn from 06:41:20):
total runs: 69 | duration min/max ms: 255 506
distinct results: {'heartbeat skipped: empty-heartbeat-file'}
2026-09-04T06:26:28 | finished | skipped | 260 ms | heartbeat skipped: empty-heartbeat-file
2026-09-04T06:47:11 | finished | skipped | 265 ms | heartbeat skipped: empty-heartbeat-file
2026-09-04T06:48:31 | finished | skipped | 260 ms | heartbeat skipped: empty-heartbeat-file
# 69 consecutive runs settled in 255–506 ms each; none deferred, none timed out.
  • Before evidence (live Gateway, unmodified origin/main, same scenario): 
    # busy window 06:36:13→06:51:13 (a-busy's model call held in flight):
2026-09-04T06:46:15 | finished | error   | 600006 ms | cron: job execution timed out
2026-09-04T06:51:15 | finished | skipped | 268499 ms | heartbeat skipped: empty-heartbeat-file
# The 06:36:15 run deferred behind the busy guards and hit the fixed 600 s
# watchdog; the post-backoff run settled only after the busy turn released
# (~4.5 min late). Both variants from the issue report reproduced live, and
# a-busy immediately opened the next busy turn at 06:51:22 — the retry storm.
# For contrast, the six runs before the busy window began (06:34:14–06:35:54)
# skipped normally in ~260 ms.
  • Observed result after fix: An empty scheduled heartbeat now resolves its terminal empty-heartbeat-file skip before the busy guards, so the wake settles immediately (that reason is not in the retryable skip set) and the owning cron automation records a skip instead of timing out; on unmodified main the same live scenario reproduces both the 600-second watchdog timeout and the minutes-late deferred settlement. The complement test confirms actionable scratch behind the same busy queue still defers with requests-in-flight.
  • Before evidence (regression test, unmodified main): AssertionError: expected { status: 'skipped', reason: 'requests-in-flight' } to deeply equal { status: 'skipped', reason: 'empty-heartbeat-file' } — the deferred-wake-then-timeout chain reported in the issue.
  • What was not tested: No live multi-agent gateway fleet reproduction (the report's 7-agent systemd setup); the live two-agent Gateway trace above covers the exact reported settlement path (empty scheduled monitor + real busy turn → 600 s timeout on main vs immediate terminal skip on this PR). Current-head hosted CI is complete; see the maintainer integration evidence below.
  • Proof tier: L2
  • Proof limitations: The Gateway traces above were supplied by the contributor; the maintainer integration preserves that production change. The original trace does not record the exact baseline commit. Current-main source inspection confirms the same preflight-ordering defect. The endpoint was a local OpenAI-compatible test server, not an authenticated provider.
  • Review state: Addressed ClawSweeper review items: real Gateway/cron settlement trace attached (before + after), and the interval rule compacted into the existing predicate with a concise invariant comment.

Maintainer Integration

The contributor's production fix is retained. The branch incorporates main 886aaa689076b7b7c2f7a4d8d0048e89f5855698 while preserving contributor ancestry and main's newer conversation/event ownership. The two regression cases now share the existing scratch fixture and both assert the recorded heartbeat outcome; this removes 15 test lines without dropping either behavior. The heartbeat docs describe immediate empty-monitor settlement before busy deferral.

Fresh independent automated review found no actionable P0–P2 findings. The integrated head 74676490d5548c658e4601eddbfbaeeef9964b88 passed CI run 33984969269, including the previously failing shared fixture. The original contributor head passed CI run 33888672199. The original empty-scratch regression failed before the fix with requests-in-flight instead of empty-heartbeat-file; actionable scratch retains busy deferral.

The first integrated CI run 33979544632 passed all jobs except an unrelated subagent authority fixture and the aggregate gate. This branch now includes the canonical fix from #139176, whose unchanged completion assertion has a proven tolerance for cold two-CPU runners. The heartbeat patch is unchanged; fresh independent automated review of the integrated candidate again found no actionable findings.

Production delta: +16/-11 (net +5), expressing the one canonical pre-busy predicate and its invariant comment. Tests: +47/-1 (net +46). Docs: +1/-1. No configuration, storage schema, timeout, or retry-policy changes.

Original fix and Gateway reproduction by @LiuwqGit; reported by @creanet-64 in #137492. Related alternative #137936 is superseded once this fix lands.

@clawsweeper

clawsweeper Bot commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@LiuwqGit LiuwqGit mentioned this pull request Sep 3, 2026
Closed
@clawsweeper clawsweeper Bot added P1 High-priority user-facing bug, regression, or broken workflow. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Sep 3, 2026
@clawsweeper

clawsweeper Bot commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed September 5, 2026, 2:49 PM ET / 18:49 UTC.

ClawSweeper review

What this changes

The PR makes scheduled heartbeats with empty stored checklists finish before busy-queue deferral, with complementary regression tests and documentation.

Merge readiness

✅ Ready for maintainer review

This remains a useful, focused repair: current main still contains the ordering defect, and the supplied Gateway traces demonstrate the corrected settlement behavior. No actionable findings remain from this review.

Priority: P1
Reviewed head: 74676490d5548c658e4601eddbfbaeeef9964b88

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused owner-boundary repair with convincing Gateway proof and complementary regression coverage.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (terminal): Contributor traces exercise the changed heartbeat preflight through real Gateway scheduling, SQLite scratch, and wake settlement: 69 patched runs skipped promptly while another agent held a real HTTP request. Baseline traces timed out or settled minutes late; the unrecorded baseline SHA is supplemented by current-main source confirmation.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): Contributor traces exercise the changed heartbeat preflight through real Gateway scheduling, SQLite scratch, and wake settlement: 69 patched runs skipped promptly while another agent held a real HTTP request. Baseline traces timed out or settled minutes late; the unrecorded baseline SHA is supplemented by current-main source confirmation.
Evidence reviewed 10 items Pinned change and review continuity: The independently inspected base-to-head delta matches the host introduction evidence. Comparing the previous reviewed head with HEAD produced no changes in the four heartbeat files; prior comment cleanup remains addressed.
Current main still needs the repair: Current main restricts early preflight to the qualifying exec-event case. Ordinary interval wakes therefore reach the busy guards before discovering empty scratch. The inspected current-main-to-head production difference is precisely the predicate extension and its caller rename.
Canonical preflight preserves real work: The expanded predicate excludes scheduled tasks. Existing preflight bypasses scratch suppression for exec, cron, explicit wake payloads, and tagged cron events; actionable or missing scratch continues through the unchanged busy guards. Session selection precedes execution, while session mutation and model invocation remain downstream. This ordering repair does not depend on a Codex runtime or protocol contract.
Findings None None.
Security None None.

How this fits together

The automation scheduler submits heartbeat wakes to the shared heartbeat runner. The runner checks stored instructions and execution activity, then either records a skip or starts an agent turn; the wake owner decides whether to settle or retry.

flowchart TD
  A[Scheduled automation] --> B[Heartbeat wake owner]
  B --> C[Scratch and event preflight]
  C -->|Empty and no due tasks| D[Recorded terminal skip]
  C -->|Work remains| E[Busy execution guards]
  E -->|Busy| B
  E -->|Available| F[Agent turn]
  D --> G[Automation settles]
  F --> G
Loading

Before merge

None.

Agent review details

Security

None.

PR surface

Source +5, Tests +46, Docs 0. Total +51 across 4 files.

View PR surface stats
Area Files Added Removed Net
Source 2 16 11 +5
Tests 1 47 1 +46
Docs 1 1 1 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 4 64 13 +51

Review metrics

Metric Value Why it matters
Production versus regression coverage Production net +5 lines; tests net +46 lines Production growth extends the existing predicate and documents its ordering invariant; test growth protects both terminal skipping and continued busy deferral.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #137492
Summary: This PR is a proven implementation candidate for the empty-monitor ordering defect; the timeout-contract repair addresses adjacent behavior.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Technical review

Best possible solution:

Keep one shared preflight predicate that settles empty monitors before contention while preserving deferral for actual work; the small growth expresses that ordering and its invariant without a parallel path.

Do we have a high-confidence way to reproduce the issue?

Yes: current-main source establishes that a scheduled empty monitor returns a retryable busy skip before scratch inspection. Contributor Gateway traces demonstrate the resulting timeout; this review did not execute a reproduction.

Is this the best way to solve the issue?

Yes: extending the existing owner-level preflight repairs the cause while preserving task/event handling. Changing watchdog duration or retry policy would leave empty monitors unnecessarily queued.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 676845210acb.

Labels

Label justifications:

  • P1: Real Gateway traces show routine empty heartbeat automations timing out and entering repeated retries during ordinary activity.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): Contributor traces exercise the changed heartbeat preflight through real Gateway scheduling, SQLite scratch, and wake settlement: 69 patched runs skipped promptly while another agent held a real HTTP request. Baseline traces timed out or settled minutes late; the unrecorded baseline SHA is supplemented by current-main source confirmation.
  • proof: sufficient: Contributor real behavior proof is sufficient. Contributor traces exercise the changed heartbeat preflight through real Gateway scheduling, SQLite scratch, and wake settlement: 69 patched runs skipped promptly while another agent held a real HTTP request. Baseline traces timed out or settled minutes late; the unrecorded baseline SHA is supplemented by current-main source confirmation.

Evidence

What I checked:

  • Pinned change and review continuity: The independently inspected base-to-head delta matches the host introduction evidence. Comparing the previous reviewed head with HEAD produced no changes in the four heartbeat files; prior comment cleanup remains addressed. (src/infra/heartbeat-runner-prompt.ts:63, 74676490d554)
  • Current main still needs the repair: Current main restricts early preflight to the qualifying exec-event case. Ordinary interval wakes therefore reach the busy guards before discovering empty scratch. The inspected current-main-to-head production difference is precisely the predicate extension and its caller rename. (src/infra/heartbeat-runner-prompt.ts:63, 676845210acb)
  • Canonical preflight preserves real work: The expanded predicate excludes scheduled tasks. Existing preflight bypasses scratch suppression for exec, cron, explicit wake payloads, and tagged cron events; actionable or missing scratch continues through the unchanged busy guards. Session selection precedes execution, while session mutation and model invocation remain downstream. This ordering repair does not depend on a Codex runtime or protocol contract. (src/infra/heartbeat-runner-prompt.ts:68, 74676490d554)
  • Caller and settlement boundary: The cron execution owner submits interval wakes and awaits heartbeat settlement. The wake owner retries requests-in-flight and cron-in-progress, but settles empty-heartbeat-file; runHeartbeatOnce returns the early skip without preparing or invoking an agent turn. (src/infra/heartbeat-wake.ts:554, 74676490d554)
  • Regression and sibling coverage: Two added runner cases distinguish empty scratch from actionable scratch under the same busy main queue and assert recorded outcomes plus no model call. Existing scratch-gating and stale-exec tests cover missing scratch, tagged cron events, triggered wakes, and coalesced task/cadence preservation. The shared fixture seeds scratch through CronService and writeCronJobScratch. (src/infra/heartbeat-runner.skips-busy-session-lane.test.ts:311, 74676490d554)
  • Contributor Gateway before-and-after proof: The supplied PR-body snapshot records two real Gateway processes with real cron scheduling, SQLite scratch, wake settlement, and an agent HTTP request held by a local compatible endpoint. On contributor head fbba43f, 69 empty-monitor runs settled in 255–506 ms during busy activity; the baseline trace recorded a 600006 ms timeout and a 268499 ms delayed skip. The baseline commit was not recorded. Captured context sourceRevision: 9b594e8f03b94218b05d66fc70f599b1413dfdf3ae845d618c1b3a18b4781df1. (fbba43ffb4aa)

Likely related people:

  • steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • Marvinthebored: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (6 earlier review cycles)
  • reviewed 2026-09-03T19:46:14.139Z sha b313018f80c8142cb2ad2a8137d802a0e0ceace4 :: needs real behavior proof before merge. :: none
  • reviewed 2026-09-04T01:43:52.827Z sha 0effee233b573e7cb342efdd07a7af159fb35375 :: blocked before merge. :: [P3] Reduce the incident comment to the ordering invariant
  • reviewed 2026-09-04T02:06:12.167Z sha df66b6da571990c44b4325d883350bbbe5ec5c8a :: blocked before merge. :: [P3] Remove the incident reference from the test comment
  • reviewed 2026-09-04T02:40:56.629Z sha dc1ad6e :: needs maintainer review before merge. :: none
  • reviewed 2026-09-04T15:44:20.581Z sha fbba43f :: blocked before merge. :: none
  • reviewed 2026-09-05T17:06:22.077Z sha 876ae2e :: needs maintainer review before merge. :: none

@LiuwqGit
LiuwqGit force-pushed the feat/issue-137492-heartbeat-empty-preflight branch 2 times, most recently from 5aaa62b to 0effee2 Compare September 4, 2026 01:35
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Sep 4, 2026
@LiuwqGit
LiuwqGit force-pushed the feat/issue-137492-heartbeat-empty-preflight branch from 0effee2 to df66b6d Compare September 4, 2026 01:57
@LiuwqGit
fix(heartbeat): skip empty scheduled heartbeats before busy queue guards
dc1ad6e 
System-owned heartbeat monitors whose monitor scratch is empty could be
deferred behind shared busy guards (main queue, cron activity, active
reply runs) before the runner resolved its empty-scratch preflight. The
wake layer treated the resulting requests-in-flight skip as retryable,
kept the settlement pending, and the owning automation waited until its
fixed 600-second watchdog expired, then entered transient-error backoff
across every configured agent.

Resolve the heartbeat preflight before the shared busy guards for
scheduled interval wakes without heartbeat tasks, so an effectively
empty scratch terminates immediately with empty-heartbeat-file instead
of retrying behind busy queues. Wakes with actionable scratch, missing
scratch, queued events, or scheduled heartbeat tasks keep ordinary
queue deferral.

Fixes openclaw#137492

AI-assisted: repaired and verified with Codex.
@LiuwqGit
LiuwqGit force-pushed the feat/issue-137492-heartbeat-empty-preflight branch from df66b6d to dc1ad6e Compare September 4, 2026 02:33
@steipete steipete self-assigned this Sep 5, 2026
@steipete @LiuwqGit
fix(heartbeat): refresh and tighten empty-monitor regression proof
876ae2e 
Merge current main while retaining the contributor's early empty-monitor
preflight. Consolidate the complementary busy-queue tests around the existing
scratch fixture and assert the recorded outcome for both cases. Document
that task-free scheduled interval monitors settle before busy deferral.

Fixes openclaw#137492
Co-authored-by: LiuwqGit <liu.weiqin@xydigit.com>
@steipete

steipete commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

@clawsweeper re-review

Integrated current main while preserving the original production repair and contributor ancestry. Head: 876ae2e. Consolidated the two regression cases around the existing scratch fixture, added the recorded-outcome assertion to both, and clarified the documented empty-monitor behavior. Independent source review and automated review found no actionable issues. The PR body retains and attributes the original Gateway before/after proof; fresh hosted CI is pending for this integrated head.

@clawsweeper

clawsweeper Bot commented Sep 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@steipete
test(heartbeat): integrate shared slow-runner CI repair
7467649 
Merge the canonical main fix for the unrelated subagent authority fixture
so fork CI can validate the heartbeat repair. Preserve the existing
heartbeat implementation, regression tests, documentation and contributor
ancestry without changing the assertions in the shared fixture.

Related: openclaw#139176
@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime labels Sep 5, 2026
@steipete

steipete commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Ready to land head 74676490d5548c658e4601eddbfbaeeef9964b88. Exact-head CI 33984969269 is green. Independent source review and fresh automated review found no actionable findings; the latest ClawSweeper review is also clear.

The heartbeat wake owner now resolves the existing empty-monitor preflight before retryable busy guards for task-free intervals. Actionable or absent scratch, due tasks and event-triggered work retain their existing handling. The two regression cases share the existing scratch fixture and assert both the returned and recorded outcomes. Production is net +5 lines; tests net +46, with no new production test seam. The docs explain the no-op ordering.

Inspected contributor Gateway proof shows 69 empty-monitor runs settling in 255–506 ms during a held model HTTP call, versus a baseline 600006 ms timeout. This proof remains explicitly attributed: the baseline commit was not recorded, and the endpoint was a local compatible test server, not an authenticated provider. Current-main source matches the reported defect; the original regression failed before the fix.

Validation also included git diff --check, the repository formatter on the changed test/docs, and pnpm docs:list on trusted main. The first integrated CI run exposed an unrelated cold-runner fixture failure; the branch now includes the canonical #139176 repair and passes CI. Contributor ancestry and @LiuwqGit's credit are preserved; @creanet-64 reported #137492.

@steipete
steipete merged commit 7f09c31 into openclaw:main Sep 5, 2026
220 checks passed
@steipete

steipete commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Merged via squash.

@github-actions github-actions Bot mentioned this pull request Sep 5, 2026
Open
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Sep 6, 2026
@LiuwqGit @steipete
fix(heartbeat): empty scheduled heartbeats time out behind busy queue…
23ab0a6 
…s and enter retry storms (openclaw#137517)

* fix(heartbeat): skip empty scheduled heartbeats before busy queue guards

System-owned heartbeat monitors whose monitor scratch is empty could be
deferred behind shared busy guards (main queue, cron activity, active
reply runs) before the runner resolved its empty-scratch preflight. The
wake layer treated the resulting requests-in-flight skip as retryable,
kept the settlement pending, and the owning automation waited until its
fixed 600-second watchdog expired, then entered transient-error backoff
across every configured agent.

Resolve the heartbeat preflight before the shared busy guards for
scheduled interval wakes without heartbeat tasks, so an effectively
empty scratch terminates immediately with empty-heartbeat-file instead
of retrying behind busy queues. Wakes with actionable scratch, missing
scratch, queued events, or scheduled heartbeat tasks keep ordinary
queue deferral.

Fixes openclaw#137492

AI-assisted: repaired and verified with Codex.

* ci: re-trigger security-fast check (transient pnpm audit 503)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
Co-authored-by: LiuwqGit <liu.weiqin@xydigit.com>
@hvhoon hvhoon mentioned this pull request Sep 9, 2026
Closed
@Finn763 Finn763 mentioned this pull request Sep 10, 2026
Closed
This was referenced Sep 14, 2026
Closed
Merged
@clawsweeper clawsweeper Bot mentioned this pull request Sep 22, 2026
Open
@LiuwqGit
LiuwqGit deleted the feat/issue-137492-heartbeat-empty-preflight branch September 22, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@steipete steipete

Labels

docs Improvements or additions to documentation gateway Gateway runtime P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Empty system-owned heartbeats wait behind busy queues, time out after 600 seconds, and enter retry storms

2 participants

@LiuwqGit @steipete