feat(agents): agents/tools/testing toolbox + deployed pi harness OOM/CPU/wall-time e2e by mattzcarey · Pull Request #2421 · cloudflare/agents

mattzcarey · 2026-09-30T16:57:54Z

Adds agents/tools/testing, a toolbox for breaking an agent's Durable Object mid-turn, and a nightly deployed e2e suite that uses it against the pi harness (examples/next/harnesses/pi).

The toolbox

agents/tools/testing has no dependencies, so any harness can wrap it in its own tool format. agents/tools/testing/ai wraps it as AI SDK tools (sleep, current_time, oom, burn_cpu).

import {
  burnCpu, fillMemory, generateBytes, // break the DO
  sleep, currentTime,                 // long turns
  longRunningPrompt, crashPrompt, secondaryTask
} from "agents/tools/testing";

await fillMemory(signal);                          // fills a buffer until the isolate passes 128 MB
await burnCpu(await fetch(`${url}/bytes`), signal); // hashes an endless generateBytes() body until the CPU limit

burnCpu hashes each byte 32 times, so it stays CPU-bound even though every read waits on I/O. CPU spent between reads counts toward the limit.
longRunningPrompt() loops sleep 60 / current_time until 30 minutes pass. crashPrompt("oom" | "burn_cpu") calls current_time, the crash tool once, then the secondary task. Both end with secondaryTask: one more current_time call and ANSWER: 893.
Local workerd enforces neither limit. Locally both tools run until aborted.

Deployed pi harness e2e (nightly)

sequenceDiagram
  participant T as deployed.test.ts
  participant W as Worker (throwaway)
  participant A as PiChaosAgent (pi harness)
  participant R as ChaosResults
  T->>W: POST /e2e/:agent/submit
  W->>A: submit(prompt)
  Note over A: current_time → crash / reset / redeploy<br/>wake alarm restarts it, pi recovers<br/>current_time → ANSWER: 893
  A->>R: progress (generating, tool starts) and final status
  loop every 10s
    T->>W: GET /e2e/:agent/result
    W->>R: get()
  end

Besides OOM, CPU and wall time, the suite simulates platform kills: ctx.abort() (a runtime reset) while the model streams and 15 s into a replay-safe sleep 60, and a redeploy at the same point. harness.test.ts adds the deterministic local version: abortAllDurableObjects() mid-stream.
The suite never touches the agent it tests, apart from the kill request. Polling the agent would start a new instance and resume pi, so the client would do the recovery. The agent pushes progress and its outcome to ChaosResults instead, so recovery has to come from the harness's wake alarm.
Crash tools run at most once per object. The marker is written with storage.sync() before the crash. A second call would be another memory-limit strike toward Lifecycle sealing the object's recovery jobs.
burn_cpu reads from the Worker's own /bytes route through ctx.exports.default.
New e2e-deployed-pi-harness job in nightly.yml (75 min timeout). PI_E2E_SCENARIOS=oom,burn_cpu runs a subset.
The playground's sleep and current_time moved to src/tools.ts so the e2e Worker reuses them.

Results from runs against a real account:

Scenario	Took	Instances	Transcript
oom	42 s	2	`current_time`, `oom` ⚠ interrupted, `current_time`, `ANSWER: 893`
burn_cpu	52 s	2	`current_time`, `burn_cpu` ⚠ interrupted, `current_time`, `ANSWER: 893`
abort-generation (`ctx.abort()` while the model streams)	95 s	2	aborted partial kept, story regenerated, `current_time`, `ANSWER: 893`
abort-tool (`ctx.abort()` 15 s into `sleep 60`)	84 s	2	one `sleep`, rerun with its memoized deadline, `current_time`, `ANSWER: 893`
redeploy (15 s into `sleep 60`)	80 s	2	same as abort-tool
wall-time	35 min	1	30 × (`sleep 60`, `current_time`), then `current_time`, `ANSWER: 893`

wrangler tail on the burn_cpu run shows the alarm invocation stopped at 32.5 s CPU (the 30 s actor top-up plus 2.5 s rollover). Each recovery came from the next 30 s heartbeat alarm.

changeset-bot · 2026-09-30T16:58:01Z

🦋 Changeset detected

Latest commit: 71035dd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
agents	Minor
@cloudflare/agent-think	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

agent-think · 2026-09-30T16:59:40Z

🟢 agents import sizes: 2 entry points changed, no growth

	Entry point	Exports	Largest gzip change	Size now
🆕	`agents/tools/testing`	9 new	—	803 B
🆕	`agents/tools/testing/ai`	1 new	—	86.6 KiB

Changed exports (10)

	Import	Gzip change	Size now
🆕	`agents/tools/testing/ai#createTestingTools`	—	86.6 KiB
🆕	`agents/tools/testing#burnCpu`	—	803 B
🆕	`agents/tools/testing#sleep`	—	692 B
🆕	`agents/tools/testing#generateBytes`	—	691 B
🆕	`agents/tools/testing#fillMemory`	—	660 B
🆕	`agents/tools/testing#currentTime`	—	596 B
🆕	`agents/tools/testing#crashPrompt`	—	547 B
🆕	`agents/tools/testing#longRunningPrompt`	—	547 B
🆕	`agents/tools/testing#secondaryTask`	—	547 B
🆕	`agents/tools/testing#testingPrompts`	—	547 B

How this works

Each runtime export is bundled on its own, minified, and gzipped. Changes smaller than 100 B, or smaller than 1% and 1 KiB, are ignored. Growth over 10% or 5 KiB is marked 🔴. This report is informational and does not fail CI. The workflow artifact contains every measurement.

_{Compared 6e1c3467 → 23b69b76 · workflow run · reported by agent-think[bot]}

pkg-pr-new · 2026-09-30T17:10:40Z

Open in StackBlitz

agents

npm i https://pkg.pr.new/agents@2421

@cloudflare/ai-chat

npm i https://pkg.pr.new/@cloudflare/ai-chat@2421

@cloudflare/codemode

npm i https://pkg.pr.new/@cloudflare/codemode@2421

hono-agents

npm i https://pkg.pr.new/hono-agents@2421

@cloudflare/shell

npm i https://pkg.pr.new/@cloudflare/shell@2421

@cloudflare/think

npm i https://pkg.pr.new/@cloudflare/think@2421

@cloudflare/voice

npm i https://pkg.pr.new/@cloudflare/voice@2421

@cloudflare/worker-bundler

npm i https://pkg.pr.new/@cloudflare/worker-bundler@2421

commit: 23b69b7

createTestingTools() returns AI SDK tools for breaking an agent's Durable Object on purpose: sleep, get_current_time, oom (allocate until the 128 MB isolate limit), and burn_cpu (spin without yielding until the CPU limit). testingPrompts/longRunningPrompt drive a 30-minute sleep loop and the destructive tools.

…i harness e2e agents/tools/testing is now dependency-free: fillMemory (fill a buffer until the isolate OOMs), burnCpu (hash a fetch() body of generateBytes() until the CPU limit), sleep, currentTime, and mid-turn prompts that end in a checkable secondary task. agents/tools/testing/ai wraps them as AI SDK tools. examples/next/harnesses/pi gains a deployed e2e suite, run nightly: an OOM and a CPU-limit kill mid-turn, and a 30-minute turn past the alarm's 15-minute wall time. Each agent reports to a separate results object, so the suite never wakes the agent and recovery must come from the harness's wake alarm.

devin-ai-integration

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 4 new potential issues.

devin-ai-integration · 2026-10-01T11:35:20Z

generateBytes

+export function createTestingTools(
+  options: CreateTestingToolsOptions = {}
+): TestingTools {
+  const bytes = options.bytes ?? (async () => generateBytes());


🔴 Local CPU injection blocks cancellation

Locally, the default burn_cpu stream never yields to an abort scheduled by a timer. generateBytes pulls synchronously, so the endless read loop monopolizes microtasks and wedges the worker.

Learn more

The default AI SDK tool uses an in-process endless stream. Its pull callback enqueues immediately in generateBytes, and burnCpu loops on resolved reads without giving timers another turn. A signal aborted by a timer cannot fire during the loop. On local workerd, there is no CPU limit to terminate it.

Example: In a local test, call createTestingTools().burn_cpu.execute with a signal scheduled to abort after one second. The timeout never runs; the tool never rejects and the test runner remains blocked.

Recommended fix: Make the default stream periodically yield to a macrotask, or require an asynchronous byte source for the AI SDK default. Verify that a scheduled abort can execute with no external bytes callback and that deployed CPU consumption remains sufficient to trigger the limit.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-01T11:35:21Z

BurnCpuResult

+      for (const byte of value) {
+        for (let round = 0; round < BURN_ROUNDS_PER_BYTE; round++) {
+          checksum = Math.imul(checksum ^ byte, 0x01000193);
+        }


🟡 CPU checksum differs from FNV-1a

For nonempty bodies, burnCpu hashes each byte 32 times instead of once. Its reported checksum differs from the documented FNV-1a hash, breaking comparisons with expected hashes.

Learn more

The exported BurnCpuResult describes checksum as an FNV-1a hash of the byte stream. FNV-1a processes each input byte once; the extra rounds change its value even for a one-byte body. The CPU-load objective can remain independent of the returned checksum.

Example: For a response containing one byte 0x00, conventional FNV-1a updates the initial value once. This implementation updates it 32 times, so its result does not match the expected hash.

Recommended fix: Maintain a separate standard FNV-1a checksum updated once per byte and a separate accumulator for the extra CPU rounds, or rename and document the returned value as a nonstandard stress-test checksum. Add a known-vector test.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-01T11:35:23Z

burnCpu

+  response: Response,
+  signal?: AbortSignal
+): Promise<BurnCpuResult> {
+  signal?.throwIfAborted();


🟡 Pre-aborted CPU streams remain open

With an already aborted signal, burnCpu rejects before canceling response.body. A streamed response stays open despite the cancellation contract.

Learn more

burnCpu promises to cancel the response body on abort. Its initial abort check runs before obtaining a reader or installing its abort listener, so it leaves an already open streaming response uncanceled.

Example: Fetch an endless /bytes response, abort the signal, then call burnCpu(response, signal). The call rejects immediately, but nothing cancels the fetched response body.

Recommended fix: When the signal is already aborted, cancel response.body before rejecting, preserving the abort reason. Cover the pre-aborted case with a stream whose cancel callback records invocation.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-01T11:35:24Z

+export function createTestingTools(
+  options: CreateTestingToolsOptions = {}
+): TestingTools {
+  const bytes = options.bytes ?? (async () => generateBytes());


🔍 Description and toolbox API diverge

The description advertises a root-level AI adapter, get_current_time, and bounded local crash tools. The implementation uses a separate adapter subpath, current_time, and unbounded local fault injection. Update the description to match the published API.

Was this helpful? React with 👍 or 👎 to provide feedback.

…harness Local: crash the object mid-stream with abortAllDurableObjects(); the wake alarm resumes it and pi regenerates the answer, keeping the streamed part as an aborted message. Deployed: ctx.abort() while the model streams, ctx.abort() 15s into a replay-safe sleep, and a redeploy 15s into the same sleep. Each agent reports progress to ChaosResults so the suite can time the interruption without waking the agent.

devin-ai-integration

Devin Review found 2 new potential issues.

devin-ai-integration · 2026-10-01T16:01:13Z

+      await waitForProgress(agent, "generating", 3 * MINUTE);
+      await kill(agent);


🔍 Mid-stream reset can race with completion

After generating appears, the model can finish before kill reaches the agent. The aborted-message assertion then fails without exercising recovery.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-01T16:01:14Z

+      await waitForProgress(agent, "generating", 3 * MINUTE);
+      await kill(agent);


🔍 Test description omits deliberate agent reset

The PR description says the suite never touches the tested agent. The new kill request does; only progress and result polling avoid it.

Was this helpful? React with 👍 or 👎 to provide feedback.

This comment was marked as resolved.

Sign in to view

mattzcarey added 2 commits October 1, 2026 11:39

mattzcarey force-pushed the feat/testing-toolbox branch from a43d449 to 23b69b7 Compare October 1, 2026 11:31

devin-ai-integration Bot reviewed Oct 1, 2026

View reviewed changes

mattzcarey changed the title ~~feat(agents): add agents/tools/testing fault-injection toolbox~~ feat(agents): agents/tools/testing toolbox + deployed pi harness OOM/CPU/wall-time e2e Oct 1, 2026

devin-ai-integration Bot reviewed Oct 1, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(agents): agents/tools/testing toolbox + deployed pi harness OOM/CPU/wall-time e2e - #2421

feat(agents): agents/tools/testing toolbox + deployed pi harness OOM/CPU/wall-time e2e#2421
mattzcarey wants to merge 3 commits into
mainfrom
feat/testing-toolbox

mattzcarey commented Sep 30, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Sep 30, 2026 •

edited

Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

agent-think Bot commented Sep 30, 2026 •

edited

Loading

Uh oh!

pkg-pr-new Bot commented Sep 30, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		await waitForProgress(agent, "generating", 3 * MINUTE);
		await kill(agent);

Conversation

mattzcarey commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The toolbox

Deployed pi harness e2e (nightly)

Uh oh!

changeset-bot Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

This comment was marked as resolved.

Uh oh!

agent-think Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 agents import sizes: 2 entry points changed, no growth

Uh oh!

pkg-pr-new Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mattzcarey commented Sep 30, 2026 •

edited

Loading

changeset-bot Bot commented Sep 30, 2026 •

edited

Loading

agent-think Bot commented Sep 30, 2026 •

edited

Loading

pkg-pr-new Bot commented Sep 30, 2026 •

edited

Loading

devin-ai-integration Bot left a comment •

edited

Loading