Skip to content

fix(adk): preserve checkpoints for canceled tool streams - #1303

Merged
shentongmartin merged 1 commit into
mainfrom
fix/v09-canceled-tool-stream-checkpoint
Sep 20, 2026
Merged

shentongmartin merged 1 commit into
mainfrom
fix/v09-canceled-tool-stream-checkpoint

Conversation

@shentongmartin

Copy link
Copy Markdown
Contributor

Problem

On v0.9.19, CancelImmediate with WithRecursive can terminate an AgentTool child while its ChatModelAgent is consuming a StreamableTool result. The resulting ErrStreamCanceled escapes as an ordinary stream failure during checkpoint materialization, so the child finishes without resumable ToolNode interrupt state. The caller then observes ErrExecutionEnded, and Runner.Resume cannot continue from the interrupted tool call.

Solution

Treat framework-driven tool stream cancellation as checkpoint control flow:

  • identify cancellation consistently across the ADK and compose boundary;
  • retain completed parallel tool results while marking canceled calls for rerun;
  • convert the canceled tool stream into a composite interrupt carrying ToolNode input and rerun state;
  • merge that nested interrupt into the graph checkpoint instead of serializing the canceled stream as output.

This is a v0.9 backport of #1224 (0fb0380e) with an end-to-end regression covering AgentTool -> ChatModelAgent -> StreamableTool cancellation and resume.

Key Insight

A framework-canceled tool stream is not a terminal tool failure. Its partial output cannot be treated as completed state; the checkpoint must preserve the original ToolNode input, retain successfully completed sibling calls, and rerun only canceled calls on resume.


问题

在 v0.9.19 中,当 AgentTool 内部的 ChatModelAgent 正在消费 StreamableTool 结果时,使用 CancelImmediate + WithRecursive 可能让 ErrStreamCanceled 在 checkpoint 物化阶段作为普通流错误返回。child 因此结束,但没有生成可恢复的 ToolNode interrupt state;调用方收到 ErrExecutionEnded,Runner.Resume 也无法从被中断的工具调用继续。

解决方案

将框架触发的工具流取消作为 checkpoint 控制流处理:

  • 在 ADK 与 compose 边界统一识别 stream cancellation;
  • 保留已经完成的并行工具结果,并将被取消的调用标记为 rerun;
  • 把 canceled tool stream 转换为携带 ToolNode 输入和 rerun state 的 composite interrupt;
  • 将该 nested interrupt 合并到 graph checkpoint,而不是把已取消的 stream 当作输出序列化。

这是 #1224(0fb0380e)面向 v0.9 的 backport,并补充了 AgentTool -> ChatModelAgent -> StreamableTool 取消与恢复的端到端回归覆盖。

关键结论

框架取消的工具流不是终态工具失败。其部分输出不能被视为已完成状态;checkpoint 必须保留原始 ToolNode 输入和已完成的 sibling tool 结果,并在恢复时只重跑被取消的调用。

Backport #1224 to v0.9 and add coverage for recursive AgentTool cancellation while its child runs a StreamableTool.

(cherry picked from commit 0fb0380)

Change-Id: Ic18c6f7d21a8a5e19ac5e992718a06d9f3b4a261
@codecov

codecov Bot commented Sep 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.90909% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.79%. Comparing base (9d983b3) to head (30e7924).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
compose/tool_node.go 91.26% 6 Missing and 5 partials ⚠️
compose/checkpoint.go 89.09% 3 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1303      +/-   ##
==========================================
+ Coverage   83.77%   83.79%   +0.02%     
==========================================
  Files         162      162              
  Lines       23940    24123     +183     
==========================================
+ Hits        20055    20215     +160     
- Misses       2625     2643      +18     
- Partials     1260     1265       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@shentongmartin
shentongmartin merged commit b9539ec into main Sep 20, 2026
19 checks passed
@shentongmartin
shentongmartin deleted the fix/v09-canceled-tool-stream-checkpoint branch September 20, 2026 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants