Repository navigation
snapshots/proxy: add optional default_timeout for proxy snapshotter calls - #14187
Conversation
There was a problem hiding this comment.
🔵 Needs a closer look
Preserve the original context for Walk callbacks while bounding only the gRPC stream.
Pull request overview
Adds configurable default deadlines for proxy snapshotter gRPC calls while preserving caller-provided deadlines.
Changes:
- Adds
default_timeoutconfiguration. - Applies optional timeouts across snapshotter methods.
- Wires configuration, tests, and documentation.
File summaries
| File | Summary |
|---|---|
docs/man/containerd-config.toml.5.md |
Documents default_timeout. |
core/snapshots/proxy/proxy.go |
Adds configurable call timeouts. |
core/snapshots/proxy/proxy_test.go |
Tests timeout behavior. |
cmd/containerd/server/server.go |
Parses and passes timeout configuration. |
cmd/containerd/server/config/config.go |
Defines the timeout setting. |
Review details
Suppressed comments (1)
core/snapshots/proxy/proxy.go:211
- The derived timeout context is also passed to
Walkcallbacks below, so callbacks receive a deadline that the caller did not provide. This changes the callback contract compared with other snapshotter implementations and can stop callback work at the proxy RPC timeout even though only the remoteListstream should be bounded. Keep the original incoming context forfnwhile using a separate derived context for the gRPC stream.
ctx, cancel := p.withTimeout(ctx)
defer cancel()
- Files reviewed: 5/5 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
bb74944 to
e4617b2
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Reject non-positive configured durations and strengthen the timeout assertion.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (1)
cmd/containerd/server/server.go:336
- A configured non-empty duration such as
0sor-1sparses successfully, butWithDefaultTimeouttreats it as disabled, so a typo can silently leave all proxy calls unbounded even thoughdefault_timeoutis set. Reject non-positive durations during config loading instead of installing an option that disables the safeguard.
if pp.DefaultTimeout != "" {
d, perr := time.ParseDuration(pp.DefaultTimeout)
if perr != nil {
return nil, fmt.Errorf("proxy plugin %q: unable to parse default_timeout %q into a time duration: %w", name, pp.DefaultTimeout, perr)
}
ssopts = append(ssopts, ssproxy.WithDefaultTimeout(d))
- Files reviewed: 5/5 changed files
- Comments generated: 1
- Review effort level: Lite
e4617b2 to
350ade8
Compare
350ade8 to
9400c17
Compare
9400c17 to
652b842
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Walk callbacks can still block beyond the RPC deadline while holding the snapshotter lock.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
Resolved since last review (1)
fuweid
left a comment
There was a problem hiding this comment.
Overall, LGTM. please address one comment. thanks
652b842 to
eacccdd
Compare
eacccdd to
e3fc501
Compare
mikebrow
left a comment
There was a problem hiding this comment.
couple nits to consider on the documentation
…alls A proxy snapshotter is a separate process, and none of its 10 gRPC methods carry a deadline unless the caller already set one. Snapshot GC calls Walk and Remove under the snapshotter-level lock, so one wedged call blocks every later mutating call on that snapshotter. Add default_timeout to the proxy_plugins config section. When set, each proxy snapshotter call gets that deadline if the incoming context has none; a context that already carries its own deadline is left alone. Empty or unset keeps today's behavior, so nothing changes for existing configs. NewSnapshotter keeps its signature and wraps the new NewSnapshotterWithOpts, so existing callers of the exported constructor are unaffected. Assisted-by: Claude Code Signed-off-by: Nahum Litvin <nahuml@wix.com>
e3fc501 to
2a843ba
Compare
Problem
A proxy snapshotter runs in its own process and none of its 10 gRPC methods get a deadline unless the caller already set one. Snapshot GC calls
Walkand thenRemoveper snapshot while holding the snapshotter-level lock ingarbageCollect, so one wedged call blocks every later mutating call on that snapshotter and no new container using it can start. #14010 lists the other daemon paths with the same shape.#13799 and #14026 each bounded one component. Per @fuweid in #14026, the bound belongs at the client instead, where it covers all 10 methods at once. @mikebrow agreed on 09/02. #13799 was reverted in #14119 ahead of 2.4 to go this route.
Fix
New
default_timeouton[proxy_plugins]. When set, each proxy snapshotter call gets that deadline if the incoming context has none. A context that already carries its own deadline is left as it is, so this does not shorten anyone's request budget. Empty or unset keeps current behavior.The option goes through a new
NewSnapshotterWithOpts, so the exportedNewSnapshottersignature is unchanged for package users.Not in here
Total pass duration is still unbounded: N snapshots each finishing just under the limit still add up while
garbageCollectholds the lock. That is @mikebrow's second point from the #14026 thread and it needs the lock itself reworked, so I left it out.The content store proxy has the same gap, but it does not fit this shape.
Writeroutlives the call that created it, so a per-call deadline would cut an ingest that is still being written. It needs its own key and its own PR.Walkgets one deadline for the whole stream, not one per received message, so time spent in callbacks counts against it and the nextRecvfails once it expires. It cannot interrupt a callback that ignores its context. The GC callback only records keys, the call that wedges is the proxy RPC. Say the word if you would ratherWalkandCleanupwere left out.Tests
TestWithTimeoutcovers both guards. Drop thedefaultTimeout <= 0check and the disabled case goes red; drop thectx.Deadline()check and the case that preserves a caller's 2h deadline goes red.TestRemoveAppliesDefaultTimeoutgoes throughNewSnapshotterWithOptsagainst a client that blocks until its context is done, so deleting the wrapper fromRemovefails it in 5s with a clear message instead of hanging the package.TestLoadPluginsRejectsMalformedProxyDefaultTimeoutcovers the config path, where a bad duration has to fail daemon startup rather than be ignored. Only the error path is exercised:LoadPluginsreturns before it reaches the global plugin registry, which panics if a test registers the same id twice.Refs #14010. Supersedes #14026.