What happened?
kubectl cp --retries=N resumes a copy only when the exec that runs tar ends with a nil error. TarPipe.initReadFrom runs the exec in a goroutine with cmdutil.CheckErr(t.o.execute(options)), so any error from the session exits kubectl before TarPipe.Read gets to retry; the resume only happens after the pipe was closed without an error.
Over WebSocket (the default since 1.30) a cut connection ends the exec with error reading from error stream: ... close 1006, so --retries has not resumed after a cut there. Over SPDY a cut ended the exec with nil, which is the client-go half of #142376 and the only reason --retries resumed. With the client-go change from that issue (part 1 of #142376 (comment)) a cut over SPDY is an error too, and --retries stops resuming there as well.
kubectl cp of a 3 GiB file from a pod on a stock kind v1.37.0 cluster, the connection between the apiserver and the kubelet killed 5 s in (ss -K on the node), one run per cell:
|
kubectl from master |
kubectl with the client-go change |
SPDY, --retries=3 |
Resuming copy at 728108032 bytes, retry 1/3, complete, exit 0 |
error: connection closed before the command's status was received; the output may be incomplete, exit 1, no resume |
WebSocket, --retries=3 |
error: error reading from error stream: next reader: websocket: close 1006 (abnormal closure): unexpected EOF, exit 1, no resume |
same |
What did you expect to happen?
--retries resumes after a cut connection on either transport; that is what it was added for in #104792 (#60140).
How can we reproduce it (as minimally and precisely as possible)?
- A kind cluster and a pod with a large file (
head -c 3221225472 /dev/zero > /tmp/big).
kubectl cp --retries=3 pod:/tmp/big ./big- A few seconds in, on the node:
ss -K sport = :10250, which kills the apiserver's connection to the kubelet.
Anything else we need to know?
The fix is in TarPipe.initReadFrom: when MaxTries != 0, close the pipe with the exec's error instead of calling CheckErr, so that Read retries; an exit status from tar (utilexec.ExitError) stays fatal. That raises the question of which errors are worth a retry, and whether --retries=-1 needs a backoff or a stop when no byte was copied since the last attempt, since a persistent error (the pod is gone) would otherwise retry in a tight loop. I can send the PR once the client-go change has landed.
Kubernetes version
kubectl built from master (e234a6f) against kind v1.37.0 (containerd 2.3.4).
What happened?
kubectl cp --retries=Nresumes a copy only when the exec that runstarends with a nil error.TarPipe.initReadFromruns the exec in a goroutine withcmdutil.CheckErr(t.o.execute(options)), so any error from the session exits kubectl beforeTarPipe.Readgets to retry; the resume only happens after the pipe was closed without an error.Over WebSocket (the default since 1.30) a cut connection ends the exec with
error reading from error stream: ... close 1006, so--retrieshas not resumed after a cut there. Over SPDY a cut ended the exec with nil, which is the client-go half of #142376 and the only reason--retriesresumed. With the client-go change from that issue (part 1 of #142376 (comment)) a cut over SPDY is an error too, and--retriesstops resuming there as well.kubectl cpof a 3 GiB file from a pod on a stock kind v1.37.0 cluster, the connection between the apiserver and the kubelet killed 5 s in (ss -Kon the node), one run per cell:--retries=3Resuming copy at 728108032 bytes, retry 1/3, complete, exit 0error: connection closed before the command's status was received; the output may be incomplete, exit 1, no resume--retries=3error: error reading from error stream: next reader: websocket: close 1006 (abnormal closure): unexpected EOF, exit 1, no resumeWhat did you expect to happen?
--retriesresumes after a cut connection on either transport; that is what it was added for in #104792 (#60140).How can we reproduce it (as minimally and precisely as possible)?
head -c 3221225472 /dev/zero > /tmp/big).kubectl cp --retries=3 pod:/tmp/big ./bigss -K sport = :10250, which kills the apiserver's connection to the kubelet.Anything else we need to know?
The fix is in
TarPipe.initReadFrom: whenMaxTries != 0, close the pipe with the exec's error instead of callingCheckErr, so thatReadretries; an exit status fromtar(utilexec.ExitError) stays fatal. That raises the question of which errors are worth a retry, and whether--retries=-1needs a backoff or a stop when no byte was copied since the last attempt, since a persistent error (the pod is gone) would otherwise retry in a tight loop. I can send the PR once the client-go change has landed.Kubernetes version
kubectl built from master (e234a6f) against kind v1.37.0 (containerd 2.3.4).