Skip to content

fix: bound VolumeSnapshot and VolumeSnapshotContent error retry by CSISnapshotTimeout - #10493

Open
webdevsamran wants to merge 2 commits into
velero-io:mainfrom
webdevsamran:fix-csi-vsc-error-retry
Open

webdevsamran wants to merge 2 commits into
velero-io:mainfrom
webdevsamran:fix-csi-vsc-error-retry

Conversation

@webdevsamran

Copy link
Copy Markdown

Please add a summary of your change

In pkg/backup/actions/csi/volumesnapshot_action.go, Progress() previously treated VolumeSnapshotContent errors (vsc.Status.Error != nil) as terminal failures on first sight (marking progress.Completed = true and progress.Err = *vsc.Status.Error.Message), causing immediate backup failure (PartiallyFailed) when the external snapshotter controller encountered retryable provider errors (such as AWS SnapshotCreationPerVolumeRateExceeded). Conversely, VolumeSnapshot errors (vs.Status.Error != nil) were unconditionally treated as temporary, allowing them to hang until itemOperationTimeout (4 hours).

This PR brings consistent, symmetrical error handling to both VolumeSnapshot and VolumeSnapshotContent:

  1. If vsc.Status.Error != nil occurs within the configured backup.Spec.CSISnapshotTimeout, Velero logs a warning and allows the external snapshotter controller to retry instead of terminating immediately.
  2. If either vs.Status.Error or vsc.Status.Error persists beyond backup.Spec.CSISnapshotTimeout, Velero marks the operation failed with a descriptive error message, failing early rather than hanging until ItemOperationTimeout.
  3. Added unit tests covering both transient (within timeout window) and persistent (beyond timeout) error scenarios for both VolumeSnapshot and VolumeSnapshotContent.
  4. Added changelog entry in changelogs/unreleased/10483-webdevsamran.

Does your change fix a particular issue?

Fixes #10483

Please indicate you've done the following:

…ISnapshotTimeout

Progress() in pkg/backup/actions/csi/volumesnapshot_action.go previously treated VolumeSnapshotContent errors as terminal on first sight, causing immediate backup failures when external snapshot controllers recorded retryable provider errors (such as SnapshotCreationPerVolumeRateExceeded). VolumeSnapshot errors were also treated as unconditionally temporary.

This change aligns both VolumeSnapshot and VolumeSnapshotContent error handling to permit retries within backup.Spec.CSISnapshotTimeout, failing only once the error persists beyond that duration.

Fixes velero-io#10483

Signed-off-by: Samran Asif <samranwebdev2000@gmail.com>
@webdevsamran
webdevsamran requested a review from a team as a code owner September 7, 2026 06:24
@netlify

netlify Bot commented Sep 7, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for velero ready!

Name Link
🔨 Latest commit 8799be4
🔍 Latest deploy log https://app.netlify.com/projects/velero/deploys/6ab6c9ab32937d00075f0534
😎 Deploy Preview https://deploy-preview-10493--velero.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

kaovilai
kaovilai previously approved these changes Sep 25, 2026

@kaovilai kaovilai left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — bounds VolumeSnapshot/VSC error retry by CSISnapshotTimeout, preventing unbounded retry loops. Has unit tests and changelog, CI green.

Signed-off-by: Tiger Kaovilai <passawit.kaovilai@gmail.com>
@codecov

codecov Bot commented Sep 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CSI][Async Backup] Progress() treats retryable VolumeSnapshotContent errors as terminal, failing backups the snapshot controller was about to retry

2 participants