Repository navigation
Velero backups failing due to blob missing from the repository #8654
Description
Activity
There is no way to recover a missing repo blob from the source volume, you can only run a new backup and create blobs for the new backup.
@Lyndon-Li thanks for the confirmation!
Running new backup will search for existing blobs and fails if it doesnt find. Is there a way to instruct kopia through velero to run a fresh backup like kopia offerskopia snapshot create --force-hash=100?And what might be possible reason for missing blob?
Running new backup will search for existing blobs and fails if it doesnt find.
What is the error when creating a new backup and failing?
It gives the same error:
error to find manifests: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID qf639e8abd6afa38f0b504f5e4643753e-sf3d16207b6fc75a5130: BLOB not foundIf a new clean backup cannot be created, then I agree we need a mitigation to force kopia to create fresh backup/repo assuming there is nothing useful still usable being overwritten.
Yes, kopia offers --force-hash=100 flag to create everything fresh, but i dont see a way to do that through the velero aws plugin.
Is there a way to instruct kopia through velero to run a fresh backup like kopia offers kopia snapshot create --force-hash=100 ?
It is out the policy of Velero to overwrite an existing repo (even though the repo cannot be used by Velero because some of blobs are missing, full/partial of the backup data may still be recovered if users desire). The recommended way to do this is --- If you are sure the repo data is never used, you can clear the objects from your object store; then you can delete the related backuprepository CR so that Velero creates a new repo for you along with the next backup.
force-hashis not used to overwrite an existing repo, but to tell the uploader to do a full backup and bypass the deduplication results. It cannot backup failure problem in an existing repo as you mentioned.And what might be possible reason for missing blob?
Velero and Kopia never delete an existing blob from the repo, so we don't see any possibility from Velero/Kopia.
We suggest you to check from your object storage or any other app that also manipulate the same bucket.@Lyndon-Li Nothing else can manipulate our bucket. How does the cleanup happen in s3 bucket if velero backup is deleted? It cleans up blobs which are not used by any snapshot right?
velero delete commands create DeleteBackupRequest object in cluster and velero controller will clean up data in object store along with marking data in kopia for deletion. After that, velero backup cr is removed from cluster by velero.
Data in kopia will not disappear right away, will take couple days over 2+ maintenance cycles.
I just realized that Velero 1.12 is being used and the missing blob is actually a snapshot manifest. Then see issue #7094 which makes an improvement to ignore the previous snapshot if it is not found.
Velero 1.12 is very old, please upgrade to the latest release.
@Lyndon-Li #7904 makes no mention of "snapshot". Is it related?
We are in velero 1.12 because of this issue.
Our data uploads take more than an hour and the token refresh currently doesnt seem to be supported by IRSAThis issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.
This issue was closed because it has been stalled for 14 days with no activity.
Have to re-open/adress this, as it keeps happening on our end using PodVolumeBackups (no CSI snapshot available) and S3; running on Chart version
12.0.2(1.18.1) andvelero-plugin-for-aws:v1.14.1Reopening because this report on Velero v1.18.1 appears to expose a path not covered by the earlier fix in #7094 / #7096.
I checked the released code tags. The #7094 mitigation has been included since v1.12.2 and is present in v1.18.1, v1.18.2, and v1.18.3-rc.1. However, it only falls back to a full backup when an explicitly supplied parent snapshot fails to load.
In this report, the failure occurs while automatically discovering previous snapshots. findPreviousSnapshotManifest() calls Kopia’s snapshot-listing code, which fails while loading a manifest whose content blob is missing. SnapshotSource() then treats that discovery error as fatal:
Failed to find previous kopia snapshot manifests ... unable to load manifest contents ... failed to get blob ... BLOB not foundConsequently, a corrupt or missing old manifest blob can prevent all subsequent PodVolumeBackups for that source, even though the source volume remains available and Velero could potentially create a new full backup.
This behavior is still present in the checked v1.18.x tags, so upgrading from v1.18.1 does not currently address this exact failure path. The AWS plugin version is unlikely to control it because the error handling is in Velero core’s pkg/uploader/kopia/snapshot.go.
We should investigate whether missing/corrupt manifests encountered during automatic parent discovery can be skipped, or whether this condition can safely fall back to a full backup. This would not recover the missing blob or repair affected historical snapshots, but it should prevent one damaged manifest from blocking future backups.
@FuxMak, could you also provide the complete node-agent/data-mover logs around the failure and confirm whether S3 versioning, lifecycle rules, object lock, or external bucket cleanup is configured?
@kaovilai Thank you for your fast response!
I just verified and there's no versioning, life cycle rules or object locks configured and velero is the only application, that has access to those buckets.
As for the logs, I didn't find anything related to missing blobs, as I'm not too familiar with the software as you are. The export includes all warnings and exports from the last 24h, where I also tried to delete a backup, causing this exact behaviour to appear.
Details
╭~/Documents/deployment main !4 ?4 ·························································· 3.12.10 10:19:27 ╰─❯ v describe backup velero-daily-backup-2-20260824001528 Name: velero-daily-backup-2-20260824001528 Namespace: velero Labels: app.kubernetes.io/instance=velero app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=velero helm.sh/chart=velero-12.0.2 velero.io/schedule-name=velero-daily-backup-2 velero.io/storage-location=test Annotations: meta.helm.sh/release-name=velero meta.helm.sh/release-namespace=velero velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.34.3+k3s1 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=34 Phase: Deleting Warnings: Velero: <none> Cluster: <none> Namespaces: lserver: resource: /pods name: /postgresql-0 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume dshm error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded seaweedfs: resource: /pods name: /seaweedfs-filer-0 message: /Skip pod volume seaweedfs-filer error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /seaweedfs-filer-1 message: /Skip pod volume seaweedfs-filer error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /seaweedfs-filer-2 message: /Skip pod volume seaweedfs-filer error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /seaweedfs-volume-0 message: /Skip pod volume mount0 error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /seaweedfs-volume-1 message: /Skip pod volume mount0 error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /seaweedfs-volume-2 message: /Skip pod volume mount0 error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /seaweedfs-volume-3 message: /Skip pod volume mount0 error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded seaweedfs-test: resource: /pods name: /seaweed-test-volume-0 message: /Skip pod volume mount0 error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /seaweed-test-volume-1 message: /Skip pod volume mount0 error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /seaweed-test-volume-2 message: /Skip pod volume mount0 error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded traefik: resource: /pods name: /traefik-f9kjf message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /traefik-f9kjf message: /Skip pod volume tmp error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /traefik-fnw9r message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /traefik-fnw9r message: /Skip pod volume tmp error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /traefik-zj9dl message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /traefik-zj9dl message: /Skip pod volume tmp error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded youtrack: resource: /pods name: /youtrack-0 message: /Skip pod volume youtrack-data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /youtrack-0 message: /Skip pod volume youtrack-config error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /youtrack-0 message: /Skip pod volume youtrack-logs error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /youtrack-0 message: /Skip pod volume youtrack-backup error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /youtrack-0 message: /Skip pod volume jmx-exporter-agent error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded keycloak: resource: /pods name: /keycloak-0 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /keycloak-0 message: /Skip pod volume keycloak-codefy-theme-share error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /keycloak-postgresql-0 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /keycloak-postgresql-0 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /keycloak-postgresql-0 message: /Skip pod volume dshm error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded release-test: resource: /pods name: /cabinet-docs-75f9b6d95d-mtr5p message: /Skip pod volume app-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /cabinet-docs-75f9b6d95d-mtr5p message: /Skip pod volume tmp-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /cabinet-docs-75f9b6d95d-mtr5p message: /Skip pod volume template-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgres-release-test-1 message: /Skip pod volume pgdata error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgres-release-test-1 message: /Skip pod volume scratch-data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgres-release-test-1 message: /Skip pod volume shm error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume dshm error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded release2-test: resource: /pods name: /cabinet-docs-ffbf4d9b9-vjzjk message: /Skip pod volume app-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /cabinet-docs-ffbf4d9b9-vjzjk message: /Skip pod volume tmp-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /cabinet-docs-ffbf4d9b9-vjzjk message: /Skip pod volume template-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume dshm error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded vault: resource: /pods name: /vault-0 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /vault-0 message: /Skip pod volume home error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded bitwarden: resource: /pods name: /bitwarden-backup-cron-29776140-ktmqs message: /Skip pod volume mssql-backups error: /pod is not in the expected status, name=bitwarden-backup-cron-29776140-ktmqs, namespace=bitwarden, phase=Pending: pod is not running elasticsearch1: resource: /pods name: /elasticsearch1-ingest-0 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-ingest-0 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-ingest-1 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-ingest-1 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-ingest-2 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-ingest-2 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-master-0 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-master-0 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-master-1 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-master-1 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-master-2 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-master-2 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-master-3 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-master-3 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch1-metrics-d764bfc8-2swjp message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded huge-data-size: resource: /pods name: /postgresql-0 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume dshm error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded longhorn-system: resource: /pods name: /longhorn-ui-7ff79dfb4-5gq62 message: /Skip pod volume nginx-cache error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /longhorn-ui-7ff79dfb4-5gq62 message: /Skip pod volume nginx-config error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /longhorn-ui-7ff79dfb4-5gq62 message: /Skip pod volume var-run error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /longhorn-ui-7ff79dfb4-c2npn message: /Skip pod volume nginx-cache error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /longhorn-ui-7ff79dfb4-c2npn message: /Skip pod volume nginx-config error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /longhorn-ui-7ff79dfb4-c2npn message: /Skip pod volume var-run error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded cattle-fleet-system: resource: /pods name: /fleet-agent-b497bff4-ck5r6 message: /Skip pod volume kube error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /fleet-agent-b497bff4-ck5r6 message: /Skip pod volume tmp error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded deployment-test: resource: /pods name: /cabinet-docs-5d6dc79c5c-qfj49 message: /Skip pod volume app-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /cabinet-docs-5d6dc79c5c-qfj49 message: /Skip pod volume tmp-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /cabinet-docs-5d6dc79c5c-qfj49 message: /Skip pod volume template-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /postgresql-0 message: /Skip pod volume dshm error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded elasticsearch0: resource: /pods name: /elasticsearch0-ingest-0 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-ingest-0 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-ingest-1 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-ingest-1 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-ingest-2 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-ingest-2 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-master-0 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-master-0 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-master-1 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-master-1 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-master-2 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-master-2 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-master-3 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-master-3 message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded resource: /pods name: /elasticsearch0-metrics-b74bf556c-kmlhf message: /Skip pod volume empty-dir error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded postgres-operator: resource: /pods name: /postgres-operator-cloudnative-pg-6549655686-nxqht message: /Skip pod volume scratch-data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded umami: resource: /pods name: /umami-postgresql-0 message: /Skip pod volume data error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded Errors: Velero: message: /failed to wait PVBs processed for the ItemBlock error: /context deadline exceeded name: /fleet-agent-b497bff4-ck5r6 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /cabinet-docs-5d6dc79c5c-qfj49 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /postgresql-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch0-ingest-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch0-ingest-1 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch0-ingest-2 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch0-master-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch0-master-1 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch0-master-2 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch0-master-3 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch0-metrics-b74bf556c-kmlhf message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch1-ingest-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch1-ingest-1 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch1-ingest-2 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch1-master-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch1-master-1 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch1-master-2 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch1-master-3 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /elasticsearch1-metrics-d764bfc8-2swjp message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /postgresql-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /keycloak-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /keycloak-postgresql-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /longhorn-ui-7ff79dfb4-5gq62 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /longhorn-ui-7ff79dfb4-c2npn message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /postgresql-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /postgres-operator-cloudnative-pg-6549655686-nxqht message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /cabinet-docs-75f9b6d95d-mtr5p message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /postgres-release-test-1 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /postgresql-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /cabinet-docs-ffbf4d9b9-vjzjk message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /postgresql-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /seaweedfs-filer-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /seaweedfs-filer-1 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /seaweedfs-filer-2 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /seaweedfs-volume-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /seaweedfs-volume-1 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /seaweedfs-volume-2 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /seaweedfs-volume-3 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /seaweed-test-volume-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /seaweed-test-volume-1 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /seaweed-test-volume-2 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /traefik-f9kjf message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /traefik-fnw9r message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /traefik-zj9dl message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /umami-postgresql-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /vault-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded name: /youtrack-0 message: /Error backing up item error: /failed to list node-agent pods: client rate limiter Wait returned an error: context deadline exceeded message: /timed out waiting for all PodVolumeBackups to complete Cluster: <none> Namespaces: <none> Namespaces: Included: * Excluded: velero, kube-system, kube-public, kube-node-lease, default, linkerd, linkerd-multicluster, system-upgrade, cattle-system, rook-ceph, minio-tenant, monitoring Resources: Included: * Excluded: events, events.events.k8s.io, volumesnapshots.snapshot.storage.k8s.io, volumesnapshotcontents.snapshot.storage.k8s.io Cluster-scoped: auto Label selector: <none> Or label selector: <none> Storage Location: test Velero-Native Snapshot PVs: auto File System Backup (Default): true Snapshot Move Data: false Data Mover: velero TTL: 168h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: <none> Backup Format Version: 1.1.0 Started: 2026-08-29 02:44:52 +0200 CEST Completed: 2026-08-29 06:45:28 +0200 CEST Expiration: 2026-09-05 02:44:52 +0200 CEST Total items to be backed up: 2377 Items backed up: 2377 Backup Volumes: Velero-Native Snapshots: <none included> CSI Snapshots: <none included> Pod Volume Backups - kopia (specify --details for more information): Completed: 19 Failed: 3 HooksAttempted: 0 HooksFailed: 0 Deletion Attempts (1 failed): 2026-09-03 10:13:09 +0200 CEST: Processed Errors: error downloading backup tarball, CSI snapshot cleanup was skipped: error copying Backup to temp file: rpc error: code = Unknown desc = error getting object backups/velero-daily-backup-2-20260824001528/velero-daily-backup-2-20260824001528.tar.gz: operation error S3: GetObject, exceeded maximum number of attempts, 3, https response error StatusCode: 503, RequestID: tx00000ba89ea60fed02f91-006a992c1b-4a89a1aa-fsn1-prod1-ceph4, HostID: 4a89a1aa-fsn1-prod1-ceph4-fsn1, api error SlowDown: UnknownError error to delete manifest e215d3fb54f3458fa2766be259274904: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "qe628b9b365ef329a427ccded60382660-s785813e0e9fe81d4143": failed to get blob with ID qe628b9b365ef329a427ccded60382660-s785813e0e9fe81d4143: BLOB not found error to delete manifest 85e3984d5cf696ab4ed47d17daafe36e: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "qe628b9b365ef329a427ccded60382660-s785813e0e9fe81d4143": failed to get blob with ID qe628b9b365ef329a427ccded60382660-s785813e0e9fe81d4143: BLOB not found error to delete manifest 579655e714c3935d5cbb559af4190d87: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "qe628b9b365ef329a427ccded60382660-s785813e0e9fe81d4143": failed to get blob with ID qe628b9b365ef329a427ccded60382660-s785813e0e9fe81d4143: BLOB not found error to delete manifest 1f6100684f81304cdf10e4af13a1fa35: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "qe628b9b365ef329a427ccded60382660-s785813e0e9fe81d4143": failed to get blob with ID qe628b9b365ef329a427ccded60382660-s785813e0e9fe81d4143: BLOB not found error to delete manifest daa4ceae596a36b4da2c2efae6f7a6e7: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "qe628b9b365ef329a427ccded60382660-s785813e0e9fe81d4143": failed to get blob with ID qe628b9b365ef329a427ccded60382660-s785813e0e9fe81d4143: BLOB not found error to delete manifest 266b68e3d361a3017f06710e31ffb8d3: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "qe628b9b365ef329a427ccded60382660-s785813e0e9fe81d4143": failed to get blob with ID qe628b9b365ef329a427ccded60382660-s785813e0e9fe81d4143: BLOB not found error to delete manifest 8884ee5ac4a0904cc9322a7a1b2f125b: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q89690e1513e87fe1e5d408f0104b4c6e-s1a2377f3b08d4944143": failed to get blob with ID q89690e1513e87fe1e5d408f0104b4c6e-s1a2377f3b08d4944143: BLOB not found error to delete manifest 62df5ac9207172e59f5001c471581ef0: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q89690e1513e87fe1e5d408f0104b4c6e-s1a2377f3b08d4944143": failed to get blob with ID q89690e1513e87fe1e5d408f0104b4c6e-s1a2377f3b08d4944143: BLOB not found error to delete manifest 3a1f966c6b1b675217a6b255af29f809: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q89690e1513e87fe1e5d408f0104b4c6e-s1a2377f3b08d4944143": failed to get blob with ID q89690e1513e87fe1e5d408f0104b4c6e-s1a2377f3b08d4944143: BLOB not found error to delete manifest fff6bf5c88430b500362b313c747af67: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q89690e1513e87fe1e5d408f0104b4c6e-s1a2377f3b08d4944143": failed to get blob with ID q89690e1513e87fe1e5d408f0104b4c6e-s1a2377f3b08d4944143: BLOB not found error to delete manifest 8b10152f9bcf99f57a25b0511117bc98: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q89690e1513e87fe1e5d408f0104b4c6e-s1a2377f3b08d4944143": failed to get blob with ID q89690e1513e87fe1e5d408f0104b4c6e-s1a2377f3b08d4944143: BLOB not found error to delete manifest c63aa45eb4493032e49e3d56f012635a: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q89690e1513e87fe1e5d408f0104b4c6e-s1a2377f3b08d4944143": failed to get blob with ID q89690e1513e87fe1e5d408f0104b4c6e-s1a2377f3b08d4944143: BLOB not found error to delete manifest b56716abb865cee365f6b09b9e6bdead: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143": failed to get blob with ID q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143: BLOB not found error to delete manifest ddeb02eaacbc6eb005c8c5361848016d: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143": failed to get blob with ID q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143: BLOB not found error to delete manifest 5cf61653897a5f713a09416418f754ae: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143": failed to get blob with ID q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143: BLOB not found error to delete manifest 79906620bbf3f603f527bdd71e6b57f9: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143": failed to get blob with ID q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143: BLOB not found error to delete manifest 7cf464cf07f4561ee723c7e1d81598d7: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143": failed to get blob with ID q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143: BLOB not found error to delete manifest 7abe3beae155b4043d94c59ba6b262f5: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143": failed to get blob with ID q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143: BLOB not found error to delete manifest 0dde889b00705ceb5d37c0a834d319b1: error to delete manifest: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143": failed to get blob with ID q94230db63166b37c3364ccc1cdb9d9c9-sa8c34705e6cd0cf8143: BLOB not found
error message: data path backup failed: Failed to run kopia backup: Failed to find previous kopia snapshot manifests for si default@default:snapshot-data-upload-download/kopia/glowroot/data-glowroot-cassandra-0: unable to find manifest entries: failed to get manifests with labels map[hostname:default path:snapshot-data-upload-download/kopia/glowroot/data-glowroot-cassandra-0 type:snapshot username:default]: error to find manifests: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID qf639e8abd6afa38f0b504f5e4643753e-sf3d16207b6fc75a5130: BLOB not found
Also tried to connect to kopia s3 repository and genuinely the blob is not available. How to recover the missing blob if the source volume is still available?
velero version: 1.12
velero plugin for aws: v1.7.0