velero-plugin-for-aws >= 1.14.1 makes S3 region mandatory

What steps did you take and what happened:
I upgraded to velero-plugin-for-aws 0.14.2. Afterwards my backupStorageLocations are in unavailable state because no Region is configured.

    config:
      checksumAlgorithm: ""
      insecureSkipTLSVerify: "true"
      s3ForcePathStyle: "true"
      s3Url: http://localhost:8088

status:
    lastSyncedTime: "2026-07-06T09:02:41Z"
    lastValidationTime: "2026-07-06T09:02:43Z"
    message: 'BackupStorageLocation "rclone-s3" is unavailable: rpc error: code =
      Unknown desc = operation error S3: ListObjectsV2, failed to resolve service
      endpoint, endpoint rule error, A region must be set when sending requests to
      S3.'

If I set the region to any string e.g. region: "dummy" the backupStorageLocation is available again.

What did you expect to happen:

I would expect that 'breaking changes' are announced in the release notes.

Anything else you would like to add:
Is it a valid solution to use a "dummy" value or does this impose other issues e.g. while restoring from backup?

Environment:

Velero version (use velero version): v1.18.2
Velero features (use velero client config get features):
Kubernetes version (use kubectl version): Server Version: v1.35.6
Kubernetes installer & version: kubeadm
Cloud provider or hardware configuration: openstack
OS (e.g. from /etc/os-release): flatcar 4593.2.3

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

👍 for "I would like to see this bug fixed as soon as possible"
👎 for "There are more important bugs to focus on right now"

@networkhell

@networkhell

This seems an error from the SDK.

What object store are you using when you see this error?
What was the version of the plugin before you upgrade?

@reasonerjt

Hi @reasonerjt,

Object Store is CEPH / RGW.
Last working Plugin version is 1.14.0, failing from >= 1.14.1

@networkhell

@networkhell

I quickly checked the code of the aws-sdk-go-v2 and it looks like the error is thrown via:
https://github.com/aws/aws-sdk-go-v2/blob/769233f0d058bff8b856a9253db9ef37070ccdfc/service/s3/endpoints.go#L3413

I don't think bump up the plugin alone will introduce this problem. Maybe in your earlier deployment the region was set via environment var or somehow and it didn't trigger this error in the SDK.

@reasonerjt

@reasonerjt sorry for the late reply... I thought I already answered your question. I never set a region in my deployments. I can reliably reproduce that when I upgrade to plugin version >= 1.14.1 it breaks. After downgrade it works again.
As I mentioned before I could introduce a "dummy" region to circumvent the error but I don't really know if this has any implication on e.g. restoring from backup.

Dummy region is safe if you force s3url

I can also confirm that config.region = "dummy" works. I have MinIO and have never set region before. This should at least be announced in release notes.

Navigation Menu

Repository navigation

velero-plugin-for-aws >= 1.14.1 makes S3 region mandatory #9963

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

velero-plugin-for-aws >= 1.14.1 makes S3 region mandatory #9963

Description

Activity

reasonerjt commented on Jul 12, 2026

networkhell commented on Jul 13, 2026

reasonerjt commented on Jul 17, 2026

networkhell commented on Jul 30, 2026

kaovilai commented on Jul 30, 2026

danielskowronski commented on Sep 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions