Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

az storage blob upload-batch returns ErrorCode:BlobAlreadyExists error v2.34.0 on upload #21477

Closed
nosvalds opened this issue Mar 1, 2022 · 16 comments
Closed

az storage blob upload-batch returns ErrorCode:BlobAlreadyExists error v2.34.0 on upload #21477

nosvalds opened this issue Mar 1, 2022 · 16 comments
Assignees
@evelyn-ys @calvinhzy
Labels
Auto-Assign Auto assign by bot customer-reported Issues that are reported by GitHub users external to the Azure organization. Storage az storage
Milestone
Backlog

Comments

@nosvalds
Copy link

nosvalds commented Mar 1, 2022

Describe the bug
Trying to upload a file to the '$web' container for a static site with az storage blob upload-batch from GitHub Actions is failing on the latest version of the cli v2.34.0 saying that the blob already exists. The errors says to use --overwrite.

ERROR: The specified blob already exists.
RequestId:5a838987-601e-002b-6577-2d31d6000000
Time:2022-03-01T14:17:46.5388178Z
ErrorCode:BlobAlreadyExists
If you want to overwrite the existing one, please add --overwrite in your command.

Adding --overwrite command results in this error:

ERROR: argument --overwrite: expected one argument

To Reproduce
Setup a GitHub Actions workflow to upload directory to a storage blob;

      - name: Login to Azure CLI
        uses: azure/login@v1
        with:
          creds: ${{ env.AZURE_CREDENTIALS }}

      - name: Upload to blob storage
        uses: azure/CLI@v1
        env:
          STORAGE_NAME: sadatastoresearchdev
        with:
          azcliversion: latest
          inlineScript: |
            az storage blob upload-batch --account-name ${{ env.STORAGE_NAME }} -d '$web' -s dist/

Expected behavior
You should be able to upload the blob and overwrite as you could before.

Environment summary

https://github.com/Azure/cli in GitHub Actions

Additional context

Specifying an older azcliversion version v2.33.1 in the workflow allow you to upload the blob.

      - name: Login to Azure CLI
        uses: azure/login@v1
        with:
          creds: ${{ env.AZURE_CREDENTIALS }}

      - name: Upload to blob storage
        uses: azure/CLI@v1
        env:
          STORAGE_NAME: sadatastoresearchdev
        with:
          azcliversion: 2.33.1
          inlineScript: |
            az storage blob upload-batch --account-name ${{ env.STORAGE_NAME }} --auth-mode key -d '$web' -s dist/

Detailed logs from GitHub Actions:

022-03-01T14:17:06.8368067Z ##[group]Run azure/login@v1
2022-03-01T14:17:06.8368444Z with:
2022-03-01T14:17:06.8370124Z   creds: ***
2022-03-01T14:17:06.8370507Z   enable-AzPSSession: false
2022-03-01T14:17:06.8370913Z   environment: azurecloud
2022-03-01T14:17:06.8371321Z   allow-no-subscriptions: false
2022-03-01T14:17:06.8371762Z   audience: api://AzureADTokenExchange
2022-03-01T14:17:06.8372170Z env:
2022-03-01T14:17:06.8373821Z   AZURE_CREDENTIALS: ***
2022-03-01T14:17:06.8374471Z   STORAGE_KEY: ***
2022-03-01T14:17:06.8374823Z ##[endgroup]
2022-03-01T14:17:15.8274876Z [command]/usr/bin/az cloud set -n azurecloud
2022-03-01T14:17:21.4767300Z Done setting cloud: "azurecloud"
2022-03-01T14:17:23.0710370Z Login successful.
2022-03-01T14:17:23.0894436Z ##[group]Run azure/CLI@v1
2022-03-01T14:17:23.0894808Z with:
2022-03-01T14:17:23.0895130Z   azcliversion: latest
2022-03-01T14:17:23.0895667Z   inlineScript: az storage blob upload-batch --account-name sadatastoresearchdev -d '$web' -s dist/

2022-03-01T14:17:23.0896146Z env:
2022-03-01T14:17:23.0897712Z   AZURE_CREDENTIALS: ***
2022-03-01T14:17:23.0898317Z   STORAGE_KEY: ***
2022-03-01T14:17:23.0898675Z   AZURE_HTTP_USER_AGENT: 
2022-03-01T14:17:23.0899158Z   AZUREPS_HOST_ENVIRONMENT: 
2022-03-01T14:17:23.0899547Z   STORAGE_NAME: sadatastoresearchdev
2022-03-01T14:17:23.0899947Z ##[endgroup]
2022-03-01T14:17:23.1710590Z Starting script execution via docker image mcr.microsoft.com/azure-cli:latest
2022-03-01T14:17:45.3477491Z WARNING: 
2022-03-01T14:17:45.3478765Z There are no credentials provided in your command and environment, we will query for account key for your storage account.
2022-03-01T14:17:45.3583025Z It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.
2022-03-01T14:17:45.3583675Z 
2022-03-01T14:17:45.3598136Z You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
2022-03-01T14:17:45.3599424Z For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.
2022-03-01T14:17:45.3599847Z 
2022-03-01T14:17:45.3600793Z In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
2022-03-01T14:17:46.6529259Z ERROR: The specified blob already exists.
2022-03-01T14:17:46.6530651Z RequestId:5a838987-601e-002b-6577-2d31d6000000
2022-03-01T14:17:46.6531594Z Time:2022-03-01T14:17:46.5388178Z
2022-03-01T14:17:46.6532370Z ErrorCode:BlobAlreadyExists
2022-03-01T14:17:46.6533513Z If you want to overwrite the existing one, please add --overwrite in your command.
2022-03-01T14:17:46.9476322Z ##[error]Error: az cli script failed.
2022-03-01T14:17:46.9487839Z cleaning up container...
2022-03-01T14:17:46.9851133Z MICROSOFT_AZURE_CLI_1646144243123_CONTAINER
2022-03-01T14:17:46.9851724Z 
2022-03-01T14:17:46.9910045Z (node:1739) UnhandledPromiseRejectionWarning: Error: az cli script failed.
2022-03-01T14:17:46.9913655Z     at /home/runner/work/_actions/azure/CLI/v1/dist/index.js:1:23964
2022-03-01T14:17:46.9914390Z     at Generator.throw (<anonymous>)
2022-03-01T14:17:46.9915105Z     at rejected (/home/runner/work/_actions/azure/CLI/v1/dist/index.js:1:20436)
2022-03-01T14:17:46.9915949Z     at processTicksAndRejections (internal/process/task_queues.js:93:5)
2022-03-01T14:17:46.9917419Z (node:1739) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
2022-03-01T14:17:46.9919329Z (node:1739) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
2022-03-01T14:17:46.9996553Z Post job cleanup.
@ghost ghost added customer-reported Issues that are reported by GitHub users external to the Azure organization. Storage az storage labels Mar 1, 2022
@ghost ghost added this to the Backlog milestone Mar 1, 2022
@ghost ghost assigned evelyn-ys Mar 1, 2022
@ghost ghost added the Auto-Assign Auto assign by bot label Mar 1, 2022
@bacongobbler
Copy link

bacongobbler commented Mar 1, 2022 •
edited
Loading

Can confirm - this is a recent change introduced in 2.34.0, breaking several of my pipelines using azure-blob-storage-upload which uses az storage blob upload-batch under the covers.

In addition to the use case @nosvalds provided earlier, I use this action to upload nightly releases of my program to Azure. With this recent change, I have no way to work around this short of deleting the object before uploading the new asset.

Reverting to 2.33.1 resolves the issue for now.

bacongobbler added a commit to fishworks/gofish that referenced this issue Mar 1, 2022
@bacongobbler
Revert "add --overwrite flag"
06be971 
This is an upstream bug with the Azure CLI - see Azure/azure-cli#21477

This reverts commit 44a7e79.
@bacongobbler bacongobbler mentioned this issue Mar 1, 2022
Closed
@yonzhan
Copy link
Collaborator

yonzhan commented Mar 1, 2022

storage

@evelyn-ys
Copy link
Member

@calvinhzy Can you help take a look? Might be a bug introduced by track2 migration

@calvinhzy calvinhzy mentioned this issue Mar 2, 2022
Merged
3 tasks
@calvinhzy
Copy link
Member

Hotfix is getting merged today. You can specify --overwrite to overwrite existing blobs. This is a breaking change as we believe overwriting silently is not a good default. Thanks.

@LynAutumn
Copy link

Overwrite is not working for me either, failing with:

ERROR: argument --overwrite: expected one argument

@DragonBale
Copy link

Overwrite is not working for me either, failing with:

ERROR: argument --overwrite: expected one argument

You need to do
--overwrite true

@ozcanbecoms
Copy link

Thanks you @DragonBale

--overwrite true
works well

@bacongobbler
Copy link

bacongobbler commented Mar 2, 2022 •
edited
Loading

I'm curious why true is a required argument for the --overwrite flag. Isn't the presence of the --overwrite flag enough to assert that the value is true, or is there an implied third value the CLI can provide?

In other words...

az storage blob upload-batch ...               # implies overwrite==false
az storage blob upload-batch --overwrite ...   # implies overwrite==true

Most CLI parsers work in this manner, so I'm curious why azure-cli chose to approach this differently.

devongovett added a commit to adobe/react-spectrum that referenced this issue Mar 2, 2022
@devongovett
Fix production deploy to azure
76f058c 
Ref: Azure/azure-cli#21477
@devongovett devongovett mentioned this issue Mar 2, 2022
Merged
devongovett added a commit to adobe/react-spectrum that referenced this issue Mar 2, 2022
@devongovett
Fix production deploy to azure (#2906)
12bb96b 
Ref: Azure/azure-cli#21477
@evelyn-ys
Copy link
Member

evelyn-ys commented Mar 3, 2022 •
edited
Loading

The hotfix is on the way, both --overwrite and --overwrite true will be accepted after hotfix

markstokan added a commit to pivotal-cf/om that referenced this issue Mar 4, 2022
@markstokan
specifiy --overwrite to overwrite existing blobs
75da144 
- az cli v2.34.0 introduced breaking change to prevent silent overwrites
  by default

- Azure/azure-cli#21477

Authored-by: Mark Stokan <stokanm@vmware.com>
@renovate renovate bot mentioned this issue Mar 7, 2022
Merged
1 task
@a-cordier a-cordier mentioned this issue Mar 9, 2022
Merged
a-cordier added a commit to gravitee-io/gravitee-api-management that referenced this issue Mar 9, 2022
@a-cordier
ci: add overwrite flag to az storage upload
ff7b051 
The storage command of azure cli needs an overwrite flag set to true
since v2.34.0

see Azure/azure-cli#21477
gaetanmaisse pushed a commit to gravitee-io/gravitee-api-management that referenced this issue Mar 9, 2022
@a-cordier @gaetanmaisse
ci: add overwrite flag to az storage upload
731f1ec 
The storage command of azure cli needs an overwrite flag set to true
since v2.34.0

see Azure/azure-cli#21477
phiz71 added a commit to gravitee-io/gravitee-docs that referenced this issue Mar 9, 2022
@phiz71
ci: add overwrite flag to az storage upload
e8d090b 
The storage command of azure cli needs an overwrite flag set to true
since v2.34.0

see Azure/azure-cli#21477
@phiz71 phiz71 mentioned this issue Mar 9, 2022
Merged
tomgeudens pushed a commit to gravitee-io/gravitee-docs that referenced this issue Mar 10, 2022
@phiz71
ci: add overwrite flag to az storage upload (#715)
4474c32 
The storage command of azure cli needs an overwrite flag set to true
since v2.34.0

see Azure/azure-cli#21477
@shaunanoordin shaunanoordin mentioned this issue Mar 10, 2022
Merged
@rin-skylight rin-skylight mentioned this issue Mar 10, 2022
Merged
1 task
@duncanworthy duncanworthy mentioned this issue Mar 10, 2022
Closed
@jeffpdotone
Copy link

Azure CLI task for release pipeline is not updated to the latest version is there a workaround to make the task use the latest one?

@evelyn-ys
Copy link
Member

Which release pipeline do you mean? Is the latest version you mentioned v2.34.1?

@evelyn-ys evelyn-ys mentioned this issue Mar 11, 2022
Closed
@zwolf zwolf mentioned this issue Mar 11, 2022
Closed
@alismx alismx mentioned this issue Mar 11, 2022
Merged
1 task
@zwolf
Copy link

zwolf commented Mar 14, 2022

Which release pipeline do you mean? Is the latest version you mentioned v2.34.1?

"latest", insofar as the azure-cli github action is concerned, is still 2.33.1 as of this post. It was 2.34.1 for at least a few hours last week, but was apparently reverted. That is, a github action such as the one in the OP shows something like this in the console:

Run azure/CLI@v1
  with:
    inlineScript: 
      az storage blob upload-batch --account-name my_account -d '$web' -s dist/

Starting script execution via docker image mcr.microsoft.com/azure-cli:2.33.1

I have pinned the older 2.33.1 version in my workflows because the update to 2.34 breaks them in several different ways (newly requiring the --overwrite flag as in this issue, and the new content-type issue described here). This is instead of keeping the azcliversion argument undefined as suggested here.

@wahyuen
Copy link

wahyuen commented Mar 15, 2022

Yet more breaking changes marked with 'patch' numbers. Is it so hard to follow SemVer and mark it as new release and not cause customers workloads to fail which have been working for YEARS!

gabrielweyer added a commit to gabrielweyer/cors that referenced this issue Apr 5, 2022
@gabrielweyer
Add overwrite to Az CLI upload-batch
9d96406 
There was a breaking change in a patch release:
Azure/azure-cli#21477.
@gabrielweyer gabrielweyer mentioned this issue Apr 5, 2022
Merged
gabrielweyer added a commit to gabrielweyer/cors that referenced this issue Apr 5, 2022
@gabrielweyer
Add overwrite to Az CLI upload-batch
106a213 
There was a breaking change in a patch release:
Azure/azure-cli#21477.
@fzhao99 fzhao99 mentioned this issue Apr 5, 2022
Merged
@calvinhzy
Copy link
Member

We will GA the --overwrite flag in May/June time frame. Thanks.

@lukecookssw lukecookssw mentioned this issue Apr 11, 2022
Merged
@VikingTristan VikingTristan mentioned this issue Apr 21, 2022
Merged
@lgmorand lgmorand mentioned this issue Apr 25, 2022
Closed
BassOfLion added a commit to avinor/tau that referenced this issue May 4, 2022
@BassOfLion
Add --overwrite flag as per breaking changes in az
382c0ce 
Azure/azure-cli#21477
@trieutran91 trieutran91 mentioned this issue May 25, 2022
Merged
@wkoeter wkoeter mentioned this issue Jun 2, 2022
Open
@JCPuerto
Copy link

Is anybody else getting the error az : WARNING: Argument '--overwrite' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus?

It overwrites the file but it throws an exception with a warning message.

Should I open a new issue?

@venikx
Copy link

venikx commented Jul 26, 2023

Adding the --overwrite doesn't seem to have any effect, as overriding the blog storage still results in a ERROR: The specified blob already exists. error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@evelyn-ys evelyn-ys

@calvinhzy calvinhzy

Labels
Auto-Assign Auto assign by bot customer-reported Issues that are reported by GitHub users external to the Azure organization. Storage az storage
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests