Skip to content

Fix e2e-test-kind and changelog-check CI flakes - #10555

Open
kaovilai wants to merge 2 commits into
velero-io:mainfrom
kaovilai:fix-e2e-kind-ci-flakes
Open

kaovilai wants to merge 2 commits into
velero-io:mainfrom
kaovilai:fix-e2e-kind-ci-flakes

Conversation

@kaovilai

@kaovilai kaovilai commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Note

Responses generated with Claude

Thank you for contributing to Velero!

Please add a summary of your change

Scanned recent Run the E2E test on kind runs on main and open PRs and found two CI-only flakes that were failing runs unrelated to the change under test, plus a third flake surfaced by this PR itself:

  1. kind binary checksum flake — helm/kind-action's own installer (kind.sh's call_curl) downloads the kind binary/checksum without curl's -f/--fail flag, so a transient HTTP error response from GitHub's release CDN gets saved as if it were the real file instead of triggering a retry. sha256sum then fails with no properly formatted checksum lines found or a checksum mismatch, aborting the job before any test runs. This repo pre-populates kind-action's cache directory itself with a properly -f/retry/checksum-verified binary, so kind-action's own (flaky) install step is skipped.
  2. AWS plugin image-pull flake — velero/velero-plugin-for-aws:latest was left to be pulled live from Docker Hub inside the kind node at pod start, with no retry, which intermittently failed with trying and failing to pull image and made BeforeSuite abort. This pre-pulls it on the runner (with retry) and kind load docker-images it before the suite starts. This is safe because pkg/install/deployment.go sets ImagePullPolicy: IfNotPresent for the plugin container whenever VELERO_IMAGE isn't tagged latest (it never is in this workflow), so kubelet uses the preloaded image instead of re-pulling.
  3. changelog-check vs. auto-labeler race — auto_label_prs.yml's file-glob labeler (.github/labeler.yml) auto-applies kind/changelog-not-required to PRs that only touch .github/** (like this one), but that workflow checks out the repo and diffs file globs, so it routinely lands the label a few seconds after pr-changelog-check.yml's much faster job has already evaluated (and failed) the check — reproduced on this PR itself, where the label landed the same second the changelog-check job completed. hack/changelog-check.sh now polls briefly (up to 30s) for the exempt label before concluding the changelog is missing, only adding latency on the path that was already about to fail without it.

A fourth flake (bitnami/containers reorganizing its MinIO Dockerfile path, hit on the release-1.18 build) is already fixed on main via the BITNAMI_CONTAINERS_COMMIT pin (8850862e9), so no change is needed here for that one.

No behavior change to the product itself — CI workflow/tooling only.

Does your change fix a particular issue?

N/A — CI flake fix, not tied to a tracked issue.

Please indicate you've done the following:

  • Accepted the DCO (commits signed off with -s).
  • /kind changelog-not-required — CI-only change, no user-facing behavior.
  • N/A — no user-facing documentation change.

🤖 Generated with Claude Code

Two failures unrelated to the PR under test were seen recurring across
multiple unrelated runs on main and PRs:

- helm/kind-action's own installer downloads the kind binary/checksum
  without curl's -f/--fail flag, so a transient HTTP error response is
  saved as if it were the real file instead of triggering a retry.
  sha256sum then fails with "no properly formatted checksum lines
  found" or a checksum mismatch, aborting the job before any test
  runs. Pre-populate kind-action's cache directory ourselves with a
  properly -f/retry/checksum-verified binary so it skips its own
  install step.

- velero/velero-plugin-for-aws:latest was left to be pulled live from
  Docker Hub inside the kind node at pod start, with no retry, which
  intermittently failed ("trying and failing to pull image") and made
  BeforeSuite abort. Pre-pull it on the runner (with retry) and
  `kind load docker-image` it before the suite starts. This is safe
  because pkg/install/deployment.go sets ImagePullPolicy:
  IfNotPresent for the plugin container whenever VELERO_IMAGE isn't
  tagged "latest" (it never is in this workflow), so kubelet uses the
  preloaded image instead of re-pulling.

A third flake (bitnami/containers reorganizing its MinIO Dockerfile
path) was found on release-1.18 but is already fixed on main via the
BITNAMI_CONTAINERS_COMMIT pin (8850862); no change needed here.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
@kaovilai
kaovilai requested a review from a team as a code owner September 21, 2026 19:04
@netlify

netlify Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for velero canceled.

Name Link
🔨 Latest commit e7f5f42
🔍 Latest deploy log https://app.netlify.com/projects/velero/deploys/6ab18d37f3bc9b0008f2872a

@github-actions github-actions Bot added the kind/changelog-not-required PR does not require a user changelog. Often for docs, website, or build changes label Sep 21, 2026
@codecov

codecov Bot commented Sep 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

auto_label_prs.yml applies kind/changelog-not-required (and other
exempt labels) from a separate workflow triggered by the same PR-open
event. That workflow checks out the repo and diffs file globs, so it
routinely takes a few seconds longer to land the label than this
job's checkout-and-run-one-script path takes to fail. This was
reproduced on PR velero-io#10555 itself: the label landed at the same second
the changelog-check job completed and failed.

Poll for the exempt label briefly (up to 30s) before concluding the
changelog is missing. This only adds latency on the path that was
already about to fail without it (no changelog file and, so far, no
exempt label); PRs that already carry a changelog file or already have
the label return immediately as before.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
@kaovilai kaovilai changed the title Fix e2e-test-kind CI flakes: kind checksum download, AWS plugin pull Fix e2e-test-kind and changelog-check CI flakes Sep 21, 2026
run: |
docker run -d --rm -p 9000:9000 -e "MINIO_ROOT_USER=minio" -e "MINIO_ROOT_PASSWORD=minio123" -e "MINIO_DEFAULT_BUCKETS=bucket,additional-bucket" bitnami/minio:local
# kind-action's own kind installer (kind.sh's call_curl) downloads without
# curl's -f/--fail flag, so a transient HTTP error page from GitHub's

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why wouldn't contribute the -f/--fail flag part to the upstream(helm/kind-action)?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/changelog-not-required PR does not require a user changelog. Often for docs, website, or build changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants