Skip to content

preload: download from github when gcs not available#21605

Merged
medyagh merged 32 commits into
kubernetes:masterfrom
medyagh:preload_gh_asset
Sep 30, 2025
Merged

preload: download from github when gcs not available#21605
medyagh merged 32 commits into
kubernetes:masterfrom
medyagh:preload_gh_asset

Conversation

@medyagh

@medyagh medyagh commented Sep 20, 2025

Copy link
Copy Markdown
Member

Summary

  • Adds option to download preloads from github https://github.com/kubernetes-sigs/minikube-preloads assets when GCS fails
  • Removes Verifying Checksum manually since it is already done by go-getter (we provide the SHA to the lib and library will check at the end of download and also verifies the local file)
    that shaves off almost 1 second off minikube download

local testing

tested the current hack/preload code compiles with these changes

$ make out/preload-tool
cd hack && go build -ldflags="-X k8s.io/minikube/pkg/version.version=v1.37.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.37.0 -X k8s.io/minikube/pkg/version.gitCommitID="95b413f8489a3a5ee8d9e478ee6a5bd0da344f2d" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -o ../out/preload-tool preload-images/*.go

also tested

 out/preload-tool -no-upload --limit 1

and worked fine.

Follow Up PRs

possible todos after this PR:

#21638

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 20, 2025
@k8s-ci-robot k8s-ci-robot requested review from nirs and prezha September 20, 2025 20:22
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 20, 2025
@medyagh

medyagh commented Sep 20, 2025

Copy link
Copy Markdown
Member Author

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Sep 20, 2025
@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot

This comment has been minimized.

@nirs

nirs commented Sep 21, 2025

Copy link
Copy Markdown
Collaborator

Summary

Adds option to download preloads from github https://github.com/kubernetes-sigs/minikube-preloads assets when GCS fails

Why do we need a fallback? Why not switch to github?

Follow Up PRs

  • Possibly we could make it configurable for the end user if they prefer Github over GCS

Why the user should care about this? we need location all resources.

It makes sense to add a config in case user cannot access GCS or GitHub, and want to download resources from local server. For example disconnected environment or dealing with local restrictions (e.g. China).

@nirs nirs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a GitHub fallback add unneeded complexity. Can we have a config option to use either GCS or GitHub and simplify the code to use the configured source?

Comment thread pkg/minikube/download/preload.go Outdated
Comment thread pkg/minikube/download/preload.go
Comment thread pkg/minikube/download/preload.go
Comment thread pkg/minikube/download/preload.go
Comment thread pkg/minikube/download/download_test.go Outdated
if !existence || checkCalled {
t.Errorf("Expected preload to exist and no check to be performed. Existence: %v, Check: %v", existence, checkCalled)
if !existence || gcsCheckCalls != 0 || ghCheckCalls != 0 {
t.Errorf("Expected preload to exist and no checks to be performed. Existence: %v, GCS Calls: %d, GH Calls: %d", existence, gcsCheckCalls, ghCheckCalls)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests are too intrusive. We need to have more high level tests that test the system behavior instead of the implementation details.

The issues exist before this change but hits change makes it worse.

@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot

This comment has been minimized.

@medyagh medyagh changed the title preload: download from github when gcs not available preload: download from github when gcs not available and remove manual checksum check Sep 25, 2025
@medyagh medyagh changed the title preload: download from github when gcs not available and remove manual checksum check preload: download from github when gcs not available Sep 25, 2025
@minikube-pr-bot

This comment has been minimized.

@medyagh

medyagh commented Sep 26, 2025

Copy link
Copy Markdown
Member Author

/retest-this-please

@minikube-pr-bot

This comment has been minimized.

@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 26, 2025
@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot

This comment has been minimized.

@nirs nirs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed up to
preload: download from github when gcs not available

But it is impossible to review in this state - we have many partial or wrong commits fixes in later commits. Can you squash the changes to we have one commit for every correct logical change?

Comment thread pkg/minikube/download/preload.go
Comment thread pkg/minikube/download/download_test.go Outdated
Comment thread pkg/minikube/download/download_test.go
@@ -276,10 +283,6 @@ func Preload(k8sVersion, containerRuntime, driverName string) error {
return errors.Wrapf(err, "download failed: %s", url)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use this in line 279:

	} else if checksum != nil {
		// add URL parameter for go-getter to automatically verify the checksum
		url += fmt.Sprintf("?checksum=md5:%s", hex.EncodeToString(checksum))
	}

This will work only for GCH when we have md5, but for GitHub we have sha256 so the library handling download will fail to verify the checksum.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you sure this is whats happening ? I moved the logic into a helper func maybe that helps easier to see the flow

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment is correct for this commit. Later commit fixed this issue, but I review the commits, expecting that every commit is a correct single logical change.

It will be better to fix this commit to make this change easier to review and keep git history clean.

Comment thread pkg/minikube/download/preload.go
Comment thread pkg/minikube/download/gh/gh.go
Comment thread pkg/minikube/download/preload.go Outdated
Comment thread pkg/minikube/download/preload.go
default:
// this should never happen
klog.Fatalf("unknown preload source: %s", source)
return string(preloadSourceNone)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This locks wrong - if we get here preloadSourceLocal or preloadSourceNone we return "".

Should we return a "file:///path/to/preload" if the preload is local?

We need to cover all possible values (GCS, GH, Local, None). If we expect only GCS and GH here the function should panic (programmer error).

Comment thread pkg/minikube/download/gh/gh.go
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

@medyagh: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-minikube-integration 7a7a8d7 link true /test pull-minikube-integration

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@minikube-pr-bot

Copy link
Copy Markdown

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21605 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 44.2s    │ 42.3s                  │
│ enable ingress │ 16.3s    │ 15.8s                  │
└────────────────┴──────────┴────────────────────────┘
Details

Times for minikube start: 43.3s 42.1s 45.0s 44.4s 46.0s
Times for minikube (PR 21605) start: 40.7s 44.0s 43.0s 41.0s 42.6s

Times for minikube ingress: 15.8s 15.8s 15.3s 18.9s 15.8s
Times for minikube (PR 21605) ingress: 15.8s 15.8s 15.8s 15.8s 15.8s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21605 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 23.2s    │ 21.9s                  │
│ enable ingress │ 12.4s    │ 12.6s                  │
└────────────────┴──────────┴────────────────────────┘
Details

Times for minikube start: 22.3s 25.2s 23.0s 22.6s 22.8s
Times for minikube (PR 21605) start: 23.6s 20.8s 21.8s 21.9s 21.4s

Times for minikube (PR 21605) ingress: 12.7s 12.7s 13.6s 10.6s 13.6s
Times for minikube ingress: 13.7s 14.6s 10.6s 10.6s 12.7s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21605 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 20.1s    │ 21.0s                  │
│ enable ingress │ 27.5s    │ 24.5s                  │
└────────────────┴──────────┴────────────────────────┘
Details

Times for minikube ingress: 23.1s 39.1s 23.1s 29.1s 23.1s
Times for minikube (PR 21605) ingress: 23.1s 24.1s 23.1s 29.2s 23.1s

Times for minikube start: 20.0s 20.3s 20.8s 19.1s 20.3s
Times for minikube (PR 21605) start: 21.2s 19.3s 20.9s 23.2s 20.3s

@minikube-pr-bot

Copy link
Copy Markdown

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment Test Name Flake Rate
Docker_Linux_crio_arm64 (6 failed) TestFunctional/parallel/ServiceCmdConnect(gopogh) 0.00% (chart)
Docker_Linux_crio_arm64 (6 failed) TestFunctional/parallel/ServiceCmd/DeployApp(gopogh) 0.00% (chart)
Docker_Linux_crio_arm64 (6 failed) TestFunctional/parallel/ServiceCmd/HTTPS(gopogh) 0.00% (chart)
Docker_Linux_crio_arm64 (6 failed) TestFunctional/parallel/ServiceCmd/Format(gopogh) 0.00% (chart)
Docker_Linux_crio_arm64 (6 failed) TestFunctional/parallel/ServiceCmd/URL(gopogh) 0.00% (chart)
KVM_Linux_crio (4 failed) TestFunctional/parallel/ImageCommands/ImageBuild(gopogh) 0.00% (chart)
KVM_Linux_crio (4 failed) TestFunctional/parallel/ImageCommands/ImageRemove(gopogh) 2.50% (chart)
Docker_Linux_crio (6 failed) TestFunctional/parallel/ServiceCmdConnect(gopogh) 0.00% (chart)
Docker_Linux_crio (6 failed) TestFunctional/parallel/ServiceCmd/DeployApp(gopogh) 0.00% (chart)
Docker_Linux_crio (6 failed) TestFunctional/parallel/ServiceCmd/HTTPS(gopogh) 0.00% (chart)
Docker_Linux_crio (6 failed) TestFunctional/parallel/ServiceCmd/Format(gopogh) 0.00% (chart)
Docker_Linux_crio (6 failed) TestFunctional/parallel/ServiceCmd/URL(gopogh) 0.00% (chart)
Docker_Windows (1 failed) TestErrorSpam/setup(gopogh) Unknown

Besides the following environments also have failed tests:

  • Docker_Linux_containerd_arm64: 12 failed (gopogh)

To see the flake rates of all tests by environment, click here.

@medyagh medyagh merged commit bbc8733 into kubernetes:master Sep 30, 2025
29 of 42 checks passed
@medyagh medyagh deleted the preload_gh_asset branch January 15, 2026 05:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants