Skip to content

verify aux drivers installed correctly and exit nicely if not#21576

Merged
medyagh merged 11 commits into
kubernetes:masterfrom
medyagh:verify_aux_driver
Sep 17, 2025
Merged

verify aux drivers installed correctly and exit nicely if not#21576
medyagh merged 11 commits into
kubernetes:masterfrom
medyagh:verify_aux_driver

Conversation

@medyagh

@medyagh medyagh commented Sep 15, 2025

Copy link
Copy Markdown
Member

improve user experince for When our aux driver is doesnt run


jenkins@ubuntu-20-agent-3:~/workspace/minikube$ docker-machine-driver-kvm2 --version
docker-machine-driver-kvm2: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by docker-machine-driver-kvm2)
docker-machine-driver-kvm2: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by docker-machine-driver-kvm2)
docker-machine-driver-kvm2: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_7.1.0' not found (required by docker-machine-driver-kvm2)
docker-machine-driver-kvm2: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_7.7.0' not found (required by docker-machine-driver-kvm2)
docker-machine-driver-kvm2: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_8.0.0' not found (required by docker-machine-driver-kvm2)
docker-machine-driver-kvm2: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_7.8.0' not found (required by docker-machine-driver-kvm2)
docker-machine-driver-kvm2: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_7.2.0' not found (required by docker-machine-driver-kvm2)
docker-machine-driver-kvm2: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_6.10.0' not found (required by docker-machine-driver-kvm2)
docker-machine-driver-kvm2: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_7.3.0' not found (required by docker-machine-driver-kvm2)

before this PR

jenkins@ubuntu-20-agent-3:~$ minikube start -d kvm
😄  minikube v1.37.0 on Ubuntu 20.04 (kvm/amd64)
✨  Using the kvm2 driver based on user configuration
💾  Downloading driver docker-machine-driver-kvm2:
    > docker-machine-driver-kvm2-...:  65 B / 65 B [---------] 100.00% ? p/s 0s
    > docker-machine-driver-kvm2-...:  15.22 MiB / 15.22 MiB  100.00% ? p/s 100
💿  Downloading VM boot image ...
    > minikube-v1.37.0-amd64.iso....:  65 B / 65 B [---------] 100.00% ? p/s 0s
    > minikube-v1.37.0-amd64.iso:  370.78 MiB / 370.78 MiB  100.00% 185.23 MiB 
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating kvm2 VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
🤦  StartHost failed, but will try again: new host: dial tcp: missing address
🔥  Creating kvm2 VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
😿  Failed to start kvm2 VM. Running "minikube delete" may fix it: new host: dial tcp: missing address

❌  Exiting due to DRV_MISSING_ADDRESS: Failed to start host: new host: dial tcp: missing address
💡  Suggestion: The machine-driver specified is failing to start. Try running 'docker-machine-driver-<type> version'
🍿  Related issues:
    ▪ https://github.com/kubernetes/minikube/issues/6023
    ▪ https://github.com/kubernetes/minikube/issues/4679

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

After this PR

jenkins@ubuntu-20-agent-3:~/workspace/minikube$ ./out/minikube start -d kvm
😄  minikube v1.37.0 on Ubuntu 20.04 (kvm/amd64)
✨  Using the kvm2 driver based on user configuration
💾  Downloading driver docker-machine-driver-kvm2:
    > docker-machine-driver-kvm2-...:  65 B / 65 B [---------] 100.00% ? p/s 0s
    > docker-machine-driver-kvm2-...:  15.22 MiB / 15.22 MiB  100.00% ? p/s 100

❌  Exiting due to DRV_AUX_NOT_HEALTHY: Aux driver kvm2: failed to execute auxiliary version command "/home/jenkins/.minikube/bin/docker-machine-driver-kvm2 --version"

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

@medyagh medyagh changed the title verify aux drivers --version exit 0 wip: verify aux drivers --version exit 0 Sep 15, 2025
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 15, 2025
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot requested review from nirs and prezha September 15, 2025 19:51
@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 15, 2025
Comment thread pkg/minikube/driver/auxdriver/install.go Outdated
@medyagh

medyagh commented Sep 15, 2025

Copy link
Copy Markdown
Member Author

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Sep 15, 2025
@minikube-pr-bot

This comment has been minimized.

@medyagh medyagh changed the title wip: verify aux drivers --version exit 0 verify aux drivers installed correctly and exit nicely if not Sep 15, 2025
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 15, 2025
Comment thread cmd/minikube/cmd/start.go Outdated
Comment thread cmd/minikube/cmd/start.go Outdated
Comment thread pkg/minikube/driver/auxdriver/install.go Outdated

output, err := exec.Command(path, "version").Output()
cmd := exec.Command(path, "version")
output, err := cmd.CombinedOutput()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not combine stdout and stderr - stdout should be the yaml:

version: xxx
commit: yyy

stderr can have anything we log in command.

We should use:

out, err := cmd.Output()
if err != nil {
    if ee, ok := err.(*exec.ExitError); ok {
        stderr = string(ee.Stderr)
    }
    stderr = ""
}

I'm using a helper to get Stderr since this is so common:
https://github.com/nirs/kubectl-gather/blob/6b34fdce4c2ffe83111b2c540a371f788979189e/e2e/commands/commands.go#L34

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that seems like it could be its own Separate PR to update all minikube referneces to that. i prefer if we change this we change it everywhere that would be a bit of an effort, welcome to make that PR

Comment thread pkg/minikube/driver/auxdriver/install.go Outdated
path, err := exec.LookPath(executable)
if err != nil {
return path, err
klog.Warningf("driver not in path : %s, %v", path, err.Error())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

klog.Warningf("driver %q not in path %q: %v", driverName, path, err)

@minikube-pr-bot

This comment has been minimized.

1 similar comment
@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot

This comment has been minimized.

@medyagh

medyagh commented Sep 16, 2025

Copy link
Copy Markdown
Member Author

/ok-to-test

@k8s-ci-robot

Copy link
Copy Markdown
Contributor

@medyagh: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-minikube-integration 50d4032 link true /test pull-minikube-integration

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@medyagh

medyagh commented Sep 16, 2025

Copy link
Copy Markdown
Member Author

/retest-this-please

@minikube-pr-bot

Copy link
Copy Markdown

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21576 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 40.5s    │ 41.0s                  │
│ enable ingress │ 15.8s    │ 16.0s                  │
└────────────────┴──────────┴────────────────────────┘
Details

Times for minikube start: 39.5s 41.8s 39.8s 39.6s 42.0s
Times for minikube (PR 21576) start: 41.3s 40.0s 41.6s 42.9s 39.3s

Times for minikube ingress: 15.8s 15.8s 15.8s 15.8s 15.8s
Times for minikube (PR 21576) ingress: 16.3s 15.8s 16.3s 15.8s 15.8s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21576 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 21.5s    │ 22.3s                  │
│ enable ingress │ 12.0s    │ 12.4s                  │
└────────────────┴──────────┴────────────────────────┘
Details

Times for minikube start: 20.6s 22.0s 23.1s 21.3s 20.7s
Times for minikube (PR 21576) start: 22.2s 21.9s 22.3s 22.1s 22.8s

Times for minikube ingress: 10.6s 11.6s 13.6s 10.6s 13.6s
Times for minikube (PR 21576) ingress: 13.6s 10.6s 13.6s 10.6s 13.6s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21576 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 20.2s    │ 20.1s                  │
│ enable ingress │ 27.7s    │ 28.1s                  │
└────────────────┴──────────┴────────────────────────┘
Details

Times for minikube start: 20.1s 19.9s 19.0s 20.2s 21.8s
Times for minikube (PR 21576) start: 19.5s 20.8s 20.0s 21.3s 18.8s

Times for minikube ingress: 23.1s 23.1s 40.1s 29.1s 23.1s
Times for minikube (PR 21576) ingress: 32.1s 23.1s 23.1s 39.1s 23.1s

@minikube-pr-bot

Copy link
Copy Markdown

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment Test Name Flake Rate
Hyper-V_Windows (10 failed) TestErrorSpam/setup(gopogh) Unknown
Hyper-V_Windows (10 failed) TestFunctional/parallel/ServiceCmd/HTTPS(gopogh) Unknown
Hyper-V_Windows (10 failed) TestFunctional/parallel/ServiceCmd/Format(gopogh) Unknown
Hyper-V_Windows (10 failed) TestFunctional/parallel/ServiceCmd/URL(gopogh) Unknown
Hyper-V_Windows (10 failed) TestMultiControlPlane/serial/PingHostFromPods(gopogh) Unknown
Hyper-V_Windows (10 failed) TestMultiControlPlane/serial/StopSecondaryNode(gopogh) Unknown
Hyper-V_Windows (10 failed) TestMultiNode/serial/PingHostFrom2Pods(gopogh) Unknown
Hyper-V_Windows (10 failed) TestMultiNode/serial/RestartKeepsNodes(gopogh) Unknown
Hyper-V_Windows (10 failed) TestNoKubernetes/serial/StartWithK8s(gopogh) Unknown
Hyper-V_Windows (10 failed) TestPause/serial/Pause(gopogh) Unknown
KVM_Linux_crio (4 failed) TestFunctional/serial/ExtraConfig(gopogh) 6.98% (chart)

Besides the following environments also have failed tests:

  • Docker_Linux: 25 failed (gopogh)

  • Docker_Linux_containerd: 13 failed (gopogh)

  • Docker_Linux_crio: 20 failed (gopogh)

  • Docker_Linux_crio_arm64: 12 failed (gopogh)

To see the flake rates of all tests by environment, click here.

@medyagh medyagh merged commit 1a676ed into kubernetes:master Sep 17, 2025
29 of 42 checks passed
@medyagh medyagh deleted the verify_aux_driver branch January 15, 2026 05:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants