vfkit: Add Rosetta support for deploying amd64 images on Apple silicon #22140

nirs · 2025-12-13T15:35:48Z

Add --rosetta flag enabling Rosetta[1] for running apps built for Intel processor on Mac with Apple silicon. An example use case is deploying open-cluster-management that do not provide arm64 builds yet.

If Rosetta is not installed on the host, it will be installed on the first time starting a cluster. When running in non-interactive mode automatic install is disabled and if Rosetta is not installed start will fail.

The --rosetta flag is ignored with a warning if enabled on Mac with Intel processor.

Testing amd64 executable on Apple silicon

% cat rosetta/test.go 
package main

import (
        "bytes"
        "fmt"
        "os/exec"
        "runtime"
)

func main() {
        uname, _ := exec.Command("uname", "-a").Output()
        fmt.Printf("✅ Running %s binary on %s\n", runtime.GOARCH, bytes.TrimSpace(uname))
}

% GOARCH=amd64 GOOS=linux go build -o rosetta/test rosetta/test.go

% file rosetta/test
rosetta/test: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=9743ccad9a20c7d535a9794cc7b22ed1147c89d8, with debug_info, not stripped

% out/minikube start --driver vfkit --rosetta --mount-string $PWD/rosetta:/mnt/rosetta --no-kubernetes
😄  minikube v1.37.0 on Darwin 26.1 (arm64)
✨  Using the vfkit driver based on user configuration
👍  Starting minikube without Kubernetes in cluster minikube
🔥  Creating vfkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
🐳  Preparing Docker 28.5.2 ...
🏄  Done! minikube is ready without Kubernetes!

% minikube ssh /mnt/rosetta/test                                  
✅ Running amd64 binary on Linux minikube 6.6.95 #1 SMP PREEMPT Thu Dec 11 21:09:52 UTC 2025 aarch64 GNU/Linux

Testing with container images

I tested the PR with ramen regional DR environment, using containerd runtime. We deploy several adm64 only container images like open-cluster-management and ramen.
RamenDR/ramen#2356

[1] https://developer.apple.com/documentation/virtualization/running-intel-binaries-in-linux-vms-with-rosetta

Fixes #20559

nirs · 2025-12-13T15:36:09Z

/ok-to-test

nirs · 2025-12-13T15:36:25Z

/cc @afbjorklund

nirs · 2025-12-13T15:42:27Z

@cfergeau can you review?

With the new vmnet network[1] and rosetta[2] in minikube we can use minikube on macOS. This change replace macOS defaults to use minikube with vfkit driver and vmnet-shared network. With this change the vm.yaml environment starts in 16 seconds, 5 times faster than lima. The regional-dr environment starts in 324 seconds, about 1.3 times faster than lima. Example run with minimal environment: % drenv start envs/vm.yaml 2025-12-13 18:58:13,469 INFO [vm] Starting environment 2025-12-13 18:58:13,547 INFO [cluster] Starting minikube cluster 2025-12-13 18:58:29,953 INFO [cluster] Cluster started in 16.41 seconds 2025-12-13 18:58:29,955 INFO [cluster/0] Running addons/example/start 2025-12-13 18:58:41,240 INFO [cluster/0] addons/example/start completed in 11.28 seconds 2025-12-13 18:58:41,241 INFO [cluster/0] Running addons/example/test 2025-12-13 18:58:41,471 INFO [cluster/0] addons/example/test completed in 0.23 seconds 2025-12-13 18:58:41,471 INFO [vm] Environment started in 28.00 seconds Example run with full environment: % drenv start envs/regional-dr.yaml 2025-12-13 19:11:14,075 INFO [rdr] Starting environment 2025-12-13 19:11:14,116 INFO [dr2] Starting minikube cluster 2025-12-13 19:11:14,116 INFO [dr1] Starting minikube cluster 2025-12-13 19:11:14,121 INFO [hub] Starting minikube cluster 2025-12-13 19:11:29,211 INFO [dr1] Cluster started in 15.09 seconds ... 2025-12-13 19:11:32,399 INFO [hub] Cluster started in 18.28 seconds ... 2025-12-13 19:11:36,296 INFO [dr2] Cluster started in 22.18 seconds ... 2025-12-13 19:16:38,669 INFO [rdr] Dumping ramen e2e config to '/Users/nir/.config/drenv/rdr' 2025-12-13 19:16:38,761 INFO [rdr] Environment started in 324.69 seconds Note: This change requires local minikube build since the rosetta support is not merged yet. [1] kubernetes/minikube#20501 [2] kubernetes/minikube#22140 Signed-off-by: Nir Soffer <nsoffer@redhat.com>

nirs · 2025-12-13T23:20:01Z

/ok-to-test

cmd/minikube/cmd/start_flags.go

pkg/drivers/vfkit/vfkit.go

nirs · 2025-12-16T19:51:49Z

Latest push rebased on master to split the rebase from the actual changes.

nirs · 2025-12-16T21:12:21Z

@cfergeau I addressed the comments:

Typos in warnings fixed
Warnings change to match better similar warnings around this code
I kept setupRosetta() as is for simplicity and ease of maintainability

nirs · 2025-12-17T13:33:38Z

This KVM_Linux fail with the known issue when TestISO are run after the context deadline was exceeded:
KVM_Linux
KVM_Linux — Jenkins: completed with 10 / 447 failures in 65.73 minutes.

The interesting part is that this test did not time out after 90 minutes - so this is not an issue with the global test timeout. Maybe this an issue with the context we use in these tests:

=== RUN   TestISOImage/PersistentMounts//data
=== PAUSE TestISOImage/PersistentMounts//data
=== CONT  TestISOImage/PersistentMounts//data
iso_test.go:97: (dbg) Run:  out/minikube-linux-amd64 -p guest-625557 ssh "df -t ext4 /data | grep /data"
iso_test.go:97: (dbg) Non-zero exit: out/minikube-linux-amd64 -p guest-625557 ssh "df -t ext4 /data | grep /data": context deadline exceeded (3.039µs)

obnoxxx · 2025-12-17T16:08:20Z

@nirs , I was trying yout patch locally, but I can't get minikube start with the commandline as in the PR description to complete successfully on my machine.

I am seeing this:

$ minikube delete --all --purge

$ minikube start ... --rosetta ..
...
🔄  Successfully unblocked bootpd process from firewall, retrying
🔥  Creating vfkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
😿  Failed to start vfkit VM. Running "minikube delete" may fix it: creating host: create host timed out in 360.000000 seconds

❌  Exiting due to DRV_CREATE_TIMEOUT: Failed to start host: creating host: create host timed out in 360.000000 seconds
💡  Suggestion: Try 'minikube delete', and disable any conflicting VPN or firewall software
🍿  Related issue: https://github.com/kubernetes/minikube/issues/7072

$ echo $?
52
$

nirs · 2025-12-17T16:20:13Z

@nirs , I was trying yout patch locally, but I can't get minikube start with the commandline as in the PR description to complete successfully on my machine.

I am seeing this:

$ minikube delete --all --purge

$ minikube start ... --rosetta ..
...
🔄  Successfully unblocked bootpd process from firewall, retrying
🔥  Creating vfkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
😿  Failed to start vfkit VM. Running "minikube delete" may fix it: creating host: create host timed out in 360.000000 seconds

❌  Exiting due to DRV_CREATE_TIMEOUT: Failed to start host: creating host: create host timed out in 360.000000 seconds
💡  Suggestion: Try 'minikube delete', and disable any conflicting VPN or firewall software
🍿  Related issue: https://github.com/kubernetes/minikube/issues/7072

You hide the details, so I must guess from the partial output. You are using vment-helper on managed Mac, and did not get IP address. minikube wrongly try to "unblock" bootpd which does not have any affect and fail with the wrong error.

If you try without --rosetta you will probably get the same result.

One known issue is connecting to your company VPN. For me after connecting to the VPN and disconnecting, the vment network is always broken and the only way to recover is to reboot.

If you did not connect to the VPN since boot, this workaround that usually works:

minikube delete
sudo killall socketfilterfw bootpd
minikube start ...

It this does not work reboot usually fix the issue.

obnoxxx · 2025-12-17T16:34:01Z

@nirs, with your hint, I got minikube start to complete now, and running the test binary in the vm succeeds:

% ./out/minikube ssh
$ /mnt/rosetta/test
✅ Running amd64 binary on Linux minikube 6.6.95 #1 SMP PREEMPT Tue Dec 16 02:19:13 UTC 2025 aarch64 GNU/Linux
$

so this seems all fine.

medyagh · 2025-12-17T18:39:10Z

cmd/minikube/cmd/start_flags.go

 	startCmd.Flags().String(qemuFirmwarePath, "", "Path to the qemu firmware file. Defaults: For Linux, the default firmware location. For macOS, the brew installation location. For Windows, C:\\Program Files\\qemu\\share")
+
+	// vfkit
+	startCmd.Flags().Bool(rosetta, false, "Enable Rosetta to support apps built for Intel processor on a Mac with Apple silicon (vfkit driver only)")


does it also support running images built for x86 on all container runtimes ?
or is it only for binaries ? if also images, should mention in the help text.

it would be nice to check or verify (
suggestion, add an integration test that pulls an x86 image and runs it on silicon)
and test be skipped on anything but macos arm64
(could be a subset of functional test)

if dont wanna add ftest, can also just add manual verification and add to the PR description.

does it also support running images built for x86 on all container runtimes ? or is it only for binaries ? if also images, should mention in the help text.

It works for running any executable, on the host or in a container. I'm using the same language used by apple to describe the feature (apps built for Intel processors). Should we replace "apps" with container images since minikube is about containers?

it would be nice to check or verify

I tested running executable in the guest (example in the message), and running ramen testing clusters (containerd runtime) which include several amd64 images (ocm, ramen).
The ramen PR RamenDR/ramen#2356 shows example run with this PR.

( suggestion, add an integration test that pulls an x86 image and runs it on silicon) and test be skipped on anything but macos arm64 (could be a subset of functional test)

if dont wanna add ftest, can also just add manual verification and add to the PR description.

I want to add a test that will do this:

created executable like the example as part of the test

run it using the virtiofs mount

create a container image including the executable by building inside minikube, or in podman

load the image into minikube

start a pod with the image

I'll open an issue for working on the test, since it more work than the actual feature.

pkg/drivers/vfkit/vfkit.go

Add --rosetta flag enabling Rosetta[1] for running apps built for Intel processor on Mac with Apple silicon. An example use case is deploying open-cluster-management that do not provide arm64 builds yet. If Rosetta is not installed on the host, it will be installed on the first time starting a cluster. When running in non-interactive mode automatic install is disabled and if Rosetta is not installed start will fail. The --rosetta flag is ignored with a warning if enabled on Mac with Intel processor. [1] https://developer.apple.com/documentation/virtualization/running-intel-binaries-in-linux-vms-with-rosetta

nirs · 2025-12-21T19:22:17Z

/ok-to-test

minikube-pr-bot · 2025-12-21T20:43:25Z

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 22140 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 39.1s    │ 39.6s                  │
│ enable ingress │ 15.7s    │ 15.8s                  │
└────────────────┴──────────┴────────────────────────┘

Details

Times for minikube start: 38.9s 37.7s 38.6s 38.9s 41.4s
Times for minikube (PR 22140) start: 38.5s 39.7s 40.3s 41.1s 38.4s

Times for minikube (PR 22140) ingress: 15.7s 15.7s 16.2s 15.7s 15.7s
Times for minikube ingress: 15.7s 15.7s 15.7s 15.7s 15.7s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 22140 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 23.8s    │ 21.8s                  │
│ enable ingress │ 10.8s    │ 10.8s                  │
└────────────────┴──────────┴────────────────────────┘

Details

Times for minikube ingress: 10.6s 11.6s 10.6s 10.6s 10.6s
Times for minikube (PR 22140) ingress: 10.6s 11.6s 10.6s 10.7s 10.6s

Times for minikube start: 20.9s 24.3s 25.0s 24.5s 24.3s
Times for minikube (PR 22140) start: 20.0s 20.7s 21.1s 23.5s 23.8s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 22140 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 19.9s    │ 19.6s                  │
│ enable ingress │ 23.5s    │ 23.1s                  │
└────────────────┴──────────┴────────────────────────┘

Details

Times for minikube start: 19.2s 19.2s 21.4s 18.2s 21.4s
Times for minikube (PR 22140) start: 21.7s 17.3s 17.8s 19.0s 22.4s

Times for minikube (PR 22140) ingress: 23.1s 23.1s 23.1s 23.1s 23.1s
Times for minikube ingress: 23.1s 23.1s 23.1s 25.1s 23.2s

k8s-ci-robot · 2025-12-21T20:44:29Z

@nirs: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
integration-docker-docker-linux-arm64	`eeabf3e`	link	true	`/test integration-docker-docker-linux-arm64`
integration-docker-docker-linux-x86-64	`eeabf3e`	link	true	`/test integration-docker-docker-linux-x86-64`
integration-kvm-crio-linux-x86-64	`eeabf3e`	link	true	`/test integration-kvm-crio-linux-x86-64`
integration-vfkit-docker-macos-arm	`5ba8afb`	link	false	`/test integration-vfkit-docker-macos-arm`
integration-none-containerd-linux-x86	`5ba8afb`	link	false	`/test integration-none-containerd-linux-x86`
integration-docker-crio-linux-x86	`5ba8afb`	link	false	`/test integration-docker-crio-linux-x86`
integration-kvm-crio-linux-x86	`5ba8afb`	link	false	`/test integration-kvm-crio-linux-x86`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

medyagh · 2025-12-22T00:45:36Z

/lgtm

k8s-ci-robot · 2025-12-22T00:45:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, nirs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

minikube-pr-bot · 2025-12-22T04:59:53Z

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate
none_Linux (2 failed)	TestFunctionalNewestKubernetes/Versionv1.35.0-rc.1/parallel/DashboardCmd(gopogh)	Unknown
none_Linux (2 failed)	TestFunctional/parallel/DashboardCmd(gopogh)	0.00% (chart)
KVM_Linux_crio (3 failed)	TestFunctionalNewestKubernetes/Versionv1.35.0-rc.1/serial/ComponentHealth(gopogh)	Unknown
KVM_Linux_crio (3 failed)	TestPreload/Restart-With-Preload-Check-User-Image(gopogh)	Unknown

Besides the following environments also have failed tests:

Docker_Linux_crio_arm64: 55 failed (gopogh)
Docker_Linux_crio: 26 failed (gopogh)
Docker_Linux_containerd_arm64: 35 failed (gopogh)

To see the flake rates of all tests by environment, click here.

k8s-ci-robot requested review from ComradeProgrammer and medyagh December 13, 2025 15:35

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 13, 2025

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Dec 13, 2025

k8s-ci-robot requested a review from afbjorklund December 13, 2025 15:36

nirs mentioned this pull request Dec 13, 2025

drenv: Use minikube on macOS RamenDR/ramen#2356

Draft

This comment has been minimized.

Sign in to view

cfergeau reviewed Dec 16, 2025

View reviewed changes

cmd/minikube/cmd/start_flags.go Outdated Show resolved Hide resolved

pkg/drivers/vfkit/vfkit.go Outdated Show resolved Hide resolved

pkg/drivers/vfkit/vfkit.go Show resolved Hide resolved

pkg/drivers/vfkit/vfkit.go Show resolved Hide resolved

nirs force-pushed the vfkit-rosetta branch from ec89ba5 to 721947c Compare December 16, 2025 19:50

This comment has been minimized.

Sign in to view

nirs force-pushed the vfkit-rosetta branch from 721947c to 3ea11c4 Compare December 16, 2025 21:09

nirs requested a review from cfergeau December 16, 2025 21:10

This comment has been minimized.

Sign in to view

nirs force-pushed the vfkit-rosetta branch from 3ea11c4 to eeabf3e Compare December 17, 2025 00:10

This comment has been minimized.

Sign in to view

medyagh reviewed Dec 17, 2025

View reviewed changes

medyagh requested changes Dec 17, 2025

View reviewed changes

pkg/drivers/vfkit/vfkit.go Show resolved Hide resolved

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 20, 2025

nirs force-pushed the vfkit-rosetta branch from eeabf3e to 5ba8afb Compare December 21, 2025 00:08

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 21, 2025

nirs requested a review from medyagh December 21, 2025 00:09

This comment has been minimized.

Sign in to view

medyagh merged commit 94f2fdf into kubernetes:master Dec 22, 2025
36 of 54 checks passed

k8s-ci-robot assigned medyagh Dec 22, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 22, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 22, 2025

vfkit: Add Rosetta support for deploying amd64 images on Apple silicon #22140

vfkit: Add Rosetta support for deploying amd64 images on Apple silicon #22140

Conversation

nirs commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing amd64 executable on Apple silicon

Testing with container images

Uh oh!

nirs commented Dec 13, 2025

Uh oh!

nirs commented Dec 13, 2025

Uh oh!

nirs commented Dec 13, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

nirs commented Dec 13, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nirs commented Dec 16, 2025

Uh oh!

This comment has been minimized.

nirs commented Dec 16, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nirs commented Dec 17, 2025

Uh oh!

obnoxxx commented Dec 17, 2025

Uh oh!

nirs commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

obnoxxx commented Dec 17, 2025

Uh oh!

medyagh Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

nirs Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

nirs commented Dec 21, 2025

Uh oh!

minikube-pr-bot commented Dec 21, 2025

Uh oh!

k8s-ci-robot commented Dec 21, 2025

Uh oh!

medyagh commented Dec 22, 2025

Uh oh!

Uh oh!

k8s-ci-robot commented Dec 22, 2025

Uh oh!

minikube-pr-bot commented Dec 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nirs commented Dec 13, 2025 •

edited

Loading

nirs commented Dec 17, 2025 •

edited

Loading