[DRAFT] Pass cloud init to shim containers#6038
Draft
zedi-pramodh wants to merge 1 commit into
Draft
Conversation
de17dae to
4d4712e
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6038 +/- ##
==========================================
+ Coverage 20.72% 21.11% +0.38%
==========================================
Files 504 514 +10
Lines 92760 94500 +1740
==========================================
+ Hits 19222 19951 +729
- Misses 71917 72745 +828
- Partials 1621 1804 +183 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
@zedi-pramodh , pls, don't forget to fix the commit message (blank) and the PR description.... the changes looks good for me... |
b008fd3 to
c2f9fd5
Compare
Container apps on eve-k run as kubevirt shim VMIs (VirtualizationMode != NOHYPER)
launched from eve-external-boot-image. Two HV=k-only gaps relative to HV=kvm
are addressed:
1. Cloud-init env vars never reached the guest. On HV=kvm/xen the snapshot-
mount path projects env vars into /environment on the rootfs via the OCI
spec. On HV=k the container rootfs is a read-only PVC at /dev/vda, so that
path is not available.
2. Customer-attached volumes were visible in the guest but never mounted, so
the container ran without its persistent data.
Pillar side:
- pkg/pillar/cmd/domainmgr/domainmgr.go: when hvTypeKube &&
VirtualizationMode != NOHYPER, emit a synthetic DiskStatus with
Devtype="cloudinit-nocloud" and the env map rendered as sorted shell
KEY="value" lines (escaping " and \). Refactor the VM-app cloud-init ISO
path into appendCloudInitDisk so its behavior is unchanged; the new
branch lives alongside it.
- pkg/pillar/hypervisor/kubevirt.go: translate the cloudinit-nocloud
DiskStatus into v1.Volume{CloudInitNoCloud: {UserDataBase64: ...}} attached
as a read-only sata cdrom. Base64 keeps the payload out of plaintext
`kubectl describe vmi`.
Initrd side (pkg/xen-tools/initrd):
- init-initrd: for root != 9p (i.e. HV=k path, not HV=kvm/xen which already
projects env vars via OCI), probe for a cdrom with LABEL=cidata, mount it
iso9660, and source user-data lines as `export KEY=value` into
/mnt/environment. Parse raw `blkid` output by hand — busybox's
`blkid -L cidata` returns empty in the eve-external-boot-image build even
when raw `blkid` enumerates the device with the correct LABEL field. No
modprobe is needed; iso9660 and the sata/sr_mod stack are built into the
eve-external-boot-image kernel.
- mount_disk.sh: skip the boot block device. On HV=k the container PVC is
itself the boot block device, already mounted at /mnt. Without the skip
it would be treated as the first extra volume — either remounting the
rootfs on top of itself, or consuming mountPoints line 1 and shifting
every subsequent device-to-path mapping down by one, leaving the
customer's actual extra volume (/dev/vdb) either unreached or mapped to
the wrong target. On HV=kvm/xen root=9p has no block device at /mnt so
the guard is a no-op there.
- mount_disk.sh: exit cleanly when /mnt/mountPoints is absent (image
declared no VOLUMEs — writeKubevirtMountpointsFile in
pkg/pillar/cas/containerd.go gates the write on len(mountpoints) > 0),
instead of erroring on every enumerated block device.
Build system (Makefile):
- Set `pkg/external-boot-image: FORCE_BUILD := --force`. The package tag
is hashed only from git-tracked files (Dockerfile.in, build.yml), so it
never changes when xen-tools changes. Without --force, `linuxkit pkg build`
short-circuits on cache-hit and keeps serving stale runx-initrd
indefinitely, regardless of init-initrd / mount_disk.sh edits.
- Set `pkg/kube: FORCE_BUILD := --force` for the same reason one level up:
pkg/kube bundles pkg/kube/external-boot-image.tar which is gitignored, so
changes to it don't change pkg/kube's hash. Without --force, the eve-kube
image in cache keeps the old tar baked in, and the rootfs ships stale
content to the device.
Signed-off-by: Pramodh Pallapothu <pramodh@zededa.com>
c2f9fd5 to
38acc3d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For shim containers on eve-k we are not passing in the cloud init. This fix addressed that.
PR dependencies
List all dependencies of this PR (when applicable, otherwise remove this
section).
How to test and validate this PR
Please describe how the changes in this PR can be validated or verified. For
example:
This will be used
The first is especially important, so, please make sure to provide as much
detail as possible.
If it's covered by an automated test, please mention it here.
Changelog notes
Text in this section will be used to generate the changelog entry for
release notes. The consumers of this are end users, not developers.
So, provide a clear and short description of what is changed in the PR from
the end user perspective. If it changes only tooling or some internal
implementation, put a note like "No user-facing changes" or "None".
PR Backports
For all current LTS branches, please state explicitly if this PR should be
backported or not. This section is used by our scripts to track the backports,
so, please, do not omit it.
Here is the list of current LTS branches (it should be always up to date):
For example, if this PR fixes a bug in a feature that was introduced in 14.5,
you can write:
Also, to the PRs that should be backported into any stable branch, please
add a label
stable.Checklist
For backport PRs (remove it if it's not a backport):
And the last but not least:
check them.
Please, check the boxes above after submitting the PR in interactive mode.