microkvm

microkvm is a step-by-step educational VMM built directly on the Linux KVM API to learn virtualization internals.

Each step introduces exactly one new virtualization concept, starting from a minimal guest that executes hlt and gradually evolving toward a full Linux boot, virtio I/O, and live migration. By step 11, microkvm boots a real Linux kernel and provides an interactive shell over an emulated 8250 serial console. From step 12 onward, the guest stays fixed and the VMM itself evolves - mirroring real-world hypervisor development.

Each step is intentionally minimal and self-contained. VM exits can be observed through KVM tracing and correlated with userspace-visible exits.

Unlike production VMMs such as QEMU, microkvm intentionally prioritizes readability, traceability, and incremental learning over performance.

• KVM ioctl workflow (/dev/kvm → VM → vCPU → KVM_RUN)
• VM exits and exit handling
• x86 CPU privilege modes (real → protected → long mode)
• x86 paging and identity-mapped page tables
• Interrupt delivery (IDT, interrupt gates, iretq)
• MMIO and device emulation
• MSR handling (TSC, synthetic MSRs)
• Multi-vCPU with pthreads
• Booting a real Linux kernel from a minimal VMM
• 8250 UART emulation (TX + RX with DLAB, IER, IIR, IRQ injection)
• Virtio-mmio transport (device discovery, feature negotiation, virtqueue setup)
• Virtio-console (TX and RX via descriptor/avail/used rings)
• Exit reduction techniques (ioeventfd, irqfd)
• EPT/MMU internals and demand paging observation
• Dirty page tracking (KVM_GET_DIRTY_LOG)
• VM snapshot and restore
• Live migration (dirty logging, iterative pre-copy, downtime reduction)
• PCI Configuration Mechanism #1 (CF8/CFC port I/O)
• BAR probing and MMIO device register access
• DMA via descriptor ring and doorbell kick
• MSI-X interrupt delivery (KVM_SIGNAL_MSI)
• PCI hotplug (config-space visibility + sysfs rescan/remove)
• Linux PCI driver development (pci_register_driver, probe/remove lifecycle)
• BAR mapping from kernel space (pci_iomap, readl/writel)
• DMA from a Linux driver (dma_alloc_coherent, descriptor submission)
• MSI-X interrupt handling (pci_alloc_irq_vectors, request_irq)
• Follow VM exits through Linux KVM source code
• Trace KVM internals with ftrace and perf
• Understand VMCS, EPT, and VM-execution controls at the hardware-virtualization level

QEMU is production-grade and feature-rich. microkvm intentionally trades completeness for readability. Each step introduces exactly one concept, making every VM exit and device interaction easy to trace and understand.

microkvm is an educational VMM intended for a cooperative Linux guest. It implements basic correctness checks but is not designed as a hardened security boundary for untrusted guests.

Each step adds exactly one concept. Part 1 steps are code milestones and are tagged in git. Part 2 is an observation/documentation phase and Part 3 mixes small observation tools with source-reading, so not every step has a tag.

Part 1 is the build phase. Each step adds one working feature to the VMM, from a minimal hlt guest up to Linux boot, virtio I/O, live migration, and a PCI device with its Linux driver.

Linux milestone: From Step 12 onward, microkvm uses Linux as the guest. The focus shifts from extending guest.S to evolving the VMM itself.

Toggle USE_IOEVENTFD=1 / USE_IRQFD=1 to compare. See docs/en/benchmark.md for details.

Example Results (Step 22 complete)

MMU stats (Step 19):
  pages_4k: 6966, pf_fixed: 6966

Dirty tracking (Step 20):
  echo hello:  137 pages
  dd 4MB:     1224 pages

Live migration (Step 22):
  Iteration 0: 32768 pages
  Iteration 1: 27 pages
  Downtime:    0.5 ms

Driver milestone: Phase E built the VMM-side PCI device model. Phase F builds the Linux guest-side driver for the same virtual PCI device. Together they demonstrate both sides of a device virtualization stack: the device model in the VMM and the driver inside the guest kernel.

Part 2 is an observation/documentation phase. It does not add VMM features; the deliverables are traces, source-reading notes, and behavioral explanations comparing microkvm with Linux KVM.

Part 3 explores VT-x through the interfaces KVM already exposes. KVM itself is not modified; VMCS/EPT behavior is observed indirectly through KVM APIs and tracepoints, then correlated with KVM source and the Intel SDM.

Part 1 code milestones are tagged in git:

git checkout step1    # view Step 1 code
git checkout step11   # view Linux boot milestone
git checkout step22   # view live migration
git checkout step27   # view PCI hotplug
git checkout step31   # view Linux PCI driver (MSI-X)
git checkout step40   # view KVM-visible guest state dump
git checkout step41   # view memory slot explorer
git checkout main     # return to latest

To see what changed between steps:

git diff step20..step21   # what snapshot added

Part 2 (Step 32–39) and Step 42 do not add VMM features — they are observation and source-reading. Their write-ups live under docs/ rather than a git tag.

make
./microkvm

For Phase B+ (Linux boot), place bzImage and initramfs.gz in the same directory:

./microkvm                                    # boot Linux
./microkvm --restore snapshot.bin             # restore from snapshot  (Step 21)
./microkvm --restore-migration migration.bin  # restore from migration (Step 22)

• Linux with KVM support (/dev/kvm)
• x86_64 Linux host. Part 3 specifically follows Intel VT-x/VMX terminology (VMCS, EPT, VM-execution controls) and expects an Intel VT-x-capable host for direct correspondence with the documented flow.
• GCC, GNU Make
• bzImage + initramfs.gz for Step 10+

Direct VMCS/EPT instrumentation, execution-control mutation, and a standalone VMX implementation are intentionally outside microkvm's completed learning path.

Released under the MIT License.

• Using the KVM API (LWN)
• Linux KVM source
• Virtio specification v1.2