Tags: lmb/vimto
Tags
init: work around the mother of all race conditions ebpf-go CI has been plagued by sporadic hangs, where tests simply time out while trying to write status information to stdout. The bug manifests when issuing blocking writes to a virtio console while also polling it. The way we trigger the bug is quite involved: - init opens tha port via os.OpenFile. This sets O_NONBLOCK on the fd, and registers the os.File with the poller. - The port is passed to the child process via exec.Cmd.Stdout. This internally calls os.File.Fd(), which clears O_NONBLOCK but doesn't remove the file from the poller. - The child process receives a blocking stdout. Writing to it will issue a blocking write to the virtio-console port, specifically port_fops_write() in virtio_console.c. - port_fops_write() calls wait_port_writable(). This puts the calling thread to sleep if the virtqueue is full, by waiting on port->waitqueue. We now enter the race window. - The host processes the guest's write, frees up some space in the virtqueue and issues an interrupt to the guest. - This interrupt races with a call to port_fops_poll() issued by the init process's Go runtime. That function invokes will_write_block(), which consumes all used buffers from the virtqueue. - The interrupt handler vring_interrupt() checks whether the virtqueue has any unused buffers via more_used(). Since all buffers have just been consumed by port_fops_poll() the interrupt is dropped. At this point we still have a writer stuck in port_fops_write() waiting for a wakeup that never comes. The workaround for this issue is to close the stdio file in init, thereby removing it from the runtime poller. Fixes: #29
Revert "work around race in virtio serial console" This reverts commit 475850d. It seems that kernels < 6.1 suffer from a separate lost wakeup problem which manifests in reads of the control message timing out: [ 1.312132] Run /home/runner/go/bin/vimto as init process Error: read command: read /dev/vport2p1: i/o timeout [ 3.144910] ACPI: Preparing to enter system sleep state S5
work around race in virtio serial console
The ebpf-go CI has been plagued by a non-deterministic hang of
unit tests. It affects all packages and manifests as a write to
stdout getting stuck, followed by the test timing out. This
triggers a goroutine dump, which in turn unblocks the stuck write
to stdout.
Its possible to reproduce this behaviour using the following
commandline:
taskset -c 0 vimto -smp cpus=2 -kernel ghcr.io/cilium/ci-kernels:6.15.3 \
exec -- sh -c 'seq 1 1000000 | while read i; do echo "line $i"; done'
After a few seconds the output will freeze. Inspecting the stack of
the executing program shows something like the following:
[<0>] wait_port_writable+0x139/0x2d0
[<0>] port_fops_write+0x88/0x130
[<0>] vfs_write+0xf3/0x450
[<0>] ksys_write+0x6d/0xe0
[<0>] do_syscall_64+0x9e/0x1a0
[<0>] entry_SYSCALL_64_after_hwframe+0x77/0x7f
1 0x1 0x7ffdf4878c80 0x9 0x0 0x0 0x0 0x7ffdf4878c20 0x7f592daed77e
As far as I can tell it is critical that execution is restricted to
a single CPU on the host side, while qemu presents two vCPU to the VM.
Passing ioeventfd=off to the serial console device works around
this problem.
See cilium/ebpf#1734 for more details.
don't extract image for each script test The script test runner sets a new TMPDIR for each script. This changes the location of the cache. Hence every invocation of vimto does a new fetch, which adds up to a bit of time. Add an explicit test for OCI behaviour and reuse the cached kernel in other tests.