Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions tools/lab/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,14 @@ Run as root on IN from the repository root. IDs 1–8 are independent labs and m
time. Every command checks the host's `/etc/resolv.conf`, `/etc/hosts` and `/etc/ppp/**` before
and after and fails loudly if anything changed.

**IDs are host-wide**, shared by every checkout and agent on the build host: `up` takes the lock
`/run/mrlab/<ID>.lock` (owner checkout + PID) until `down`, and every command refuses an ID another
checkout holds. Check first, then pick a free one:

```sh
L=tools/lab/lab.sh; ID=1
trap '$L down $ID' EXIT # always clean up
$L status $ID # "lock": free | ours | held (owner, owner_pid) | stale
trap '$L down $ID' EXIT # always clean up (also releases the ID)

# 1. a new router: never-booted Alpine 3.24 + the real install.sh (2–3 min)
$L reset $ID # empty lab: namespaces + a fresh disk
Expand All @@ -32,6 +37,8 @@ $L lan $ID exec 'curl -fsS http://example.com/ >/dev/null && ping6 -c2 2001:db8:
$L api $ID GET status
$L change $ID '[{"op":"set","path":"system.hostname","value":"r1"}]' # apply + confirm
$L change $ID '[{"op":"set","path":"dhcp.lease","value":"1h"}]' --no-confirm # leave it pending
$L change $ID '[{"op":"set","path":"lan.ipv4","value":"192.168.2.1/24"}]' # LAN moves: the client renews,
# the change is confirmed on the new address with the one-time token, later commands use it

# 5. inside the router, power
$L console $ID 'mr status; tail -20 /var/log/messages' # root shell, works with the network down
Expand All @@ -40,10 +47,15 @@ $L reboot $ID; $L powercut $ID # clean reboot / kill -9 QEMU an
# 6. clean up
$L down $ID
ip -br link | grep -c mrlab # must print 0

# an ID whose owner died without `down` (holder process gone, "lock":"stale"):
MRLAB_FORCE=1 $L reset $ID # only after checking that nobody uses it any more
```

Scenarios (TAP output, log in `/tmp/mrlab<ID>-<name>.log`): `$L run $ID 01-fresh-dhcp`,
`02-pppoe-pd`, `03-move-dhcp-samesubnet`, `04-powercut-confirm`, `05-isp-switch-live`.
`02-pppoe-pd`, `03-move-dhcp-samesubnet`, `04-powercut-confirm`, `05-isp-switch-live`,
`06-lan-renumber` (192.168.1.1 → .6 → 192.168.2.1 via the API), `07-require-public-ip` (PPPoE behind
CGNAT, `wan[].require_public_ip`).
Each starts with `reset` and removes its lab when it ends (passed, failed or interrupted;
`MRLAB_KEEP=1` leaves it up for inspection), so give every parallel scenario its own ID. A failed
setup step prints `Bail out!` and stops the scenario.
Expand Down
121 changes: 117 additions & 4 deletions tools/lab/lab.sh
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,86 @@ sw(){ echo "mrlab$1-sw"; }; isp(){ echo "mrlab$1-isp$2"; }; lan(){ echo "mrlab$1
router_ip(){ cat "$(state "$1")/lan.ip" 2>/dev/null || echo 192.168.1.1; }
run_ns(){ n=$1; shift; ip netns exec "$n" "$@"; }
pidfile(){ echo "$(state "$1")/qemu.pid"; }
# ---- host-wide lab id lock (#152)
# A lab's namespaces, taps and QEMU are host-wide while its state is per checkout. `up` starts a holder
# process that flocks /run/mrlab/<id>.lock until `down` and writes its PID and the checkout into it;
# every command that touches lab <id> refuses an id another checkout holds.
LOCKDIR=/run/mrlab
lockfile(){ echo "$LOCKDIR/$1.lock"; }
# lock_read ID: L_PID, L_CHECKOUT (empty: free) and L_HELD (1: a holder has the flock)
# Only a process that has the flock writes or empties the file. A held lock whose record is empty or
# names a dead PID is being taken over (the new holder has not written yet): readers wait for it.
# lock_read ID: L_PID, L_CHECKOUT (empty: no record), L_HELD (1: someone has the flock)
lock_read(){
_lf=$(lockfile "$1"); L_PID=; L_CHECKOUT=; L_HELD=0
[ -e "$_lf" ] || return 0
L_PID=$(sed -n 's/^pid=//p' "$_lf" | head -1); L_CHECKOUT=$(sed -n 's/^checkout=//p' "$_lf" | head -1)
flock -n "$_lf" true 2>/dev/null || L_HELD=1
}
# lock_settle ID: lock_read, waiting up to 3 s while the lock is held without a live owner record
lock_settle(){
_s=0
while :; do
lock_read "$1"
[ "$L_HELD" = 1 ] && { [ -z "$L_CHECKOUT" ] || [ ! -d "/proc/${L_PID:-0}" ]; } || return 0
[ "$_s" -lt 30 ] || return 0
sleep 0.1; _s=$((_s + 1))
done
}
# a stale record is emptied under the flock, so a holder that just won the lock is never touched
lock_clear_stale(){
_lf=$(lockfile "$1")
# shellcheck disable=SC2016
if [ -e "$_lf" ]; then flock -n "$_lf" sh -c ': > "$1"' sh "$_lf" 2>/dev/null || true; fi
}
id_guard(){
lock_settle "$1"
if [ "$L_HELD" = 1 ]; then
[ "$L_CHECKOUT" = "$LABDIR" ] && return 0
[ -n "$L_CHECKOUT" ] || die "lab $1: the lock is held but names no owner yet (another checkout is taking it): try again"
die "lab $1 is in use by $L_CHECKOUT (pid $L_PID): pick another id (lab.sh status <id> shows the owner)"
fi
if [ -z "$L_CHECKOUT" ] || [ "$L_CHECKOUT" = "$LABDIR" ]; then return 0; fi
if [ "${MRLAB_FORCE:-0}" = 1 ]; then
echo "lab: taking over lab $1 from $L_CHECKOUT (its lock holder, pid $L_PID, is gone)" >&2
lock_clear_stale "$1"; return 0
fi
die "lab $1 was left by $L_CHECKOUT, whose lock holder (pid $L_PID) is gone: MRLAB_FORCE=1 lab.sh reset $1 takes it over"
}
id_acquire(){
mkdir -p "$LOCKDIR"; _lf=$(lockfile "$1"); _spawned=0; _n=0
while [ "$_n" -lt 30 ]; do
lock_read "$1"
if [ "$L_HELD" = 1 ]; then
[ "$L_CHECKOUT" = "$LABDIR" ] && [ -d "/proc/${L_PID:-0}" ] && return 0
# a live owner from another checkout: refuse; no record / a dead PID: the winner is writing
[ -n "$L_CHECKOUT" ] && [ "$L_CHECKOUT" != "$LABDIR" ] && [ -d "/proc/${L_PID:-0}" ] && die "lab $1 is in use by $L_CHECKOUT (pid $L_PID): pick another id"
elif [ "$_spawned" = 0 ]; then
# append-open: a contender never truncates; the winner writes its record in one go
# shellcheck disable=SC2016
setsid sh -c 'exec 9>>"$1" && flock -n 9 || exit 3
ts=$(date "+%F %T"); printf "pid=%s\ncheckout=%s\nsince=%s\n" "$$" "$2" "$ts" > "$1"
exec sleep 2147483647' sh "$_lf" "$LABDIR" </dev/null >/dev/null 2>&1 &
_spawned=1
fi
sleep 0.1; _n=$((_n + 1))
done
die "lab $1: could not take the lock (held by ${L_CHECKOUT:-?}, pid ${L_PID:-?})"
}
id_release(){
lock_read "$1"; _lf=$(lockfile "$1")
if [ "$L_CHECKOUT" != "$LABDIR" ]; then
# MRLAB_FORCE down of another checkout's stale lab: its record goes as well
[ "$L_HELD" = 0 ] && [ "${MRLAB_FORCE:-0}" = 1 ] && lock_clear_stale "$1"
return 0
fi
if [ "$L_HELD" = 0 ]; then lock_clear_stale "$1"; return 0; fi
# our holder has the flock: empty the record first (nobody else writes), then stop the holder
: > "$_lf"
if [ "$(cat "/proc/$L_PID/comm" 2>/dev/null)" = sleep ] && [ "$(readlink "/proc/$L_PID/fd/9")" = "$_lf" ]; then
stop_pids "$L_PID"
fi
}
# `ip netns exec` bind-mounts /etc/netns/<ns>/* over /etc/*: every lab namespace gets its own
# resolv.conf (and the ISP its own /etc/ppp) so no daemon can write the host's files.
ns_etc_setup(){
Expand Down Expand Up @@ -315,6 +395,7 @@ up(){
die "the host service $s runs; the lab only needs the binary: systemctl disable --now $s (or MRLAB_ALLOW_HOST_SERVICES=1)"
fi
done
id_acquire "$id"
if [ -s "$d/flavor" ] && [ -e "$d/overlay.qcow2" ]; then use_flavor "$(cat "$d/flavor")"; else echo "$FLAVOR" > "$d/flavor"; fi
lock_images; ensure_base; unlock_images
[ -e "$d/overlay.qcow2" ] || { qemu-img create -f qcow2 -F qcow2 -b "$BASE" "$d/overlay.qcow2" >/dev/null; qemu-img resize "$d/overlay.qcow2" 1G >/dev/null; }
Expand Down Expand Up @@ -370,13 +451,17 @@ teardown(){
for n in "$(lan "$id")" "$(isp "$id" 2)" "$(isp "$id" 1)" "$(sw "$id")"; do ns_del "$n"; ns_etc_remove "$n"; done
rm -rf "$(state "$id")"
}
down(){ id=$1; valid_id "$id"; teardown "$id"; echo "down $id"; }
reset(){ id=$1; valid_id "$id"; teardown "$id"; up "$id"; }
down(){ id=$1; valid_id "$id"; teardown "$id"; id_release "$id"; echo "down $id"; }
reset(){ id=$1; valid_id "$id"; id_acquire "$id"; teardown "$id"; up "$id"; }
status(){
id=$1; valid_id "$id"; p=$(pidfile "$id"); alive=false; [ -s "$p" ] && kill -0 "$(cat "$p")" 2>/dev/null && alive=true
rc=$(sed -n 's/.*MRLAB-INSTALL-RC=\([0-9]*\).*/\1/p' "$(state "$id")/console.log" 2>/dev/null | tail -1)
isp1=$(cat "$(state "$id")/isp1/mode" 2>/dev/null || echo none)
printf '{"id":%s,"qemu":%s,"install_rc":"%s","isp1":"%s","router_ip":"%s"}\n' "$id" "$alive" "$rc" "$isp1" "$(router_ip "$id")"
lock_settle "$id"; lk=free
[ -n "$L_CHECKOUT" ] && { if [ "$L_HELD" = 1 ]; then lk=held; else lk=stale; fi; }
[ "$lk" = held ] && [ "$L_CHECKOUT" = "$LABDIR" ] && lk=ours
printf '{"id":%s,"lock":"%s","owner":"%s","owner_pid":"%s","qemu":%s,"install_rc":"%s","isp1":"%s","router_ip":"%s"}\n' \
"$id" "$lk" "$L_CHECKOUT" "$L_PID" "$alive" "$rc" "$isp1" "$(router_ip "$id")"
}

# ---------------------------------------------------------------- ISP simulator
Expand Down Expand Up @@ -669,7 +754,32 @@ change_cmd(){
need jq
rev=$(api_cmd "$id" GET config | jq -r .rev) || die "change: cannot read config"
body=$(jq -cn --arg rev "$rev" --argjson patch "$patch" --argjson sec "$secrets" '{base_rev: $rev, patch: $patch, secrets: $sec, confirm: 120, comment: "lab change"}')
api_cmd "$id" POST apply "$body" >/dev/null || die "change: apply refused"
resp=$(api_cmd "$id" POST apply "$body") || die "change: apply refused: $resp"
cip=$(printf '%s' "$resp" | jq -r '.confirm_ip // empty'); ctok=$(printf '%s' "$resp" | jq -r '.confirm_token // empty')
if [ -n "$cip" ]; then
# the change moves the router's LAN address (#153): like the web UI, follow it — the client
# renews (a wired PC's port flaps), then the one-time token confirms on the new address. The lab
# switches to the new address only once that worked (--no-confirm: once the router answers there).
d=$(state "$id"); old_ip=$(router_ip "$id"); moved=; end=$(($(date +%s) + 100))
while [ "$(date +%s)" -lt "$end" ]; do
sleep 4
lan_cmd "$id" dhcp >/dev/null 2>&1 || continue
if [ "$confirm" = 1 ]; then
out=$(run_ns "$(lan "$id")" curl -sS --max-time 8 -H X-MR:1 -H 'Content-Type: application/json' \
-d "{\"token\":\"$ctok\"}" "http://$cip/cgi-bin/api?a=confirm" 2>/dev/null) || continue
case $out in *'"confirmed":true'*) moved=1; break;; esac
else
run_ns "$(lan "$id")" curl -fsS -o /dev/null --max-time 5 "http://$cip/" 2>/dev/null && { moved=1; break; }
fi
done
if [ -z "$moved" ]; then
die "change: LAN move to $cip not $([ "$confirm" = 1 ] && echo confirmed || echo reachable) within 100 s; the router rolls back to $old_ip by itself, the lab stays on $old_ip"
fi
printf '%s\n' "$cip" > "$d/lan.ip"; rm -f "$d/api.cookies"
if [ "$confirm" = 1 ]; then echo "{\"lan_moved_to\":\"$cip\",\"confirmed\":true,\"via\":\"lan-token\"}"
else echo "{\"lan_moved_to\":\"$cip\",\"confirmed\":false,\"note\":\"pending: lab.sh api $id POST confirm '{}' keeps it\"}"; fi
return 0
fi
i=0; st=running
while [ "$st" = running ] && [ "$i" -lt 180 ]; do
sleep 2; i=$((i + 2))
Expand All @@ -687,6 +797,9 @@ powercut_cmd(){ id=$1; valid_id "$id"; kill_qemu "$id" 9; up "$id" >/dev/null; w

usage(){ die "usage: lab.sh up|down|reset|status <id> | isp <id> <wan> dhcp|pppoe|static|down [opts] | install <id> [opts] | lan <id> dhcp|static|check|exec | console|ssh <id> <cmd> | api <id> GET|POST <action> [json] | change <id> <patch-json> [--no-confirm] | reboot|powercut <id> | run <id> <scenario>"; }
cmd=${1:-}; shift || true
case $cmd in
up|down|reset|isp|install|lan|ssh|console|api|change|run|reboot|powercut) valid_id "${1:-}"; id_guard "$1";;
esac
case $cmd in
up) up "$@";; down) down "$@";; status) status "$@";; reset) reset "$@";;
isp) isp_cmd "$@";; install) install_cmd "$@";; lan) lan_cmd "$@";;
Expand Down
29 changes: 29 additions & 0 deletions tools/lab/scenarios/06-lan-renumber.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#!/bin/sh
# Changing the LAN subnet from the web UI (#153): 192.168.1.1/24 -> 192.168.1.6/24 (same subnet) ->
# 192.168.2.1/24. The change is confirmed on the new address with the one-time token, the client
# renews, and the change must still be there after the 120 s confirm window.
NAME=06-lan-renumber
# shellcheck source=lib.sh
. "$(dirname "$0")/lib.sh"
# shellcheck disable=SC2016
panel_on() { lab lan "$ID" exec "test \"\$(curl -s -o /dev/null -w %{http_code} --max-time 5 http://$1/)\" = 200"; }
lan_is() { lab console "$ID" "grep -q '^ ipv4: $1\$' /etc/mini-router/router.yaml && ip -4 -o addr show br-lan | grep -q ' $1 '"; }
step lab reset "$ID"
step lab isp "$ID" 1 dhcp
check "install.sh completes (DHCP WAN)" lab install "$ID" --wan-proto dhcp
check "baseline: client online via 192.168.1.1" sh -c "\"$LAB_SH\" lan $ID dhcp && \"$LAB_SH\" lan $ID exec 'curl -4 -fsS -o /dev/null --max-time 10 http://example.com/'"
check "API: LAN 192.168.1.1 -> 192.168.1.6 confirmed on the new address" lab change "$ID" '[{"op":"set","path":"lan.ipv4","value":"192.168.1.6/24"}]'
check ".6: panel answers on 192.168.1.6" retry 30 panel_on 192.168.1.6
check ".6: 192.168.1.1 is gone from br-lan" lab console "$ID" '! ip -4 -o addr show br-lan | grep -q " 192.168.1.1/"'
check ".6: client renewed, gateway 192.168.1.6, online" retry 30 sh -c "\"$LAB_SH\" lan $ID dhcp >/dev/null && \"$LAB_SH\" lan $ID exec 'ip -4 route show default | grep -q \"via 192.168.1.6 \" && curl -4 -fsS -o /dev/null --max-time 10 http://example.com/'"
sleep 130
check ".6: still applied after the confirm window (no rollback)" lan_is 192.168.1.6/24
check "API: LAN 192.168.1.6 -> 192.168.2.1 confirmed on the new address" lab change "$ID" '[{"op":"set","path":"lan.ipv4","value":"192.168.2.1/24"}]'
check "2.1: client lease in 192.168.2.0/24, gateway 192.168.2.1" retry 30 sh -c "\"$LAB_SH\" lan $ID dhcp | grep -q '\"addr\":\"192.168.2.' && \"$LAB_SH\" lan $ID exec 'ip -4 route show default | grep -q \"via 192.168.2.1 \"'"
check "2.1: panel answers on 192.168.2.1" retry 30 panel_on 192.168.2.1
check "2.1: client online and resolves names" retry 30 sh -c "\"$LAB_SH\" lan $ID exec 'getent hosts example.com >/dev/null && curl -4 -fsS -o /dev/null --max-time 10 http://example.com/'"
sleep 130
check "2.1: still applied after the confirm window (no rollback)" lan_is 192.168.2.1/24
check "2.1: no old address left on br-lan" lab console "$ID" '! ip -4 -o addr show br-lan | grep -qE " 192\.168\.1\.(1|6)/"'
check "2.1: API works on the new address (login + status)" sh -c "\"$LAB_SH\" api $ID GET status | grep -q '\"wan\"'"
finish
33 changes: 33 additions & 0 deletions tools/lab/scenarios/07-require-public-ip.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/bin/sh
# wan[].require_public_ip (#150): the ISP hands out CGNAT addresses (100.64.<id>.x); with the option
# on, the router redials after the gap (cron re-check), stops after 5 private addresses with a risk
# event, and takes a public address again when the ISP offers one.
NAME=07-require-public-ip
# shellcheck source=lib.sh
. "$(dirname "$0")/lib.sh"
sessions() { _c=$(grep -c "^remote IP address" "$(dirname "$LAB_SH")/state/mrlab$ID/isp1/pppd.log" 2>/dev/null) || _c=0; echo "$_c"; }
more_sessions() { [ "$(sessions)" -gt "$1" ]; }
step lab reset "$ID"
step lab isp "$ID" 1 pppoe --cgnat
check "install.sh completes (PPPoE WAN behind CGNAT)" lab install "$ID" --wan-proto pppoe
step lab lan "$ID" dhcp
check "router has a CGNAT address (100.64.$ID.x)" lab console "$ID" "ip -4 addr show pppoe-wan | grep -q 'inet 100.64.$ID.'"
before=$(sessions)
note "PPPoE sessions before: $before"
check "API: require_public_ip on, gap 30 s (confirmed)" lab change "$ID" '[{"op":"set","path":"wan[wan].require_public_ip","value":true},{"op":"set","path":"wan[wan].require_public_ip_gap","value":30}]'
check "crontab has the public-check line" lab console "$ID" 'grep -q "mr wan public-check --cron" /etc/crontabs/root'
check "router redials (a new PPPoE session at the ISP within 150 s)" retry 150 more_sessions "$before"
check "syslog: 'cgnat IPv4 address; redial N/4'" retry 60 lab console "$ID" 'grep -qE "wan wan: .*IPv4 address; redial [0-9]/4" /var/log/messages'
check "after 5 private addresses: redial stops with a risk event" retry 420 lab console "$ID" 'mr event list 2>&1 | grep -q "automatic redial stopped"'
n1=$(sessions)
sleep 90
no_more_sessions() { [ "$(sessions)" -le "$1" ]; }
check "stopped: no further redials (sessions stay at $n1)" no_more_sessions "$n1"
# a CGNAT session that passes the health check is "ok" in mr doctor by design (#150); the stop is
# recorded as the risk event above
info lab console "$ID" 'mr doctor 2>&1 | grep -A1 "wan.wan"'
info lab console "$ID" 'grep -E "public|redial" /var/log/messages | tail -12; cat /run/mini-router/*public* 2>/dev/null'
step lab isp "$ID" 1 pppoe
check "ISP offers public addresses again: router gets 203.0.113.x" retry 120 lab console "$ID" "ip -4 addr show pppoe-wan | grep -q 'inet 203.0.113.'"
check "LAN client online over the public session" retry 60 sh -c "\"$LAB_SH\" lan $ID dhcp >/dev/null && \"$LAB_SH\" lan $ID exec 'curl -4 -fsS -o /dev/null --max-time 10 http://example.com/'"
finish
Loading