A production-like NOC/SOC monitoring lab built on top of a self-hosted K3s HA cluster: two isolated networks, a dedicated firewall, and a full observability + security stack deployed from scratch, then automated with Ansible.
Prerequisite: K3s-lab, the 6-node K3s HA cluster this lab monitors.
- The k3snet interface with zero firewall rules : 8 Wazuh agents stuck in
pending, a routing fix that looked complete but wasn't the real cause, and howtcpdump+pfctlstate tables ruled out four wrong suspects before finding the actual one. - 20 hours lost to one Wazuh default : four unrelated bugs stacked on top of each other, each masking the next; why "the scanner reports success but finds nothing" is worth checking what it's scanning before how it writes results.
- Reversing a router migration cleanly : a FortiGate cutover halted after checking license limits against the topology before cutover, documented as a correctly diagnosed constraint, not a failed attempt
This lab extends the K3s cluster with a fully isolated monitoring infrastructure:
- NOC: host metrics via Zabbix, K8s metrics via Prometheus (kube-state-metrics + kubelet/cAdvisor with explicit RBAC), unified in Grafana
- SOC: Wazuh SIEM validated end-to-end: FIM (realtime), SSH brute-force active response, CVE detection, VirusTotal integration, email alerting
- Network isolation: monitoring VMs live on a separate network, routed through OPNsense
- Automation: Zabbix + Wazuh agent rollout and host registration across all 13 VMs via Ansible
Everything is documented phase by phase, including architecture decisions (ADR) and troubleshooting post-mortems.
k3s-net (10.10.0.0/24) monitoring-net (10.20.0.0/24)
K3s-srv-1 10.10.0.11 Zabbix-srv 10.20.0.10
K3s-srv-2 10.10.0.12 Wazuh-srv 10.20.0.11
K3s-srv-3 10.10.0.13 Prometheus-srv 10.20.0.12
K3s-agent-1 10.10.0.31 Grafana-srv 10.20.0.13
K3s-agent-2 10.10.0.32 Loki-srv 10.20.0.14
K3s-agent-3 10.10.0.33 Ansible-srv 10.20.0.125
K3s-db 10.10.0.20
Load-srvs 10.10.0.10
OPNsense WAN 10.10.0.254 ────────────── OPNsense LAN 10.20.0.254
The two networks are fully isolated at L2. OPNsense handles all inter-network routing and provides firewall logs, packet capture, and traffic visibility.
| Tool | Role |
|---|---|
| Zabbix 7.4 | Host metrics (CPU, RAM, disk, network) via agent + infrastructure alerting |
| Prometheus 3.12 | K8s metrics: kube-state-metrics, kubelet, cAdvisor |
| Grafana 13.1 OSS | Unified NOC dashboards: Zabbix + Prometheus datasources |
| Loki | Log aggregation, monolithic deployment, local FS storage |
| Grafana Alloy | Log/metric shipping agent, deployed via Ansible across 12 nodes |
| Wazuh 4.14 | SOC: SIEM, FIM, brute force active response, CVE scan, VirusTotal |
| OPNsense 26.1 | Router/firewall between k3s-net and monitoring-net |
| Ansible | Agent deployment + Zabbix host registration (configs/ansible) |
| Phase | Description | Status |
|---|---|---|
| 0 | GitHub setup, repo structure, aliases | ✅ Done |
| 1 | Network, VMs, OPNsense, persistent routes | ✅ Done |
| 2 | Install Zabbix · Wazuh · Prometheus · Grafana | ✅ Done |
| 3 | NOC: Grafana datasources · Prometheus self-monitoring dashboard | ✅ Done |
| 4 | SOC: Wazuh FIM, active response, CVE, VirusTotal, email alerting | ✅ Done |
| 5 | Automation: Ansible agent rollout · Prometheus ↔ K3s scraping | ✅ Done |
| 6 | Dashboard suite: Golden Signals, USE/RED, cross-source panels | 🔄 In progress |
| 7 | Router migration: OPNsense → FortiGate | 🚫 Blocked (license limits, see ADR-013 amendment) |
| 8 | Logging: Loki + Grafana Alloy | ✅ Done |
| Dashboard | Scope |
|---|---|
| Prometheus Self-Monitoring: published on Grafana.com (ID 25537) | Process-level metrics (memory, CPU, TSDB cardinality, query engine health) for the Prometheus server itself: zero external dependencies |
# Start the K3s cluster
k3s-start
# Start the monitoring stack (Wazuh, Prometheus, Grafana, Zabbix)
monitoring-start
# Stop
monitoring-stop
k3s-stop| Doc | Description |
|---|---|
| Architecture Decisions | Why this stack, why these choices: full ADR log |
| Troubleshooting: Ansible rollout war stories | 13 real-world issues hit during the agent rollout: symptom → root cause → fix |
| Troubleshooting: Wazuh SOC | Post-mortems: silent vulnerability scanner, inert XML config blocks, mail relay |
| Troubleshooting: Zabbix Apache auth | Post-mortem: Apache stripping the Authorization header before PHP-FPM |
| Troubleshooting: Grafana schema v2 export | Post-mortem: why the native "Export as code" flow produces an incompatible schema |
- Host: ThinkPad E14 Gen 5, i7-13700H, 31GB RAM, 476GB NVMe
- Hypervisor: KVM/libvirt on Arch Linux
- VM OS: Ubuntu 26.04 LTS Server · Alpine Linux (Ansible control node)