GitOps makes a specific promise: the cluster matches git. When a change is merged, it deploys. When a resource is deleted from git, it’s deleted from the cluster. Merge = deploy is the core guarantee the entire workflow rests on.

It’s not always true. This is a story about finding three cases where it wasn’t - two discovered during a drift audit, one that would have been inert even if ArgoCD had synced it correctly.

View the complete homelab infrastructure source on GitHub 🐙

directory.includeThe cluster uses ArgoCD’s app-of-apps pattern: a root Application watches a directory, creates child Applications for each manifest it finds, and those child Applications sync the actual workloads. The root Application uses aglob to decide which files to track.

The problem: that glob is set once, at bootstrap time, and files added to the directory later are only tracked if they match the existing glob. A file that doesn’t match is invisible to ArgoCD - it won’t create an Application for it, won’t sync it, and won’t alert about it. From ArgoCD’s perspective, the file doesn’t exist.

application.ymlmanifests-application.ymlkubernetes/system/monitoring/directory.includeThe monitoring stack had this problem. Two files -and- had been added to thedirectory but were missing from the parent Application’sglob:

configSecretapplication.ymlA fix for a brokenreference had been merged to. It was valid YAML, it would have worked - but ArgoCD never picked it up. The live cluster was still running the broken version from initial bootstrap, indefinitely.

kubectl get applications -n argocdSynced/HealthyThe standard ArgoCD health check () doesn’t catch this. All Applications show. The bootstrapped-but-untracked files don’t generate an Application at all, so there’s nothing to show as out-of-sync.

kubernetes/system/*/The audit that found it was manual: compare the contents of eachdirectory against the tracked files in the corresponding bootstrap Application, and flag any file not in the glob:

The audit found approximately 14 directories with the same untracked-after-bootstrap gap - no current drift in most of them, but nothing preventing drift from accumulating silently if files were added there in the future.

SyncedThis is the same class of gap that a GitHub Actions Terraform-plan-to-Slack alert is designed to close on the Terraform side — a green pipeline or abadge tells you the command ran, not that the running state matches intent.

allowCrossNamespaceargo.woitzik.devmonitoring.woitzik.devstatus.woitzik.devTraefik’sfix had been merged to git on 2026-07-04. The change allowed IngressRoutes in one namespace to reference Services in another - without it,,, andreturned bare 404s.

SyncedThe fix was in git. ArgoCD showed the Traefik Application as. But the endpoints were still 404ing.

SyncedThe cause: the Traefik Application’s last successful sync was before the fix merged. ArgoCD’s sync status reflects the last successful reconciliation, not a continuous comparison. If the sync was clean before the change and nothing triggered a re-sync after (no new commit to the watched path, no manual refresh), the Application can showwhile being hours or days behind.

After manual sync, all three endpoints came back to 302 within seconds of the rollout completing.

Velerohad a merged fix for repository maintenance frequency - too-frequent Kopia maintenance jobs were running during backup windows and causing timeouts. The fix set a custom maintenance frequency via Helm values.

The fix was synced correctly. It also did nothing, for two independent reasons:

Reason 1: Wrong Helm values structure.

extraArgsconfigurationThekey was placed at the top level of the Helm values instead of nested under:

Helm silently drops unknown top-level keys. No error, no warning - the extra args simply weren’t passed to the binary. Velero started with its defaults.

Reason 2: Wrong flag name.

--default-repo-maintenance-frequency--default-repo-maintain-frequencyThe flagdoesn’t exist in Velero. The real flag is(no “ance”). Even if the values structure had been correct, the binary would have printed an unknown flag error and ignored it.

Two independently-wrong things, each of which would have prevented the fix from taking effect on its own. Caught during the drift audit by checking the running Velero container’s actual args:

The correct fix, both issues resolved:

All three failures share the same shape:

In each case, the developer closed the PR with the reasonable belief that the fix was live. In each case, it wasn’t. The merge was the easy part; the verification that the change actually took effect in the running cluster is the part that was skipped.

For any change where “merged” isn’t sufficient proof of “deployed,” there’s one check that works:

None of these are expensive. They’re all one-liners. The cost of skipping them - in this case, Traefik returning 404 for three production subdomains, Velero running backup maintenance at the wrong frequency indefinitely, and monitoring fixes sitting inert - was substantially higher.

The same GitOps verification gap exists in Azure DevOps pipelines and GitHub Actions that deploy Terraform or Helm. A pipeline that reports “succeeded” proves that the deployment command ran without error - it doesn’t prove that the cluster’s running state matches the manifest. Checking the running state directly, after deployment, is the only check that closes the gap.

The Phoenix Project*is still the sharpest account I know of exactly this gap between “the change was made” and “the change took effect” - fiction, but painfully accurate about how it actually happens.

How the Gap Opens: ArgoCD’s App-of-Apps Bootstrap Problem

Finding the Gap: What the Drift Audit Actually Checks

The First Live Finding: Traefik allowCrossNamespace

The Second Finding: Velero’s Fix That Couldn’t Have Worked

The Three-Part Pattern

The Verification Step That Prevents It

# monitoring-bootstrap/application.yml (the root)spec:source:directory:include:"alertmanager-config.yml"# ← only this file tracked# application.yml and manifests-application.yml: invisible

# For each subdirectory, list tracked files vs actual filesfordirinkubernetes/system/*/;doecho"===$dir==="# What ArgoCD tracks (from the bootstrap Application's directory.include)# vs what's actually in the directoryls"$dir"*.yml2>/dev/nulldone

# Force ArgoCD to re-compare against current git HEADargocdappsynctraefik--force# or via annotation:kubectlannotateapplicationtraefik-nargocd\argocd.argoproj.io/refresh=hard--overwrite

# What was merged (broken)velero:extraArgs:-"--default-repo-maintain-frequency=168h"# What Helm actually needsvelero:configuration:extraArgs:-"--default-repo-maintain-frequency=168h"

kubectlgetpod-nvelero-lapp.kubernetes.io/name=velero-ojsonpath=\'{.items[0].spec.containers[0].args}'|tr',''\n'# → no --default-repo-maintain-frequency flag present

velero:configuration:extraArgs:-"--default-repo-maintain-frequency=168h"

# For a Helm values change: verify the flag is in the running containerkubectlgetpod-n<namespace>-l<label-selector>\-ojsonpath=&#39;{.items[0].spec.containers[0].args}&#39;# For a Kubernetes manifest change: verify the field is livekubectlget<resource>-n<namespace><name>\-ojsonpath=&#39;{.spec.<changed-field>}&#39;# For an ArgoCD Application bootstrap: verify the file is trackedargocdappget<bootstrap-app>--outputjson|\jq&#39;.spec.source.directory.include&#39;

CPU contention causes for etcdleaderElectionRetryPeriodKyverno’s cleanup-controller was restarting every few hours. The leader-election timeout was expiring under load — the same class of leader-election instability that— the controller lost its lock, and the new leader took over, causing a brief gap in policy enforcement. The fix looked simple: increase. The fix crash-looped the very pod it was meant to stabilize.

The pod started, immediately crashed, and Kubernetes restarted it. Three crashes in under two minutes. The Kyverno dashboard showed the cleanup controller as unavailable, and policy reports stopped updating.

The crash-loop meant the binary was rejecting its own flags. Checking the pod’s logs:

--0=--leaderElectionRetryPeriod=10sThe binary receivedas a single flag. That’s not a valid argument — it’s a rendered artifact of wrong YAML structure.

cleanupController.extraArgsmapThe Helm chart’sexpects a, not a list:

range$key, $valueHelm’sfunction iterates over maps aspairs. When you pass a list, Helm treats each list item as a map entry with an integer index as the key. The rendered result:

--0The binary seesas a flag name, doesn’t recognize it, and exits with a non-zero code. Kubernetes restarts the pod. Crash-loop.

This wasn’t a staging environment. The Kyverno cleanup-controller handles:

causedThe irony: the fix for leader-election instabilitya worse instability. The cleanup-controller restarted 3 times before I caught it — and during those restarts, policy reports accumulated without cleanup.

1. Dry-run the Helm template:

--0=...If the rendered YAML showsin the container args, the structure is wrong.

2. Check the chart’s values.yaml:

extraArgsThe chart documents whetheris a map or list. Trust the chart, not your muscle memory from other Helm charts.

3. Use Kyverno’s policy reports to validate:

After deploying, check the cleanup controller’s pod status immediately:

CrashLoopBackOffIf it showswithin the first minute, the flags are wrong.

One line change. The pod started cleanly, the crash-loop stopped, and policy reports resumed normal cleanup.

This is a common Helm gotcha across many charts, not just Kyverno:

helm templateThe fix is always the same: if the chart expects a map, use map syntax. If you’re unsure,will show you exactly what the binary receives.

Further reading:Designing Data-Intensive Applications by Martin Kleppmann*For a deeper dive into distributed systems failure modes,explains why leader-election timeouts behave the way they do under I/O pressure.

Kyverno policies are only effective when the controller is running. A crash-looping cleanup controller means policy reports accumulate without cleanup, and namespace selector caches go stale. Pin your Helm values correctly, and dry-run before deploying to production.

The Symptom

The Investigation

The Root Cause: List vs Map

Why This Is a Production Risk

How to Catch This Before It Hits Production

The Fix

The Pattern

cleanup-controller-7f8b9c4d6-xk2p4   0/1     CrashLoopBackOff   3 (42s ago)

Error: unknown flag: --0=--leaderElectionRetryPeriod=10s

# WRONG — list syntax, renders as --0=--leaderElectionRetryPeriod=10scleanupController:extraArgs:-"--leaderElectionRetryPeriod=10s"# CORRECT — map syntax, renders as --leaderElectionRetryPeriod=10scleanupController:extraArgs:leaderElectionRetryPeriod:"10s"

# What the binary actually receives:--0=--leaderElectionRetryPeriod=10s# What it expected:--leaderElectionRetryPeriod=10s

helmtemplatekyvernokyverno/kyverno-nkyverno\--setcleanupController.extraArgs.leaderElectionRetryPeriod=10s\|grep-A5cleanup-controller

helmshowvalueskyverno/kyverno|grep-A10extraArgs

kubectlgetpods-nkyverno-lapp.kubernetes.io/name=kyverno-cleanup-controller

# kubernetes/system/kyverno/application.ymlcleanupController:extraArgs:leaderElectionRetryPeriod:"10s"

• Policy report cleanup— old reports pile up without it
• Namespace selector caching— stale cache entries cause policy enforcement delays
• Leader election— the restart cycle itself destabilizes the leader lock

postgres:16postgres:18PostgreSQL major version upgrades are not swap-the-image operations. You can’t changetoin a Deployment and expect it to work. The data directory format changes, system catalogs are incompatible, and extensions need to be rebuilt.

AutheliaWhen CNPG (CloudNativePG) upgraded from PG16 to PG18, the procedure required a fresh PVC, a full dump and restore, and a role discovery bug that nearly locked me out of thedatabase.

View the complete homelab infrastructure source on GitHub 🐙

PG_VERSIONPostgreSQL stores data in a format specific to its major version. Thefile in the data directory tells PostgreSQL which version created it:

PG_VERSION = 16When PostgreSQL 18 starts and finds, it refuses to start:

pg_upgradeis the official tool for in-place major version upgrades. It copies data files from the old format to the new format, rewriting system catalogs and tuple headers. It works well on bare-metal PostgreSQL where you have direct filesystem access.

pg_upgradeOn Kubernetes with CNPG,is not the right approach because:

-Fcpg_restoreTheflag produces a custom-format dump that’s compressed and can be restored with. The dump includes all data, schemas, roles, and extensions.

Wait — the image is still PG16. That’s intentional. CNPG creates the cluster with PG16 first, then upgrades the image to PG18 after the data is restored. This ensures the PVC and the operator agree on the initial state.

After the cluster is running with PG16, update the image to PG18:

CNPG detects the image change, creates a new pod with PG18, and the old PG16 pod is terminated. The data directory is still PG16 format, so the new pod fails to start — which is expected.

Now restore the dump into the fresh PG18 data directory:

pg_restore--clean --if-existshandles the format conversion — it reads PG16-format data and writes it in PG18 format. Theflags drop existing objects before restoring, ensuring a clean state.

Update the Authelia deployment to use the new Postgres 18 connection string (same host, same port, same database — the connection string doesn’t change):

pg_restoreDuring the restore,reported:

The dump file was at a different path than expected. The real problem: the restore was running from a pod that had a different filesystem layout than the dump pod.

After fixing the path, the restore completed but Authelia failed to start:

oc_dwpg_dumpTherole was a leftover from the original CNPG cluster initialization — CNPG creates a default operator role that isn’t visible inoutput because it’s a replication role, not a regular database role.

The fix: create the missing role before restoring:

oc_dwstreaming_replicapg_dumpThe lesson: CNPG creates internal roles (,) that aren’t included inoutput. After a dump-and-restore migration, these roles must be recreated manually.

you need a fresh PVC.pg_restoreThe critical step that most tutorials skip:You can’t restore a PG16 dump into a PG16 data directory and then expect PG18 to read it. The data directory must be empty — PG18 creates its own data directory format on first start, andpopulates it.

In CNPG, this means:

The PVC deletion is the scary part. If the dump is corrupted or incomplete, the data is gone. The verification before deletion:

PostgreSQL major version upgrades on Kubernetes are the same challenge as Azure Database for PostgreSQL Flexible Server upgrades: Azure handles the in-place upgrade automatically, but the same data directory format incompatibility exists. The difference is that Azure abstracts the dump-and-restore behind a API call, while Kubernetes requires you to do it manually. The underlying PostgreSQL constraint is identical: major versions are not backward-compatible at the storage layer.

The Linux Command Line*pg_dumppg_restorekubectl cpis worth having on the shelf for exactly this kind of migration -,,, and a dozen other shell tools chained together under time pressure go a lot smoother when the shell itself isn’t also something you’re learning in the moment.

Why You Can’t Swap the Image

The Migration Procedure

The Role Discovery Bug

The Fresh PVC Requirement

What I’d Change

cat/var/lib/postgresql/data/PG_VERSION# → 16

FATAL: data directory has wrong ownershipHINT: The data directory was initialized by PostgreSQL 16.Upgrade by running pg_upgrade.

# Exec into the CNPG podkubectlexec-ndatabase-itpostgres-authelia-0--bash# Dump the databasepg_dump-Upostgres-Fcauthelia>/tmp/authelia.dump# Copy the dump to a temporary locationkubectlcpdatabase/postgres-authelia-0:/tmp/authelia.dump./authelia.dump

# kubernetes/system/postgres/cluster.ymlapiVersion:postgresql.cnpg.io/v1kind:Clustermetadata:name:postgres-authelianamespace:databasespec:imageName:ghcr.io/cloudnative-pg/postgresql:16.4# temporary — will be updatedinstances:1storage:size:2GistorageClass:local-path

spec:imageName:ghcr.io/cloudnative-pg/postgresql:18.4

# Port-forward to the CNPG servicekubectlport-forward-ndatabasesvc/postgres-authelia5432:5432&# Restore the dumppg_restore-Upostgres-dauthelia--clean--if-exists./authelia.dump

# Verify Authelia connects to the new databasekubectl logs -n apps -l app=authelia --tail=20# → "Successfully connected to PostgreSQL 18.4"

pg_restore: error: could not open input file "/tmp/authelia.dump": No such file or directory

FATAL: role "oc_dw" does not exist

CREATEROLEoc_dwWITHREPLICATIONLOGIN;

# Verify dump is completepg_restore--list./authelia.dump|wc-l# → Should show hundreds of objects (tables, sequences, functions)# Verify dump integritypg_restore--verbose--no-owner--no-privileges--dry-run./authelia.dump# → Should complete without errors

1. CNPG manages the data directory through its operator — manual modifications are reverted
2. The PVC is bound to the cluster definition — changing the PostgreSQL version in the CRD doesn’t automatically upgrade the data
3. CNPG’s recommended migration path is dump-and-restore, not in-place upgrade

1. Delete the existing PVC (data loss — you need the dump)
2. Let CNPG create a new PVC with the PG18 image
3. Restore the dump into the fresh cluster

1. Use CNPG’s Backup/Restore instead of manual dump.pg_basebackupoc_dwCNPG supportsand WAL archiving to S3. A CNPG backup includes the operator roles and can be restored directly without thegotcha. The manual dump approach was chosen because the existing cluster wasn’t configured for CNPG backups at the time of migration.

2. Test the migration on a non-production cluster first.oc_dwTherole discovery happened during the Authelia migration — the only SSO for every service. If the restore had failed, every service would be unreachable.

Step 1: Dump from PG16

Step 2: Create Fresh PG18 Cluster

Step 3: Restore into PG18

Step 4: Update Authelia

the operating modelRenovate is the dependency update bot that keeps my k3s cluster current, and it’s the “auto-update” half ofthat decides what gets merged automatically vs. what waits for a human. It runs as a Kubernetes CronJob every 2 hours, checks for new container images, Helm chart versions, and Terraform provider releases, and opens PRs on GitHub.

successfulJobsHistoryLimitFor two weeks, it was failing silently on every run. The CronJob reported “Completed.” The GitHub commits showed no new PRs. The logs showed nothing — because Renovate exits cleanly on config validation errors, and the CronJob’skept only the last successful run.

View the complete homelab infrastructure source on GitHub 🐙

renovate.jsonThe first failure was a config error. Thefile referenced an invalid preset:

:enableHelpfulPre-commitis not a real Renovate preset. Renovate’s config validation caught the error and exited with a zero exit code — because config validation errors are treated as “nothing to do,” not “something broke.”

The CronJob’s exit code was 0. Kubernetes considered the job successful. No alert fired. Renovate did nothing on every run.

config:baseconfig:recommendedThe fix: remove the invalid preset and replace the deprecatedwith.

After fixing the config, Renovate started running — and immediately OOM’d.

The container limit was 512Mi. Renovate’s Node.js runtime, plus the GitHub API client, plus the Terraform provider registry client, plus the container image metadata parser, consumed more than 512Mi during a full scan.

OOMKilledfailedJobsHistoryLimit: 3The symptom: the Pod restarted withstatus. Kubernetes restarted it. The next run OOM’d again. After 3 failures, the CronJob marked the job as failed — butmeant only the last 3 failures were visible, and they were being garbage-collected faster than I checked.

NODE_OPTIONSThe fix: raise the memory limit to 1Gi and setto limit the V8 heap:

NODE_OPTIONSThelimit of 896MB (leaving ~100MB for native code and runtime overhead) prevents V8 from consuming the entire container memory during garbage collection cycles. Without it, V8 allocates memory up to the container limit and gets OOMKilled before the next GC cycle.

The third failure was the most subtle. Renovate was consuming 6GB of memory during Terraform provider update checks. The reason: Renovate downloads Terraform provider binaries to verify version compatibility.

routerosproxmoxcloudflaregarageFor each Terraform provider in the repo (,,,), Renovate:

Step 2 is the memory problem. Terraform provider binaries are 50-200MB compressed. With 4 providers, plus their dependencies, Renovate downloads ~500MB of provider binaries during a full scan. The decompressed binaries and their metadata consume 2-3x that in memory.

The fix was two-part:

concurrentRequestLimit: 2With, Renovate downloads at most 2 provider binaries simultaneously, reducing peak memory usage from ~6GB to ~3GB. The scan takes longer, but it completes within the 6Gi limit.

The common thread across all three failures: Renovate exited cleanly. The CronJob reported success. No alert fired. The only evidence of failure was the absence of new PRs on GitHub.

This is the silent failure pattern:

The fix: add a post-run check that verifies Renovate actually did something:

restartPolicy: NeverbackoffLimit: 2Thewithmeans the CronJob retries twice on failure, then gives up. Combined with the memory limit and heap size cap, Renovate completes its scan within resource bounds.

Synced/HealthySilent failures in CronJobs are the most dangerous failure mode in a Kubernetes cluster — the same “reports success but did nothing real” pattern as ArgoCD showingon an Application that never actually applied. The job “succeeds,” the operator doesn’t check, and the dependency update pipeline silently stops working. Weeks pass without updates, security patches don’t apply, and nobody notices until a vulnerability is disclosed for a package that Renovate would have updated.

The fix isn’t just more memory — it’s observability on the pipeline itself. Every CronJob that performs a critical function should have a post-run verification that confirms it actually did its job, not just that it exited cleanly.

CronJob observability is the same problem in Azure DevOps: a pipeline that succeeds but produces no artifact is indistinguishable from a pipeline that didn’t run. Azure Monitor’s Pipeline Analytics tracks success rate, not “did the pipeline produce meaningful output.” The fix in both cases is the same: add a verification step that checks for the expected outcome (PRs opened, artifacts published, deployments completed) and alerts on absence.

The Phoenix Project*tells the same story from the human side of this exact failure mode - the automated system that everyone assumes is working, until someone finally checks.

Failure 1: Invalid Preset

Failure 2: OOM at 512Mi

Failure 3: Terraform Provider Downloads

The Silent Failure Pattern

The Current Configuration

The Lesson

{"extends": ["config:base",":enableHelpfulPre-commit"]}

{"extends": ["config:recommended"]}

kubectlgetjobs-napps-lapp=renovate# NAME              COMPLETIONS   DURATION   AGE# renovate-28197    0/1           ...        2m# renovate-28196    0/1           ...        2h# renovate-28195    0/1           ...        4h

env:-name:NODE_OPTIONSvalue:"--max-old-space-size=896"resources:limits:memory:6Girequests:memory:1Gi

{"terraform": {"concurrentRequestLimit":2}}

# Post-run check: verify PRs were opened-name:Verify Renovate activityscript:|PR_COUNT=$(gh pr list --repo dwoitzik/homelab-infrastructure \--author "app/renovate" --state open --json number --jq 'length')if [ "$PR_COUNT" -eq 0 ] && [ "$(date +%u)" -le 5 ]; thenecho "WARNING: No open Renovate PRs on a weekday"# Alert via Discord webhookfi

# kubernetes/apps/renovate/renovate.ymlapiVersion:batch/v1kind:CronJobmetadata:name:renovatenamespace:appsspec:schedule:"0 */2 * * *"# every 2 hoursjobTemplate:spec:template:spec:containers:-name:renovateimage:ghcr.io/renovatebot/renovate:43.245.0env:-name:RENOVATE_TOKENvalueFrom:secretKeyRef:name:renovate-tokenkey:github-pat-name:NODE_OPTIONSvalue:"--max-old-space-size=896"resources:limits:memory:6Gicpu:2000mrequests:memory:1Gicpu:100mrestartPolicy:NeverbackoffLimit:2

1. Queries the Terraform Registry API for available versions
2. Downloads the provider binary for the current and latest versions
3. Compares binary compatibility (provider schema, API version)
4. Opens a PR if a newer version is available

1. Raise the memory limit to 6Gi— enough for the full scan including provider downloads
2. Limit concurrent requeststo the Terraform Registry:

1. The application fails internally but exits with code 0
2. Kubernetes considers the job successful
3. successfulJobsHistoryLimitThepreserves the “successful” job
4. Nobody checks because the job “succeeded”

Longhorn is designed for multi-node Kubernetes clusters. It replicates PersistentVolume data across nodes, so if one node dies, the data survives on the other two. It’s a great distributed storage system.

I run three k3s nodes on a single physical host. All three nodes share the same NVMe. When Longhorn replicates a PVC across three nodes, all three replicas live on the same physical disk. There is no distribution. There is no redundancy. And when a node failed, Longhorn’s own high-availability logic created a problem that wouldn’t exist without it.

View the complete homelab infrastructure source on GitHub 🐙

When a k3s node becomes unreachable (OomKill, kubelet crash, network blip), Longhorn marks its replicas as “rebuilding” and tries to attach the volume to a healthy node. But Longhorn’s RWO (ReadWriteOnce) volumes can only be attached to one node at a time.

The sequence:

The pod that needs the volume (Postgres, for example) can’t start on either node because Longhorn refuses to mount a volume that’s in a Multi-Attach state. The fix requires manually detaching the volume from both nodes and letting Longhorn re-attach it cleanly — a manual step during an outage, exactly when automation should be working.

On a multi-host cluster, this is a real HA scenario — the volume legitimately needs to failover to a different physical disk. On a single-host cluster, the “failover” is to the same NVMe, and the Multi-Attach error is pure overhead.

NFS (Network File System) doesn’t have a Multi-Attach problem because it’s not a block storage system. NFS exports a directory over the network. Any number of clients can mount it simultaneously. There’s no “attachment” state to conflict.

ct-srv-nfs-01The NFS server runs on a dedicated LXC () with a ZFS-backed dataset:

The k3s NFS provisioner creates PVCs on this NFS server:

When k3s-12 goes down, the pods reschedule to k3s-11 or k3s-13, mount the same NFS export, and continue where they left off. No Multi-Attach error, no manual detachment, no volume attachment state machine. NFS handles concurrent mounts natively.

NFS gives up replication and node-failure tolerance. If the NFS server dies, every PVC it serves is unavailable. On a single physical host, this is the same risk as Longhorn — both storage systems live on the same NVMe. Longhorn’s replication doesn’t help when the underlying disk is the single point of failure.

After migrating from Longhorn to NFS, I defined the rule:

embedded databases go on local-path (file-locking), everything else goes on NFS (survivability).The principle:PostgreSQL doesn’t fit either category because CNPG manages its own PV independently.

Moving PVCs from Longhorn to NFS required:

Step 5 is the nerve-wracking part — deleting a PVC that contains production data. The verification before deletion:

After the migration, Longhorn was uninstalled. The cluster went from three storage replicas on one disk to a single NFS server on the same disk — simpler, more predictable, and without the Multi-Attach false alarm.

Distributed storage on a single host is the same anti-pattern as Azure Zone-Redundant Storage across availability zones that share the same power source. If the underlying infrastructure isn’t actually distributed, the replication layer adds complexity without adding resilience. The fix in both cases: match the storage topology to the actual infrastructure topology. Single host = single NFS server. Multi-host = distributed storage.

Designing Data-Intensive Applications*has the clearest explanation I’ve read of why replication only buys you resilience when the replicas are actually independent - which is the whole lesson of this migration in one sentence.

The Multi-Attach Error

The NFS Alternative

The Trade-offs

The Storage Decision Matrix

The Migration

1. k3s-12 becomes unreachable (kubelet lost lease)
2. Longhorn detaches the PVC from k3s-12
3. Longhorn tries to attach the PVC to k3s-13
4. k3s-12 recovers and comes back online
5. Longhorn sees both nodes claiming the volume
6. Multi-Attach error: the volume is “attached” to two nodes simultaneously

1. Deploy the NFS server LXC with the correct ZFS dataset
2. nfs-subdir-external-provisionerInstall theHelm chart
3. kubectl cpCopy data from Longhorn volumes to NFS (usingor a temporary pod)
4. storageClassNamelonghornnfs-clientUpdate each application’sfromto
5. Delete the old Longhorn PVC (data is already on NFS)
6. Remove Longhorn from the cluster

# terraform/stacks/proxmox/lxc.tfresource"proxmox_virtual_machine""ct_srv_nfs_01"{vm_id=220name="ct-srv-nfs-01"memory{ dedicated=2048}# ...}

# kubernetes/system/nfs-provisioner/application.ymlapiVersion:argoproj.io/v1alpha1kind:Applicationspec:source:helm:values:nfs:server:10.0.20.100path:/archivestorageClass:name:nfs-clientdefaultClass:true

# Verify data exists on NFSkubectlexec-ittemp-pod--ls-la/mnt/nfs/authelia-data/# → Confirm database files, config, etc.# Verify application starts with NFS PVCkubectldeletepodauthelia-xxxxx# force reschedule# → Pod starts, mounts NFS, passes health checks

What NFS Gives Up

What NFS Gives Back

• No Multi-Attach errors.Pods reschedule freely without volume attachment conflicts.
• Simpler failure modes.NFS is either up or down. No “rebuilding,” “degraded,” or “reverted” states.
• ZFS snapshots for backup.PBS backs up the NFS server’s ZFS dataset, giving point-in-time recovery without Longhorn’s snapshot overhead.
• Lower resource usage.Longhorn runs a per-node process (manager + driver) consuming ~200MB RAM per node. NFS is a single process on one LXC.

A homelab that requires daily attention isn’t a homelab — it’s a job. The goal: a cluster that runs for weeks without intervention, alerts when something breaks, self-heals what it can, and waits for a human only when a human is actually needed.

This is the operating model that makes that work. Not aspirational — documented from what actually runs in production, including the deliberate gaps and the non-goals that keep the system honest.

View the complete homelab infrastructure source on GitHub 🐙

Renovate runs every 2 hours via CronJob, tracking container images, Helm charts, and Terraform providers. But not everything gets the same treatment:

selfHealStateless workloads that can be rolled back by ArgoCD’sif they break:

syncPolicy.automated.selfHeal: trueRenovate opens a PR, CI runs, and after 3 days of green the PR auto-merges. If the new version breaks, ArgoCD’sreverts to the previous version on the next sync.

Stateful and critical services where a bad upgrade can cause data loss or auth outage:

Every update — patch, minor, or digest — gets a PR that requires manual review. The failure mode for these isn’t “pod restarts” — it’s “database schema mismatch” or “OIDC keys rotated.”

Every major version bump, any package, regardless of tier. Major versions break APIs, change configuration formats, and introduce migration steps that no automated tool can reliably handle.

atlantis applyEvery Terraform change goes through Atlantis: PR → plan →comment → apply. No exceptions. Terraform changes physical and VM state on a single-point-of-failure host — that always gets a human in the loop.

Prometheus + Alertmanager route to Discord via webhook. ArgoCD’s notifications-controller sends app-state events through a separate path.

severity: criticalEverything with:

Only explicitly named warnings that matter at steady state:

Routine warnings that would make Discord noisy without being actionable:

an alert nobody acts on trains people to ignore Discord.The principle:Every alert must have a corresponding action in the runbook. If there’s no action, there’s no alert.

syncPolicy.automated.selfHeal: truekubectlAll 41 live Applications have. A merged manifest change deploys itself. Manualdrift on anything ArgoCD tracks gets reverted automatically within minutes.

This is the primary self-healing mechanism. If someone manually patches a Deployment (during debugging, for example), ArgoCD reverts it on the next sync cycle. The cluster always converges to the git state.

restart: unless-stoppedMedia stack, Minecraft, AdGuard, Unbound — all run within Docker Compose. Crashes and host reboots self-recover the process. The data recovery is handled by NFS/ZFS, not Docker.

Kyverno runs in Audit mode — it reports policy violations in PolicyReports but does not block or auto-remediate. This is deliberate:

require-resource-limitsdisallow-latest-tagdisallow-privileged-containersThe policies exist (,,) but they’re watching, not enforcing. When I’m confident they won’t cause false positives, they’ll flip to Enforce.

atlantis applyAlways through Atlantis. A human comments. Never auto-applied, regardless of what changed. This is the one rule with no exceptions.

Authelia, Vault, CNPG, Garage — reviewed before merge. The failure mode is data loss or auth outage, not “reroll the pod.”

Proxmox VM/CT snapshot or Velero PV snapshot, taken manually before applying anything that touches running state. The judgment of “is this change state-affecting” doesn’t belong to a script.

Running, tested successfully at least once per namespace. But not yet exercised across every service. A backup that’s only partly restore-tested is closer to hope than guarantee.

DISASTER-RECOVERY.mdDocumented in. Not automated. The target is fast, well-documented recovery — not zero-touch failover, because true HA isn’t achievable on one physical host.

notThese are explicitlytargets:

This document is the contract between the infrastructure and the operator. Everything not listed here is assumed to be working. If it’s not working and it’s not in this document, it’s a gap — not an expected manual task.

automate what’s safe, alert what’s important, and leave everything else to a human with context.The operating model is a living document. As the cluster evolves (Cilium CNI, R2 offsite backup, Kyverno enforcement), the categories shift. But the principle stays:

applyThis operating model maps directly to enterprise SRE practices: SLO-based alerting replaces threshold alerting, runbooks define the human response for each alert type, and change management gates (like Atlantis’srequirement) prevent unreviewed infrastructure changes. The difference is that enterprise environments have teams; a homelab has one person who needs to sleep through the night without Discord notifications.

The Phoenix Project*is the book that most shaped this document - it’s fiction, but the underlying argument (know exactly what’s automated, what’s gated, and why) is the same one this operating model tries to make explicit instead of leaving implicit.

What Auto-Updates

What Alerts (and Where)

What Self-Heals

What Stays Manual (On Purpose)

The Non-Goals

The Contract

Auto-Merge (Patch/Minor/Digest, 3-Day Soak)

PR-Only, Always (Manual Review Required)

PR-Only, Always (Major Version Bumps)

Terraform: Never Auto-Applied

Critical Alerts (Always Discord)

Warning Alerts (Selective)

Deliberately NOT Alerted

ArgoCD: Automatic Sync + Self-Heal

Docker Containers: restart: unless-stopped

Kyverno: Audit-Only (Deliberately)

Terraform Apply

Stateful Service Bumps

Snapshots Before State-Affecting Changes

Velero Restore Testing

Full Cluster Rebuild

• Dashboard widgets (Homepage, Uptime Kuma display)
• Development tools (pre-commit hooks, linters)
• Non-critical utilities (SearXNG, Mealie, MySpeed)

• Databases:CNPG Postgres, Redis/Valkey
• Auth:AutheliaVault,
• Storage:Garage S3, Velero
• Apps with data migrations:Nextcloud, Paperless, Gitea, Immich

• Proxmox host temperature
• k3s control-plane down
• Storage >93% full
• Postgres replication broken
• Certificate expiry soon
• Velero backup failed

• KubePodCrashLooping— a pod is crash-looping
• KubeJobFailed— a CronJob failed (covers Renovate itself)
• KubeNodeNotReady— a node is unreachable
• VeleroBackupPartialFailure— backup completed with warnings

• KubeMemoryQuotaOvercommit— LXC soft ceilings, not real pressure
• KubeCPUThrottling— normal behavior for bursty workloads
• KubePodNotReadyduring rolling updates — temporary, self-resolving

• A bad policy in Enforce mode silently blocks legitimate deploys
• On a single-operator homelab, there’s nobody to notice the block quickly
• Audit mode provides the visibility without the blast radius

• Zero-downtime HA.Not achievable with one physical host. Not attempted.
• Fully unattended major-version upgrades.Stateful service upgrades have repeatedly needed human judgment mid-migration (Postgres mount-point gotcha, capability-drop regression). Automating past that trades a 10-minute manual step for a much longer unattended-failure cleanup.
• Alerting on everything.Deliberately tuned to steady-state-relevant signals only.

Basierend auf einem realen Betrieb: der Übergang eines Acmebot-Deploys zu einer vollständig netzwerkisolierten Zero-Trust-Architektur, mitgebaut für ISO 27001 und NIS2-Anforderungen.

Azure AcmebotLet’s-Encrypt-Zertifikate mit Azure Key Vault zu automatisieren ist ein gelöstes Problem. Tools wiemachen den Deploy lächerlich einfach.

ErreichbarkeitISO 27001, KRITIS oder NIS2Das Problem: Der Standard-Deploy ist aufoptimiert, nicht auf Isolation. Drei Komponenten öffnen eine öffentliche Angriffsfläche - und sobald du ein Audit gegenfährst, klickst du dir genau damit die Findings zusammen, egal wie stark deine Auth am Ende ist.

Default-Deny auf Netzwerk-Ebene, nicht nur auf Identity-Ebene.Für jede ernsthafte Compliance-Liste muss dieses Modell komplett umgedreht werden. Das Zielprinzip:

Das öffentliche Internet-Routing ersetzen wir durch das interne Azure-Backbone, über drei Stellschrauben:

Microsoft.Web/serverFarms/27Die Function App braucht ein eigenes Subnetz mit Delegation zu. Ein-Präfix reicht für diese Workload und verschwendet keine Adressen.

Jeder Endpoint braucht seinen eigenen Private Endpoint samt DNS-Zone-Verankerung. Das Pattern wiederholt sich dreimal - hier der Storage Account exemplarisch:

aktivDer wichtigste Schritt ist der destruktive: Nachdem alle Private Endpoints stehen, wird der öffentliche Zugriffabgeschaltet - nicht nur per Firewall-Rule kaschiert, sondern im Terraform-Code verboten. Damit ist die Öffnung weg, nicht bloß versteckt:

Für den Key Vault gilt dasselbe, plus ein Härtungs-Panel, das öffentliche Zugriffe per Default ausschließt:

Damit die Function App die privaten IPs auch findet, braucht es die drei Zonen und ihre Verkettung ans Hub-VNet:

privatelink.blob.core.windows.netprivatelink.vaultcore.azure.netZonen, die du brauchst:,und fürs App-Subnetze die Function-App-DNS-Platte.

kein öffentlicher Endpoint mehr im LandschaftsbildabwesendnmapNach dem Umbau existiert für die Acmebot-Platte. Ein Scanner (z.B.oder ein Compliance-Scan wie bei einer ISO-27001-Begutachtung) findet schlicht nichts, das er attackieren könnte. Nicht „abgesichert”, sondern.

Zusammen mit den anderen NIS2-Säulen - Zero-Trust-Netzwerk, durchgängige Terraform-IaC (kein Click-Ops), Secrets via Key Vault statt in Code - erfüllst du genau die Netzwerk-Kontrollen, die in der NIS2-Artikel-21-Rechnung und in ISO-27001-Annex-A gefragt sind:

Verwandt:„NIS2 Artikel 21 in Azure implementieren”Wie du die vollständigen NIS2-Artikel-21-Pflichten mit Terraform baust, steht bereits im Blog - der Praxisteil ist innachzulesen.

Die drei offenen Flächen im Standard-Deploy

Zielarchitektur

Schritt 1: VNet Integration

Schritt 2: Private Endpoints

Schritt 3: Öffentliche Endpoints abschalten

Schritt 4: Private DNS auf zusammenwachsen lassen

Warum das fürs Audit zählt

🇬🇧 English:Hardening Azure Acmebot for ISO 27001 & NIS2 Compliance →Die englische Fassung dieses Artikels:.

1. Storage Account:Die Function App braucht einen Storage Account für Zustand und WebJobs. Default: nimmt Verkehr aus allen Netzwerken an.
2. Key Vault:Dein Zertifikatsspeicher hängt per Default am öffentlichen Endpoint.
3. Function App:Das Acmebot-Dashboard und der ACME-Webhook sind ohne Netzwerk-Restriktion öffentlich erreichbar.

• VNet Integration:Die Function App wird in ein dediziertes, delegiertes Subnetz injiziert. Sämtlicher ausgehender Verkehr startet in deinem Virtual Network.
• Azure Private Link:Storage Account und Key Vault bekommen private IPs in deinem VNet über Private Endpoints. Ihre öffentlichen Endpoints werden komplett abgeschaltet.
• Private DNS Zones:*.blob.core.windows.net*.vaultcore.azure.netInterne Auflösung sorgt dafür, dass die Function Appundüber private IPs auflöst - nicht über öffentliche.

• Privates Netz statt öffentlicher Endpoints → A.13/Netzwerksegmentierung
• IaC statt manueller Änderungen → A.8/Configuration Management
• Secrets im Vault, nie im Repo → A.10/Kryptographie

resource"azurerm_subnet""acmebot_integration"{name="snet-acmebot-integration"resource_group_name=azurerm_resource_group.app.namevirtual_network_name=azurerm_virtual_network.core.nameaddress_prefixes=["10.0.4.0/27"]delegation{name="acmebot-fn-delegation"service_delegation{name="Microsoft.Web/serverFarms"actions=["Microsoft.Network/virtualNetworks/subnets/action"]}}}

resource"azurerm_private_endpoint""sa_blob"{name="pe-sa-blob"location=azurerm_resource_group.app.locationresource_group_name=azurerm_resource_group.app.namesubnet_id=azurerm_subnet.acmebot_integration.idprivate_service_connection{name="psc-sa-blob"private_connection_resource_id=azurerm_storage_account.acmebot.idis_manual_connection=falsesubresource_names=["blob"]}private_dns_zone_group{name="pdns-blob"private_dns_zone_ids=[azurerm_private_dns_zone.blob.id]}}

resource"azurerm_storage_account""acmebot"{...public_network_access_enabled=falsemin_tls_version="TLS1_2"}

resource"azurerm_key_vault""acmebot"{...public_network_access_enabled=falseenable_rbac_authorization=true}

resource"azurerm_private_dns_zone""blob"{name="privatelink.blob.core.windows.net"resource_group_name=azurerm_resource_group.app.name}resource"azurerm_private_dns_zone_virtual_network_link""blob"{name="link-hub"resource_group_name=azurerm_resource_group.app.nameprivate_dns_zone_name=azurerm_private_dns_zone.blob.namevirtual_network_id=azurerm_virtual_network.hub.id}

Basierend auf einem real betriebenen Homelab mit ~80 Diensten (Kubernetes, Proxmox, eigene Infra) und den echten Memory-Zahlen, die ein solcher Betrieb über Wochen produziert.

„Für was kaufe ich das Ding eigentlich?”Wenn du selbstständig bist und überlegst, einen eigenen Server zu kaufen (statt Cloud), dann stehst du vor genau einer Frage, die kaum ein Review beantwortet:

Die Marketing-Maschine sagt „Kerne, Kerne, GHz”. Die Betriebsrealität sagt etwas anderes. Ich zeige dir die Rechnung, die in meinem eigenen Betrieb über Monate entstanden ist.

nichtRAMMemory-ProblemDie häufigsten Selbstständigen-Setups (Dokumenten-Papierkram, Nextcloud, ein kleines CRM, Monitoring) landenbeim CPU-Bottleneck — sie landen beim. Mein eigener Betrieb: ein K3s-Cluster auf einem Node mit 4 GiB RAM war bei 99% belegt, noch bevor die CPU 10% erreicht hatte. Der API-Server des Clusters (etcd dahinter) bekam p99-Latenzen von über 8 Sekunden. Kein CPU-Problem — ein.

Die Daumenregel für Selbstständige:

1,1 GiBWarum so viel? Ein einziger Node des kube-prometheus-Stacks (Monitoring!) braucht auf einem Node mit knappem RAMallein für Prometheus. Dazu kommen Immich und weitere Homelab-Apps mit insgesamt 1,3 GiB. Das ist kein Overengineering, das ist die Baseline moderner Stack-Betriebe.

Gaming/PeakDauerlast6 W TDP und deutlich weniger Multi-Core-LeistungDer CPU-Kauffehler bei Selbstständigen ist: Reviewer kaufen für, du kaufst für. Ein N100 ist zwar günstig, aber unter Dauerlast (Backup-Jobs, nächtliches Transcoding, Crawler) bietet er nurals ein echter Desktop-Chip. Die Antwort ist nicht „mehr GHz”, sondern „genug Kerne für die Job-Lücke”:

6-Kern-Prozessor mit integrierter GrafikEmpfehlung für 2026 ohne Blödsinn: Ein(iGPU für Transcoding) und 16–32 GiB RAM. Die iGPU ist der unterschätzte Kaufpunkt: Intel Arc/Xe oder AMD iGPU transkodiert 4K ohne extra GPU-Karte — das ist der Unterschied zwischen „Media-Server mit Shopping-Budget” und „Media-Server, der Rechenleistung totbrennt”.

>3 VMs mit 24h-LastDer reine Cloud-Rechenknoten lohnt sich erst bei— darunter zahlst du dich dumm und dusselig. Mein eigener Betrieb: ~10 VMs würden in der Cloud €300–500+/Monat kosten; mein Server läuft mit Stromkosten unter €20.

6 Kerne + 16–32 GiB RAM + iGPU + NVMeRAM ist der einzige Bottleneck, der dir real den Stack wegstirbtFür Selbstständige 2026:ist der Sweet Spot, egal ob du €700 oder €1.400 ausgibst.— mehr CPU kauft dir nichts, wenn der Node bei 99% Memory steht und der API-Server 8 Sekunden p99 fährt. Kauf nach dieser Rechnung, nicht nach GHz.

Die Wahrheit über RAM — der einzige Bottleneck, der zählt

CPU: der Wert, der nicht abstürzt, wenn du Meter machst

Proxmox vs. Cloud: die echte Selbstständigen-Rechnung

Die 3 Zukäufe, die wirklich Geld sparen (statt teuer sind)

Fazit (Kurzversion, ehrlich)

• 8 GiB4–5 Dienste (Nextcloud, Paperless, Search, ein Dashboard):reicht knapp.
• 16 GiB10+ Dienste oder Container-Virtualisierung (Proxmox + VMs):ist der Einstieg, 32 GiB die Komfortzone.
• ab 32 GiB.Du willst LLMs (homelab-scale) oder Media-Transcoding stabil laufen lassen:

• Tägliche Backups (rclone/restic) + Monitoring: meist 2 Kerne busy, Rest warte.
• Weekly full-transcode (Media-Server): 4–6 Kerne unter Vollast, 15 min.
• Ein LLM-Inference worker (optional): 8+ Kerne, Dauerlast.

1. RAM-Erweiterung statt neuer Server:Wenn dein Server bei 80% RAM hängt, ist RAM-Adding der kleinste Eingriff. In meinem Betrieb: Node von 4GiB → 8GiB war die RAM-Wurzel aller Ausfälle. Prüfe zuerst die RAM-Module, nicht die CPU.
2. NVMe statt SATA-SSD als Boot-Medium:Der Unterschied bei Kubernetes/Docker-IO ist spürbar. NVMe-NVMe macht selber Backups in Minuten statt in Stunden.
3. Proxmox statt Docker-only:Wenn du mehr als 3 Dienste hast, wird Containers-Mixing zum Wartungs-Albtraum. Proxmox (LXC/VM) gibt dir Snapshots, Migration, Clone — das ist der Unterschied zwischen „ich kann testen” und „ich fürchte mich vor dem Test”.

Peers und Access nicht mehr von Hand, sondern als Terraform-Code, versioniert und per Rolle verwaltet.WireGuard auf MikroTik (RouterOS 7) ist inzwischen stabil genug für den Produktivbetrieb - aber es ist immer noch die „ich füge einen Peer per Winbox hinzu”-Falle, die mich zurückgeholt hat. Die Route aus meinem Betrieb:

Statt jedes Gerät einzeln zu pflegen, definiere ich drei Rollen - die decken 95% aller Homelab-Zugriffe ab:

admin@mikrotik:/interface/wireguard/peers addDas Modul macht dasselbe, was du per CLI beimachen würdest - nur eben reproduzierbar:

RouterOSDer Peer selbst darf nichts — die-Firewall erzwingt die Rollen:

Default-Drop am Ende der Kette, jede Rolle bekommt explizite Allow-Regeln davor.Das Muster, das hier funktioniert:Wer in der Kette weiter unten liegt, ist raus - egal wie viele Peers du noch hinzufügst.

homelab-infrastructure/mikrotik/Ich verwalte die RouterOS-Konfig nicht mehr über Winbox — die liegt als Terraform-HCL in, und ArgoCD/Atlantis (dein GitOps-Setup) spielt sie aus. Das gibt dir eine saubere Audit-Trail: Welcher Peer wurde wann und von wem eingefügt, steht im Git-Commit - und nicht im „ich hatte mal ein Skript”-Ordner.

Verdrahtung:Zero-Trust MikroTik Firewall mit TerraformWireGuard-Site-to-Site mit Role-Based AccessDer perfekte Begleitartikel dazu ist— dort ist die komplette Access-Kette mit Default-Drop inklusive NAT-Details aufgebaut. Und diezeigt, wie du das ganze als modulare Terraform-Einheit deployst.

Die Rollen-Kette (ohne sie gerätst du in Access-Hölle)

Terraform-Modul für den Peer

WireGuard, das dir die Doku nicht sagt

Die Access-Kette: Feuerwehr per Firewall, nicht per Peer

Der ArgoCD-Gedanke: Dein Router als Code

🇬🇧 English:Automating MikroTik WireGuard VPN with Role-Based Access via Terraform →Die englische Fassung dieses Artikels:.

• Admin: dein Laptop, dein Handy mit externem Zugriff - darf alles im Netz (10.0.0.0/16).
• App: die LXC-Container, die VPN-Gegenstellen sind (z.B. der Tunnel zum Proxmox-Standort).
• IoT: Smart-Home-Geräte mit starker Segmentierung - dürfen nur zu ihrem eigenen Broker (10.0.40.0/24), sonst: Drop.

# Peers als Datenstruktur statt als 15 Blöcke zu kopierenvariable"wireguard_peers"{type=map(object({role=stringpublic=stringallowed=list(string)}))default={laptop={role="admin"public="wRXr..."# aus der Generierung, nie im Repo hardcodenallowed=["10.0.0.0/16"]}}}

resource"routeros_interface_wireguard_peers""peer"{interface="wireguard1"public_key=var.peer.publicallowed_address=var.peer.allowed[0]comment="${var.peer.role}-${each.key}"for_each=var.wireguard_peers}

# role "iot" → nur zum MQTT-Brokerresource"routeros_ip_firewall_filter""iot_allow_broker"{chain="forward"src_address="10.0.40.0/24"dst_address="10.0.50.10"dst_port="1883"protocol="tcp"action="accept"place_before="drop-all"}

homelab-infrastructure/└── mikrotik/├── wireguard.tf        # interface + globale settings├── peers.tf            # role-basierte Peers└── firewall.tf         # Default-Drop + Rollen-Access

1. listen-portinterfaceundändern sich nicht per Peer/interface/wireguard/peers addallowed_address— die sind global. Der Fehler, den ich stundenlang gemacht habe: pro Peer eine eigene IP-Addresse auszu ziehen, wo es bei WireGuard nur diegibt, die entscheidet, welches Subnetz durch den Tunnel darf.
2. Kein NAT nötig für Site-to-Site: Der Proxmox-Standort (10.0.20.0/24) spricht das Hauptnetz direkt über die Peer-Route an. NAT wäre hier die falsche Abkürzung - sie zerstört genau die Rollen-Segmentierung, die du aufgebaut hast.

kubeconformkubectl --dry-runcapabilities.drop: [ALL]runAsNonRoot: trueallowPrivilegeEscalation: falsesecurityContextselfHeal: truelief grün.lief grün. Der PR sah exakt danach aus, was jede Kubernetes-Security-Checkliste empfiehlt:,,- über alle Container, denen einfehlte. Schema-validiert, reviewed, gemerged. Wegen ArgoCD mitwar der Merge die Ausrollung. Neun Container waren in Minuten down - zwei davon Postgres, die Paperless und Nextcloud trugen. Das ist kein degradierter Neben-Dienst, das ist ein Ausfall.

securityContextHier ist die Fehleranalyse: die zwei falschen Annahmen, die Falle, die mich beim Wiederherstellen gebissen hat, und die Lektion für jede pauschale-Passade.

chownchmodsu-execsetprivCAP_CHOWNCAP_SETUIDCAP_SETGIDdrop: [ALL]Richtig ist: Ein riesiger Anteil an Container-Images folgt demselben Muster - als root starten, das Datenverzeichnis per/an einen unprivilegierten User übergeben, dann peroderdie Rechte vor dem eigentlichen Prozess fallen lassen. Genau dieser Fall braucht selbst,,- Capabilities, dieentfernt, bevor das Entrypoint-Skript überhaupt läuft.

nichtcommand: redis-server ...securityContextDas brach gitea, authelia, headscale, mealie und beide Postgres-Instanzen (paperless + nextcloud) - alle laufen exakt dieses root-then-drop-Muster im Entrypoint. Es brach auch die Paperless- und Nextcloud-Redis - aberdie Authelia-Redis, weil die explizit mitdie eigene Entrypoint-Kette umschiffte. Gleiches Image, gleicher, anderes Ergebnis - weil der tatsächliche Codepfad ein anderer ist.

runAsNonRootvault-unsealcloudflare-ddnsAdmission-Zeit-ChecknieFalsch, in die andere Richtung.ändert nichts daran, wie der Container läuft - es ist ein, der failt, wenn das Image als root startet und nichts im Pod-Spec das überschreibt.(hashicorp/vault), die Nextcloud- und Paperless-Redis und(curlimages/curl) laufen alle als root. Die crashten nicht im Loop - die starteten:

CreateContainerConfigErrorEin sauberermit klarer Meldung - deshalb war diese Kategorie leicht zu diagnostizieren. Die Crash-Looping-Container aus Annahme 1 waren die härtere Hälfte.

kubectl get application -n argocdSyncedHealthyDer erste Instinkt bei Problemen ist ArgoCD.zeigteund. Das war veraltet - ArgoCDs Poll-Intervall hatte die App-Ansicht noch nicht aktualisiert, obwohl die neuen Pods drunter schon am Failen waren.

selfHealden alten, gehärteten Stand wieder reinNach dem Rollback-Hebel (securityContext wieder raus, alten Pod-Spec via Git restore) passierte fast das Zweite: der Re-Reconcile von ArgoCD zog beim nächsten, wenn der Fix im Repo nicht sauber committed war, während der Live-Cluster schon wieder lief. Der Fix im Repo ist die Wahrheit - nicht der Pod im Cluster.

securityContextcapabilities.drop: [ALL]fsGroupkeineEine pauschale-Passade über alle Container hinweg istHärtung, sie ist ein Zahlendreher-Wartungsrisiko. Sicher härtest du nur, wenn du für jedes Image weißt, ob es das root-then-drop-Muster im Entrypoint nutzt. Die Reihenfolge: erst pro-Image dokumentieren, welcher Codepfad läuft, dann einschränken - nie andersrum. Für Verzeichnis-Mounts, die der Container anlegen muss, bleibt stattoft nur: dem vorhandenen User explizit die Verzeichnisse mitgeben (), statt die Capabilities zu amputieren.

capabilities.drop: [ALL]Falsche Annahme 1:ist sicher, wenn der Container keine Runtime-Rechte braucht

runAsNonRoot: trueFalsche Annahme 2:ist auf jedem Container “sicher zu setzen”

Warum “Application: Synced/Healthy” log - und uns in die Falle lief

Die Wiederherstellungs-Falle: der zweite Merge

Die Lektion

🇬🇧 English:I Hardened Pod securityContext and Broke 9 Containers in Production →Die englische Fassung dieses Artikels:.

# Sah nach sicherem, empfohlenem Härtung aus:securityContext:allowPrivilegeEscalation:falsecapabilities:drop: ["ALL"]

Error: container has runAsNonRoot and image will run as root

# Der Zustand, der alles zeigte:kubectlgetpods-A-owide|grep-E"CrashLoopBackOff|CreateContainerConfigError"

Backup-Automatisierung auf Kubernetes klingt nach einer dieser Sachen, die man „irgendwann mal” macht, bis der erste Node stirbt. Dann merkst du, dass „irgendwann mal” ein Jahr zu spät ist.

Velero + einem S3-kompatiblen Backend, das du selbst betreibstDie gute Nachricht: Mit, brauchst du weder AWS noch eine externe Cloud - und die Kette ist kürzer, als du denkst. Das ist die vollständige Anleitung aus meinem Betrieb.

Garage ist ein selbstgehosteter, S3-kompatibler Object-Store, der auf Ressourcen-Effizienz ausgelegt ist. Deploy via Helm in deinem GitOps-Repo:

garage.woitzik.devWichtig beim Setup: Garage braucht ein eigenes DNS/subdomain () und eine feste Node-Verteilung - es funktioniert am besten, wenn jede Replica auf einem anderen Cluster-Node liegt.

Velero deployed man am saubersten über ArgoCD - nicht manuell:

TrugbildernDie Kette läuft seit Monaten. Der Punkt, der Backups zumacht, ist nicht der Schedule - es ist der Moment, an dem jemand die Sicherung testet. Dazu unten mehr.

Der größte Fehler, den ich selbst gemacht habe: Jeden Tag „erfolgreiche” Backups in Velero sehen, die Details nie geprüft, und den Restore erst dann versucht, als eine Datenbank weg war.

Completedwöchentlich einen echten Restore anstößtmit detail versteckten Fehlern - ein Restore aus einem solchen Backup versagt an genau der Stelle, an der du ihn brauchst. Mein Fix: Ein zweiter Velero-Schedule, der(in eine Wegwerf-Namespace), damit die Wiederherstellbarkeit getestet ist, bevor sie gefordert wird.

Fazit:Die Architektur (Garage + Velero + ArgoCD) ist der kleinere Teil. Der wichtigere Teil ist der regelmäßige Restore-Test - ohne den ist dein „ich mache Backups” nur eine Art, Daten zu lagern, die du nie zurückbekommst.

Die Architektur

Schritt 1: Garage auf k3s deployen

Schritt 2: Velero via ArgoCD

Schritt 3: Der Schedule-Schlüssel: täglich, nicht „wenn ich dran denke”

Das Backup, das keiner geprüft hat

🇬🇧 English:k3s Backup Without the Complexity: Velero + Garage S3 on Longhorn →Die englische Fassung dieses Artikels:.

+-------------------+       +-------------------+|   Longhorn-RVs    |  ===> |   Velero Pulumi   ||  (deine Daten)    |       |  + Garage S3      |+-------------------+       +-------------------+

# values-garage.yaml (gekürzt)replicas:3backends:-node:k3s-1-node:k3s-2-node:k3s-3storage:20Gi

apiVersion:argoproj.io/v1alpha1kind:Applicationmetadata:name:veleronamespace:argocdspec:destination:namespace:veleroserver:https://kubernetes.default.svcsource:repoURL:https://github.com/dwoitzik/homelab-infrastructurepath:kubernetes/velerotargetRevision:mainsyncPolicy:automated:selfHeal:true

apiVersion:velero.io/v1kind:Schedulemetadata:name:daily-backupnamespace:velerospec:schedule:"0 1 * * *"template:includedNamespaces:-nextcloud-paperless-vaultttl:720h

$ velero backup describe daily-backup-20260922...Phase: Completed           # Komfort-LügeErrors:  3               # die WahrheitWarnings: 1

spec:schedule:"0 3 * * 0"template:…hooks:resources:-name:verify-restore...

• Longhornspeichert die PersistentVolumes deines Clusters (etcd-Volumes, DB-Daten, Nextcloud-Daten, etc.).
• Garageläuft als S3-Backend direkt im Cluster - object storage ohne Cloud-Anbieter, mit Replikation über die Cluster-Nodes.
• Velerosichert die Longhorn-Volumes als Snapshots + Backups nach Garage (S3), mit täglichem Schedule.
• ArgoCDkubectlmanaged den ganzen Velero-Stack via Git - Deklaration statt manuelle-Rufe.

Hinweis: Keine Rechtsberatung. Das hier ist die öffentlich zugängliche Einordnung von BSI, Bundesrats-Drucksachen und der NIS2-Umsetzung (NIS2UmsuCG), laienhaft aufgeschlüsselt. Für deinen konkreten Fall: Anwalt.

Bin ich überhaupt von NIS2 betroffen?Wenn du einen Homelab betreibst und in Deutschland selbstständig bist, kommt irgendwann die Frage:

Wahrscheinlich nicht. Aber die Ausnahme ist genau da, wo dein Geld liegt.Die ehrliche Antwort, nachdem ich die Umsetzungsverordnung und die öffentlichen BSI-Papiere durchgearbeitet habe:

OrganisationenNIS2 richtet sich an— juristische und natürliche Personen, die einen kritischen bzw. wichtigen Dienst anbieten. Der Anwendungsbereich hängt an zwei Hebeln:

fällt in der Regel nicht in den Anwendungsbereich.Ein privater Homelab, eine kleine Agentur, ein Einzelunternehmer ohne nennenswerte Belegschaft:Punkt.

Aber drei Fälle ändern das — und genau die sind für selbstständige Tech-Betreiber relevant:

DienstLieferketten-ParagrafenWer alsRessourcen für den kritischen Sektor bereitstellt (z.B. ein Managed-Hosting-Betreiber, auf den Energie- oder Gesundheitseinrichtungen aufbauen), kann über den(§ 38 NIS2UmsuCG: Sicherheitspflichten für wesentliche Lieferanten und Cloud-Dienstleister) mit hineingezogen werden — selbst wenn HOST klein bist.

Nicht deine eigene Betriebsgröße entscheidet, sondern ob ein kritischer Kunde auf deiner Infrastruktur aufbaut.Das ist der Fall, der für dich praktisch relevant wird:

Lieferkette / ihre Dienstleister angemessen abgesichertDu wirst als Dienstleister geprüft — nicht von der Behörde, sondern im Vertrag deines Kunden.Je größer dein Kunde, desto mehr wirkt NIS2 auch vertraglich auf dich. Die Verordnung verlangt von betroffenen Organisationen, sich zu vergewissern, dass ihresind (Art. 21 Abs. 2 lit. d). Übersetzung für dich:

Nur relevant, wenn du erkennbar der Kritikalitätsdefinition unterfällst (§§ 42 ff.). Für die allermeisten Homelab-Kleinbetreiber: nein.

Falls einer der Fälle greift, sind es praktisch drei Dinge:

NIS2 ist im Kern eine Nachweis-Pflicht, keine Vorschriften-Pflicht.Punkt 3 ist der Teil, den die meisten übersehen:Du musst nicht alternativlos bestimmte Tools nutzen — du musst zeigen können, was du tust.

Die meisten Selbstständigen und Homelab-Betreiber sind sauber nachweislich abgesichert, wenn sie die Grund-Hygiene dokumentieren.Das Schöne an dieser Einordnung:Du brauchst keinen teuren Zertifizierungs-Katalog — du brauchst die drei Säulen nachweisbar:

man hat die Firewall-Rules, aber keine Doku, dass die Rules das geplant tun.Genau das ist der Teil, wo ein Homelab-Betrieb ohne Tooling schnell brüchig wird:Und genau dafür gibt es automatisierbare Terraform-/Proxmox-Module, die aus der Infrastruktur heraus einen Conformance-Nachweis erzeugen.

Möchtest du wissen, ob dein konkreter Fall in den Anwendungsbereich fällt? Zu den drei Pflichten gibt es eine kompakte Checkliste — sag Bescheid.

Wo NIS2 überhaupt ansetzt

Fall 1: Du bietest einen „wichtigen“ oder „kritischen“ Dienst an

Fall 2: Du bist Kunde eines Betroffenen

Fall 3: Du bist als „wesentliche Einrichtung“ eingestuft

Die drei Pflichten, die dann gelten

Was das für deinen Betrieb bedeutet

1. Sektor-Zugehörigkeit(Anlage 1 NIS2UmsuCG: Energie, Transport, Gesundheit, digitale Infrastruktur, u.a.)
2. GrößenkriteriumMittelstandsschwelle überschrittenoder— in der Regel(per Definition der Empfehlung 2003/361/EG: >50 Beschäftigte>10 Mio. € Jahresumsatz)

1. Risikobasierte Sicherheitsmaßnahmen(Art. 21 Abs. 2): Inventory, Access-Management (Least Privilege), incident-Handling, Backups, Verschlüsselung, Supply-Chain-Management.
2. Meldepflicht bei erheblichen Vorfällen(Art. 23): IT-Ausfall mit erheblichen Auswirkungen auf den Dienst → Meldung an die zuständige Behörde bzw. das BSI, gestaffelt nach Schwere.
3. Rechenschaftspflichtnachweisbar: Diese Maßnahmenmachen (Audit-Logs, Richtlinien, Schulungen). Wer nicht nachweisen kann, dass er Maßnahmen hat, bekommt bei einer Prüfung Probleme — selbst wenn die Maßnahmen faktisch da sind.

• Zugriff nur nach Bedarf (Least Privilege) + Protokollierung
• Backup & Recoverability (getestet, nicht nur konfiguriert)
• Incident-Plan (wer, wie, wann wird informiert)

Update:cpu.unitsthis cluster has since moved off etcd entirely — a single-server k3s setup uses the embedded SQLite datastore by default, and the multi-server HA path that would have used etcd was never turned on here. The CPU-contention problem and thefix below were real while etcd was in play, and the underlying lesson (latency-sensitive, write-heavy workloads need scheduling priority when they share a host with bursty ones like LLM inference or game servers) still holds for anyone running real multi-server etcd today. Left as a historical/general reference rather than a description of this cluster’s current architecture.

etcd is the brain of Kubernetes. Every API call, every configmap update, every pod scheduling decision goes through etcd. When etcd is slow, everything is slow. When etcd times out, the API server becomes unreachable.

On a single Proxmox host running k3s VMs alongside Ollama LLM inference, Minecraft game servers, and Docker media workloads, etcd shares CPU with everything else. Without scheduling priority, a Minecraft player join and an Ollama model load can delay etcd’s fdatasync calls enough to trigger leader-election timeouts.

cpu.units = 2048staggered boot orderingThe fix:on k3s VMs, giving them 2x the CPU scheduling priority over every other workload on the host — the same VMs that getto prevent I/O storms in the first place.

View the complete homelab infrastructure source on GitHub 🐙

cpu.unitsProxmox uses the Linux CFS (Completely Fair Scheduler) with added weight controls. Each VM gets avalue that determines its share of CPU time when multiple VMs compete for the same physical cores.

units = 1024units = 2048units = 1024The default is. When two VMs with equal units compete for a single core, they each get 50% of the CPU time. When one VM hasand another has, the first gets 2/3 and the second gets 1/3.

This is not CPU pinning (which restricts a VM to specific cores). It’s CPU weighting (which determines priority when cores are shared). On a host with 16 threads and 12+ VMs/LXCs, most cores are shared between multiple workloads.

etcd’s performance depends on write latency. Every key-value operation (lease renewal, configmap update, secret sync) requires an fdatasync to the WAL (Write-Ahead Log) on disk. etcd’s leader-election timeout is 5 seconds — if the leader can’t renew its lease within that window, controller-runtime terminates the process.

On a single NVMe shared by all VMs, etcd’s fdatasync competes with:

Under normal load, this is fine — NVMe IOPS are high enough to handle all of them. But when Ollama starts loading a 26B model (Gemma 4) and Minecraft generates terrain simultaneously, the NVMe queue depth spikes, fdatasync latency increases, and etcd’s lease renewal window shrinks.

Without CPU priority, etcd’s fdatasync also competes for CPU time with these workloads. The kernel’s CFS scheduler doesn’t know that etcd’s fdatasync is more important than Ollama’s matrix multiplication — it just sees two processes requesting CPU time and allocates it equally.

units = 20481024The k3s VMs get. Everything else stays at the default. When etcd and Ollama compete for the same CPU cycle, etcd gets 2/3 of the time and Ollama gets 1/3.

This doesn’t reduce Ollama’s throughput under normal conditions — when there’s no contention, Ollama still gets 100% of the CPU it requests. The priority only kicks in when multiple workloads compete for the same cores simultaneously.

cpu.units = 2048Before, the CNPG operator was restarting due to leader-election timeouts:

310 restarts in 21 days. Each restart’s log showed:

cpu.units = 2048--leader-renew-deadline=50After applyingand the corresponding leader-election timeout increase (), restarts dropped to zero.

The CPU priority alone didn’t fix it — the leader-election timeout increase was also necessary. But the CPU priority reduced the frequency of fdatasync delays enough that the 50-second deadline is never challenged under normal load.

cpu.unitsmemory.dedicatedmemory.floatingCPU priority and memory priority are separate in Proxmox.affects CPU scheduling;andaffect RAM allocation.

On this host, both matter:

The RAM allocation is a hard reservation — 12 GB is always available for the k3s-11 VM. The CPU priority is a soft weighting — it only matters when cores are shared. Both are necessary: without RAM priority, the k3s VM could be ballooned down to 4 GB under pressure; without CPU priority, etcd could lose the fdatasync race to Ollama.

cpu.unitsCPU scheduling priority is the same concept as Azure VM series selection: E-series VMs are memory-optimized, F-series are compute-optimized, and Dv5-series offer balanced resources. Choosing the wrong series for etcd (a latency-sensitive, write-heavy workload) produces the same performance degradation as running it withouton Proxmox. The difference is that Azure makes the choice at VM creation time, while Proxmox lets you adjust it dynamically.

Designing Data-Intensive Applications*is the best resource I know for understanding why a consensus system like etcd is so much more latency-sensitive than an ordinary stateless workload in the first place.

How Proxmox CPU Scheduling Works

The etcd Problem

The Fix

The Evidence

The RAM Priority Interaction

What I’d Change

• Ollama reading model weights from disk (14GB qwen2.5 model)
• Minecraft world writes from the DMZ game server
• NFS serving PVC data to k3s pods
• ZFS txg commits flushing dirty data

• cpu.unitsetcd needs low-latency CPU for fdatasync (solved by)
• memory.dedicated = 12284the overcommit guard writeupk3s control-plane needs guaranteed RAM for API server and scheduler (solved by, the same VM-vs-LXC memory model covered in)

# terraform/stacks/proxmox/vm.tf# k3s VMs — 2x scheduling priorityresource"proxmox_virtual_machine""vm_srv_k3s_11"{cpu{cores=4units=2048# 2x default}}# Ollama LXC — default priorityresource"proxmox_virtual_machine""ct_srv_ai_01"{cpu{cores=6units=1024# default}}# Minecraft LXC — default priorityresource"proxmox_virtual_machine""ct_dmz_games_01"{cpu{cores=2units=1024# default}}

kubectl get pods -n cnpg-system# NAME                                    RESTARTS# cnpg-cloudnative-pg-5f8b9c4d6-xk2p4   310

Leader election retry deadline exceededcontext deadline exceeded

1. Pin etcd to specific cores.CPU pinning would guarantee etcd always has CPU available, rather than just having priority. But pinning reduces overall CPU utilization — a pinned core can’t be used by other VMs even when etcd is idle. On a 16-thread host with 12+ workloads, the utilization loss isn’t worth it.

2. Monitor etcd fdatasync latency directly.etcd_disk_wal_fsync_duration_secondsPrometheus can scrape etcd’smetric. An alert on p99 > 100ms would catch CPU contention before it triggers leader-election timeouts.

the Garage S3 target Velero backs intoThe 3-2-1 backup rule is simple: 3 copies of your data, on 2 different media types, with 1 copy offsite. My homelab implements all three layers. In theory. In practice, each layer has its own failure mode that I discovered only when I needed it — starting with, which is where layer 1 lives.

View the complete homelab infrastructure source on GitHub 🐙

Velero backs up k3s namespaces (apps, vault, database, argocd) to Garage S3 — a self-hosted S3-compatible object store running inside the same cluster.

defaultVolumesToFsBackup: truewas the critical fix. Without it, Velero only captured Kubernetes manifests — not the actual PVC data. The backups “completed” for weeks with zero data.

veleroAfter the fix, Kopia sidecars run alongside each pod, copying PVC contents to the Garage S3bucket. Daily backups, 30-day retention.

The catch:Garage runs inside the cluster. If the cluster dies, both Velero and Garage are gone. Layer 1 is useful for recovering individual PVCs or namespaces within a running cluster — not for full cluster recovery.

Seagate 2TB External HDD*Proxmox Backup Server (PBS) backs up VM and LXC disk images to aconnected via USB.

PBS handles deduplication, compression, and incremental backups. The USB HDD provides local, offline backup that survives cluster failures.

The gotcha:PBS and the k3s VMs share the same physical host. A host-level failure (PSU, NVMe death) takes out both the primary data and the PBS backup. Layer 2 is protection against software failure (corruption, accidental deletion), not hardware failure.

The offsite layer was supposed to be rclone syncing from Garage S3 to Google Drive. In practice, Google’s API throttled the sync to 1.6 KiB/s:

Google Drive’s API rate limiting for server-to-server transfers (no user interaction) is aggressive. For 4 GB of data, the sync would take days. Combined with Google’s 750 GB/day upload limit for personal accounts, and the fact that the sync would need to run continuously to keep up with daily backups, the offsite layer was disabled.

The replacement (Cloudflare R2) is scaffolded but not active:

Waiting on a Cloudflare R2 account with real API credentials. R2 has no egress fees and no API rate limiting for the volume this backup produces (~4 GB/day). Activating it also fixes a second problem beyond throttling: Garage runs inside the same cluster Velero is protecting, so an in-cluster-only backup target is circular regardless of how fast it uploads.

The first time I tested a full restore, PBS hit an IP/MAC conflict:

The issue: PBS restores the VM’s network configuration exactly as it was, including the MAC address. If the original VM is still running (or its MAC is cached in the bridge), the restored VM can’t start because Proxmox detects a MAC conflict on the virtual bridge.

The fix: restore to a temporary VMID, verify the restore is complete, then shut down the original and rename the restored VM. This is a manual multi-step process — not something you want to do under time pressure during a real disaster.

The honest assessment: I have two functional backup layers, both on the same physical host. True offsite backup doesn’t exist yet. The 3-2-1 rule is aspirational, not achieved.

a backup you haven’t restored is not a backup.The most important lesson:

defaultVolumesToFsBackupVelero backups run daily, PBS runs daily, and until recently neither had been fully restored. The Velerogap existed for weeks because nobody tested a restore. The PBS IP conflict was discovered during the first full restore test.

After these discoveries, I added:

Backup verification is the same compliance requirement in Azure: ISO 27001 and NIS2 both require documented, tested restore procedures. Azure Backup reports “Completed” for VM snapshots, but the snapshot might not include the data disk if the backup policy was misconfigured. The only way to verify is to actually restore a test VM and confirm its contents — the same monthly drill I now run against Velero and PBS.

The Three Layers

The Restore Gotcha

What Actually Works

The Verification Gap

Layer 1: Velero → Garage S3 (in-cluster, daily 05:00 UTC)↓Layer 2: PBS → External HDD (local, daily 03:00 UTC)↓Layer 3: rclone → Google Drive (offsite, DISABLED)

# kubernetes/system/velero/schedule.ymlspec:schedule:"0 5 * * *"template:defaultVolumesToFsBackup:trueincludedNamespaces: ["apps","vault","database","argocd"]excludedResources: ["events","events.events.k8s.io"]ttl:720h# 30 days

# PBS backup job — runs daily at 03:00 UTC# Backs up all VMs and LXCs to /mnt/backup (USB HDD)proxmox-backup-clientbackup\--repositorylocal:/mnt/backup\--nshomelab\vm/110/pct/200/pct/210/...

Transferred:   1.6 KiB / 4.2 GiB,  0.00%Elapsed time:  2h 30mTransfer rate: 1.6 KiB/s

# kubernetes/system/velero/offsite-schedule.ymlapiVersion:velero.io/v1kind:Schedulemetadata:name:daily-offsitespec:schedule:"0 4 * * *"template:includedNamespaces: ["apps","vault","database","argocd"]storageLocation:r2-offsitettl:168h# 7 days

Error: VM 211 is running on a different node (10.0.20.11)Proxmox cannot start the VM because the MAC address is already in use

Layer 1: Velero → Garage S3

Layer 2: PBS → External HDD

Layer 3: rclone → Google Drive (DISABLED)

1. Monthly Velero restore testdatabase— restore thenamespace to a temporary namespace, verify Postgres starts and contains expected data
2. Quarterly PBS restore test— restore one VM to a temporary VMID, verify it boots and services are functional
3. Post-backup verificationvelero backup describe --details | grep "Pod Volume Backups"—after every scheduled backup

My media acquisition stack — SABnzbd for Usenet downloads, Sonarr for TV, Radarr for movies, Bazarr for subtitles, NZBHydra2 for indexer search — used to run as k3s Deployments. It worked, but three problems made it a bad fit for Kubernetes: GPU passthrough for Jellyfin transcoding, per-flow traffic isolation for indexer queries versus the actual download path, and the NFS file-locking trap for media libraries.

The solution: move the media stack out of k3s entirely. Jellyfin runs in its own GPU-passthrough LXC. The acquisition stack runs in a second LXC with Docker Compose. A Python CronJob in k3s bridges the two via Traefik Service+Endpoints.

Update:gluetunthe first version of this stack wrapped the whole acquisition LXC in a Mullvad WireGuard tunnel via, on the assumption that every flow out of that box needed VPN protection equally. Two days later I tore that back out — see “The Traffic-Isolation Rethink” below for why a single blanket tunnel was the wrong model for what these five apps actually do.

View the complete homelab infrastructure source on GitHub 🐙

BMAX Mini PC*Jellyfin needs GPU access for hardware video transcoding. Thehas an AMD Radeon Vega iGPU that supports VAAPI hardware transcoding. Proxmox GPU passthrough requires IOMMU group isolation — the GPU is passed to a single container or VM exclusively.

nvidia-device-plugin/dev/dri/renderD128Kubernetes doesn’t natively support GPU passthrough for LXCs. Theworks for NVIDIA GPUs on specific cloud providers, but for AMD iGPU passthrough on bare-metal Proxmox, you need a dedicated LXC withmapped directly.

ct-srv-jellyfin-01Jellyfin runs inwith:

vm-srv-k3s-11/12/13amdgpuvfio-pciVMsk3s’s own nodes here () are Proxmox, not LXCs — a VM doesn’t share the host kernel, so there’s no “just bind-mount the device node” path the way there is for an LXC. Getting a k3s pod real GPU access on this hardware would mean classic VFIO passthrough of the iGPU to one specific VM: unbindingfrom the Proxmox host and handing the whole device toinstead. I looked at this seriously before ruling it out, because “GPU-in-Kubernetes” device plugins exist and I wanted to know if they’d apply.

rocm/k8s-device-plugin/dev/driamdgpuentiredifferentThey don’t, for this hardware. AMD’s owntargets ROCm compute (HIP/OpenCL) — a materially heavier stack than what VAAPI hardware transcoding actually needs, which is justvisibility. And a Kubernetes device plugin can’t manufacture GPU access a node’s kernel doesn’t already have — for a VM, that access only exists after real hypervisor-level VFIO passthrough, which a plugin doesn’t do. Worse, this is a single consumer Ryzen APU, not a data-center part with SR-IOV or mediated-device support for splitting one GPU across VMs — passthrough would hand theGPU to exactly one of the three k3s VMs, and the Proxmox host itself (which usesfor its own display/telemetry) would permanently lose access to it. There’s also no portability payoff to offset that cost: k3s’s scheduler can’t move a pod needing a passed-through device to anode than the one VFIO was bound to, so the usual “GPU follows the pod” reason people move transcoding into Kubernetes never materializes on a single-host, single-iGPU homelab. It would just be Jellyfin running on a VM instead of an LXC, at the permanent cost of the GPU being unavailable to anything else on the host.

amdgpu/dev/dri/renderD128The LXC path avoids all of that: an LXC shares the host’s kernel, so the host keeps thedriver bound and simply grants the container access to the resultingdevice node. Non-exclusive from the host’s perspective, already proven working, no PCI device binding to get wrong. If this box ever gets a GPU with real SR-IOV support, this is worth revisiting — the constraint here is the specific hardware, not a principled objection to GPU workloads in Kubernetes.

doesThe acquisition LXC has two genuinely different outbound flows, and my first pass treated them as one problem. SABnzbd connects to Eweka (my Usenet provider) over NNTPS on port 563 — already encrypted end-to-end, and Usenet copyright enforcement works exclusively via BitTorrent peer-list monitoring, so there’s no mechanism by which an ISP or rights-holder observes or reports Usenet downloads in the first place. NZBHydra2’s indexer search queries are a completely different flow: plain HTTP/HTTPS lookups against third-party indexer sites, whichexpose the home IP to whoever’s on the other end, the same as browsing any site directly.

Wrapping the whole LXC in gluetun/Mullvad “solved” both at once, but it was the wrong tool for either: VPN on the download path halves throughput for no privacy benefit Eweka’s own SSL doesn’t already provide, and it risks Eweka flagging the account for apparent multi-subscriber IP sharing. I tore gluetun out two days after standing it up and replaced it with a model that actually matches the two flows:

sabnzbd.inisocks5_proxy_url = ""ssl = 1ssl_verify = 2nzbhydra.ymlproxyType: SOCKSproxyIgnoreDomainsI verified this configuration directly rather than trusting the design intent on paper:shows(empty — SABnzbd was never routed through Tor) with,confirming the direct-to-Eweka SSL path is real;showspointed at the Tor container with no fallback option enabled. One live exception I noticed and haven’t chased down yet: one specific indexer bypasses Tor and connects directly () — possibly a site that blocks Tor exit nodes, possibly a leftover exception from before I understood this stack properly. Worth revisiting.

Tor’s bandwidth genuinely can’t handle bulk transfers, and routing downloads through it would be abusive to a network that exists for people who need anonymity for safety — never route the actual download path through Tor, only small metadata lookups.

ct-srv-media-acq-01The acquisition LXC () runs all five apps this way — no blanket tunnel, no kill-switch sidecar to maintain, no shared failure mode between “is Eweka’s SSL up” and “is the VPN provider’s WireGuard endpoint reachable today.”

the SQLite trap articleMedia libraries (downloaded files, metadata databases) live on NFS. As I wrote about in, NFS file-locking semantics don’t work with embedded databases. Sonarr and Radarr use SQLite internally for their media databases — and those databases corrupt on NFS under concurrent access.

Moving the acquisition stack to a local LXC with local storage eliminates the NFS lock problem. The media files themselves (downloaded episodes, movies) still live on NFS for sharing, but the application databases stay local.

Endpointsk3s Traefik routes external traffic to services inside the cluster. But the media stack isn’t in the cluster — it’s in LXCs. The bridge: Traefik IngressRoutes point at Kubernetes Services, which useobjects with hardcoded IP addresses pointing at the LXC containers.

selectorThis is the same pattern for all external services. The Service has no— it’s a “headless” Service where the Endpoints are manually maintained. Traefik doesn’t know or care that the backend is an LXC instead of a pod.

The IngressRoute for Jellyfin:

AutheliaCloudflare TunnelThe media stack getsprotection, wildcard TLS, andexternal access — the same as every in-cluster service. The only difference is the backend IP.

A Python CronJob runs every 10 minutes inside k3s, bridging the gap between the media stack and the cluster:

The script:

Without this CronJob, stuck downloads sit indefinitely. Sonarr and Radarr don’t have built-in queue monitoring — they trust the download client to report status, and SABnzbd sometimes silently fails without notifying the *arr stack.

The media stack’s departure from Kubernetes is the same pattern as running stateful workloads outside AKS: GPU workloads go to dedicated VMs with GPU passthrough, traffic-sensitive workloads get network-level isolation instead of a sidecar, and file-locking workloads go to local SSDs. Kubernetes excels at stateless, horizontally-scalable workloads. Media transcoding and acquisition are neither — they’re stateful, single-instance, and hardware-dependent. The right platform for them is the bare metal underneath, not the orchestration layer on top.

Why Kubernetes Was Wrong for Media

The Architecture

The Traefik Bridge

The Python CronJob

What I’d Change

GPU Passthrough

The Traffic-Isolation Rethink

NFS File Locking

# terraform/stacks/proxmox/lxc.tflxc_conf{desc="Jellyfin - GPU passthrough"# GPU device mapped via pct set}

┌─────────────────────────────────────────────┐│  k3s Cluster                                ││  ┌─────────────────────────────────────┐    ││  │ Python CronJob (every 10 min)       │    ││  │ - Checks Sonarr/Radarr queue        │    ││  │ - Clears stuck items                │    ││  │ - Triggers Jellyfin library scan    │    ││  └──────────┬──────────────────────────┘    ││             │ HTTP via Traefik              ││  ┌──────────▼──────────────────────────┐    ││  │ Traefik IngressRoutes               │    ││  │ - sabnzbd.woitzik.dev               │    ││  │ - sonarr.woitzik.dev                │    ││  │ - radarr.woitzik.dev                │    ││  │ - bazarr.woitzik.dev                │    ││  └─────────────────────────────────────┘    │└──────────────────┬──────────────────────────┘│ Traefik Service+Endpoints┌──────────────────▼──────────────────────────┐│  ct-srv-media-acq-01 (LXC, Tor for indexers)││  ┌─────────┐ ┌────────┐ ┌────────┐         ││  │ SABnzbd │ │ Sonarr │ │ Radarr │         ││  └─────────┘ └────────┘ └────────┘         ││  ┌─────────┐ ┌──────────────┐              ││  │ Bazarr  │ │ NZBHydra2    │              ││  └─────────┘ └──────────────┘              │└─────────────────────────────────────────────┘┌─────────────────────────────────────────────┐│  ct-srv-jellyfin-01 (LXC, GPU passthrough)  ││  ┌─────────┐ ┌──────────────┐              ││  │ Jellyfin│ │ /dev/dri/    │              ││  │         │ │ renderD128   │              ││  └─────────┘ └──────────────┘              │└─────────────────────────────────────────────┘

# kubernetes/apps/jellyfin/jellyfin.ymlapiVersion:v1kind:Servicemetadata:name:jellyfinnamespace:appsspec:ports:-port:8096targetPort:8096---apiVersion:v1kind:Endpointsmetadata:name:jellyfinnamespace:appssubsets:-addresses:-ip:10.0.20.254# ct-srv-jellyfin-01ports:-port:8096

apiVersion:traefik.io/v1alpha1kind:IngressRoutemetadata:name:jellyfinnamespace:appsspec:entryPoints: [websecure]routes:-match:Host(`media.woitzik.dev`)kind:Rulemiddlewares: [{name:authelia}]services:-name:jellyfinport:8096tls:secretName:wildcard-woitzik-dev-tls

# CronJob that monitors media acquisitionschedule:"*/10 * * * *"jobTemplate:spec:template:spec:containers:-name:media-watchdogimage:python:3.12-slimcommand:-python-/scripts/media-watchdog.py

• SABnzbd → Eweka, direct, over SSL.No VPN. The ISP sees “connected to news.eweka.nl,” never content — that’s the whole job done by NNTPS alone.
• NZBHydra2 → indexers, through a Tor SOCKS5 proxyno direct-connection fallbacktor172.28.1.10:9050,container at, configured with. Search queries are small and latency-tolerant, a good fit for Tor’s limited bandwidth, and Tor is a better fit than a commercial VPN for metadata-only queries — no single operator to trust, no throughput to throttle. If Tor is down, indexer queries fail outright instead of silently leaking the home IP.

1. Queries Sonarr/Radarr APIs for stuck queue items (downloads stuck for >30 minutes)
2. Clears stuck items and re-triggers the download
3. Checks Jellyfin’s library scan status
4. Posts status to Discord via webhook

1. Use a proper service mesh for cross-boundary traffic.The manual Endpoints pattern works but is fragile — if the LXC IP changes, the Endpoints must be updated manually. A DNS-based service discovery (Headscale DNS entries, for example) would be more resilient.

2. Move the CronJob to a native LXC cron.The Python script runs in a k3s Pod but talks to LXC-hosted services. It has no business being in the cluster. A systemd timer on the media acquisition LXC would be simpler.

Garage S3SQLITE_CORRUPTterraform-state, my self-hosted S3-compatible object store, uses SQLite for its metadata database. One morning, Terraform state operations started failing with. The bucket metadata was gone. Thebucket, the Atlantis lock table, the Velero backup index — all of it stored in a SQLite file that was now corrupted.

The root cause: Garage was running on an NFS-backed PersistentVolume. NFS doesn’t support the file-locking primitives that SQLite’s WAL (Write-Ahead Logging) mode requires. Under concurrent access — Garage’s metadata writer and a Velero backup reading the same database — the NFS lock delegation failed silently, and SQLite wrote to overlapping pages.

View the complete homelab infrastructure source on GitHub 🐙

My k3s cluster has two StorageClasses:

nfs-clientNFS () is the default. It’s backed by a dedicated LXC running an NFS server on ZFS, providing storage that survives pod rescheduling — a pod on k3s-12 can access the same PVC as a pod on k3s-13 because the NFS server is independent of any specific node.

local-pathLocal-path () pins the PV to whichever node created it. If the pod reschedules to a different node, the PVC is inaccessible until the pod returns to the original node. This is a limitation, but it’s the right trade-off for certain workloads.

fcntl()F_SETLKF_SETLKWSQLite’s WAL mode requiresfile locks — specifically,(non-blocking lock) and(blocking lock). These locks coordinate access between concurrent processes writing to the same database file.

NFS handles file locks differently:

The result: SQLite thinks it has an exclusive lock, but another process (or the same process on a different connection) also has a lock. Both write to the database file. Pages overlap. The database corrupts.

Garage runs with two storage mounts:

datametaalsoThevolume (bucket objects) is on NFS — fine, because S3 object storage doesn’t use file locks for individual files. Thevolume (SQLite database) wason NFS initially. This worked until a Velero backup and a Terraform state write happened simultaneously.

db.sqliteVelero reads Garage’s S3 API to enumerate backup objects. Terraform reads the same database to verify state file existence. Both hit SQLite through Garage’s metadata layer. Under NFS, the concurrent reads triggered the lock delegation failure, and SQLite corrupted pages 169–184 of.

.recoverSQLite has acommand that can extract data from a corrupted database:

.recoverThecommand scanned every page of the corrupted file and extracted whatever data it could read. For pages 169–184 (the corrupted range), it found partial data — enough to reconstruct the bucket and key metadata, but not enough to guarantee referential integrity.

After recovery, the missing objects (terraform-state bucket and Atlantis lock key) had to be re-inserted manually via Python:

The data was restored, but the trust was gone. The database could corrupt again under the same conditions.

local-pathMove every embedded database to:

local-pathThe apps that need:

local-pathReadWriteOnceReadWriteManyPostgreSQL (via CNPG) doesn’t have the SQLite lock problem because it uses its own file locking, but CNPG requiresor a CSI driver that supports— NFS’ssemantics can confuse CNPG’s WAL archiving.

local-pathmeans the PVC is pinned to one node. If the pod reschedules, it loses access to the data. For Garage, this is acceptable: Garage runs on a single node, and if that node goes down, the S3 data is unavailable regardless (it’s on the same host).

local-pathFor databases that need HA (Postgres, Redis), the solution isn’tor NFS — it’s a managed operator (CNPG for Postgres) that handles replication and failover independently of the storage layer.

if the application uses file-level locking (SQLite, BoltDB, LMDB), it goes on local-path. If it uses network-level locking (PostgreSQL, MySQL), it goes on NFS or a managed operator.The principle:

SQLite on NFS is the same failure mode as running SQLite on an SMB share in a Windows domain: the file-locking semantics are fundamentally incompatible. In Azure, this maps to Azure Files (SMB-backed) vs. Azure Disk (block storage). Azure Files supports SMB locks but has the same delegation latency issues under concurrent access — any application that needs tight file-level locking should use Azure Disks, not Azure Files. The principle is identical: embedded databases need local, low-latency storage with native file-locking support.

Designing Data-Intensive Applications*db.sqlitecovers exactly this class of correctness assumption — what a storage layer actually guarantees about concurrent access versus what an application silently assumes it guarantees — in far more depth than a corruptedfile teaches you in the moment.

The Storage Classes

Why SQLite and NFS Don’t Work

The Garage Incident

The Recovery

The Fix

The Trade-off

# NFS — for most workloadsapiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:nfs-clientprovisioner:nfs-subdir-external-provisionerparameters:server:10.0.20.100path:/archivereclaimPolicy:Retain# Local-path — for embedded databasesapiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:local-pathprovisioner:rancher.io/local-pathreclaimPolicy:Retain

# kubernetes/apps/garage/garage.ymlvolumes:-name:datapersistentVolumeClaim:claimName:garage-data# NFS — bucket objects-name:metapersistentVolumeClaim:claimName:garage-meta# local-path — SQLite metadata

# Dump recoverable data from corrupted SQLitesqlite3db.sqlite".recover">recovered.sql# Recreate the database from the dumpsqlite3db_clean.sqlite<recovered.sql

importsqlite3importmsgpackconn=sqlite3.connect("db_clean.sqlite")# Garage uses msgpack-encoded metadata with 'G2key'/'G2bkt' prefixes# Re-insert the terraform-state bucketconn.execute("INSERT INTO buckets (name, ...) VALUES (...)")conn.commit()

# Garage — meta volume moved to local-pathvolumes:-name:metapersistentVolumeClaim:claimName:garage-meta# NOW: local-path (was: nfs-client)

1. NFSv3fcntl(): No native lock support.calls return success but locks are local to the client — two NFS clients can both acquire an “exclusive” lock on the same file simultaneously.

2. NFSv4LOCK: Hasoperations, but the lock delegation model introduces latency and failure modes that SQLite’s tight locking loop doesn’t tolerate. If the NFS server is slow to respond to a lock request, SQLite’s default 5-second busy timeout can expire, causing the application to retry — and the retry can conflict with the lock held by another client.

3. Kubernetes NFS provisionernfs-subdir-external-provisionerrpc.lockdlockd: Theuses NFSv4, but the lock delegation is handled by the NFS server’sdaemon, which runs in a separate process space. Under concurrent load,can lose track of which client holds which lock.

az aks createRunwith the defaults and you get a public API server FQDN, a Standard Load Balancer giving every node its own path to the internet, and — unless you go out of your way to stop it — an admin kubeconfig that works from anywhere with the right token. None of that is a bug. It’s the fast path, and for a demo cluster it’s fine.

For a cluster inside a governed Hub & Spoke, it’s three separate audit findings. This is the Terraform to close all three at once: a private API server, forced-tunneled egress through a firewall you already control, and private-endpoint-only ACR and Key Vault.

Get the base template free on GitHub 🐙

Hub & Spoke moduleThree engineering decisions drive this, same as thethis one is designed to sit behind:

private_cluster_public_fqdn_enabled = falseprivate_cluster_enabled = trueaz aks showis the setting people miss.alone still publishes a public FQDN that resolves to nothing reachable — a fingerprintable artifact for no benefit. Turning it off meansdoesn’t leak a hostname at all.

privatelink.<region>.azmk8s.iolifecycle.ignore_changesHub & Spoke module’sThe cluster needs its own Private DNS Zone for the API server (), linked to the VNet — same DINE-policy-safepattern as thecentralized zones, because Azure Policy tagging fights this resource too.

network_plugin = "azure"network_policy = "azure"outbound_typeandget most of the attention in AKS networking guides. The setting that actually determines whether traffic is inspectable is.

loadBalancerLeft at its default (), AKS provisions its own Standard Load Balancer outbound rule and every node gets a direct, un-inspected path to the internet — a private API server with fully public node egress is a common half-measure that looks locked down in the portal and isn’t.

outbound_type = "userDefinedRouting"depends_onbeforetells AKS to trust this route table instead of provisioning its own path. The Route Table has to exist and be associated to the node subnetthe cluster is created —enforces the ordering, because AKS validates the UDR is actually in place at cluster-creation time and fails otherwise.

admin_enabled = truekubectl get secret -o yamlNodes still need to pull images and (often) read secrets. The naive fix —on the registry, a shared pull secret in a Kubernetes Secret — is a credential that outlives any single deployment and shows up infor anyone with namespace read access.

AcrPullpublic_network_access_enabled = falseKey Vault Secrets UserThe kubelet identity — the identity that actually pulls images, not the cluster’s control-plane identity — getsdirectly. No credential to rotate, nothing in a Secret, nothing that outlives the node it’s bound to. Key Vault follows the identical shape: RBAC-authorized,, kubelet identity gets, ready for the CSI Secrets Store driver to mount without ever touching a static key.

kubectlazure_active_directory_role_based_access_controlsku_tierFreeStandardAzure Firewall Forced Tunneling moduleIt’s a starting cluster, not a finished platform. No Entra ID–integratedauth (addyourself), no multi-pool topology, no cluster autoscaler wiring,defaults to(no uptime SLA — setfor production). And it assumes you already have a firewall to hand it a private IP — pair it with theif you don’t.

GitOps and Kubernetes*For the operational reality of running production Kubernetes past the “it deployed” stage — the workflows and failure modes that don’t show up in a getting-started guide —is the reference I’ve actually kept open while doing this.

Target Architecture

Step 1: The API Server Has No Public FQDN

userDefinedRoutingStep 2:— the Setting That Actually Matters

Step 3: ACR and Key Vault Without a Public Endpoint

What This Doesn’t Do

┌─────────────────────┐│   Azure Firewall     │  ← you supply the IP│  (existing / NVA)     │└──────────┬───────────┘│ UDR: 0.0.0.0/0┌──────────┴───────────┐│   vnet-aks            ││                       │┌──────────┴──────────┐  ┌─────────┴──────────┐│  snet-aks-nodes     │  │ snet-private-       ││  (default-deny NSG) │  │ endpoints            ││                     │  │  - ACR (Premium)     ││  Private AKS        │  │  - Key Vault         ││  (no public FQDN)   │  │                      │└─────────────────────┘  └─────────────────────┘

resource"azurerm_kubernetes_cluster""this"{# ...private_cluster_enabled=trueprivate_dns_zone_id=azurerm_private_dns_zone.aks.idprivate_cluster_public_fqdn_enabled=falsenetwork_profile{network_plugin="azure"network_policy="azure"outbound_type="userDefinedRouting"}}

resource"azurerm_route_table""aks_egress"{route{name="default-via-firewall"address_prefix="0.0.0.0/0"next_hop_type="VirtualAppliance"next_hop_in_ip_address=var.firewall_private_ip}}resource"azurerm_subnet_route_table_association""aks_nodes"{subnet_id=azurerm_subnet.aks_nodes.idroute_table_id=azurerm_route_table.aks_egress.id}

resource"azurerm_container_registry""this"{sku="Premium"# required for Private Linkadmin_enabled=falsepublic_network_access_enabled=false}resource"azurerm_role_assignment""aks_acr_pull"{scope=azurerm_container_registry.this[0].idrole_definition_name="AcrPull"principal_id=azurerm_kubernetes_cluster.this.kubelet_identity[0].object_idskip_service_principal_aad_check=true}

• No public control-plane endpoint— reachable only from inside the VNet, a peered network, or a VPN
• Forced tunneling, not default outbound— every node packet leaves through a firewall you control
• Private-endpoint-only dependencies— ACR and Key Vault never touch the public internet either