The centralized observability platform for FUTO services. A single Talos Kubernetes cluster per environment, stretched across multiple OVH datacentres and managed by Flux. It hosts the metrics, dashboards, and alerting stack, and is the central metrics store that other FUTO clusters remote-write into.
Built for geographic resilience with single-cluster operational simplicity: three control planes in low-latency DCs hold one etcd quorum, three bare-metal workers carry the observability workload, and everything talks over a private OVH vRack.
Status: both environments are built and running — o11y-staging and o11y-production.
| Guide | Covers |
|---|---|
| 01 — Bootstrap | Standing up an environment: one-time prep, Terragrunt apply order, operator access, common operations |
| 02 — Infrastructure architecture | OVH compute, hardware specs, vRack networking, IPLB ingress, cost |
| 03 — Cluster architecture | Talos, Kubernetes, host firewall, and Flux GitOps |
| 04 — Application architecture | Envoy ingress, VictoriaMetrics central store, Grafana, CloudNativePG, supporting operators |
| Runbook: tenant cutover | Moving a store's metrics from tenant 0 onto per-cluster tenants, staging first |
deployment/modules/
├── ovh/account/ # cloud project, vRack, private network, CPs, workers, IPLB, DNS
├── netbird/cluster/ # per-env mesh: vRack route, mesh-gateway VIP + DNS, pod egress, policies
├── grafana/cluster/ # Grafana service account for the Rootly integration
├── netbox/cluster/ # IPAM registration of the ranges the other modules allocate
├── talos/cluster/ # machine secrets, CP + worker configs, bootstrap, ingress firewall
└── kubernetes/helm/ # CoreDNS, Flux Operator + Instance, bootstrap-settings, env secrets
kubernetes/
├── apps/
│ ├── base/<app>/ # ks.yaml (the app's Flux Kustomization) + app/ (chart source + manifests)
│ └── <env>/<ns>/ # namespace overlay: which base apps run; deltas from base as patches/<app>.yaml
├── components/ # shared kustomize Components (targetNamespace replacement)
└── clusters/
└── <env>/
├── apps.yaml # cluster-apps entry point (the Flux Instance points here)
└── cluster-settings.yaml # per-env CLUSTER_* ConfigMap (BOOTSTRAP_* comes from Terraform)
State lives in S3 under yucca/o11y/v3/<module>/<env>. Secrets and OVH/NetBird tokens come from 1Password via op run and deployment/.env.