Observability stack on top of nixos-homelab: Grafana, long-term metrics storage, and the exporters/collectors that feed it, for both cluster and host-level monitoring.
For the full list of module options, see the documentation, under Options.
{
inputs = {
...
homelab-monitoring = {
url = "github:nixos-homelab/monitoring";
inputs.nixpkgs.follows = "nixpkgs";
};
...
};
}{ inputs, ... }:
{
imports = [
inputs.homelab-monitoring.nixosModules.grafana
inputs.homelab-monitoring.nixosModules.mimir
];
config = {
homelab.grafana.enable = true;
homelab.mimir.enable = true;
};
}- grafana: dashboards, provisioned with datasources for Mimir.
homelab.grafana.dashboards.<name>provisions a dashboard (read-only in the UI), andhomelab.grafana.homeDashboardmakes one Grafana's home. The modules' tiles (homelab.grafana.home.tiles) make a Home dashboard, which is the home by default. - mimir: long-term Prometheus-compatible metrics storage.
- alloy: the Grafana Alloy agent
that scrapes and forwards metrics/logs, with a per-service
integrationsallow-list likehomepage's. - kube-state-metrics: metrics about the cluster's objects (Deployments, pods, nodes, ...), Secrets excepted.
- node-exporter: host-level metrics (CPU, memory, disk, network).
- smartctl-exporter: disk S.M.A.R.T. health metrics, from a NixOS service on each host.
- systemd-exporter: the state of each host's systemd units, failed ones included, from a NixOS service on each host.
- zfs-exporter: ZFS pool and dataset metrics, from a NixOS service on each host.
lib.grafana builds dashboards in Grafana's v2 resource format, with the
Mimir datasource set on every query. Panels go where they are shown, in
the layout; dashboard numbers them.
{ inputs, ... }:
{
homelab.grafana.dashboards.nas.dashboard =
with inputs.homelab-monitoring.lib.grafana;
dashboard {
name = "nas";
title = "NAS";
layout = rows [
(row { title = "CPU"; } (autoGrid { } [
(timeseries {
title = "Busy";
unit = "percentunit";
queries = [
(prom ''1 - avg(rate(node_cpu_seconds_total{mode="idle"}[$__rate_interval]))'' { })
];
})
]))
];
};
}Builders: dashboard; layouts grid, autoGrid, rows/row,
tabs/tab; panels timeseries, stat, gauge, bargauge, table,
stateTimeline, text, or panel "<plugin>" for any other; prom for a
query, queryVariable, link, and whenVariable to show a row, tab or
item only for a variable's value. A dashboard can also be a JSON file
(dashboards.<name>.dashboard = ./board.json;).
lib.screenshots renders dashboards without a cluster: the pinned Grafana
on a dashboard's JSON, its Mimir datasource pointed at a Prometheus that
serves samples generated for the dashboard's queries, and headless Chromium
taking the whole page as a PNG. Each dashboard is a derivation of its own,
so a changed one renders again on its own, in seconds.
$ nix build .#screenshots.nas # the mixed scenario
$ nix build .#screenshots.nas.failing # or healthy
$ nix build .#screenshots.all # every dashboard, mixedNext to <name>.png, <name>.log lists every query the page ran, with
what came back (frames and rows, no data, or the error), and what the page
reported.
The data comes from a scenario (lib.screenshots.scenarios: healthy,
mixed, failing): label tables that keep a workload's namespace, pod and
job together across the metrics a query joins, and per-metric specs where a
metric's name says too little. scenarios.nix documents the format; a repo
with metrics of its own extends one with lib.recursiveUpdate. A machine's
own dashboards:
legacyPackages.screenshots = inputs.homelab-monitoring.lib.screenshots.forDashboards {
inherit pkgs;
inherit (self.nixosConfigurations.nas.config.homelab.grafana) dashboards;
# A render with the dashboard's variables set
variants.workload-sonarr = {
dashboard = "workload";
variables.workload = "sonarr";
};
};