Go applications expose a useful set of runtime metrics, but raw /metrics output does not make it easy to spot GC pressure, scheduler latency, memory growth, or file descriptor exhaustion. This post covers a go-mixin for Prometheus and Grafana that adds a dashboard and alerts for the Go runtime.

The mixin is available on GitHub. It currently ships with one Grafana dashboard and three alert rules:

• Go / Overview - A dashboard for runtime CPU usage, scheduler latency, garbage collection, heap churn, mutex contention, cgo activity, and file descriptor pressure.
• GoHighGcCpu - Alerts when a Go process spends too much CPU time in garbage collection.
• GoHighSchedulerLatency - Alerts when runnable goroutines wait too long to be scheduled.
• GoHighFdUsage - Alerts when a Go process is close to its file descriptor limit.

The repo also includes generated dashboard JSON and Prometheus rule files, so you can either vendor the mixin into your Jsonnet setup or import the generated files directly.

I use Ansible to manage my entire OS setup, with a playbook that installs all necessary software, sets up my dotfiles, and configures my system. This playbook can be run on a fresh installation of my OS and have my system set up exactly how I like it. Recently, I shifted from the Linux Arch AUR to go install for Go binaries, as it felt simpler and ensured up-to-date dependencies - without having to depend on the authors creating Arch AUR packaging for it. However, I still couldn’t find a way to pin packages to specific GitHub tags/releases. To solve this, I created a simple Ansible playbook to install Go binaries directly from GitHub releases, allowing me to pin and auto update the version of the Go binaries I want to install.