Skip to content

containeroo/heartbeats

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Heartbeats

heartbeats.png

Go Report Card Go Doc Release GitHub tag Tests Build license


A lightweight HTTP service for monitoring periodic "heartbeat" pings ("bumps") and notifying configured receivers when a heartbeat goes missing or recovers. Includes an in‐browser read-only dashboard showing current heartbeats, receivers, and historical events.

πŸš€ Features

  • Heartbeat monitoring with configurable interval & grace periods
  • Pluggable notifications via Slack, Email, or MS Teams
  • In-memory history of recent events (received, failed, state changes, notifications, API requests)
    • Stores up to --history-size entries (default: 10,000)
    • Not persisted across restarts (~1.73β€―MB at max size)
  • Dashboard with:
    • Heartbeats: status, URL, last bump, receivers, quick-links
    • Receivers: type, destination, last sent, status
    • History: timestamped events, filter by heartbeat
    • Text-search filters & copy-to-clipboard URLs
  • /healthz and /metrics endpoints for health checks & Prometheus
  • YAML configuration with variable resolution via containeroo/resolver

🏁 Flags

Flag Shorthand Default Environment Variable Description
--config -c heartbeats.yml HEARTBEATS_CONFIG Path to configuration file
--listen-address -a :8080 HEARTBEATS_LISTEN_ADDRESS Address to listen on (host:port)
--route-prefix `` HEARTBEATS_ROUTE_PREFIX ath prefix to mount the app (e.g., /heartbeats).
--site-root -r http://localhost:8080 HEARTBEATS_SITE_ROOT Base URL for dashboard and link rendering
--history-size -s 10000 HEARTBEATS_HISTORY_SIZE Number of historical events to keep in memory (default ~1.73β€―MB for 10,000). Not persisted.
--skip-tls - false HEARTBEATS_SKIP_TLS Skip TLS verification for all receivers (can be overridden per receiver)
--debug -d false HEARTBEATS_DEBUG Enable debug-level logging
--debug-server-port -p 8081 HEARTBEATS_DEBUG_SERVER_PORT Port for the debug server
--log-format -l text HEARTBEATS_LOG_FORMAT Log format (json or text)
--retry-count - 3 HEARTBEATS_RETRY_COUNT Number of times to retry a failed notification. Use -1 for infinite.
--retry-delay - 2s HEARTBEATS_RETRY_DELAY Delay between retries. Must be β‰₯ 1s.
--help -h - - Show help and exit
--version - - - Print version and exit

Proxy Environment Variables

  • HTTP_PROXY: URL of the proxy server to use for HTTP requests
  • HTTPS_PROXY: URL of the proxy server to use for HTTPS requests

🌐 Endpoints

Path Method Description
/ GET Dashboard home page
/bump/{id} POST, GET Create a new heartbeat
/bump/{id}/fail POST, GET Manually mark heartbeat as failed
/healthz GET Liveness probe
/metrics GET Prometheus metrics endpoint

βš™οΈ Configuration

heartbeats and receivers must be defined in your YAML file (default config.yaml).

Examples (config.yaml)

---
receivers:
  dev-crew-int:
    slack_configs:
      - channel: integration
        token: env:SLACK_TOKEN
heartbeats:
  prometheus-int:
    description: "Prometheus β†’ Alertmanager test"
    interval: 15m
    grace: 90s
    receivers:
      - dev-crew-int

Heartbeats

A heartbeat waits for periodic pings ("bumps"). If no bump arrives within interval + grace, notifications are sent.

To reduce noise from race conditions (e.g. pings arriving milliseconds after grace timeout), Heartbeats adds a short internal delay before transitioning to grace or missing. This ensures smoother handling of near-expiry bumps without affecting responsiveness.

Key Type Description
description string (optional) Human-friendly description
interval duration Required. Go duration (e.g. 15m, 90s) for expected interval between pings
grace duration Required. Go duration after interval before marking missing
receivers []string Required. List of receiver IDs (keys under receivers:) to notify upon missing

Example

heartbeats:
  prometheus-int:
    description: "Prometheus β†’ Alertmanager test"
    interval: 15m
    grace: 90s
    receivers:
      - dev-crew-int

Receivers

Each receiver can have multiple notifier configurations. Supported under receivers::

  • slack_configs
  • email_configs
  • msteams_configs

You may use any template variable from the heartbeat (e.g. {{ .ID }}, {{ .Status }}), and these helper functions:

  • upper: {{ upper .ID }}
  • lower: {{ lower .ID }}
  • formatTime: {{ formatTime .LastBump "2006-01-02 15:04:05" }}
  • ago: {{ ago .LastBump }}
  • isRecent: {{ isRecent .LastBump }} // isRecent returns true if the last bump was less than 2 seconds ago
  • join: {{ join .Tags ", " }}

Variable Resolution

Heartbeats uses containeroo/resolver for variable resolving.

Resolver supports:

  • Plain: literal value
  • Environment: env:VAR_NAME
  • File: file:/path/to/file
  • Within-file: file:/path/to/file//KEY, also supported yaml:,json:,ini: and toml:. For more details see containeroo/resolver.

Slack

Defaults:

  • subject_tmpl: [{{ upper .Status }}] {{ .ID }}"
  • text_tmpl: {{ .ID }} is {{ .Status }} (last bump: {{ ago .LastBump }})"
receivers:
  dev-crew-int:
    slack_configs:
      - channel: "#integration"
        token: env:SLACK_TOKEN
        # optional custom templates:
        title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
        text_tmpl: "{{ .ID }} status: {{ .Status }}"
        # optional: override global skip TLS
        skip_tls: true

Heartbeats adds a custom User-Agent: Heartbeats/<version> header to all outbound HTTP requests. The Content-Type header is also set to application/json.

Email

Defaults:

  • subject_tmpl: "[HEARTBEATS]: {{ .ID }} {{ upper .Status }}"
  • body_tmpl: "<b>Description:</b> {{ .Description }}<br>Last bump: {{ ago .LastBump }}"
email_configs:
  - smtp:
      host: smtp.gmail.com
      port: 587
      from: admin@example.com
      username: env:EMAIL_USER
      password: env:EMAIL_PASS
      # optional
      start_tls: true
      # optional: override global skip TLS
      skip_insecure_verify: true
    email:
      is_html: true
      to: ["ops@example.com"]
      # optional custom templates:
      subject_tmpl: "[HB] {{ .ID }} {{ upper .Status }}"
      body_tmpl: "Last bump: {{ ago .LastBump }}"

MS Teams (incomming webhook)

Defaults:

  • title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
  • text_tmpl: "{{ .ID }} is {{ .Status }} (last bump: {{ ago .LastBump }})"
msteams_configs:
  - webhook_url: file:/secrets/teams/webhook//prod
    # optional custom templates:
    title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
    text_tmpl: "{{ .ID }} status: {{ .Status }}"
    # optional: override global skip TLS
    skip_tls: true

Heartbeats adds a custom User-Agent: Heartbeats/<version> header to all outbound HTTP requests. The Content-Type header is also set to application/json.

MS Teams (Graph API) (NOT TESTED)

Defaults:

  • title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
  • text_tmpl: "{{ .ID }} is {{ .Status }} (last bump: {{ ago .LastBump }})"
msteamsgraph_configs:
  - channel_id: env:MSTEAMSGRAPH_CHANNEL_ID
    team_id: env:MSTEAMSGRAPH_TEAM_ID
    # optional custom templates:
    title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
    text_tmpl: "{{ .ID }} status: {{ .Status }}"
    # optional: override global skip TLS
    skip_tls: true

Heartbeats adds a custom User-Agent: Heartbeats/<version> header to all outbound HTTP requests. The Content-Type header is also set to application/json.

πŸ“ˆ Metrics

Heartbeats exposes the following Prometheus metrics via the /metrics endpoint:

Heartbeat Metrics

Name Type Labels Description
heartbeats_heartbeat_last_status gauge heartbeat Most recent status of each heartbeat (0 = DOWN, 1 = UP)
heartbeats_heartbeat_received_total counter heartbeat Total number of received heartbeats per ID
heartbeats_receiver_last_status counter receiver, type, target Reports the status of the last notification attempt (1 = ERROR, 0 = SUCCESS)

History Store Metrics

Name Type Description
heartbeats_history_byte_size gauge Current size of the in-memory history store in bytes

πŸ§ͺ Development

Download the binary and update the example config.yaml according your needs. If you prefer to run heartbeats in docker, you find a docker-compose.yaml & config.yaml here. For a kubernetes deployment you find the manifests here.

πŸ”§ Debugging

Heartbeats includes optional internal endpoints for testing receiver notifications and simulating heartbeats. These are only enabled when the --debug flag is set.

Internal Endpoints

Path Method Description
/internal/receiver/{id} GET Sends a test notification to the receiver
/internal/heartbeat/{id} GET Simulates a bump for the given heartbeat

These endpoints listen only on 127.0.0.1.

Kubernetes

Forward the debug port to your local machine:

kubectl port-forward deploy/heartbeats 8081:8081

curl http://localhost:8081/internal/receiver/{id}
curl http://localhost:8081/internal/heartbeat/{id}

Docker

Bind the debug port only to your host's loopback interface:

docker run \
  -p 127.0.0.1:8081:8081 \
  -v ./config.yaml:/config.yaml \
  containeroo/heartbeats

curl http://localhost:8081/internal/receiver/{id}
curl http://localhost:8081/internal/heartbeat/{id}

βœ… This ensures /internal/* endpoints are only reachable from your local machine, not from other containers or the network.

⚠️ Warning: These endpoints are meant for local testing and debugging only. Never expose them in production.

πŸ“ License

This project is licensed under the Apache 2.0 License. See the LICENSE file for details.