Complete stack to demo how grafana attach an alert image to webhook notification.
- grafana
- grafana-renderer
- postgres
- prometheus
- alertmanager
- minio(s3)
- mc(s3 tools)
- webhook
expect imageUrl in webhook notifications
docker-compose logs -f webhook{
title: '[Alerting] grafana alerts alert',
ruleId: 1,
ruleName: 'grafana alerts alert',
state: 'alerting',
evalMatches: [
{
value: 1,
metric: 'grafana_alerting_active_alerts{instance="grafana:3000", job="grafana"}',
tags: [Object]
}
],
orgId: 1,
dashboardId: 1,
panelId: 5,
tags: {},
ruleUrl: 'http://localhost:3000/d/BeEFMkOnz/overview?tab=alert&viewPanel=5&orgId=1',
imageUrl: 'http://s3:9000/grafana/LQFw3BOJF2WsPURU1XzD.png'
}
- grafana alert triggered
- grafana call renderer to render an image using embed chromium
- renderer visit grafana url to generate an image
- image is uploaded to s3 bucket by grafana
- image url is included in notification payload
- notification sent to webhook address
reload prometheus configuration
curl -X POST http://127.0.0.1:9090/-/reload
get a mc shell
docker-compose run --rm --entrypoint /bin/bash mc