Added alert labels cannot be used in annotation #2992
Description
What did you do?
When I add a label in an alert via LABELS I would like to be able to use it immediately in the annotation, however that doesn't seem to be possible.
ALERT InstanceDemo
IF up == 1
FOR 10s
LABELS {
where = "{{ .Labels.instance | toUpper }}",
severity = "demo" ,
}
ANNOTATIONS {
summary = "Instance {{ $labels.where }} down",
description = "{{ $labels.where }} of job {{ $labels.job }} has been down for more than 5 minutes.",
}
What did you expect to see?
I would expect $labels.where in the annotation description to be replaced with the value I just set in the LABELS section.
What did you see instead? Under which circumstances?
It gets replaced with an empty string.
You can see "where" in the list of labels in the output of curl -sg '<prom-url>/api/v1/query?query=ALERTS, but the contents posted in the Slack channel is "Instance down".
Environment
-
System information:
Linux 4.4.0-83-generic x86_64
-
Prometheus version:
prometheus, version 1.6.3 (branch: master, revision: c580b60c67f2c5f6b638c3322161bcdf6d68d7fc)
build user: root@a6410e65f5c7
build date: 20170522-09:15:06
go version: go1.8.1
- Alertmanager version
alertmanager, version 0.6.2 (branch: master, revision: b011c0a32ce887c1a10f7d34d52fd8cce485c1cf)
build user: root@e3ca4de32142
build date: 20170509-08:56:14
go version: go1.8.1
- Prometheus configuration file:
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
external_labels:
monitor: 'demo-2454'
rule_files:
- "alert-demo-2454.rules"
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9091']
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9094- Alertmanager configuration file:
global:
slack_api_url: https://hooks.slack.com/services/XXXX/YYYY
route:
receiver: demo-2454
group_by: ['alertname', 'origin']
group_wait: 10s
group_interval: 30s
# If an alert has successfully been sent, wait 'repeat_interval' to
# resend them.
repeat_interval: 1m
receivers:
- name: demo-2454
slack_configs:
- channel: 'alert-testing'
text: |
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
*Description:* {{ .Annotations.description }}
*Details:*
{{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
{{ end }}
{{ end }}