实现报警功能 #1161

kailongapple · 2024-04-18T03:28:07Z

kailongapple
Apr 18, 2024

前言

官方文档
https://clickvisual.net/zh/clickvisual/03funcintro/alarm-function-configuration-description.html#%E6%9E%B6%E6%9E%84%E8%AF%B4%E6%98%8E

报警流程

在 ClickHouse 中产生对应数据表的 Materialized View，作用是将数据按照配置的规则以秒为单位录入 metrics.samples 表中
产生的 Prometheus 告警规则写入配置的文件或者 configmap 中
Prometheus 读取 metrics.samples 的数据，并根据告警规则产生告警，推送到 Prometheus AlertManager
Prometheus AlertManager 通过 webhook 推送到 clickvisual，再由 clickvisual 分发到钉钉

告警触发流程图

部署

部署prometheus 和alertmanager

使用github的yaml
#修改连接地址

clickvisual/data/k8s/prometheus/
cd clickvisual
sed -i 's/api\/v1\/status/api\/v2\/status/' data/k8s/prometheus/alertmanager-deployment.yaml
sed -i 's/prom2click:9201/clickvisual:9021/' data/k8s/prometheus/prometheus-configmap.yaml

#去掉默认命名空间配置

sed -i '/^ *namespace: .*$/d' data/k8s/prometheus/alertmanager-deployment.yaml
sed -i '/^ *namespace: .*$/d' data/k8s/prometheus/alertmanager-configmap.yaml
sed -i '/^ *namespace: .*$/d' data/k8s/prometheus/alertmanager-deployment.yaml
sed -i '/^ *namespace: .*$/d' data/k8s/prometheus/alertmanager-service.yaml
sed -i '/^ *namespace: .*$/d' data/k8s/prometheus/prometheus-configmap.yaml
sed -i '/^ *namespace: .*$/d' data/k8s/prometheus/prometheus-deployment.yaml
sed -i '/^ *namespace: .*$/d' data/k8s/prometheus/prometheus-rules.yaml
sed -i '/^ *namespace: .*$/d' data/k8s/prometheus/prometheus-service.yaml

#部署

kubectl apply -f data/k8s/prometheus/alertmanager-deployment.yaml -n ilogtail
kubectl apply -f data/k8s/prometheus/alertmanager-configmap.yaml -n ilogtail
kubectl apply -f data/k8s/prometheus/alertmanager-deployment.yam -n ilogtail
kubectl apply -f data/k8s/prometheus/alertmanager-service.yaml -n ilogtail
kubectl apply -f data/k8s/prometheus/prometheus-configmap.yaml -n ilogtail
kubectl apply -f data/k8s/prometheus/prometheus-deployment.yaml -n ilogtail
kubectl apply -f data/k8s/prometheus/prometheus-rules.yaml -n ilogtail
kubectl apply -f data/k8s/prometheus/prometheus-service.yaml -n ilogtail

告警环境初始化

接入k8s
系统管理-->集群管理-->新增集群
配置prometheus和指定告警规则写入的Configmap
报警-->环境检测-->编辑告警环境

clickhouse

追加配置

<yandex>
  <graphite_rollup>
    <path_column_name>tags</path_column_name>
    <time_column_name>ts</time_column_name>
    <value_column_name>val</value_column_name>
    <version_column_name>updated</version_column_name>
    <default>
      <function>sum</function>
      <retention>
        <age>0</age>
        <precision>1</precision>
      </retention>
      <retention>
        <age>360</age>
        <precision>60</precision>
      </retention>
      <retention>
        <age>3600</age>
        <precision>300</precision>
      </retention>
      <retention>
        <age>86400</age>
        <precision>3600</precision>
      </retention>
    </default>
  </graphite_rollup>
</yandex>

初始化告警的需要的数据库和表

#配置报警规则
报警-->报警列表-->新增报警--
这个会写入prometheus的告警规则的confmap

检查报警规则是否正常

关于告警模式
将符合查询条件的数据写入告警表

默认是普通统计单位时间内的数据的数量
聚合查询告警标准的val数值

目前问题
clickvisual 会自动重启

kailongapple · 2024-05-05T10:04:30Z

kailongapple
May 5, 2024
Author

使用外部prom2click 可以解决重启问题

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

实现报警功能 #1161

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

实现报警功能 #1161

Uh oh!

kailongapple Apr 18, 2024

前言

部署

部署prometheus 和alertmanager

告警环境初始化

clickhouse

Replies: 1 comment

Uh oh!

kailongapple May 5, 2024 Author

kailongapple
Apr 18, 2024

kailongapple
May 5, 2024
Author