一、基于企业微信的报警媒介¶
- 实时告警通知：企业微信/钉钉等即时通信工具能够实现实时的告警通知，使得团队 成员能够及时响应和解决问题。
- 通知范围更广：基于企业微信/钉钉的告警通知可以通过群组和@某人的方式，将告 警通知发送给更广泛的接收者，避免出现漏报的情况。
- 告警信息更直观：企业微信/钉钉等通信工具提供了更丰富的告警信息呈现方式，例如文本消息、链接、图片、语音等，使得告警信息更加直观和易于理解。
1.1 获取企业微信相关信息¶
企业微信管理后台：https://work.weixin.qq.com/wework_admin/frame#contacts
to_party获取位置
agent_id获取位置
api_secret获取位置
corp_id获取位置
访问令牌获取
curl -H "Content-Type: application/json" \
-d '{"corpid": "wwe897498df31cc026", "corpsecret": "kqi1OyauWVz5HJvNSV268ORZQ8bANHlAV4l5j9BqWqc"}' \
"https://qyapi.weixin.qq.com/cgi-bin/gettoken"
# 回显内容
{"errcode":0,"errmsg":"ok","access_token":"hgDdVMMPDvoIIoJkOiLfwg7aFmtOj03xmEupfZujxPeYdg4WmFNOs8rkdf3_0yGgFFPAOUbhOn9r6J_dw3rQ5PLu_5dJm7u6tezsvx0F1cO1EHwAKHBIoearwP-B4y1xgDyErVgy-x6GMSOPHnD4m8L8vMcXkhso0M-voiHLeF9hX8zrR1PLtxL0Otsk6WDSaj-LhwgQhSmt2cWgiMIzxg","expires_in":7200}
企业可信IP获取
curl -H "Content-Type: application/json" -d '{
"toparty": 4,
"msgtype": "text",
"agentid": 1000005,
"text": {
"content": "告警测试"
},
"safe": 0
}' "https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=hgDdVMMPDvoIIoJkOiLfwg7aFmtOj03xmEupfZujxPeYdg4WmFNOs8rkdf3_0yGgFFPAOUbhOn9r6J_dw3rQ5PLu_5dJm7u6tezsvx0F1cO1EHwAKHBIoearwP-B4y1xgDyErVgy-x6GMSOPHnD4m8L8vMcXkhso0M-voiHLeF9hX8zrR1PLtxL0Otsk6WDSaj-LhwgQhSmt2cWgiMIzxg"
# 回显内容
{"errcode":60020,"errmsg":"not allow to access from your ip, hint: [1744441094419152176073175], from ip: 36.112.118.106, more info at https://open.work.weixin.qq.com/devtool/query?e=60020"}
最终汇总获取信息：
- to_party：4
- agent_id：1000005
- api_secret：kqi1OyauWVz5HJvNSV268ORZQ8bANHlAV4l5j9BqWqc
- corp_id：wwe897498df31cc026
- 企业可信IP：36.112.118.106
- 访问令牌：hgDdVMMPDvoIIoJkOiLfwg7aFmtOj03xmEupfZujxPeYdg4WmFNOs8rkdf3_0yGgFFPAOUbhOn9r6J_dw3rQ5PLu_5dJm7u6tezsvx0F1cO1EHwAKHBIoearwP-B4y1xgDyErVgy-x6GMSOPHnD4m8L8vMcXkhso0M-voiHLeF9hX8zrR1PLtxL0Otsk6WDSaj-LhwgQhSmt2cWgiMIzxg
1.2 告警配置¶
[root@master01 7]# vim alertmanager-configmap-wechat.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: alertmanager-config
namespace: monitor
data:
alertmanager.yml: |-
global:
resolve_timeout: 1m
#邮箱服务器的SMTP主机配置，个人邮箱使用smtp.qq.com:465，企业邮箱使用smtp.exmail.qq.com:465
smtp_smarthost: 'smtp.qq.com:465'
#发送邮件主题
smtp_from: 'wechat@wechat.cn'
#登录用户名
smtp_auth_username: '1904763431@qq.com'
#此处的auth password是邮箱的第三方登录授权密码，而非用户密码
smtp_auth_password: 'xdjdwczivdfpcbhj'
#有些邮箱需要开启此配置，这里使用的是企微邮箱，仅做测试，不需要开启此功能。
smtp_require_tls: false
templates:
- '/etc/alertmanager/*.tmpl'
route:
group_by: ['env','instance','type','group','job','alertname','cluster']
group_wait: 10s
group_interval: 2m
repeat_interval: 10m
receiver: 'email'
routes:
- receiver: 'wechat'
match:
severity: critical
receivers:
- name: 'email'
email_configs:
- to: 'xiaozhang_vip123@163.com'
send_resolved: true
html: '{{ template "email.to.html" . }}'
- name: 'wechat'
wechat_configs:
- corp_id: 'wwe897498df31cc026'
to_party: '4'
to_user: '@all'
agent_id: 1000005
api_secret: 'kqi1OyauWVz5HJvNSV268ORZQ8bANHlAV4l5j9BqWqc'
send_resolved: true
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']
wechat.tmpl: |-
{{ define "wechat.default.message" }}
{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $index, $alert := .Alerts -}}
{{- if eq $index 0 }}
========= 监控报警 =========
告警状态：{{ .Status }}
告警级别：{{ .Labels.severity }}
告警类型：{{ $alert.Labels.alertname }}
故障主机: {{ $alert.Labels.instance }}
告警主题: {{ $alert.Annotations.summary }}
告警详情: {{ $alert.Annotations.message }}{{ $alert.Annotations.description}};
触发阀值：{{ .Annotations.value }}
故障时间: {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
========= = end = =========
{{- end }}
{{- end }}
{{- end }}
{{- if gt (len .Alerts.Resolved) 0 -}}
{{- range $index, $alert := .Alerts -}}
{{- if eq $index 0 }}
========= 告警恢复 =========
告警类型：{{ .Labels.alertname }}
告警状态：{{ .Status }}
告警主题: {{ $alert.Annotations.summary }}
告警详情: {{ $alert.Annotations.message }}{{ $alert.Annotations.description}};
故障时间: {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
恢复时间: {{ ($alert.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
{{- if gt (len $alert.Labels.instance) 0 }}
实例信息: {{ $alert.Labels.instance }}
{{- end }}
========= = end = =========
{{- end }}
{{- end }}
{{- end }}
{{- end }}
email.tmpl: |-
{{ define "email.from" }}xxx.com{{ end }}
{{ define "email.to" }}xxx.com{{ end }}
{{ define "email.to.html" }}
{{- if gt (len .Alerts.Firing) 0 -}}
{{ range .Alerts }}
========= 监控报警 =========<br>
告警程序: prometheus_alert <br>
告警级别: {{ .Labels.severity }} <br>
告警类型: {{ .Labels.alertname }} <br>
告警主机: {{ .Labels.instance }} <br>
告警主题: {{ .Annotations.summary }} <br>
告警详情: {{ .Annotations.description }} <br>
触发时间: {{ .StartsAt.Format "2006-01-02 15:04:05" }} <br>
========= = end = =========<br>
{{ end }}{{ end -}}
{{- if gt (len .Alerts.Resolved) 0 -}}
{{ range .Alerts }}
========= 告警恢复 =========<br>
告警程序: prometheus_alert <br>
告警级别: {{ .Labels.severity }} <br>
告警类型: {{ .Labels.alertname }} <br>
告警主机: {{ .Labels.instance }} <br>
告警主题: {{ .Annotations.summary }} <br>
告警详情: {{ .Annotations.description }} <br>
触发时间: {{ .StartsAt.Format "2006-01-02 15:04:05" }} <br>
恢复时间: {{ .EndsAt.Format "2006-01-02 15:04:05" }} <br>
========= = end = =========<br>
{{ end }}{{ end -}}
{{- end }}
# 应用
[root@master01 7]# kaf alertmanager-configmap-wechat.yaml
测试验证：
## 匹配如上webhook标签：severity":"critical"
$ curl -XPOST -H 'Content-Type: application/json' http://alertmanager.zhang-qing.com/api/v1/alerts -d '[{"labels":{"severity":"critical"},"annotations":{"summary":"This is a test alert"}}]'
# 回显内容如下，同时企业微信会有告警信息
{"status":"success"}
首次告警触发后，等待1分钟后，会自动恢复
告警触发：
#关掉node-exporter的容器
[root@iZ2zei8khbswwoi09foupeZ ~]# docker ps | grep node
726b455c0f35 registry.cn-hangzhou.aliyuncs.com/abroad_images/node-exporter:latest "/bin/node_exporter …" 19 hours ago Up 19 hours 0.0.0.0:9100->9100/tcp, [::]:9100->9100/tcp flamboyant_mccarthy
[root@iZ2zei8khbswwoi09foupeZ ~]# docker stop flamboyant_mccarthy
