Prometheus 监控分为两种：
- 白盒监控
- 黑盒监控
白盒监控：是指我们日常监控主机的资源用量、容器的运行状态的运行数据。
黑盒监控：常见的黑盒监控包括 HTTP探针 、 TCP探针 、 Dns、 Icmp等用于检测站点、 服务的可访问性、服务的连通性，以及访问效率等。
两者比较：
- 黑盒监控是以故障为导向当故障发生时，黑盒监控能快速发现故障。
- 白盒监控则侧重于主动发现或者预测潜在的问题。
一个完善的监控目标是要能够从白盒的角度发现潜在问题，能够在黑盒的角度快速发现 已经发生的问题。
目前支持的应用场景：
- ICMP 测试
- 主机探活机制
- TCP 测试
- 业务组件端口状态监听
- 应用层协议定义与监听
- HTTP 测试
- 定义 Request Header 信息
- 判断 Http status / Http Respones Header / Http Body 内容
- POST 测试
- 接口联通性
- SSL 证书过期时间
环境准备工作
下面监控service时会用到该应用
cat java.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: springboot
spec:
replicas: 2
selector:
matchLabels:
app: springboot
template:
metadata:
labels:
app: springboot
spec:
containers:
- name: springboot
image: registry.cn-hangzhou.aliyuncs.com/abroad_images/springboot:v1-aming
resources:
limits:
memory: "1Gi"
cpu: "1"
requests:
memory: "128Mi"
cpu: "100m"
ports:
- containerPort: 8080
name: web
livenessProbe:
httpGet:
port: web
path: /apptwo
timeoutSeconds: 2
periodSeconds: 30
readinessProbe:
tcpSocket:
port: web
initialDelaySeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: springboot
spec:
type: ClusterIP
selector:
app: springboot
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
# 应用
kaf java.yaml
一、Blackbox Exporter 部署¶
Exporter Configmap 定义，可以参考下面两个链接
https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md
https://github.com/prometheus/blackbox_exporter/blob/master/example.yml
首先得声明一个 Blackbox 的 Deployment，并利用 Configmap 来为 Blackbox 提供配置文件。
Configmap：
参考 BlackBox Exporter 的 Github 提供的 示例配置文件
[root@master01 7]# vim blackbox-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: blackbox-exporter
namespace: monitor
labels:
app: blackbox-exporter
data:
blackbox.yml: |-
modules:
## ----------- DNS 检测配置 -----------
dns_tcp:
prober: dns
dns:
transport_protocol: "tcp"
preferred_ip_protocol: "ip4"
query_name: "kubernetes.default.svc.cluster.local" # 用于检测域名可用的网址
query_type: "A"
## ----------- TCP 检测模块配置 -----------
tcp_connect:
prober: tcp
timeout: 5s
## ----------- ICMP 检测配置 -----------
icmp:
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: "ip4"
## ----------- HTTP GET 2xx 检测模块配置 -----------
http_get_2xx:
prober: http
timeout: 10s
http:
method: GET
preferred_ip_protocol: "ip4"
valid_http_versions: ["HTTP/1.1","HTTP/2"]
valid_status_codes: [200] # 验证的HTTP状态码,默认为2xx
no_follow_redirects: false # 是否不跟随重定向
## ----------- HTTP GET 3xx 检测模块配置 -----------
http_get_3xx:
prober: http
timeout: 10s
http:
method: GET
preferred_ip_protocol: "ip4"
valid_http_versions: ["HTTP/1.1","HTTP/2"]
valid_status_codes: [301,302,304,305,306,307] # 验证的HTTP状态码,默认为2xx
no_follow_redirects: false # 是否不跟随重定向
## ----------- HTTP POST 监测模块 -----------
http_post_2xx:
prober: http
timeout: 10s
http:
method: POST
preferred_ip_protocol: "ip4"
valid_http_versions: ["HTTP/1.1", "HTTP/2"]
#headers: # HTTP头设置
# Content-Type: application/json
#body: '{}' # 请求体设置
Deployment：
[root@master01 7]# vim blackbox-exporter.yaml
apiVersion: v1
kind: Service
metadata:
name: blackbox-exporter
namespace: monitor
labels:
k8s-app: blackbox-exporter
spec:
type: ClusterIP
ports:
- name: http
port: 9115
targetPort: 9115
selector:
k8s-app: blackbox-exporter
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: blackbox-exporter
namespace: monitor
labels:
k8s-app: blackbox-exporter
spec:
replicas: 1
selector:
matchLabels:
k8s-app: blackbox-exporter
template:
metadata:
labels:
k8s-app: blackbox-exporter
spec:
containers:
- name: blackbox-exporter
image: registry.cn-hangzhou.aliyuncs.com/abroad_images/blackbox-exporter:v0.21.0
imagePullPolicy: IfNotPresent
args:
- --config.file=/etc/blackbox_exporter/blackbox.yml
- --web.listen-address=:9115
- --log.level=info
ports:
- name: http
containerPort: 9115
resources:
limits:
cpu: 200m
memory: 256Mi
requests:
cpu: 100m
memory: 50Mi
livenessProbe:
tcpSocket:
port: 9115
initialDelaySeconds: 5
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 9115
initialDelaySeconds: 5
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
volumeMounts:
- name: config
mountPath: /etc/blackbox_exporter
volumes:
- name: config
configMap:
name: blackbox-exporter
defaultMode: 420
# 部署
[root@master01 7]# kubectl apply -f blackbox-configmap.yaml
[root@master01 7]# kubectl apply -f blackbox-exporter.yaml
# 查看部署后的资源
[root@master01 7]# kg all -nmonitor |grep blackbox
pod/blackbox-exporter-8765d9478-8tl4z 1/1 Running 0 37s
service/blackbox-exporter ClusterIP 192.168.208.118 <none> 9115/TCP 37s
deployment.apps/blackbox-exporter 1/1 1 1 37s
replicaset.apps/blackbox-exporter-8765d9478 1 1 1 37s
定义 BlackBox 在 Prometheus 抓取设置
下面抓取设置，都存放在
prometheus-config.yaml文件中，设置可参考https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml