目录
一.云原生应用监控
1.1云原生组件 etcd 集群监控
1.1.1查看etcd证书
1.1.2创建etcd service
1.1.3创建etcd证书
1.1.4创建ServiceMonitor
1.1.5查看监控数据
1.1.6grafana监控界面展示
1.2云原生 kube-proxy 监控
1.2.1查看端口
1.2.2修改监听地址
1.2.3创建kube-proxy Service
1.2.4创建 kube-proxy servicemonitor 资源
1.2.5查看监控数据
1.2.6grafana监控界面展示
1.3云原生 calico 网络插件监控
1.3.1开启 Metrics 端口
1.3.2创建podMonitor对象
1.3.3查看监控数据
1.3.4grafana监控界面展示
1.4云原生nginx-ingress监控
1.4.1开启 Metrics 端口
1.4.2修改service monitor 资源
1.4.3查看监控数据
1.4.4grafana监控界面展示
二.非云原生应用监控
2.1 Linux 主机监控
2.1.1检查暴露服务
2.1.2查看监控数据
2.1.3grafana监控界面展示
2.2非云原生应用 Redis 集群监控
2.2.1部署redis集群
2.2.2配置rbac权限
2.2.3查看监控数据
2.2.4grafana监控界面展示
2.3非云原生应用 NGINX 监控
2.3.1编辑configmap文件
2.2.2上传nginx-exporter镜像
2.3.3部署Nginx资源
2.3.4创建service
2.3.5创建servicemonitor
2.3.6查看监控数据
2.3.7grafana监控界面展示
一.云原生应用监控
1.1云原生组件 etcd 集群监控
1.1.1查看etcd证书
说明：在控制面板查看证书
grep -E "key-file|cert-file" /etc/kubernetes/manifests/etcd.yaml查看端口：
ss -tunlp | grep etcd测试etcd能否通过metric接口暴露出去
curl -s \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://192.168.7.163:2379/metrics -k | tail -11.1.2创建etcd service
1.编写yaml文件
vim etcd-service.yml添加以下内容：
--- apiVersion: v1 kind: Endpoints metadata: labels: app: etcd-prom name: etcd-prom namespace: kube-system subsets: - addresses: - ip: 192.168.7.163 ports: - name: https-metrics port: 2379 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app: etcd-prom name: etcd-prom namespace: kube-system spec: ports: - name: https-metrics port: 2379 protocol: TCP targetPort: 2379 type: ClusterIP2.运行yaml文件
kubectl apply -f etcd-service.yml kubectl get svc etcd-prom -o wide -n kube-system3.测试通过svc抓取监控
curl -s --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key https://10.101.115.69:2379/metrics -k | tail -11.1.3创建etcd证书
说明：确保k8s集群安全
kubectl -n monitoring create secret generic etcd-ssl \ --from-file=/etc/kubernetes/pki/etcd/ca.crt \ --from-file=/etc/kubernetes/pki/etcd/server.crt \ --from-file=/etc/kubernetes/pki/etcd/server.key查看证书：
kubectl get secret etcd-ssl -n monitoring绑定证书
vim prometheus-prometheus.yaml添加以下内容：
secrets: - etcd-ssl运行yaml文件：
kubectl apply -f prometheus-prometheus.yaml kubectl exec prometheus-k8s-0 -c prometheus -n monitoring -- ls /etc/prometheus/secrets/etcd-ssl1.1.4创建ServiceMonitor
说明：ServiceMonitor 是 Prometheus Operator 定义的 CRD，用来以"声明式"的方式告诉 Prometheus 该抓取哪些目标（target），取代了手动改prometheus.yml的scrape_configs
1.编写yaml文件
vim etcd-servicemonitor.yaml添加以下内容：
--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: etcd namespace: monitoring labels: app: etcd #标签 app=etcd spec: jobLabel: k8s-etcd-cluster-apps #用 Service 上的哪个 label 作为 job 名 endpoints: - interval: 30s port: https-metrics scheme: https tlsConfig: caFile: /etc/prometheus/secrets/etcd-ssl/ca.crt certFile: /etc/prometheus/secrets/etcd-ssl/server.crt keyFile: /etc/prometheus/secrets/etcd-ssl/server.key insecureSkipVerify: true selector: matchLabels: app: etcd-prom #匹配标签 namespaceSelector: matchNames: - kube-system运行yaml文件
kubectl apply -f etcd-servicemonitor.yaml kubectl get servicemonitor etcd -n monitoring1.1.5查看监控数据
1.1.6grafana监控界面展示
grafana导入etcd监控界面展示地址URL：
https://grafana.com/grafana/dashboards/9733-etcd-for-k8s-cn/1.2云原生 kube-proxy 监控
说明：kube-proxy默认暴露两个端口，一个是10249，在/metrics端点暴露监控指标，另一个是10256，默认监听127.0.0.1
1.2.1查看端口
ss -tunlp | grep kube-proxy测试能否访问资源
curl 127.0.0.1:10249/metrics1.2.2修改监听地址
说明：修改监听端口，监听所有端口。端口为本地，Prometheus如果没有部署在这台机子上，就无法监听到该端口。
kubectl edit configmap kube-proxy -n kube-system添加以下内容：
metricsBindAddress: "0.0.0.0:10249"重启Pod
说明：kube-proxy采用副本的方式构建，一旦删除，会立即重建
kubectl -n kube-system get pod|grep kube-proxy|awk '{system("kubectl -n kube-system delete pod "$1"")}' ss -tunlp | grep kube-proxy1.2.3创建kube-proxy Service
vim kube-proxy-service.yaml添加以下内容：
--- apiVersion: v1 kind: Service metadata: labels: k8s-app: kube-proxy name: kube-proxy namespace: kube-system spec: selector: k8s-app: kube-proxy #筛选标签 type: ClusterIP ports: - name: https-metrics port: 10249 targetPort: 10249 protocol: TCP运行yaml
kubectl apply -f kube-proxy-service.yaml kubectl get svc kube-proxy -n kube-system尝试抓取服务：
curl 10.110.8.62:10249/metrics1.2.4创建kube-proxy servicemonitor 资源
1.编写yaml文件
vim kube-proxy-servicemonitor.yaml添加以下内容：
--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: kube-proxy namespace: monitoring labels: app: kube-proxy #Service Monitor标签 spec: jobLabel: kube-proxy endpoints: - interval: 30s port: https-metrics scheme: http selector: matchLabels: k8s-app: kube-proxy #匹配标签 namespaceSelector: matchNames: - kube-system2.运行yaml文件
kubectl apply -f kube-proxy-servicemonitor.yaml kubectl get servicemonitor kube-proxy -n monitoring1.2.5查看监控数据
1.2.6grafana监控界面展示
说明：prometheus本身自带granfan界面
1.3云原生 calico 网络插件监控
说明：Calico 中最核心的组件就是Felix，它负责设置路由表和 ACL规则等，以便为该主机上的 endpoints 资源正常运行提供所需的网络连接。同时它还负责提供有关网络健康状况的数据（例如，报告配置其主机时发生的错误和问题），这些数据会被写入etcd，以使其对网络中的其他组件和操作人员可见。由此可见，监控calico的核心便是监控felix，felix相当于calico的大脑。
1.3.1开启 Metrics 端口
说明：calico在启动时，默认没有打开Metrics监听
kubectl edit ds calico-node -n kube-system添加以下内容：
spec: containers: - env: - name: FELIX_PROMETHEUSMETRICSENABLED value: "True" - name: FELIX_PROMETHEUSMETRICSPORT value: "9091" ports: - containerPort: 9091 name: http-metrics protocol: TCP1.3.2创建podMonitor对象
说明：ServiceMonitor 通过 Service 发现目标，PodMonitor 直接选 Pod。前者适合有 Service 的服务，后者适合没有 Service 或不想建 Service 的场景。
1.创建yml文件
vim prometheus-podMonitorCalico.yaml添加以下内容：
--- apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: labels: k8s-app: calico-node name: calico-node namespace: monitoring spec: podMetricsEndpoints: - interval: 15s path: /metrics port: http-metrics namespaceSelector: matchNames: - kube-system selector: matchLabels: k8s-app: calico-node #匹配标签2.运行yml
kubectl apply -f prometheus-podMonitorCalico.yaml kubectl get podmonitor calico-node -n monitoring1.3.3查看监控数据
1.3.4grafana监控界面展示
https://grafana.com/grafana/dashboards/12175-felix-dashboard-calico/1.4云原生nginx-ingress监控
1.4.1开启 Metrics 端口
说明：helm部署时默认没有开启metrics采集服务
vim ingress-nginx.yml添加以下内容：
global: image: registry: reg.westos.org controller: image: image: ingress-nginx/controller tag: "v1.13.3" digest: "" digestChroot: "" ingressClassResource: name: nginx default: true service: type: LoadBalancer admissionWebhooks: patch: image: registry: reg.westos.org image: ingress-nginx/kube-webhook-certgen tag: v1.6.3 digest: "" metrics: #开启metrics采集 enabled: true service: enabled: true labels: app: "ingress-nginx-metrics" serviceMonitor: enabled: true namespace: "monitoring" defaultBackend: enabled: true name: defaultbackend image: registry: reg.westos.org image: ingress-nginx/defaultbackend-amd64 tag: "1.5"2.使用helm部署Ingress-nginx
helm -n ingress-nginx upgrade ingress-nginx -f ingress-nginx.yml ingress-nginx-4.13.3.tgz kubectl -n ingress-nginx get svc ingress-nginx-controller-metrics测试
curl 10.107.130.97:10254/metrics1.4.2修改service monitor 资源
说明：helm部署service monitor，修改匹配标签
kubectl -n monitoring edit servicemonitors ingress-nginx-controller添加以下内容：
spec: endpoints: - interval: 30s port: metrics scheme: http namespaceSelector: matchNames: - ingress-nginx selector: matchLabels: app: ingress-nginx-metrics在prometheus监控部署之后，安装的服务会出现权限的问题，需要配置rbac授权(补充)：
kubectl apply -f rbac.yml文件内容如下：
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus-endpoints-view namespace: ingress-nginx rules: - apiGroups: [""] resources: ["endpoints", "pods", "services"] verbs: ["get", "list", "watch"] - apiGroups: ["discovery.k8s.io"] resources: ["endpointslices"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus-k8s-endpoints namespace: ingress-nginx subjects: - kind: ServiceAccount name: prometheus-k8s namespace: monitoring roleRef: kind: Role name: prometheus-endpoints-view apiGroup: rbac.authorization.k8s.io1.4.3查看监控数据
1.4.4grafana监控界面展示
https://grafana.com/grafana/dashboards/21336-nginx-ingress-controller/二.非云原生应用监控
说明：非云原生应用不能使用metric暴露服务端口，需要使用外部的 exporter暴露
2.1 Linux 主机监控
说明：通过kube-prometheus技术栈安装的prometheus默认会以ds的形式在每一个节点上启动一个node-exporter pod，并且会监控一个9100端口。该端口用来采集宿主机的相关监控指标。
2.1.1检查暴露服务
ss -tunlp | grep 9100 kubectl get svc node-exporter -n monitoring -o yaml kubectl get servicemonitor node-exporter -n monitoring -o yaml2.1.2查看监控数据
2.1.3grafana监控界面展示
默认面板：
中文面板
131052.2非云原生应用 Redis 集群监控
2.2.1部署redis集群
1.编辑配置文件
vim redis-values-metrics.yml添加以下内容：
global: imageRegistry: "reg.westos.org" security: allowInsecureImages: true redis: password: "westos" metrics: enabled: true serviceMonitor: enabled: true namespace: "monitoring"2.上传镜像
docker pull bitnami/redis-exporter docker tag bitnami/redis-exporter reg.westos.org/bitnami/redis-exporter3.部署redis集群
helm install redis-cluster -f redis-values-metrics.yml redis-23.2.6.tgz -n redis kubectl get all -n redis测试是否暴露服务：
curl 10.103.174.171:9121/metrics2.2.2配置rbac权限
vim rbac.yml添加以下内容：
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus-endpoints-view namespace: redis rules: - apiGroups: [""] resources: ["endpoints", "pods", "services"] verbs: ["get", "list", "watch"] - apiGroups: ["discovery.k8s.io"] resources: ["endpointslices"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus-k8s-endpoints namespace: redis subjects: - kind: ServiceAccount name: prometheus-k8s namespace: monitoring roleRef: kind: Role name: prometheus-endpoints-view apiGroup: rbac.authorization.k8s.io运行yml
kubectl apply -f rbac.yml2.2.3查看监控数据
2.2.4grafana监控界面展示
7632.3非云原生应用 NGINX 监控
说明：nginx有自己的监控模块stub_status。
2.3.1编辑configmap文件
vim default.conf添加以下内容：
server { listen 80; server_name localhost; location / { root /usr/share/nginx/html; index index.html index.htm; } location /status { stub_status on; } error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/share/nginx/html; } }运行yml文件
kubectl create cm nginx-conf --from-file=./default.conf2.2.2上传nginx-exporter镜像
docker pull nginx/nginx-exporter docker tsg nginx/nginx-exporter reg.westos.org/library/nginx-exporter docker push reg.westos.org/library/nginx-exporter2.3.3部署Nginx资源
vim deploy-nginx.yml添加以下内容：
--- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-server spec: selector: matchLabels: app: nginx replicas: 3 template: metadata: labels: app: nginx annotations: prometheus.io/scrape: 'true' prometheus.io/port: '9113' spec: containers: - name: nginx image: reg.westos.org/nginx/nginx:1.29 ports: - containerPort: 80 volumeMounts: - name: nginxconf mountPath: /etc/nginx/conf.d - name: nginx-exporter image: reg.westos.org/nginx/nginx-prometheus-exporter:1.5.1 args: - '-nginx.scrape-uri=http://localhost/status' resources: limits: memory: 128Mi cpu: 500m ports: - containerPort: 9113 volumes: - name: nginxconf configMap: name: nginx-confkubectl apply -f deploy-nginx.yml kubectl get pod2.3.4创建service
vim service-nginx.yml添加以下内容：
--- apiVersion: v1 kind: Service metadata: name: webapp labels: app: nginx spec: selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 name: web - protocol: TCP port: 9113 targetPort: 9113 name: metrics运行yml文件
kubectl apply -f service-nginx.yml kubectl get svc webapp查看抓取数据
curl 10.108.234.20/status curl 10.108.234.20:91132.3.5创建servicemonitor
vim nginx-servicemonitor.yml添加以下内容：
--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: nginx-exporter namespace: monitoring labels: k8sapp: nginx-exporter namespace: monitoring spec: jobLabel: k8s-app endpoints: - port: metrics interval: 30s scheme: http path: /metrics selector: matchLabels: app: nginx namespaceSelector: matchNames: - default运行yml文件
kubectl apply -f nginx-servicemonitor.yml2.3.6查看监控数据
2.3.7grafana监控界面展示
https://github.com/nginx/nginx-prometheus-exporter/blob/main/grafana/dashboard.json
