Kubernetes Metric Server学习参考Metric Server环境准备rootmaster30:~# kubectl create ns metricrootmaster30:~# kubectl config set-context --current --namespace metricMetrics-Server 概述我们在使用 Kubernetes 中过程中面临的问题如何监控 node 计算资源使用情况如何监控 pod 计算资源使用情况如何根据 pod 计算资源使用情况自动扩展我们可以使用 Metrics-Server监控node 和 pod 计算资源使用情况。Metrics Server 是 Kubernetes 内置自动缩放管道的可扩展、高效的容器资源指标来源。Metrics Server 通过 Kubelet 收集资源指标并通过 Metrics API 将它们公开在 Kubernetes apiserver 中供 Horizontal Pod Autoscaler 和 Vertical Pod Autoscaler 使用。Metrics API 还可以通过 kubectl top 访问从而更轻松地调试自动缩放管道。Metrics Server 不适用于非自动缩放目的。 请勿将其用于将指标转发到监控解决方案或作为监控解决方案指标的来源。 在这种情况下请直接从 Kubelet /metrics/resource 端点收集指标。指标服务器提供适用于大多数集群的单一部署请参阅要求。快速自动缩放每 15 秒收集一次指标。资源效率集群中每个节点使用 1 mili 核心 CPU 和 2 MB 内存。可扩展支持多达 5,000 个节点集群。Metrics-Server 部署项目地址 kubernetes-sigs/metrics-server# 下载 Metrics-Serverrootmaster30:~# wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.7.1/components.yaml#内网下载地址rootmaster30 ~17:02:45# wget -O components.yaml http://192.168.46.200/class/course-materials/softwares/stage03/metrics-server-components-v0.7.1.yaml# 修改 Metrics-Server不校验tls内网下载的yaml文件已经修改完成rootmaster30:~# sed -i /metric-resolution/a\ - --kubelet-insecure-tls components.yaml# 下载镜像rootmaster30 ~17:03:15# grep image: components.yamlimage: hub.laoma.cloud/metrics-server/metrics-server:v0.7.1#修改镜像为内网的rootmaster30 ~17:03:48# sed -i s/registry.k8s.io/hub.laoma.cloud/g components.yamlrootmaster30 ~17:04:26# grep image: components.yamlimage: hub.laoma.cloud/metrics-server/metrics-server:v0.7.1# 部署 Metrics-Serverrootmaster30 ~17:04:29# kubectl apply -f components.yamlserviceaccount/metrics-server created clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created clusterrole.rbac.authorization.k8s.io/system:metrics-server created rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created service/metrics-server created deployment.apps/metrics-server created apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created# 查看 Metrics-Server 状态rootmaster30 ~17:10:38# kubectl get pods -n kube-system |grep metricsmetrics-server-5b98f887f4-qw57k1/1 Running06m 45sMetrics-Server 使用查看node状态rootmaster30 ~17:10:50# kubectl top nodeNAME CPU(cores)CPU% MEMORY(bytes)MEMORY% master30.ggg.cloud 114m5% 1893Mi50% worker31.ggg.cloud 35m1% 1053Mi27% worker32.ggg.cloud 66m3% 1121Mi29%查看 pod 状态rootmaster30 ~17:11:05# kubectl top pods -n kube-systemNAME CPU(cores)MEMORY(bytes)calico-kube-controllers-5b778fd8ff-8mvb8 3m 78Mi calico-node-6bltt 32m 178Mi calico-node-lv2hk 24m 176Mi calico-node-p4pp7 25m 194Mi coredns-5485cd7479-8cswc 1m 50Mi coredns-5485cd7479-rbt7z 1m 22Mi etcd-master30.ggg.cloud 14m 96Mi kube-apiserver-master30.ggg.cloud 32m 371Mi kube-controller-manager-master30.ggg.cloud 9m 127Mi kube-proxy-8dppw 7m 71Mi kube-proxy-cbzbt 1m 72Mi kube-proxy-z8zv5 8m 71Mi kube-scheduler-master30.ggg.cloud 2m 65Mi metrics-server-5b98f887f4-qw57k 2m 16Mi单位1cpu1000m扩容和减容控制扩容和缩容时间参数1. 指标采样周期--horizontal-pod-autoscaler-sync-period duration默认15sHPA 每 15 秒拉取一次 metrics-server 指标。2. 启动就绪窗口期--horizontal-pod-autoscaler-initial-readiness-delay duration默认30sPod 刚启动前 30 秒视为 “启动中”不参与 HPA 计算3. 缩容稳定窗口期--horizontal-pod-autoscaler-downscale-stabilization duration默认5m0s5 分钟缩容冷却时间。持续高负载 → 扩容每15s采集一次 CPU/内存指标Pod 刚启动前 30 秒视为 “启动中”不参与 HPA 计算。一次扩容动作完成后再次等待 45s 冷却才能下一次扩容 结论K8s 1.30 扩容最小间隔 45s持续低负载 → 缩容同样 15s 采样指标长期低于阈值需要满足300s5分钟稳定低位才会触发缩容每次缩容后再次锁定 5 分钟设置扩容和减容时间参数扩容和减容pod数量由kube-controller-manager管理如需调快/调慢直接修改 controller-manager 启动参数即可。扩容和缩容pod是有冷却时间的目的是防止流量抖动、瞬间峰值导致频繁炸裂扩容。为了演示扩容和缩容效果这里设置扩容冷却时间为30s1020缩容冷却时间为60秒。rootmaster30:~# vim /etc/kubernetes/manifests/kube-controller-manager.yamlspec:containers:-command:-kube-controller-manager---allocate-node-cidrstrue......# 增加下面三行参数---horizontal-pod-autoscaler-sync-period10s---horizontal-pod-autoscaler-initial-readiness-delay20s---horizontal-pod-autoscaler-downscale-stabilization60s保存退出静态 Pod 会自动重启生效。Horizontal Pod Autoscaler学习参考hpaHPA 作用控制器将从一系列APImetrics.k8s.io、custom.metrics.k8s.io和external.metrics.k8s.io中获取度量值。metrics.k8s.ioAPI 通常由Metrics Server提供。根据HPA中定义的指标动态实现控制器中pod伸缩支持replication controller、deployment和replicaset。适用场景Web/API 服务流量波动明显​ 前端、Nginx、网关、后端接口。白天流量高、夜间低突发秒杀、活动引流。​ → 负载上来自动加 Pod 扛流量低谷自动缩副本省钱。消息消费类任务MQ 消费者​ Kafka/RabbitMQ 消息堆积量波动大消息多了扩容 consumer堆积下降后缩容。​ → 一般用**自定义指标消息队列 pending 数量**做 HPA而不是单纯 CPU。计算型短任务、并发 Job​ 并发请求、轻量计算服务请求并发量随时变化不需要人工改 replicas。多租户 / 弹性 SaaS 平台​ 租户访问量随时间起伏自动弹性保障 SLA避免固定副本过多浪费资源。核心特点不修改 Pod 配置只改副本数实时、秒级扩缩最常用、最稳定支持指标CPU 使用率内存使用率自定义指标Prometheus 对接QPS、延迟等基于 CPU 使用率伸缩准备资源rootmaster30 ~17:11:20# kubectl create deployment web --imagehub.laoma.cloud/library/nginxrootmaster30 ~17:11:47# kubectl get podsNAME READY STATUS RESTARTS AGE web-759dfd9847-tcqkp1/1 Running06s创建 hpaUsage: kubectl autoscale(-f FILENAME|TYPE NAME|TYPE/NAME)[--minMINPODS]--maxMAXPODS[--cpu-percentCPU][options]rootmaster30 ~17:11:53# kubectl autoscale deployment web --max5 --min2 --cpu-percent80horizontalpodautoscaler.autoscaling/web autoscaled rootmaster30 ~17:12:32# kubectl get hpa web -o yaml | tee hpa-cpu.yaml# 省略部分不重要配置apiVersion:autoscaling/v2kind:HorizontalPodAutoscalermetadata:name:webspec:maxReplicas:5metrics:-resource:name:cputarget:averageUtilization:80type:Utilizationtype:ResourceminReplicas:2scaleTargetRef:apiVersion:apps/v1kind:Deploymentname:web# 稍等片刻创建一个新的podrootmaster30 ~17:13:06# kubectl get podsNAME READY STATUS RESTARTS AGE web-759dfd9847-cr2xh1/1 Running036s web-759dfd9847-tcqkp1/1 Running096s rootmaster30 ~17:13:23# kubectl get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE web Deployment/web cpu:unknown/80%25266s# CPU目标为unknown要想看到 HPA 的 TARGETS 值必须满足2个条件安装 metric server。为 pod 设定资源限制。rootmaster30:~# kubectl edit deployments.apps web# 修改spec.template.spec.containers.[N].resources属性添加limit属性如下resources: limits: cpu: 100m memory: 200Mi# 再次查看hparootmaster30 ~17:14:50# kubectl get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE web Deployment/web cpu:0%/80%2522m23s rootmaster30 ~17:14:55# kubectl expose deployment web --port80 --target-port80rootmaster30 ~17:15:16# kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)AGE web ClusterIP10.99.48.67none80/TCP 5s压力测试# 打开一个监控窗口rootmaster30:~# \whiletruedoclear;kubectl get hpa;echokubectl get pod;echokubectltoppodssleep1done# 上 CPU 压力rootworker31 ~11:20:08# apt install -y apache2-utilsrootworker31 ~17:17:08# while true ;do ab -n 300000 -c 100 http://10.99.48.67/;sleep 1;done# -n 300000总请求数# -c 100每次并发数CPU负载 cpu: 98%NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE web Deployment/web cpu:98%/80%2557m8s NAME READY STATUS RESTARTS AGE web-7cf9fd6bcb-2slz71/1 Running05m16s web-7cf9fd6bcb-6njpj1/1 Running052s web-7cf9fd6bcb-98vg41/1 Running022s web-7cf9fd6bcb-cwvw41/1 Running05m17s web-7cf9fd6bcb-tgkpc1/1 Running082s NAME CPU(cores)MEMORY(bytes)web-7cf9fd6bcb-2slz7 101m 3Mi web-7cf9fd6bcb-6njpj 100m 3Mi web-7cf9fd6bcb-98vg4 93m 3Mi web-7cf9fd6bcb-cwvw4 96m 3Mi web-7cf9fd6bcb-tgkpc 97m 3Mi观察过程随着pod CPU使用率上升自动扩展pod数量。正常情况30秒左右创建一个新pod即使pod CPU使用率超过阈值但pod数量不会超过5。停止压力测试负载降低后pod数量会逐步减少为2。清理资源# 删除 hparootmaster30:~# kubectl delete hpa web# 删除 deploymentrootmaster30:~# kubectl delete deployment web# 保留 svc后续使用基于 Mem 使用率伸缩我们仍然以nginx应用实践。想要Nginx 内存涨要访问会占用内存的页面。最简单方法让 Nginx 返回一个超大响应体。准备资源# worker节点创建big.imgrootworker31 ~10:01:36# mkdir /wwwrootworker31 ~10:13:13# dd if/dev/zero of/www/big.img bs1M count200rootworker32 ~10:01:38# mkdir /wwwrootworker32 ~10:13:19# dd if/dev/zero of/www/big.img bs1M count200rootworker32 ~10:13:19# vim deployment.yamlapiVersion:apps/v1kind:Deploymentmetadata:name:webspec:replicas:1selector:matchLabels:app:webtemplate:metadata:labels:app:webspec:containers:-name:nginximage:nginxports:-containerPort:80volumeMounts:-name:big-filemountPath:/usr/share/nginx/htmlresources:limits:cpu:100mmemory:200Mivolumes:-name:big-filehostPath:path:/wwwrootmaster30 ~10:13:34# kubectl apply -f deployment.yaml创建 hparootmaster30 ~10:13:42# vim hpa-mem.yamlapiVersion:autoscaling/v2kind:HorizontalPodAutoscalermetadata:name:webspec:minReplicas:2maxReplicas:5scaleTargetRef:apiVersion:apps/v1kind:Deploymentname:webmetrics:-resource:name:memorytarget:type:Utilization#若扩容效果不明显可以把60改低一点averageUtilization:60type:Resourcerootmaster30 ~10:13:57# kubectl apply -f hpa-mem.yamlrootmaster30 ~10:14:12# kubectl get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE web Deployment/web memory:unknown/60%2506s# 目标值为空# 稍等片刻创建一个新的podrootmaster30 ~10:14:29# kubectl get podsNAME READY STATUS RESTARTS AGE web-88cff78bb-227l21/1 Running048s web-88cff78bb-gfx741/1 Running03s# 稍等一会再次查看rootmaster30 ~10:14:45# kubectl get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE horizontalpodautoscaler.autoscaling/web Deployment/web memory:7%/60%2523m4s压力测试# 打开一个监控窗口rootmaster30 ~10:21:08# watch -n 4 kubectl get hpa;echo;kubectl top pods;echo;kubectl get pods# 上 MEM 压力rootmaster30 ~10:19:33# kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)AGE web ClusterIP10.103.195.186none80/TCP 5s rootmaster30 ~10:19:38# while true ;do ab -n 300000 -c 100 http://10.103.195.186/big.img;sleep 1;done负载上来了Every4.0s: kubectl get hpa;echo;kubectltoppods;echo;kubectl get pods master30.ggg.cloud: Thu Aug1310:21:572026NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE web Deployment/web memory:86%/60%2547m45s NAME CPU(cores)MEMORY(bytes)web-88cff78bb-227l2 8m 98Mi web-88cff78bb-gfx74 6m 102Mi NAME READY STATUS RESTARTS AGE web-88cff78bb-227l21/1 Running08m16s web-88cff78bb-gfx741/1 Running07m31s直接新建了两个podsEvery 4.0s: kubectl get hpa;echo;kubectl top pods;echo;kubectl get pods master30.ggg.cloud: Thu Aug 13 10:21:15 2026 NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE web Deployment/web memory: 47%/60% 2 5 4 7m4s NAME CPU(cores) MEMORY(bytes) web-88cff78bb-227l2 11m 188Mi web-88cff78bb-gfx74 12m 184Mi web-88cff78bb-lfz77 0m 3Mi web-88cff78bb-tbp7w 0m 3Mi NAME READY STATUS RESTARTS AGE web-88cff78bb-227l2 1/1 Running 0 7m34s web-88cff78bb-gfx74 1/1 Running 0 6m49s web-88cff78bb-lfz77 1/1 Running 0 48s web-88cff78bb-tbp7w 1/1 Running 0 48s观察过程随着pod MEM使用率上升自动扩展pod数量。即使pod MEM使用率超过阈值但pod数量不会超过5。清理资源# 删除 hparootmaster30:~# kubectl delete hpa web# 删除 deploymentrootmaster30:~# kubectl delete deployment web# 删除 svcrootmaster30:~# kubectl delete svc webVertical Pod AutoscalerVPA 使用场景不多这里不做实践演示。VPA 作用控制器将从一系列APImetrics.k8s.io、custom.metrics.k8s.io和external.metrics.k8s.io中获取度量值metrics.k8s.ioAPI 通常由Metrics Server提供。根据VPA中定义的指标动态调整 Pod 的 requests 和 limits。VPA 适用场景资源配置不合理的服务长期运行、流量稳定的服务Java、Go 等内存占用固定的应用不适合频繁扩缩的服务。VPA 核心特点不改变 Pod 数量自动给 Pod推荐 / 设置 CPU / 内存生效需要重启 Pod默认比 HPA 少用因为有侵入性HPA 与 VPA 对比项目HPA 横向扩缩容VPA 纵向扩缩容全称HorizontalVertical扩缩方式增加 / 减少 Pod 数量调整 CPU / 内存 request/limit是否重启 Pod❌ 不重启✅ 一般需要重启适合负载流量波动大资源配置不合理生效速度快慢稳定性高中生产使用非常普遍较少能否一起开不能同时开会冲突-支持资源CPU、内存、自定义CPU、内存环境清理rootmaster30:~# kubectl delete ns metric