ARTICLE · INTELLIGENCE

战地情报 · 详情页

来自尧图项目组的一线实战观察与深度解析

rancher v2.6.5 部署rancher-monitor (已监控k8s v1.23.12)

rancher v2.6.5 部署rancher-monitor (已监控k8s v1.23.12) 安装说明rancher需要安装的rancher monitoring和prometheus的node exporter rancher monitoring用来监控ranhcer里各个pods的运行状态node exporter则用来监控裸机的运行状态 rancher monitoring和prometheus的node exporter相关镜像及安装步骤开发者本地安装时借用claude code完成 未实际手动安装过输出的手册可以丢给大模型通读并根据指导安装 详见文件夹“rancher montoring及prometheus export”部署操作现场操作手册 —— Rancher Monitoring 离线安装(无外网 / 无 AI 辅助版)每一步都给了:做什么、敲什么命令、正常应该看到什么、不对怎么办。全部命令可整行复制。示例里的harbor.10.0.0.5:30003一律换成现场 Harbor 真实地址。0. 先弄清三个角色角色要求干什么A 机(推镜像机)有 docker,能访问 Harbor只干一件事:跑load-push.sh推 20 个镜像B 机(操作机)能连上 Rancher 集群跑预检、helm 安装、验收裸机们能被 B 机 ssh(或人肉拷文件)装 node_exporterA 机和 B 机可以是同一台。B 机推荐有 kubectl;没有也能干(手册里标了代替办法)。1. 交付包结构(拷贝时别落下东西)rancher-monitoring-offline/ ├── bin/ │ └── helm # ★ 自带 helm(Linux 静态版,现场不用装) ├── chart/ │ ├── rancher-monitoring-crd-103.1.1up45.31.1.tgz # CRD 包(先装) │ ├── rancher-monitoring-103.1.1up45.31.1.tgz # 主包(后装) │ └── values-offline.yaml # 唯一要改的配置(填 Harbor 地址) ├── images/ │ ├── monitoring-images.tar # 2.6G,20 个镜像(拷贝耗时正常) │ ├── imagelist.txt │ └── load-push.sh # A 机跑 ├── baremetal/ # 裸机 node_exporter 三件套 └── sop/ ├── 现场操作手册.md # 本文件 ├── preflight.sh # 预检脚本 └── README.md # 快速版流程(熟手看这个)三条铁律:文件落到 Linux 后永远不要用 Windows 记事本改(会把换行符改坏,报$\r: command not found)。要改就用vi或sed。敲命令前cd到对应目录,路径别带空格。每一步的预期输出对不上就停下翻排错表,不要凭感觉连敲命令。2. 阶段一:预检(B 机,5 分钟)cdrancher-monitoring-offlinechmodx bin/helm# 给自带 helm 加执行权限(必做,Linux 拷贝后默认没有)bashsop/preflight.sh# 基础预检HARBORharbor.10.0.0.5:30003bashsop/preflight.sh# 带 Harbor 一起查(推荐)实际操作记录-0-自查### 0、自查[rootxx rancher-monitoring-offline]# HARBOR10.xx.xx.xx9:7000 bash sop/preflight.sh1/5 本机工具✅ helm(交付包自带): ✅ kubectl(系统已装)✅ docker(系统已装): Docker version20.10.18, build b40c2f62/5 交付包完整性✅ 存在 chart/rancher-monitoring-103.1.1up45.31.1.tgz ✅ 存在 chart/rancher-monitoring-crd-103.1.1up45.31.1.tgz ✅ 存在 chart/values-offline.yaml ✅ 存在 images/monitoring-images.tar ✅ 存在 images/imagelist.txt ✅ 存在 images/load-push.sh ⚠️ values-offline.yaml 里还是占位符 __HARBOR__(正常,第3步安装前才替换)3/5 集群连通性sop/preflight.sh: line69: NODE: unbound variable[rootxx rancher-monitoring-offline]##这里报错了但还是继续往下走了。预期:结尾汇总一行❌ 0 项失败。有 ❌ 就照脚本输出里的提示处理,处理不了直接拍照进排错表对号入座。B 机没有 kubectl 也能继续:预检里的集群检查改在Rancher UI → 集群 → 右上角 kubectl 图标(网页终端)里人工确认所有节点 Ready。开工前还差两件人工事:现场 Harbor 上建好名为rancher的公开项目(浏览器进 Harbor → 项目 → 新建)Harbor 账号密码拿到手(A 机 docker login 要用)3. 阶段二:推镜像(A 机,约 10~20 分钟)# 1. 登录(输密码)dockerlogin harbor.10.0.0.5:30003# 2. 推送(在 images/ 目录下)cdimagesHARBORharbor.10.0.0.5:30003bashload-push.sh预期输出:每行一个镜像,20 行全OK,结尾✅ 全部推送完成。验证(30 秒,别跳过):从 Harbor 拉回一个最小的——dockerpull harbor.10.0.0.5:30003/rancher/mirrored-library-busybox:1.31.1能拉下来 推送真实落库了。实际操作记录-推送镜像到harbor###1、[rootxx images]# docker login 10.xx.xx.xx9:7000Authenticating with existing credentials... WARNING!Your password will be stored unencryptedin/root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded[rootxx images]####2、导入推送镜像到harbor[rootxx images]# HARBOR10.xx.xx.xx9:7000 bash load-push.shtxt1)登录 Harbor:10.xx.xx.xx9:7000Authenticating with existing credentials... WARNING!Your password will be stored unencryptedin/root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded2)载入离线镜像包 monitoring-images.tar(可能要几分钟)Loaded image: registry.rancher.com/rancher/mirrored-ingress-nginx-kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6 Loaded image: registry.rancher.com/rancher/mirrored-kiwigrid-k8s-sidecar:1.24.6 Loaded image: registry.rancher.com/rancher/mirrored-prometheus-prometheus:v2.45.0 Loaded image: registry.rancher.com/rancher/mirrored-thanos-thanos:v0.30.2 Loaded image: registry.rancher.com/rancher/kubectl:v1.20.2 Loaded image: registry.rancher.com/rancher/mirrored-curlimages-curl:7.85.0 Loaded image: registry.rancher.com/rancher/mirrored-grafana-grafana:9.1.5 Loaded image: registry.rancher.com/rancher/mirrored-prometheus-adapter-prometheus-adapter:v0.10.0 Loaded image: registry.rancher.com/rancher/mirrored-prometheus-node-exporter:v1.3.1 Loaded image: registry.rancher.com/rancher/pushprox-client:v0.1.3-rancher2-client Loaded image: registry.rancher.com/rancher/pushprox-proxy:v0.1.3-rancher2-proxy Loaded image: registry.rancher.com/rancher/shell:v0.1.25 Loaded image: registry.rancher.com/rancher/mirrored-grafana-grafana-image-renderer:3.8.0 Loaded image: registry.rancher.com/rancher/mirrored-kube-rbac-proxy:v0.14.0 Loaded image: registry.rancher.com/rancher/mirrored-library-busybox:1.31.1 Loaded image: registry.rancher.com/rancher/mirrored-library-nginx:1.24.0-alpine Loaded image: registry.rancher.com/rancher/mirrored-prometheus-operator-prometheus-config-reloader:v0.65.1 Loaded image: registry.rancher.com/rancher/mirrored-kube-state-metrics-kube-state-metrics:v2.10.1 Loaded image: registry.rancher.com/rancher/mirrored-prometheus-alertmanager:v0.26.0 Loaded image: registry.rancher.com/rancher/mirrored-prometheus-operator-prometheus-operator:v0.65.13)重打 tag 并推送到10.xx.xx.xx9:7000/rancher/...registry.rancher.com/rancher/kubectl:v1.20.2 -10.xx.xx.xx9:7000/rancher/kubectl:v1.20.2... OK registry.rancher.com/rancher/mirrored-curlimages-curl:7.85.0 -10.xx.xx.xx9:7000/rancher/mirrored-curlimages-curl:7.85.0... OK registry.rancher.com/rancher/mirrored-grafana-grafana-image-renderer:3.8.0 -10.xx.xx.xx9:7000/rancher/mirrored-grafana-grafana-image-renderer:3.8.0... OK registry.rancher.com/rancher/mirrored-grafana-grafana:9.1.5 -10.xx.xx.xx9:7000/rancher/mirrored-grafana-grafana:9.1.5... OK registry.rancher.com/rancher/mirrored-ingress-nginx-kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6 -10.xx.xx.xx9:7000/rancher/mirrored-ingress-nginx-kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6... OK registry.rancher.com/rancher/mirrored-kiwigrid-k8s-sidecar:1.24.6 -10.xx.xx.xx9:7000/rancher/mirrored-kiwigrid-k8s-sidecar:1.24.6... OK registry.rancher.com/rancher/mirrored-kube-rbac-proxy:v0.14.0 -10.xx.xx.xx9:7000/rancher/mirrored-kube-rbac-proxy:v0.14.0... OK registry.rancher.com/rancher/mirrored-kube-state-metrics-kube-state-metrics:v2.10.1 -10.xx.xx.xx9:7000/rancher/mirrored-kube-state-metrics-kube-state-metrics:v2.10.1... OK registry.rancher.com/rancher/mirrored-library-busybox:1.31.1 -10.xx.xx.xx9:7000/rancher/mirrored-library-busybox:1.31.1... OK registry.rancher.com/rancher/mirrored-library-nginx:1.24.0-alpine -10.xx.xx.xx9:7000/rancher/mirrored-library-nginx:1.24.0-alpine... OK registry.rancher.com/rancher/mirrored-prometheus-adapter-prometheus-adapter:v0.10.0 -10.xx.xx.xx9:7000/rancher/mirrored-prometheus-adapter-prometheus-adapter:v0.10.0... OK registry.rancher.com/rancher/mirrored-prometheus-alertmanager:v0.26.0 -10.xx.xx.xx9:7000/rancher/mirrored-prometheus-alertmanager:v0.26.0... OK registry.rancher.com/rancher/mirrored-prometheus-node-exporter:v1.3.1 -10.xx.xx.xx9:7000/rancher/mirrored-prometheus-node-exporter:v1.3.1... OK registry.rancher.com/rancher/mirrored-prometheus-operator-prometheus-config-reloader:v0.65.1 -10.xx.xx.xx9:7000/rancher/mirrored-prometheus-operator-prometheus-config-reloader:v0.65.1... OK registry.rancher.com/rancher/mirrored-prometheus-operator-prometheus-operator:v0.65.1 -10.xx.xx.xx9:7000/rancher/mirrored-prometheus-operator-prometheus-operator:v0.65.1... OK registry.rancher.com/rancher/mirrored-prometheus-prometheus:v2.45.0 -10.xx.xx.xx9:7000/rancher/mirrored-prometheus-prometheus:v2.45.0... OK registry.rancher.com/rancher/mirrored-thanos-thanos:v0.30.2 -10.xx.xx.xx9:7000/rancher/mirrored-thanos-thanos:v0.30.2... OK registry.rancher.com/rancher/pushprox-client:v0.1.3-rancher2-client -10.xx.xx.xx9:7000/rancher/pushprox-client:v0.1.3-rancher2-client... OK registry.rancher.com/rancher/pushprox-proxy:v0.1.3-rancher2-proxy -10.xx.xx.xx9:7000/rancher/pushprox-proxy:v0.1.3-rancher2-proxy... OK registry.rancher.com/rancher/shell:v0.1.25 -10.xx.xx.xx9:7000/rancher/shell:v0.1.25... OK✅ 全部推送完成。下一步去有 helm 的机器执行 helm install(见 sop/README.md)。[rootxx images]#常见翻车:docker login报server gave HTTP response to HTTPS client→ Harbor 走的 http,要给 A 机 docker 配 insecure(见排错表 #6)个别镜像FAIL→ 网络抖,重跑一遍脚本(已推过的会快速跳过/覆盖,无害)4. 阶段三:改配置(B 机,1 分钟)cdchartsed-is/__HARBOR__/harbor.10.0.0.5:30003/gvalues-offline.yamlgrep-n__HARBOR__values-offline.yamlecho❌ 还有残留!||echo✅ 替换干净预期:打印✅ 替换干净。5. 阶段四:安装(B 机,3 分钟命令 3 分钟等起)cd..# 回到 rancher-monitoring-offline 根目录# 1. 装 CRD(带 --create-namespace,连命名空间一起建了,不需要 kubectl)bin/helminstallrancher-monitoring-crd\chart/rancher-monitoring-crd-103.1.1up45.31.1.tgz\-fchart/values-offline.yaml\-ncattle-monitoring-system --create-namespace# 2. 装主 chartbin/helminstallrancher-monitoring\chart/rancher-monitoring-103.1.1up45.31.1.tgz\-fchart/values-offline.yaml\-ncattle-monitoring-system预期:两条命令各自最后打STATUS: deployed。实际操作过程记录-安装rancher-monitoring-crd和rancher-monitoring [rootxx rancher-monitoring-offline]##以前操作有问题没有引用values-offline.yaml导致无法拉取镜像。需要去rancher的workload - job模块删掉任务然后uninstall取消。[rootxx rancher-monitoring-offline]# bin/helm install rancher-monitoring-crd \chart/rancher-monitoring-crd-103.1.1up45.31.1.tgz\-fchart/values-offline.yaml\-ncattle-monitoring-system --create-namespace Error: INSTALLATION FAILED: cannot re-use a name that is stillinuse[rootxx rancher-monitoring-offline]#[rootxx rancher-monitoring-offline]# bin/helm list -n cattle-monitoring-system -aNAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION rancher-monitoring-crd cattle-monitoring-system12026-09-0115:04:55.76004109 0800 CST failed rancher-monitoring-crd-103.1.1up45.31.1[rootxx rancher-monitoring-offline]#[rootxx rancher-monitoring-offline]# bin/helm get values rancher-monitoring -n cattle-monitoring-systemError: release: not found[rootxx rancher-monitoring-offline]# bin/helm uninstall rancher-monitoring-crd -n cattle-monitoring-systemreleaserancher-monitoring-crduninstalled[rootxx rancher-monitoring-offline]#[rootxx rancher-monitoring-offline]#[rootxx rancher-monitoring-offline]#[rootxx rancher-monitoring-offline]#[rootxx rancher-monitoring-offline]# bin/helm list -n cattle-monitoring-system -aNAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION[rootxx rancher-monitoring-offline]#[rootxx rancher-monitoring-offline]#[rootxx rancher-monitoring-offline]# bin/helm install rancher-monitoring-crd \chart/rancher-monitoring-crd-103.1.1up45.31.1.tgz\-fchart/values-offline.yaml\-ncattle-monitoring-system --create-namespace NAME: rancher-monitoring-crd LAST DEPLOYED: Tue Sep115:43:552026NAMESPACE: cattle-monitoring-system STATUS: deployed REVISION:1TEST SUITE: None[rootxx rancher-monitoring-offline]##创建了命名空间 cattle-monitoring-system[rootxx rancher-monitoring-offline]# kubectl get nsNAME STATUS AGE cattle-impersonation-system Active 3y325d cattle-monitoring-system Active 40m cattle-system Active 3y325d default Active 3y326d xxx-vxx Active 3y322d kube-node-lease Active 3y326d kube-public Active 3y326d kube-system Active 3y326dlocalActive 3y325d[rootxx rancher-monitoring-offline]# bin/helm install rancher-monitoring \chart/rancher-monitoring-103.1.1up45.31.1.tgz\-fchart/values-offline.yaml\-ncattle-monitoring-system W090115:45:42.853896106880warnings.go:70]policy/v1beta1 PodSecurityPolicy is deprecatedinv1.21, unavailableinv1.25 W090115:45:56.745211106880warnings.go:70]policy/v1beta1 PodSecurityPolicy is deprecatedinv1.21, unavailableinv1.25 NAME: rancher-monitoring LAST DEPLOYED: Tue Sep115:45:392026NAMESPACE: cattle-monitoring-system STATUS: deployed REVISION:1TEST SUITE: None NOTES: rancher-monitoring has been installed. Check its status by running: kubectl--namespacecattle-monitoring-system get pods-lreleaserancher-monitoringVisit https://github.com/prometheus-operator/kube-prometheusforinstructions on how to createconfigure Alertmanager and Prometheus instances using the Operator.[rootxx rancher-monitoring-offline]#安装好后pod的运行信息[rootxx rancher-monitoring-offline]# kubectl get pod -n cattle-monitoring-systemNAME READY STATUS RESTARTS AGE alertmanager-rancher-monitoring-alertmanager-02/2 Running030m prometheus-rancher-monitoring-prometheus-03/3 Running029m rancher-monitoring-grafana-fc6764fc5-4c9jn4/4 Running030m rancher-monitoring-kube-state-metrics-77b7d595dd-xd54j1/1 Running030m rancher-monitoring-operator-6df4b4fd-lgrcs1/1 Running030m rancher-monitoring-prometheus-adapter-77b6c85c59-g7rlx1/1 Running030m rancher-monitoring-prometheus-node-exporter-lzbt51/1 Running030m rancher-monitoring-prometheus-node-exporter-szxxg1/1 Running030m rancher-monitoring-prometheus-node-exporter-w4fdx1/1 Running030m rancher-monitoring-prometheus-node-exporter-wqcww1/1 Running030m rancher-monitoring-prometheus-node-exporter-x65j21/1 Running030m[rootxx rancher-monitoring-offline]#5.1 先做拉取链路预验证(强烈建议,30 秒拦住 90% 的坑)大安装里 20 多个 pod 任何一个拉不动镜像都会转圈,先单拉一个最小的试试:kubectl run pull-test--restartNever--imageharbor.10.0.0.5:30003/rancher/mirrored-library-busybox:1.31.1 --sleep60kubectl get pod pull-test# 等 30 秒再查kubectl delete pod pull-test# 无论成败,验完就删预期:STATUS Running(或已 Completed)。若失败→ 看排错表 #2,把镜像拉取问题先修好再继续,否则后面 20 个 pod 陪葬。5.2 等全部起来kubectl get pods-ncattle-monitoring-system-w# CtrlC 退出观察预期:2~3 分钟内全部Running或Completed,pod 名大致是:pod(结尾带随机串)个数prometheus-rancher-monitoring-prometheus-01alertmanager-rancher-monitoring-alertmanager-01rancher-monitoring-grafana-xxxxx1rancher-monitoring-operator-xxxxx1rancher-monitoring-kube-state-metrics-xxxxx1rancher-monitoring-prometheus-adapter-xxxxx1rancher-monitoring-prometheus-node-exporter-xxxxx每节点 1 个(可能)pushprox 相关 client每节点最多 1 个总数 ≈ 6 节点数 × 1~2。有ImagePullBackOff/CrashLoopBackOff/Pending→ 排错表。6. 阶段五:验收(打开 Grafana 看图)kubectl port-forward svc/rancher-monitoring-grafana-ncattle-monitoring-system3000:80浏览器开http://localhost:3000,账号admin/prom-operator(首次登录可能强制改密,改了就记住)。左边 Dashboards 里打开Kubernetes / Compute Resources / Cluster,有曲线图 安装成功。这条 port-forward 命令会一直占着前台,验收完 CtrlC 即可,不影响监控本身运行。7. 阶段六:裸机接入(可选,每台 2 分钟 一次 helm upgrade)7.1 每台裸机装 node_exporter(B 机远程装,或拷文件到裸机本机跑)cdbaremetalSERVER10.110.141.15bashinstall-node-exporter.sh# 换成裸机真实 IP,多台循环预期:最后打印自测输出一行# HELP ...之类的内容、状态active。7.2 告诉 Prometheus 去抓谁viscrape-baremetal.yaml# 把 targets 改成真实裸机 IP,多台照加7.3 重新传入配置(不会重装,只是升级)cd..bin/helm upgrade rancher-monitoring chart/rancher-monitoring-103.1.1up45.31.1.tgz\-fchart/values-offline.yaml-fbaremetal/scrape-baremetal.yaml\-ncattle-monitoring-system7.4 验证裸机被抓到了kubectl port-forward svc/rancher-monitoring-prometheus-ncattle-monitoring-system9090:9090浏览器开http://localhost:9090/targets,找baremetal这组,每台裸机一条且STATEUP。(或在 Grafana 里查询node_load1{baremetaltrue}有数据。)8. 排错速查表(对号入座)#症状一条命令确认原因 → 修法1podImagePullBackOffkubectl describe pod pod名 -n cattle-monitoring-system | tail -20看 Events 里的镜像名:①含__HARBOR__→ values 没替换,回阶段三,重跑 install 命令;②not found→ 该镜像没推进 Harbor,回阶段二;③x509/HTTP response to HTTPS client→ 节点不认 Harbor,见 #62预验证 pod pull-test 起不来同上 describe同 #1 的③:所有 k8s 节点配 insecure 或装 Harbor CA。docker 节点改/etc/docker/daemon.json;RKE2/k3s 节点写/etc/rancher/rke2(或 k3s)/registries.yaml,改完重启节点上对应服务。改完删掉 pull-test 重试3podPending同上 describeEvents 说资源不足 → 节点太满,清理或换集群;说 PVC → 不该出现(默认无持久化),拍照求助4podCrashLoopBackOffkubectl logs pod名 -n cattle-monitoring-system --previous | tail -50多半是版本/参数问题,拍照发回5helm 报Error: non-absolute paths should be in a chart/ 找不到 tgzls chart/目录不对或文件名抄错,cd 到包根目录、照手册原文敲6docker login/push 报 HTTPS/x509 错—Harbor 是 http 或自签证书:推镜像的 A 机配/etc/docker/daemon.json的insecure-registries后systemctl restart docker;k8s 节点也要配(见 #2),两边都配才行7报$\r: command not found—文件被 Windows 记事本改坏了:sed -i s/\r$// 文件名修一下,以后别用记事本8裸机 targetDOWN在 B 机kubectl run nm --rm -it --restartNever --imageharbor.10.0.0.5:30003/rancher/mirrored-curlimages-curl:7.85.0 -- curl -s -m 5 http://10.110.141.15:9100/metrics | head -1集群内不通 → 裸机防火墙没开 9100 或路由不通,找网络组;集群内通但 target DOWN → scrape-baremetal.yaml 的 IP 写错/helm upgrade 没带-f baremetal/scrape-baremetal.yaml9Grafana admin 登不进kubectl get secret rancher-monitoring-grafana -n cattle-monitoring-system -o jsonpath{.data.admin-password} | base64 -d打出来的就是当前密码;不行就删 pod 重建前先拍照求助10helm 报 release 已存在bin/helm list -n cattle-monitoring-system之前装过。全新重来:先按附录卸载,再从阶段四开始9. 常见后续问题监控数据/自建面板重启会丢吗?会(Prometheus 和 Grafana 默认没挂持久盘)。本次验收不受影响;现场要长期保留历史,回来后我们补一份带 PVC 的 values。装完 Rancher UI 上能看到吗?能,集群里会出现 monitoring 相关图标/菜单;若 UI 没马上刷新,等几分钟或重登。怎么卸载重来?bin/helm uninstall rancher-monitoring-ncattle-monitoring-system bin/helm uninstall rancher-monitoring-crd-ncattle-monitoring-system kubectl delete namespace cattle-monitoring-system(最后一条可能卡 Terminating 几分钟,正常。)10. 打电话求助前,先收集这 5 样(拍照/截图)bash sop/preflight.sh的完整输出kubectl get pods -n cattle-monitoring-system -o wide问题 pod 的kubectl describe pod 名 -n cattle-monitoring-system | tail -30问题 pod 的kubectl logs 名 -n cattle-monitoring-system --previous | tail -50bin/helm list -n cattle-monitoring-system最终效果【看样子他把这些监控组件都嵌入在rancher里面了并不会对外暴露没有nodePort】0、Monitoring :1、alertManager:2、grafana3、Prometheus Graph4、Prometheus Rules5、Prometheus Targets其他# CC回答记录全部完成。三个问题挨个回答 --- 一、kube-prometheus-stack 和我们的方案是什么关系——它就是我们方案的亲爹不用调向 看 chart 文件名里的 up45.31.1upupstream。rancher-monitoring103.1.1 就是 Rancher 把 kube-prometheus-stack45.31.1 原样打包、换了镜像源和加了 UI 集成。同一套东西同一个 Prometheus Operator、同一个 Grafana/Alertmanager/node-exporter。你同事的建议和我们已经跑通的方案在架构上是同一个只是包装层不同 ┌────────────┬──────────────────────────────────────────────────────┬─────────────────────────────────────────────┐ │ │ 我们:rancher-monitoring │ 同事:kube-prometheus-stack(裸 upstream)│ ├────────────┼──────────────────────────────────────────────────────┼─────────────────────────────────────────────┤ │ │ 单一 registry.rancher.com,一行 systemDefaultRegistry │ quay.io docker.io registry.k8s.io │ │ 镜像来源 │ 全部改写到现场 Harbor(values-offline.yaml │ 多个源,离线要逐组件改十几个 image repo │ │ │ 就是这么工作的)│ │ ├────────────┼──────────────────────────────────────────────────────┼─────────────────────────────────────────────┤ │ Rancher UI │ 有(集群菜单、图标、告警)│ 无,Rancher 不认 │ │ 集成 │ │ │ ├────────────┼──────────────────────────────────────────────────────┼─────────────────────────────────────────────┤ │ 版本 │ Rancher 官方测试过和2.8的兼容矩阵 │ 更新更快 │ ├────────────┼──────────────────────────────────────────────────────┼─────────────────────────────────────────────┤ │ 本地验证 │ 已跑通 │ 从零再来 │ └────────────┴──────────────────────────────────────────────────────┴─────────────────────────────────────────────┘ 你们现场就是 Rancher2.8管的集群rancher-monitoring 是官方支持路径换 upstream 等于把已验证的活推翻、离线改造量翻几倍、丢 UI 集成收益为零。结论不调向可以放心回同事“我们用的就是 kube-prometheus-stack 的 Rancher 官方打包版”。 二、现场没有 Claude 会遇到什么问题——我刚把包补成了“无 AI 也能跑” 排查了一遍交付包补了三样东西1. bin/helm新增——原 SOP 附录只说“带个 helm 进去”但包里根本没有内网现场会卡死。现在是 v3.14.3 静态二进制拷过去chmodx 就能用。2. sop/preflight.sh新增——开工前一键自检helm/kubectl/docker 在不在、交付包全不全、集群连不连得上、节点架构是不是 amd64、Harbor 是 http 还是 httpshttp 会导致节点拉镜像全挂这是内网现场第一大坑脚本会直接打印修法。3. sop/现场操作手册.md新增主文档——按你“k8s 经验少”写的每步都有命令预期输出长什么样对不上翻排错表第几条。10 条排错速查表覆盖了 ImagePullBackOff 三种原因判别、http Harbor 修法、CRLF 坏文件修复、裸机防火墙等最后还有“求助前先收集这5样”清单。 另外两个原来可能翻车的点也化解了kubectl 不再是硬依赖helm --create-namespace 建命名空间验收可用 Rancher UI 网页 kubectl新增了“拉取链路预验证”步骤——正式安装前先单拉一个 busybox 镜像试pod30 秒拦住90% 的镜像源问题不让20个 pod 陪葬。我试过下载 kubectl 二进制一起打包但几个下载源全被墙了helm 拿到了清华镜像kubectl 靠 UI 兜底够用。 三、现场实际怎么干——6 个阶段全程照抄 手册里每步都有预期输出大骨架是1. 预检B 机chmod x bin/helm →HARBOR现场Harborbashsop/preflight.sh全 ✅ 才继续2. 推镜像A 机有docker那台Harbor 建公开项目 rancher →dockerlogin →HARBOR现场Harborbashload-push.sh20 行全 OK3. 改配置B 机一条sed把 __HARBOR__ 换成真地址grep 确认无残留4. 安装B 机两条 bin/helminstall→ 先跑 busybox 拉取预验证 → 等3分钟 pod 全 Running总数 ≈6 节点数5. 验收port-forward 打开 Grafanaadmin / prom-operator看到集群仪表盘即成功6. 裸机可选每台SERVERIPbashinstall-node-exporter.sh → 改 scrape-baremetal.yaml 的 targets → 一次 helm upgrade → Prometheus /targets 页面看STATEUP 三条铁律写在手册开头别用 Windows 记事本改文件会引入 CRLF、命令照原文敲、预期输出对不上就停下翻表别连敲。 唯一需要你知道的已知限制默认没配持久化Prometheus 历史数据和 Grafana 自建面板在 pod 重启后会丢不影响本次验收。现场如果要长期保留监控历史回来我给你补一份带 PVC 的 values。 ✻ Churnedfor11m 35s ※ recap: 内网离线部署监控方案已收尾确认无需改用kube-prometheus-stack本方案即其Rancher打包版并补齐了自带helm、预 检脚本和现场操作手册。下一步你通读sop/现场操作手册.md确认无误后拷包去现场。(disable recapsin/config)随记传统普通运维要被AI淘汰咯。 2026年9月3日20:44:23
RELATED READING

延伸阅读

更多一线实战笔记与深度复盘,助您持续精进