1. 高可用Kubernetes集群架构解析

在容器编排领域，构建生产级高可用(HA)Kubernetes集群是每个DevOps工程师的必修课。今天我将分享基于kubeadm工具部署HA集群的完整架构设计，这套方案已经在多个金融级生产环境稳定运行超过两年。

高可用集群与单节点集群的核心区别在于控制平面的冗余设计。传统单Master架构存在单点故障风险，当API Server不可用时，整个集群的调度和管理功能将瘫痪。而HA架构通过多Master节点+负载均衡的组合，确保即使某个控制平面节点宕机，集群仍能持续提供服务。

2. 核心组件与拓扑设计

2.1 基础架构组成

典型的HA集群包含以下核心组件：

• 3个Master节点（推荐奇数个以实现quorum）
• 2+个Worker节点（根据业务需求扩展）
• 负载均衡器（通常采用Keepalived+HAProxy方案）
• etcd集群（可独立部署或与Master节点共存）

2.2 网络拓扑方案

我们采用分层式网络设计：

[外部负载均衡器] | [Keepalived VIP] | [HAProxy Layer] → [Master Node 1 API Server] | → [Master Node 2 API Server] | → [Master Node 3 API Server] | [Worker Nodes]

这种架构下，客户端请求首先到达负载均衡层，由HAProxy将流量分发到健康的API Server实例。Keepalived负责维护虚拟IP(VIP)的高可用，当主负载均衡器故障时自动切换备用节点。

3. 关键配置参数详解

3.1 etcd集群配置

etcd作为Kubernetes的"大脑"，其配置直接影响集群稳定性。关键参数包括：

# etcd配置示例 name: etcd1 listen-peer-urls: https://192.168.1.101:2380 listen-client-urls: https://192.168.1.101:2379,https://127.0.0.1:2379 advertise-client-urls: https://192.168.1.101:2379 initial-advertise-peer-urls: https://192.168.1.101:2380 initial-cluster: etcd1=https://192.168.1.101:2380,etcd2=https://192.168.1.102:2380,etcd3=https://192.168.1.103:2380

注意：etcd节点间通信必须使用TLS加密，生产环境严禁使用HTTP明文协议。

3.2 kubeadm配置模板

kubeadm-config.yaml是部署的核心配置文件：

apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.26.0 controlPlaneEndpoint: "k8s-api.example.com:6443" apiServer: certSANs: - "k8s-api.example.com" - "192.168.1.100" # VIP地址 extraArgs: audit-log-path: /var/log/kubernetes/audit.log etcd: external: endpoints: - "https://192.168.1.101:2379" - "https://192.168.1.102:2379" - "https://192.168.1.103:2379" caFile: /etc/kubernetes/pki/etcd/ca.crt certFile: /etc/kubernetes/pki/etcd/client.crt keyFile: /etc/kubernetes/pki/etcd/client.key networking: podSubnet: "10.244.0.0/16" serviceSubnet: "10.96.0.0/12"

4. 部署流程关键步骤

4.1 前置准备阶段

1. 系统配置标准化：

# 所有节点执行 swapoff -a sed -i '/ swap / s/^/#/' /etc/fstab setenforce 0 sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

2. 内核参数调优：

cat > /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF sysctl --system

4.2 负载均衡器部署

HAProxy配置示例：

frontend k8s-api bind *:6443 mode tcp option tcplog default_backend k8s-api-servers backend k8s-api-servers mode tcp balance roundrobin option tcp-check server k8s-master-1 192.168.1.101:6443 check fall 3 rise 2 server k8s-master-2 192.168.1.102:6443 check fall 3 rise 2 server k8s-master-3 192.168.1.103:6443 check fall 3 rise 2

Keepalived配置要点：

vrrp_script chk_haproxy { script "killall -0 haproxy" interval 2 weight 2 } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 priority 101 virtual_ipaddress { 192.168.1.100/24 } track_script { chk_haproxy } }

5. 常见问题与解决方案

5.1 证书相关错误

现象：kubelet报错"x509: certificate signed by unknown authority"

排查步骤：

1. 检查各节点时间是否同步
2. 确认kubeadm init时使用的证书SAN列表包含所有访问地址
3. 验证证书有效期：

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -A 2 Validity

5.2 网络插件问题

现象：Pod间网络不通

解决方案：

1. 确认选择的CNI插件与内核版本兼容
2. 检查网络策略是否阻止通信
3. 验证iptables规则：

iptables-save | grep -i calico

5.3 etcd集群健康检查

定期监控etcd集群状态：

ETCDCTL_API=3 etcdctl \ --endpoints=https://192.168.1.101:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ endpoint health

预期输出：

https://192.168.1.101:2379 is healthy: successfully committed proposal: took = 12.345678ms https://192.168.1.102:2379 is healthy: successfully committed proposal: took = 11.234567ms https://192.168.1.103:2379 is healthy: successfully committed proposal: took = 10.123456ms

6. 性能优化建议

6.1 API Server调优

修改kube-apiserver配置：

apiServer: extraArgs: default-not-ready-toleration-seconds: "30" default-unreachable-toleration-seconds: "30" event-ttl: "1h" max-mutating-requests-inflight: "500" max-requests-inflight: "1500"

6.2 kubelet资源预留

确保系统关键进程有足够资源：

kubelet: kubeReserved: cpu: "500m" memory: "500Mi" ephemeral-storage: "1Gi" systemReserved: cpu: "500m" memory: "500Mi" ephemeral-storage: "1Gi"

7. 升级维护策略

7.1 滚动升级流程

1. 隔离第一个Master节点：

kubectl drain <master-node-1> --ignore-daemonsets --delete-emptydir-data

2. 升级kubeadm：

apt-get update && apt-get install -y kubeadm=1.26.0-00 kubeadm upgrade apply v1.26.0

3. 解除隔离：

kubectl uncordon <master-node-1>

7.2 备份恢复方案

定期备份etcd数据：

ETCDCTL_API=3 etcdctl snapshot save snapshot.db \ --endpoints=https://192.168.1.101:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key

恢复etcd集群：

ETCDCTL_API=3 etcdctl snapshot restore snapshot.db \ --initial-cluster-token=etcd-cluster-1 \ --initial-advertise-peer-urls=https://192.168.1.101:2380 \ --name=etcd1 \ --data-dir=/var/lib/etcd-from-backup