巧用 shell-operator 配置 K8s Pod 保护策略
巧用 shell-operator 配置 K8s Pod 保护策略
SRE运维进阶之路
发布于 2024-04-23 14:45:52
发布于 2024-04-23 14:45:52
保障 Pod 高可用分为以下几个方面
- 资源池划分 ：打污点、独占、隔离
- Qos 分级 & 优先级：按应用等级保障稳定性
- 自动弹性伸缩 HPA
- Pod 干扰预算 PDB
本篇主要介绍第四点：巧用 Shell-operator 配置 K8s Pod 保护策略
什么是 Shell-operator ？
Shell-operator 是一个在 Kubernetes 集群中运行事件驱动脚本的工具。
该操作器不是针对特定软件产品的操作器，例如 prometheus-operator 或 kafka-operator。Shell-operator 通过将脚本视为由事件触发的钩子，提供了 Kubernetes 集群事件和 Shell 脚本之间的集成层。您可以将其视为 operator-sdk，但用于脚本。
Shell-operator 用作更高级的 addon-operator 的基础，该 operator 支持 Helm charts 和值存储。
Shell-operator 提供了以下功能：
- Kubernetes 集群的简易管理：使用运维人员熟悉的工具。可以是 bash、python、kubectl 等，真是运维人的实用工具???。
- Kubernetes 对象事件：钩子可以由添加、更新或删除事件触发。
- 对象选择器和属性过滤器：shell-operator 可以监视特定的对象集，并检测其属性的更改。
- 简单的配置：钩子绑定定义是脚本的标准输出中的 JSON 或 YAML 文档。
- 验证 Webhook 机制：钩子可以处理 Kubernetes 资源的验证。
- 转换 Webhook 机制：钩子可以处理 Kubernetes 资源的版本转换。
借助 Shell-operator 配置 PDB
? 说明：以下是变量，可结合自身需要进行修改 配置 PDB 需要用到便签选择器 selector，示例中选择 AppID 作为标签匹配 Shell-operator 需要监控一个对象的事件，来更新 PDB 的状态，示例中选择 rollouts.argoproj.io 对象 namespace 选择 prod
创建基础模版 pdb-init.sh
#!/usr/bin/env bash
NAMESPACE="prod"
pdb_templates() {
cat <<EOF
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: clay-test
namespace: ${NAMESPACE}
spec:
minAvailable: 1
selector:
matchLabels:
appid: clay-test
EOF
}
replace_or_create() {
object=$(cat)
if ! kubectl get -f - <<< "$object" >/dev/null 2>/dev/null; then
kubectl create -f - <<< "$object" >/dev/null
else
kubectl replace --force -f - <<< "$object" >/dev/null
fi
}
init() {
echo "Init PDB templates"
pdb_templates | replace_or_create
for resourceName in $(kubectl -n ${NAMESPACE} get rollouts.argoproj.io | grep default | awk '{print $1}'); do
appid=${resourceName%-default}
echo "Init Add PDB '${appid}'"
kubectl -n ${NAMESPACE} get pdb clay-test -o json | \
jq -r ".metadata.name=\"${appid}\" | .spec.selector.matchLabels[\"appid\"]=\"${appid}\" |
.metadata |= with_entries(select([.key] | inside([\"name\", \"namespace\", \"labels\"])))" \
| replace_or_create
done
}
init "$@"
bash pdb-init.sh
编写钩子脚本pdb-hooks.sh
#!/usr/bin/env bash
NAMESPACE="prod"
ARRAY_COUNT=$(jq -r '. | length-1' $BINDING_CONTEXT_PATH)
run_hook() {
if [[ $1 == "--config" ]] ; then
config
else
trigger
fi
}
config() {
cat <<EOF
configVersion: v1
kubernetes:
- apiVersion: argoproj.io/v1alpha1
kind: Rollout
executeHookOnEvent:
- Added
- Deleted
namespace:
nameSelector:
matchNames:
- ${NAMESPACE}
EOF
}
trigger() {
for IND in `seq 0 $ARRAY_COUNT`; do
resourceEvent=$(jq -r ".[$IND].watchEvent" $BINDING_CONTEXT_PATH)
resourceName=$(jq -r ".[$IND].object.metadata.name" $BINDING_CONTEXT_PATH)
appid=${resourceName%-default}
if [[ $resourceEvent == "Added" ]] ; then
echo "Add PDB '${appid}'"
kubectl -n ${NAMESPACE} get pdb clay-test -o json | \
jq -r ".metadata.name=\"${appid}\" | .spec.selector.matchLabels[\"appid\"]=\"${appid}\" |
.metadata |= with_entries(select([.key] | inside([\"name\", \"namespace\", \"labels\"])))" \
| replace_or_create
elif [[ $resourceEvent == "Deleted" ]]; then
echo "Delete PDB '${appid}'"
kubectl -n ${NAMESPACE} delete pdb ${appid}
else
echo "Do nothing"
fi
done
}
replace_or_create() {
object=$(cat)
if ! kubectl get -f - <<< "$object" >/dev/null 2>/dev/null; then
kubectl create -f - <<< "$object" >/dev/null
else
kubectl replace --force -f - <<< "$object" >/dev/null
fi
}
run_hook "$@"
打包镜像，Dockerfile 如下
FROM ghcr.io/flant/shell-operator:latest
ADD pdb-hooks.sh /hooks
# 打包上传镜像
docker build -t wangzhichidocker/shell-operator-pdb:v1.0 .
docker push wangzhichidocker/shell-operator-pdb:v1.0
编写 RBAC、Deployment等shell-operator-pdb.yaml，部署到 K8s 集群中
apiVersion: v1
kind: ServiceAccount
metadata:
name: shell-operator-pdb
namespace: monitor
labels:
app: shell-operator-pdb
appid: shell-operator-pdb
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app: shell-operator-pdb
appid: shell-operator-pdb
name: shell-operator-pdb
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- watch
- list
- apiGroups:
- argoproj.io
resources:
- '*'
verbs:
- get
- list
- watch
- apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- get
- create
- patch
- update
- delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: shell-operator-pdb
labels:
app: shell-operator-pdb
appid: shell-operator-pdb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: shell-operator-pdb
subjects:
- kind: ServiceAccount
name: shell-operator-pdb
namespace: monitor
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: shell-operator-pdb
labels:
app: shell-operator-pdb
appid: shell-operator-pdb
namespace: monitor
spec:
replicas: 1
selector:
matchLabels:
app: shell-operator-pdb
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: shell-operator-pdb
appid: shell-operator-pdb
spec:
containers:
- image: wangzhichidocker/shell-operator-pdb:v1.0
imagePullPolicy: IfNotPresent
name: shell-operator-pdb
resources:
limits:
cpu: 500m
memory: 1000Mi
requests:
cpu: 100m
memory: 200Mi
dnsPolicy: ClusterFirst
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: shell-operator-pdb
# 部署
kubectl apply -f deploy/shell-operator-pdb.yaml
观察日志查看运行状态 kubectl -n monitor logs -f shell-operator-pdb-xxx
image-20240318133226486
快捷体验：
git clone https://github.com/clay-wangzhi/shell-operator-pdb
bash pdb-init.sh
kubectl apply -f deploy/shell-operator-pdb.yaml
详见：https://github.com/clay-wangzhi/shell-operator-pdb
参考链接：
- shell-operator：https://github.com/flant/shell-operator
本文参与?腾讯云自媒体分享计划，分享自微信公众号。
原始发表：2024-03-18，如有侵权请联系?cloudcommunity@tencent.com 删除
评论
登录后参与评论
推荐阅读