K8s Ingress的日志切割
原创
提醒：本文是在minikube上实验的，不同环境的ingress请参考官方文档?
ingress的安装
ingress的安装部署，不是这里的重点，忽略。 我实验用的是minikube，和云上环境可能差异较大。
在宿主机上创建日志路径并授权
mkdir /data/logs/nginx
chmod 777 /data/logs/nginx -R修改ingress-nginx-controller的deployment的文件
1、增加initContainer配置
2、增加hostpath配置
3、增加localtime配置
$ kubectl get deployments.apps -n ingress-nginx
NAME READY UP-TO-DATE AVAILABLE AGE
ingress-nginx-controller 1/1 1 1 24h
具体内容如下：
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/version: 0.44.0
helm.sh/chart: ingress-nginx-3.23.0
name: ingress-nginx-controller
namespace: ingress-nginx
resourceVersion: "16309799"
selfLink: /apis/apps/v1/namespaces/ingress-nginx/deployments/ingress-nginx-controller
uid: 6b3045e1-b8a6-47bb-b9b6-ca6e0d2fb964
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
spec:
containers:
- args:
- /nginx-ingress-controller
- --election-id=ingress-controller-leader
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
image: k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: controller
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
- containerPort: 8443
name: webhook
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 100m
memory: 90Mi
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
runAsUser: 101
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/local/certificates/
name: webhook-cert
readOnly: true
- mountPath: /etc/localtime
name: localtime
readOnly: true
- mountPath: /var/log/nginx/
name: logdir
dnsPolicy: ClusterFirst
initContainers:
- command:
- /bin/sh
- -c
- |
mount -o remount rw /proc/sys
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.ip_local_port_range="1024 65535"
sysctl -w fs.file-max=1048576
sysctl -w fs.inotify.max_user_instances=16384
sysctl -w fs.inotify.max_user_watches=524288
sysctl -w fs.inotify.max_queued_events=16384
image: busybox
imagePullPolicy: IfNotPresent
name: init-sysctl
resources: {}
securityContext:
capabilities:
add:
- SYS_ADMIN
drop:
- ALL
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
nodeSelector:
kubernetes.io/os: linux
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: ingress-nginx
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
defaultMode: 420
secretName: ingress-nginx-admission
- hostPath:
path: /etc/localtime
type: File
name: localtime
- hostPath:
path: /data/logs/nginx/
type: ""
name: logdir优化下ingress-nginx的configmap配置
1、增加了些常用的nginx配置，具体如下
apiVersion: v1
data:
allow-backend-server-header: "true"
enable-underscores-in-headers: "true"
generate-request-id: "true"
ignore-invalid-headers: "true"
log-format-upstream: $remote_addr - [$remote_addr] - $remote_user [$time_local]
"$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length
$request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length
$upstream_response_time $upstream_status $req_id $host
max-worker-connections: "65536"
proxy-body-size: 20m
proxy-connect-timeout: "10"
reuse-port: "true"
server-tokens: "false"
ssl-redirect: "false"
worker-cpu-affinity: auto
kind: ConfigMap
metadata:
annotations:
creationTimestamp: "2021-02-12T07:08:25Z"
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/version: 0.44.0
helm.sh/chart: ingress-nginx-3.23.0
name: ingress-nginx-controller
namespace: ingress-nginx
resourceVersion: "16310580"
selfLink: /api/v1/namespaces/ingress-nginx/configmaps/ingress-nginx-controller
uid: 12b1b526-fa3c-4680-b4fb-e7f0d9ebbb94ingress-nginx的日志切割脚本
#!/bin/bash
# 最多保留日志文件个数，可根据需求进行调整。
keep_log_num=5
# 阿里云提供的ingress是这个写法，其它的云厂商的可能也不太一样
# ingress_nginx_container_ids=$(docker ps | grep nginx-ingress-controller | grep -v pause | awk '{print $1}')
# 我在minikube里面用的nginx-ingress是用的下面这个写法
ingress_nginx_container_ids=$(docker ps | grep ingress-nginx-controller | grep -v pause | awk '{print $1}')
if [[ -z "$ingress_nginx_container_ids" ]]; then
echo "error: failed to get ingress nginx container ids"
exit 1
fi
#随机睡眠5~10秒
sleep $(( RANDOM % (10 - 5 + 1 ) + 5 ))
for id in $ingress_nginx_container_ids; do
docker exec $id bash -c "cd /var/log/nginx; if [[ \$(ls access.log-* | wc -l) -gt $keep_log_num ]]; then rm -f \$(ls -t access.log-* | tail -1); fi ; mv access.log access.log-\$(date +%F_%H_%M_%S) ; kill -USR1 \$(cat /tmp/nginx.pid)"
done
# 这里的历史文件，没有压缩。大家可以根据自己情况看是否要改改。
脚本定时执行
# 在每台宿主机上都跑一个cronjob定时任务（或者各种调度平台也行，只要能到宿主机即可），负责执行这个切割日志的脚本附：一个ingress的demo
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/client-body-buffer-size: 1M
nginx.ingress.kubernetes.io/configuration-snippet: |
access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
error_log /var/log/nginx/error.log info;
nginx.ingress.kubernetes.io/connection-proxy-header: keep-alive
nginx.ingress.kubernetes.io/enable-rewrite-log: "true"
nginx.ingress.kubernetes.io/limit-connections: "10000"
nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
nginx.ingress.kubernetes.io/proxy-buffering: "on"
nginx.ingress.kubernetes.io/proxy-max-temp-file-size: 1024m
nginx.ingress.kubernetes.io/server-alias: shanghai.example.com
nginx.ingress.kubernetes.io/server-snippet: |
set $agentflag 0;
if ($http_user_agent ~* "(Mobile)" ){
set $agentflag 1;
}
if ( $agentflag = 1 ) {
return 301 https://m.example.com;
}
nginx.ingress.kubernetes.io/ssl-ciphers: ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP
nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers: "true"
nginx.ingress.kubernetes.io/upstream-hash-by: $request_uri
creationTimestamp: "2021-02-12T03:58:02Z"
generation: 5
name: gray-release
namespace: default
resourceVersion: "16130870"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/gray-release
uid: 8b732b00-a1a9-4d74-bfe4-9210a0094b9f
spec:
rules:
- host: www.example.com
http:
paths:
- backend:
serviceName: new-nginx
servicePort: 80
path: /
pathType: ImplementationSpecificingrss的日志采集
流程：
1、这里是把全部vhost的日志都打到了 access.log 中
2、部署在k8s的ds模式的fluentd会将日志都采集发送到kafka中
3、logstash消费kafka的数据，并写到ES中
4、在kibana中，根据域名来查不同的域名的访问记录
这样处理起来比较简单，但是在日志量大的情况下，会造成es的index过大，每次查询都需要加载很多不相关的vhost的日志。
这种情况下，有2个解决方法供参考：
1、在logstash层做处理，对访问量大的某些vhost，单独写到独立的es index里，
其余体量小的vhost还是共同写到一个es index中
2、在ingress层处理，在配置access_log的时候，就单独打到独立的文件中。
然后再fluentd中也做个判断，写到不同kafka topic中，logstash单独写个pipeline去处理。
原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。
如有侵权，请联系 cloudcommunity@tencent.com 删除。
原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。
如有侵权，请联系 cloudcommunity@tencent.com 删除。
评论
登录后参与评论
推荐阅读