Kubernetes 集群网络 Benchmark

高楼Zee

发布于 2020-12-29 12:03:38

6970

发布于 2020-12-29 12:03:38

文章被收录于专栏：7DGroup7DGroup

一、准备工作

1、测试环境

在以下几种环境下进行测试：

Kubernetes 集群 node 节点上通过 Cluster IP 方式访问
Kubernetes 集群内部通过 service 访问
Kubernetes 集群外部通过 nginx ingress 暴露的地址访问

2、部署Web应用

sample-webapp 目录下包含一个简单的web测试应用。我们将其构建为 docker 镜像，在 kubernetes中运行。你可以自己构建，也可以直接用这个我构建好的镜像registry.cn-beijing.aliyuncs.com/zuozewei/sample-webapp:1.0。

在 kubernetes 上部署 sample-webapp：

cd kubernetes-config
kubectl create -f sample-webapp-controller.yaml
kubectl create -f kubectl create -f sample-webapp-service.yaml

sample-webapp-controller.yaml:

kind: ReplicationController
apiVersion: v1
metadata:
  name: sample-webapp
  labels:
    name: sample-webapp
spec:
  selector:
    name: sample-webapp
  replicas: 1
  template:
    metadata:
      labels:
        name: sample-webapp
    spec:
      containers:
      - name: sample-webapp
        image: registry.cn-beijing.aliyuncs.com/zuozewei/sample-webapp:1.0
        ports:
        - containerPort: 8000

sample-webapp-service.yaml：

kind: Service
apiVersion: v1
metadata:
  name: sample-webapp
  labels:
    name: sample-webapp
spec:
  ports:
    - port: 8000
  selector:
    name: sample-webapp
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: sample-webapp
  namespace: default
spec:
  rules:
    - host: sample-webapp.demo.com
      http:
        paths:
          - path: /
            backend:
              serviceName: sample-webapp
              servicePort: 8000

3、测试地址

Cluster IP: 10.96.73.68
Service name：sample-webapp
Service Port：8000
Ingress Host：sample-webapp.demo.com

4、测试工具

curl
iperf
测试程序：sample-webapp

5、测试说明

通过向 sample-webapp 发送 curl 请求获取响应时间，直接 curl 后的结果为：

$ curl "http://10.96.73.68:8000/"
Welcome to the "Distributed Load Testing Using Kubernetes" sample web app

二、网络延迟测试

场景一、 Kubernetes集群node节点上通过Cluster IP访问

测试命令

curl -o /dev/null -s -w '%{time_connect} %{time_starttransfer} %{time_total}' "http://10.96.73.68:8000/"

10 组测试结果：

No	time_connect	time_starttransfer	time_total
1	0.003	0.006	0.006
2	0.002	0.005	0.005
3	0.001	0.004	0.004
4	0.002	0.004	0.004
5	0.001	0.004	0.004
6	0.001	0.004	0.004
7	0.001	0.004	0.004
8	0.001	0.004	0.004
9	0.001	0.004	0.004
10	0.002	0.004	0.004

平均响应时间：「4.3」 ms 时间指标说明：

单位：秒
time_connect：建立到服务器的 TCP 连接所用的时间
time_starttransfer：在发出请求之后，Web 服务器返回数据的第一个字节所用的时间
time_total：完成请求所用的时间

场景二、Kubernetes集群内部通过service访问

测试命令

curl -o /dev/null -s -w '%{time_connect} %{time_starttransfer} %{time_total}' "http://sample-webapp:8000/"```

10 组测试结果：

No	time_connect	time_starttransfer	time_total
1	0.005	0.007	0.007
2	0.013	0.015	0.015
3	0.005	0.007	0.007
4	0.006	0.008	0.008
5	0.006	0.009	0.009
6	0.014	0.016	0.016
7	0.015	0.017	0.017
8	0.006	0.008	0.008
9	0.005	0.007	0.007
10	0.005	0.007	0.007

平均响应时间：「10.1」 ms

场景三、外部通过 nginx ingress 暴露的地址访问

测试命令

curl -o /dev/null -s -w '%{time_connect} %{time_starttransfer} %{time_total}' "http://sample-webapp.demo.com"

10 组测试结果：

No	time_connect	time_starttransfer	time_total
1	0.015	0.026	0.033
2	0.017	0.025	0.025
3	0.010	0.019	0.019
4	0.010	0.025	0.025
5	0.014	0.022	0.022
6	0.006	0.014	0.014
7	0.006	0.016	0.016
8	0.016	0.044	0.045
9	0.006	0.016	0.016
10	0.009	0.032	0.032

平均响应时间：「24.7」 ms

测试结果

在这三种场景下的响应时间测试结果如下：

Kubernetes集群node节点上通过Cluster IP方式访问：「4.3」 ms
Kubernetes集群内部通过service访问：「10.1」 ms
Kubernetes集群外部通过ingress暴露的地址访问：「24.7」 ms

?注意：执行测试的 node 节点/Pod 与 serivce 所在的 pod 的距离（是否在同一台主机上），对前两个场景可以能会有一定影响。 ?

三、网络性能测试

网络使用 Calico 的 ipip 模式。

使用iperf进行测试。服务端命令：

iperf -s -p 12345 -i 1 -M

客户端命令：

iperf -c ${server-ip} -p 12345 -i 1 -t 10 -w 20K

场景一、主机之间

------------------------------------------------------------
Client connecting to 172.16.106.56, TCP port 12345
TCP window size: 40.0 KByte (WARNING: requested 20.0 KByte)
------------------------------------------------------------
[  3] local 172.16.106.100 port 41508 connected with 172.16.106.56 port 12345
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  99.5 MBytes   835 Mbits/sec
[  3]  1.0- 2.0 sec  92.2 MBytes   774 Mbits/sec
[  3]  2.0- 3.0 sec  89.4 MBytes   750 Mbits/sec
[  3]  3.0- 4.0 sec  92.1 MBytes   773 Mbits/sec
[  3]  4.0- 5.0 sec  91.1 MBytes   764 Mbits/sec
[  3]  5.0- 6.0 sec  92.8 MBytes   778 Mbits/sec
[  3]  6.0- 7.0 sec  91.8 MBytes   770 Mbits/sec
[  3]  7.0- 8.0 sec  92.4 MBytes   775 Mbits/sec
[  3]  8.0- 9.0 sec  90.5 MBytes   759 Mbits/sec
[  3]  9.0-10.0 sec  87.8 MBytes   736 Mbits/sec
[  3]  0.0-10.0 sec   920 MBytes   771 Mbits/sec

场景二、不同主机的 Pod 之间

------------------------------------------------------------
Client connecting to gateway-mall-gateway, TCP port 12345
TCP window size: 40.0 KByte (WARNING: requested 20.0 KByte)
------------------------------------------------------------
[  3] local 10.100.18.250 port 54160 connected with 10.96.155.246 port 12345
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  62.2 MBytes   522 Mbits/sec
[  3]  1.0- 2.0 sec  43.1 MBytes   362 Mbits/sec
[  3]  2.0- 3.0 sec  80.4 MBytes   674 Mbits/sec
[  3]  3.0- 4.0 sec  84.6 MBytes   710 Mbits/sec
[  3]  4.0- 5.0 sec  80.6 MBytes   676 Mbits/sec
[  3]  5.0- 6.0 sec  76.4 MBytes   641 Mbits/sec
[  3]  6.0- 7.0 sec  77.4 MBytes   649 Mbits/sec
[  3]  7.0- 8.0 sec  77.2 MBytes   648 Mbits/sec
[  3]  8.0- 9.0 sec  81.9 MBytes   687 Mbits/sec
[  3]  9.0-10.0 sec  80.6 MBytes   676 Mbits/sec
[  3]  0.0-10.0 sec   744 MBytes   624 Mbits/sec

场景三、Node与非同主机的Pod之间

------------------------------------------------------------
Client connecting to 10.96.155.246, TCP port 12345
TCP window size: 40.0 KByte (WARNING: requested 20.0 KByte)
------------------------------------------------------------
[  3] local 172.16.106.130 port 36676 connected with 10.96.155.246 port 12345
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  71.6 MBytes   601 Mbits/sec
[  3]  1.0- 2.0 sec  75.8 MBytes   635 Mbits/sec
[  3]  2.0- 3.0 sec  73.8 MBytes   619 Mbits/sec
[  3]  3.0- 4.0 sec  65.2 MBytes   547 Mbits/sec
[  3]  4.0- 5.0 sec  58.5 MBytes   491 Mbits/sec
[  3]  5.0- 6.0 sec  62.5 MBytes   524 Mbits/sec
[  3]  6.0- 7.0 sec  66.4 MBytes   557 Mbits/sec
[  3]  7.0- 8.0 sec  62.6 MBytes   525 Mbits/sec
[  3]  8.0- 9.0 sec  68.8 MBytes   577 Mbits/sec
[  3]  9.0-10.0 sec  66.1 MBytes   555 Mbits/sec
[  3]  0.0-10.0 sec   671 MBytes   563 Mbits/sec