Kubernetes上部署rabbitmq集群

Posted on | In | |

概述

之前文章讲述了如何通过docker命令来手动创建一个rabbitmq集群；可以通过其了解rabbitmq构建的步骤和基本原理；

随着Kubernetes的逐渐流行，如何通过Kubernetes的方式来部署一个rabbitmq集群呢？下面讲述其步骤。

创建rabbitmq集群

这里我们使用的docker image有一个：

  1. rabbitmq:3.7-rc-management

创建rabbitmq集群的yaml文件

下面yaml中涉及的Kubernetes组件有：

  1. Service

    • 提供http服务的service
    • 提供StatfulSet的headless service

  2. StatefulSet

    有状态pods的集合，真正提供rabbitmq的服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
root@kmaster:~/yangguanjun# cat rabbitmq.yaml
---
apiVersion: v1
kind: Service
metadata:
  # Expose the management HTTP port on each node
  name: rabbitmq-management
  labels:
    app: rabbitmq
spec:
  ports:
  - port: 15672
    name: http
  selector:
    app: rabbitmq
  type: NodePort # Or LoadBalancer in production w/ proper security
---
apiVersion: v1
kind: Service
metadata:
  # The required headless service for StatefulSets
  name: rabbitmq
  labels:
    app: rabbitmq
spec:
  ports:
  - port: 5672
    name: amqp
  - port: 4369
    name: epmd
  - port: 25672
    name: rabbitmq-dist
  clusterIP: None
  selector:
    app: rabbitmq
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: rabbitmq
spec:
  serviceName: "rabbitmq"
  replicas: 3
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      containers:
      - name: rabbitmq
        image: rabbitmq:3.7-rc-management
        lifecycle:
          postStart:
            exec:
              command:
              - /bin/sh
              - -c
              - >
                if [ -z "$(grep rabbitmq /etc/resolv.conf)" ]; then
                  sed "s/^search \([^ ]\+\)/search rabbitmq.\1 \1/" /etc/resolv.conf > /etc/resolv.conf.new;
                  cat /etc/resolv.conf.new > /etc/resolv.conf;
                  rm /etc/resolv.conf.new;
                fi;
                until rabbitmqctl node_health_check; do sleep 1; done;
                if [ -z "$(rabbitmqctl cluster_status | grep rabbitmq-0)" ]; then
                  touch /gotit
                  rabbitmqctl stop_app;
                  rabbitmqctl reset;
                  rabbitmqctl join_cluster rabbit@rabbitmq-0;
                  rabbitmqctl start_app;
                else
                  touch /notget
                fi;
        env:
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: RABBITMQ_ERLANG_COOKIE
          value: "YZSDHWMFSMKEMBDHSGGZ"
        - name: RABBITMQ_NODENAME
          value: "rabbit@$(MY_POD_NAME)"
        ports:
        - containerPort: 5672
          name: amqp

通过Pod的postStart来执行命令，把后面的两个Pod加入rabbitmq集群。

Pod的生命周期里有两个个hook部分：PostStart，PreStop

创建不同的rabbitmq集群，需要把yaml文件的很多处rabbitmq关键字替换掉。。。

也可以通过参数来生成yaml文件

创建 rabbitmq 集群

通过kubectl命令，依据上章节的rabbitmq.yaml文件创建rabbitmq集群；

1
2
3
4
root@kmaster:~/yangguanjun# kubectl create -f rabbitmq.yaml
service "rabbitmq-management" created
service "rabbitmq" created
statefulset "rabbitmq" created

检查rabbitmq 集群

检查刚才创建的rabbitmq集群是否正常？出于什么状态？可以使用如下命令：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
root@kmaster:~/yangguanjun# kubectl get service
NAME                     CLUSTER-IP     EXTERNAL-IP   PORT(S)                       AGE
kubernetes               10.96.0.1      <none>        443/TCP                       9d
rabbitmq                 None           <none>        5672/TCP,4369/TCP,25672/TCP   21m
rabbitmq-management      10.96.158.11   <nodes>       15672:32219/TCP               21m
 
 
root@kmaster:~/yangguanjun# kubectl get pods
NAME            READY     STATUS              RESTARTS   AGE
rabbitmq-0      1/1       Running             0          21m
rabbitmq-1      1/1       Running             0          21m
rabbitmq-2      1/1       Running             0          21m
 
 
root@kmaster:~/yangguanjun# kubectl exec -it rabbitmq-0 bash
root@rabbitmq-0:/# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbitmq-0 ...
[{nodes,[{disc,['rabbit@rabbitmq-0','rabbit@rabbitmq-1',
                'rabbit@rabbitmq-2']}]},
 {running_nodes,['rabbit@rabbitmq-2','rabbit@rabbitmq-1','rabbit@rabbitmq-0']},
 {cluster_name,<<"rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local">>},
 {partitions,[]},
 {alarms,[{'rabbit@rabbitmq-2',[]},
          {'rabbit@rabbitmq-1',[]},
          {'rabbit@rabbitmq-0',[]}]}]

rabbitmq集群的域名

上面我们通过Statefulset来创建的rabbitmq集群，但是我们并没有在镜像中指定几个Pods之间如何访问，它们之间的解析是通过Headless service提供的，可以通过下面命令获取Pod里的域名解析：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
root@rabbitmq-0:/# cat /etc/resolv.conf
nameserver 10.96.0.10
search rabbitmq.default.svc.cluster.local default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
 

root@rabbitmq-0:/# dig rabbitmq.default.svc.cluster.local
 
; <<>> DiG 9.10.3-P4-Debian <<>> rabbitmq.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55697
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
 
;; QUESTION SECTION:
;rabbitmq.default.svc.cluster.local. IN A
 
;; ANSWER SECTION:
rabbitmq.default.svc.cluster.local. 30 IN A 192.168.239.175
rabbitmq.default.svc.cluster.local. 30 IN A 192.168.150.235
rabbitmq.default.svc.cluster.local. 30 IN A 192.168.152.171
 
;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri May 04 08:56:40 UTC 2018
;; MSG SIZE  rcvd: 100
 
 

root@rabbitmq-0:/# nslookup rabbitmq-1
Server:     10.96.0.10
Address:    10.96.0.10#53
 
Name:   rabbitmq-1.rabbitmq.default.svc.cluster.local
Address: 192.168.239.175

 
root@rabbitmq-0:/# nslookup rabbitmq.default.svc.cluster.local
Server:     10.96.0.10
Address:    10.96.0.10#53
 
Name:   rabbitmq.default.svc.cluster.local
Address: 192.168.150.235
Name:   rabbitmq.default.svc.cluster.local
Address: 192.168.152.171
Name:   rabbitmq.default.svc.cluster.local
Address: 192.168.239.175
 
 
root@rabbitmq-0:/# nslookup rabbitmq-management.default.svc.cluster.local
Server:     10.96.0.10
Address:    10.96.0.10#53
 
Name:   rabbitmq-management.default.svc.cluster.local
Address: 10.96.158.11

访问rabbitmq集群

创建好rabbitmq集群后，如何访问它呢？

如之前我们介绍使用的image为：rabbitmq:3.7-rc-management，它包含了rabbitmq的management组件，另外我们也创建了rabbitmq集群对应的service服务，提供了基于http端口的访问，通过下面命令获取其信息：

1
2
3
4
5
6
7
8
9
10
11
12
13
root@kmaster:~/yangguanjun# kubectl describe service/rabbitmq-management
Name:           rabbitmq-management
Namespace:      default
Labels:         app=rabbitmq
Annotations:        <none>
Selector:       app=rabbitmq
Type:           NodePort
IP:         10.96.158.11
Port:           http    15672/TCP
NodePort:       http    32219/TCP
Endpoints:      192.168.150.235:15672,192.168.152.171:15672,192.168.239.175:15672
Session Affinity:   None
Events:         <none>

则可以通过http指定IP和Port直接访问：http://10.96.158.11:15672

当然这里的IP是内网IP，若公司内外不能访问的话，可以绑定外网IP后再通过公网+port访问。

参考文档

https://blog.csdn.net/zhaohuabing/article/details/78673329
https://wesmorgan.svbtle.com/rabbitmq-cluster-on-kubernetes-with-statefulsets
https://github.com/xarg/rabbitmq-statefulset

支持原创