Kubernetes 部署配置中心 Apollo 1.4.0
系统环境:
- Apollo 版本:1.4.0
- Kubernetes 版本:1.14.0
参考:
- Apollo Github 地址:https://github.com/ctripcorp/apollo
注意:部署比较复杂,极容易出错,请严格按步骤执行!
一、Apollo 简介
Apollo(阿波罗)是携程框架部门研发的分布式配置中心,能够集中化管理应用不同环境、不同集群的配置,配置修改后能够实时推送到应用端,并且具备规范的权限、流程治理等特性,适用于微服务配置管理场景。
服务端基于 Spring Boot 和 Spring Cloud 开发,打包后可以直接运行,不需要额外安装Tomcat等应用容器。且 Java 客户端不依赖任何框架,能够运行于所有 Java 运行时环境,同时对 Spring/Spring Boot 环境也有较好的支持。
二、Apollo 特性
- 部署简单
- 灰度发布
- 版本发布管理
- 提供开放平台API
- 客户端配置信息监控
- 提供Java和.Net原生客户端
- 配置修改实时生效(热发布)
- 权限管理、发布审核、操作审计
- 统一管理不同环境、不同集群的配置
三、Apollo 环境描述
Apollo 提供了四套环境配置:
- DEV:开发环境
- FAT:功能验收测试环境,用于软件测试者测试使用
- UAT:用户验收测试环境,用于生产环境下的软件测试者测试使用
- PRO:生成环境
每个环境都需要安装相应组件,这里为了简单,只部署 DEV 与 PRO 环境。
四、获取 apollo 压缩包
从 https://github.com/ctripcorp/apollo/releases 下载预先打好的 zip 包:
- apollo-1.4.0.zip
- apollo-portal-1.4.0-github.zip
- apollo-adminservice-1.4.0-github.zip
- apollo-configservice-1.4.0-github.zip
#下载 Apollo 源码$ wget https://github.com/ctripcorp/apollo/archive/v1.4.0.zip
#下载 Apollo Admin Server$ wget https://github.com/ctripcorp/apollo/releases/download/v1.4.0/apollo-adminservice-1.4.0-github.zip
#下载 Apollo Config Service$ wget https://github.com/ctripcorp/apollo/releases/download/v1.4.0/apollo-configservice-1.4.0-github.zip
#下载 Apollo Portal$ wget https://github.com/ctripcorp/apollo/releases/download/v1.4.0/apollo-portal-1.4.0-github.zip五、解压 apollo 压缩包
将源码包中部署文件复制到自定义目录:
#解压源码包$ unzip v1.4.0.zip
#创建用于部署的自定义文件夹$ mkdir ./deploy
#将部署文件和脚本复制到自定义 deploy 目录$ cp -R ./apollo-1.4.0/scripts/apollo-on-kubernetes/* ./deploy解压三个组件并放置在 deploy 目录下对应的文件夹下:
#解压各个组件压缩包$ unzip -o apollo-configservice-1.4.0-github.zip -d ./deploy/apollo-config-server/$ unzip -o apollo-adminservice-1.4.0-github.zip -d ./deploy/apollo-admin-server/$ unzip -o apollo-portal-1.4.0-github.zip -d ./deploy/apollo-portal-server/
#应用 Jar 重命名$ mv ./deploy/apollo-config-server/apollo-configservice-1.4.0.jar ./deploy/apollo-config-server/apollo-configservice.jar$ mv ./deploy/apollo-admin-server/apollo-adminservice-1.4.0.jar ./deploy/apollo-admin-server/apollo-adminservice.jar$ mv ./deploy/apollo-portal-server/apollo-portal-1.4.0.jar ./deploy/apollo-portal-server/apollo-portal.jar六、构建 Docker 镜像
#构建 alpine 镜像$ docker build -t alpine-bash:3.8 ./deploy/alpine-bash-3.8-image
#构建 config-service 镜像$ docker build -t apollo-config-server:v1.4.0 ./deploy/apollo-config-server
#构建 admin-server$ docker build -t apollo-admin-server:v1.4.0 ./deploy/apollo-admin-server
#构建 portal 镜像$ docker build -t apollo-portal-server:v1.4.0 ./deploy/apollo-portal-server七、推送镜像到镜像仓库
推送镜像到镜像仓库,方便在 Kubernetes 中部署应用。
#打标签$ docker tag alpine-bash:3.8 mydlqclub/alpine-bash:3.8$ docker tag apollo-config-server:v1.4.0 mydlqclub/apollo-config-server:v1.4.0$ docker tag apollo-admin-server:v1.4.0 mydlqclub/apollo-admin-server:v1.4.0$ docker tag apollo-portal-server:v1.4.0 mydlqclub/apollo-portal-server:v1.4.0
#推送镜像到镜像仓库$ docker push mydlqclub/alpine-bash:3.8$ docker push mydlqclub/apollo-config-server:v1.4.0$ docker push mydlqclub/apollo-admin-server:v1.4.0$ docker push mydlqclub/apollo-portal-server:v1.4.0八、Kubernetes 中部署 Mysql
部署 Mysql 数据库
创建 Mysql 的 PV/PVC 存储资源,这里是通过静态方式创建 PV 与 PVC 绑定,用的是 NFS 存储,创建 PV 时需要指定 NFS Server 地址。
mysql-storage.yaml
apiVersion: v1kind: PersistentVolumemetadata: name: mysql-pv labels: app: mysql-pvspec: accessModes: - ReadWriteOnce capacity: storage: 5Gi mountOptions: #NFS挂在选项 - hard - nfsvers=4 nfs: #NFS设置 server: 192.168.2.11 #替换nfs服务器地址 path: /nfs/data/mysql #替换nfs存储目录---kind: PersistentVolumeClaimapiVersion: v1metadata: name: mysql-pvcspec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi #设置大小为5G selector: matchLabels: app: mysql-pv创建 Mysql 的配置文件,注意,配置文件中一定要设置最大连接数参数,否则 Apollo 启动时连接池会创建大量与 Mysql 的连接,如果设置最大连接数太小,很可能导致 Apollo 重启。
kind: ConfigMapapiVersion: v1metadata: name: mysql-config labels: app: mysqldata: my.cnf: |- [client] default-character-set=utf8mb4 [mysqld] max_connections = 2000创建 Service 与 Deployment 部署文件
mysql-deploy.yaml
apiVersion: v1kind: Servicemetadata: name: mysql labels: app: mysqlspec: type: NodePort ports: - name: mysql port: 3306 targetPort: 3306 nodePort: 30336 selector: app: mysql---apiVersion: apps/v1kind: Deploymentmetadata: name: mysql labels: app: mysqlspec: replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: "mysql:5.7.26" imagePullPolicy: "IfNotPresent" ports: - name: mysql containerPort: 3306 env: - name: MYSQL_ROOT_PASSWORD #root用户密码 value: "123456" resources: limits: cpu: 1000m memory: 512Mi requests: cpu: 100m memory: 256Mi livenessProbe: #活性探针 initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 3 exec: command: ["mysqladmin", "-uroot", "-p${MYSQL_ROOT_PASSWORD}", "ping"] readinessProbe: #就绪探针 initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 3 exec: command: ["mysqladmin", "-uroot", "-p${MYSQL_ROOT_PASSWORD}", "ping"] volumeMounts: - name: data mountPath: /var/lib/mysql - name: config mountPath: /etc/mysql/my.cnf subPath: my.cnf volumes: - name: data persistentVolumeClaim: claimName: mysql-pvc #绑定上面创建的 PVC - name: config configMap: name: mysql-config #使用 ConfigMap 中的 Mysql 配置kubernetes 集群部署 Mysql
- -n:指定应用在 Kubernetes 中部署的 Namespace
# 部署存储$ kubectl apply -f mysql-storage.yaml -n mydlqclub
# 部署 Mysql$ kubectl apply -f mysql-deploy.yaml -n mydlqclub导入 SQL 文件
进入 sources/apollo-1.4.0/scripts/apollo-on-kubernetes/db 文件夹下,内容如下:
.├── config-db-dev│ └── apolloconfigdb.sql├── config-db-prod│ └── apolloconfigdb.sql├── config-db-test-alpha│ └── apolloconfigdb.sql├── config-db-test-beta│ └── apolloconfigdb.sql└── portal-db └── apolloportaldb.sql将 config-db-dev、config-db-prod 与 portal-db 目录下的 sql 文件导入上面创建的数据库中。
九、Kubernetes 中部署 Apollo
注意:请把下面所有的
mydlqcloud为本人部署Apollo的Namespace,所以这里要将全部的mydlqcloud改成你做自己要部署Apollo的Kubernetes中Namespace名称。另一个就是,部署前确保数据库中已经导入 SQL 文件,否则会就绪探针探活失败错误。
1、部署 Dev 环境 Apollo 组件
(1)、apollo-config-server-dev 部署文件
apollo-config-server-dev.yaml
# Configmap 文件,注意修改数据库配置、用户名、密码参数kind: ConfigMapapiVersion: v1metadata: name: configmap-apollo-config-server-devdata: application-github.properties: | spring.datasource.url = jdbc:mysql://mysql.mydlqcloud:3306/DevApolloConfigDB?characterEncoding=utf8 spring.datasource.username = root spring.datasource.password = 123456 eureka.service.url = http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:8080/eureka/---kind: ServiceapiVersion: v1metadata: name: service-apollo-meta-server-dev labels: app: service-apollo-meta-server-devspec: type: ClusterIP clusterIP: None sessionAffinity: ClientIP ports: - port: 8080 targetPort: 8080 selector: app: pod-apollo-config-server-dev---kind: ServiceapiVersion: v1metadata: name: service-apollo-config-server-dev labels: app: service-apollo-config-server-devspec: type: NodePort sessionAffinity: ClientIP ports: - port: 8080 targetPort: 8080 nodePort: 30002 #NodePort方式暴露配置服务端地址,用于Kubernetes集群外部访问、测试 selector: app: pod-apollo-config-server-dev---kind: StatefulSetapiVersion: apps/v1metadata: name: statefulset-apollo-config-server-dev labels: app: statefulset-apollo-config-server-devspec: serviceName: service-apollo-meta-server-dev replicas: 3 selector: matchLabels: app: pod-apollo-config-server-dev updateStrategy: type: RollingUpdate template: metadata: labels: app: pod-apollo-config-server-dev spec: containers: - image: mydlqclub/apollo-config-server:v1.4.0 name: container-apollo-config-server-dev ports: - containerPort: 8080 env: - name: APOLLO_CONFIG_SERVICE_NAME value: "service-apollo-config-server-dev.mydlqcloud" #修改namespace名称 resources: limits: cpu: 2000m memory: 512Mi requests: cpu: 500m memory: 256Mi readinessProbe: tcpSocket: port: 8080 initialDelaySeconds: 10 periodSeconds: 5 livenessProbe: tcpSocket: port: 8080 initialDelaySeconds: 120 periodSeconds: 10 volumeMounts: - name: volume-configmap-apollo-config-server-dev mountPath: /apollo-config-server/config/application-github.properties subPath: application-github.properties volumes: - name: volume-configmap-apollo-config-server-dev configMap: name: configmap-apollo-config-server-dev items: - key: application-github.properties path: application-github.properties(2)、appllp-admin-server-dev 部署文件
appllp-admin-server-dev.yaml
kind: ConfigMapapiVersion: v1metadata: name: configmap-apollo-admin-server-devdata: application-github.properties: | spring.datasource.url = jdbc:mysql://mysql.mydlqcloud:3306/DevApolloConfigDB?characterEncoding=utf8 spring.datasource.username = root spring.datasource.password = 123456 eureka.service.url = http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:8080/eureka/---kind: ServiceapiVersion: v1metadata: name: service-apollo-admin-server-dev labels: app: service-apollo-admin-server-devspec: type: ClusterIP sessionAffinity: ClientIP ports: - port: 8090 targetPort: 8090 selector: app: pod-apollo-admin-server-dev---kind: DeploymentapiVersion: apps/v1metadata: name: deployment-apollo-admin-server-dev labels: app: deployment-apollo-admin-server-devspec: replicas: 3 selector: matchLabels: app: pod-apollo-admin-server-dev strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 template: metadata: labels: app: pod-apollo-admin-server-dev spec: initContainers: - image: mydlqclub/alpine-bash:3.8 name: check-service-apollo-config-server-dev #注意修改namespace command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-config-server-dev.mydlqcloud:8080"] containers: - image: mydlqclub/apollo-admin-server:v1.4.0 name: container-apollo-admin-server-dev ports: - containerPort: 8090 env: - name: APOLLO_ADMIN_SERVICE_NAME value: "service-apollo-admin-server-dev.mydlqcloud" # 注意修改 Namespace resources: limits: cpu: 2000m memory: 512Mi requests: cpu: 500m memory: 256Mi readinessProbe: tcpSocket: port: 8090 initialDelaySeconds: 10 periodSeconds: 5 livenessProbe: tcpSocket: port: 8090 initialDelaySeconds: 120 periodSeconds: 10 volumeMounts: - name: volume-configmap-apollo-admin-server-dev mountPath: /apollo-admin-server/config/application-github.properties subPath: application-github.properties volumes: - name: volume-configmap-apollo-admin-server-dev configMap: name: configmap-apollo-admin-server-dev items: - key: application-github.properties path: application-github.properties(3)、kubernetes 部署 Dev 环境的 admin、config
- -n:指定应用在 Kubernetes 中部署的 Namespace
# 部署 apollo-config-server-dev$ kubectl apply -f apollo-config-server-dev.yaml -n mydlqclub
# 部署 appllp-admin-server-dev$ kubectl apply -f appllp-admin-server-dev.yaml -n mydlqclub2、部署 Prod 环境 Apollo 组件
(1)、apollo-config-server-prod 部署文件
apollo-config-server-prod.yaml
kind: ConfigMapapiVersion: v1metadata: name: configmap-apollo-config-server-proddata: application-github.properties: | spring.datasource.url = jdbc:mysql://mysql.mydlqcloud:3306/ProdApolloConfigDB?characterEncoding=utf8 spring.datasource.username = root spring.datasource.password = 123456 eureka.service.url = http://statefulset-apollo-config-server-prod-0.service-apollo-meta-server-prod:8080/eureka/,http://statefulset-apollo-config-server-prod-1.service-apollo-meta-server-prod:8080/eureka/,http://statefulset-apollo-config-server-prod-2.service-apollo-meta-server-prod:8080/eureka/---kind: ServiceapiVersion: v1metadata: name: service-apollo-meta-server-prod labels: app: service-apollo-meta-server-prodspec: type: ClusterIP clusterIP: None sessionAffinity: ClientIP ports: - protocol: TCP port: 8080 targetPort: 8080 selector: app: pod-apollo-config-server-prod---kind: ServiceapiVersion: v1metadata: name: service-apollo-config-server-prod labels: app: service-apollo-config-server-prodspec: type: NodePort sessionAffinity: ClientIP ports: - port: 8080 targetPort: 8080 nodePort: 30005 selector: app: pod-apollo-config-server-prod---kind: StatefulSetapiVersion: apps/v1metadata: name: statefulset-apollo-config-server-prod labels: app: statefulset-apollo-config-server-prodspec: serviceName: service-apollo-meta-server-prod replicas: 3 selector: matchLabels: app: pod-apollo-config-server-prod updateStrategy: type: RollingUpdate template: metadata: labels: app: pod-apollo-config-server-prod spec: containers: - image: mydlqclub/apollo-config-server:v1.4.0 name: container-apollo-config-server-prod ports: - containerPort: 8080 env: - name: APOLLO_CONFIG_SERVICE_NAME value: "service-apollo-config-server-prod.mydlqcloud" # 注意修改 Namespace readinessProbe: tcpSocket: port: 8080 initialDelaySeconds: 10 periodSeconds: 5 livenessProbe: tcpSocket: port: 8080 initialDelaySeconds: 120 periodSeconds: 10 volumeMounts: - name: volume-configmap-apollo-config-server-prod mountPath: /apollo-config-server/config/application-github.properties subPath: application-github.properties volumes: - name: volume-configmap-apollo-config-server-prod configMap: name: configmap-apollo-config-server-prod items: - key: application-github.properties path: application-github.properties(2)、appllp-admin-server-prod 部署文件
appllp-admin-server-prod.yaml
kind: ConfigMapapiVersion: v1metadata: name: configmap-apollo-admin-server-proddata: application-github.properties: | spring.datasource.url = jdbc:mysql://mysql.mydlqcloud:3306/ProdApolloConfigDB?characterEncoding=utf8 spring.datasource.username = root spring.datasource.password = 123456 eureka.service.url = http://statefulset-apollo-config-server-prod-0.service-apollo-meta-server-prod:8080/eureka/,http://statefulset-apollo-config-server-prod-1.service-apollo-meta-server-prod:8080/eureka/,http://statefulset-apollo-config-server-prod-2.service-apollo-meta-server-prod:8080/eureka/---kind: ServiceapiVersion: v1metadata: name: service-apollo-admin-server-prod labels: app: service-apollo-admin-server-prodspec: ports: - protocol: TCP port: 8090 targetPort: 8090 selector: app: pod-apollo-admin-server-prod type: ClusterIP sessionAffinity: ClientIP---kind: DeploymentapiVersion: apps/v1metadata: name: deployment-apollo-admin-server-prod labels: app: deployment-apollo-admin-server-prodspec: replicas: 3 selector: matchLabels: app: pod-apollo-admin-server-prod strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 template: metadata: labels: app: pod-apollo-admin-server-prod spec: initContainers: - image: mydlqclub/alpine-bash:3.8 name: check-service-apollo-config-server-prod # 注意修改 Namespace command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 50 --retry-delay 1 --retry-max-time 120 service-apollo-config-server-prod.mydlqcloud:8080"] containers: - image: mydlqclub/apollo-admin-server:v1.4.0 name: container-apollo-admin-server-prod ports: - containerPort: 8090 env: - name: APOLLO_ADMIN_SERVICE_NAME value: "service-apollo-admin-server-prod.mydlqcloud" # 注意修改 Namespace readinessProbe: tcpSocket: port: 8090 initialDelaySeconds: 10 periodSeconds: 5 livenessProbe: tcpSocket: port: 8090 initialDelaySeconds: 120 periodSeconds: 10 volumeMounts: - name: volume-configmap-apollo-admin-server-prod mountPath: /apollo-admin-server/config/application-github.properties subPath: application-github.properties volumes: - name: volume-configmap-apollo-admin-server-prod configMap: name: configmap-apollo-admin-server-prod items: - key: application-github.properties path: application-github.properties(3)、kubernetes 部署 Dev 环境的 admin、config
- -n:指定应用在 Kubernetes 中部署的 Namespace
# 部署 apollo-config-server-prod$ kubectl apply -f apollo-config-server-prod.yaml -n mydlqclub
# 部署 appllp-admin-server-prod$ kubectl apply -f appllp-admin-server-prod.yaml -n mydlqclub3、部署 Apollo Portal
(1)、service-apollo-portal-server 部署文件
下面是 Portal 的部署文件,里面有个初始化镜像,是用于进行检测 Config Server、Admin Server 是否正常启动。这里需要注意,如果你上面只部署了 DEV 环境,那么请看部署文件中的注释,删除 Pro 环境的检测配置,否则服务将不能正常启动。
service-apollo-portal-server.yaml
# Configmap 中配置要使用的环境,这里只用 DEV 与 PRO 两个环境, 注意修改下面的参数中的 Namespacekind: ConfigMapapiVersion: v1metadata: name: configmap-apollo-portal-serverdata: application-github.properties: | spring.datasource.url = jdbc:mysql://mysql.mydlqcloud:3306/ApolloPortalDB?characterEncoding=utf8 spring.datasource.username = root spring.datasource.password = 123456 apollo-env.properties: | dev.meta=http://service-apollo-config-server-dev.mydlqcloud:8080 pro.meta=http://service-apollo-config-server-prod.mydlqcloud:8080---kind: ServiceapiVersion: v1metadata: name: service-apollo-portal-server labels: app: service-apollo-portal-serverspec: type: NodePort sessionAffinity: ClientIP # portal session 保持 ports: - protocol: TCP port: 8070 targetPort: 8070 nodePort: 30011 selector: app: pod-apollo-portal-server---kind: DeploymentapiVersion: apps/v1metadata: name: deployment-apollo-portal-server labels: app: deployment-apollo-portal-serverspec: replicas: 3 selector: matchLabels: app: pod-apollo-portal-server strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 template: metadata: labels: app: pod-apollo-portal-server spec: initContainers: # 测试 admin-service 服务是否能正常提供服务,如果执行失败,Portal 服务将不能正常启动,注意修改下面的 Namespace #### ----- 测 Dev 环境的配置 ------- - image: mydlqclub/alpine-bash:3.8 name: check-service-apollo-admin-server-dev command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-admin-server-dev.mydlqcloud:8090"] #### ------- 检测 Prod 环境的配置,如果你没有创建 Prod 环境,请去掉下面这段配置 ------- - image: mydlqclub/alpine-bash:3.8 name: check-service-apollo-admin-server-prod command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-admin-server-prod.mydlqcloud:8090"] ### ----------------------------------------------------------------------------- containers: - image: mydlqclub/apollo-portal-server:v1.4.0 name: container-apollo-portal-server ports: - containerPort: 8070 env: - name: APOLLO_PORTAL_SERVICE_NAME value: "service-apollo-portal-server.mydlqcloud" # 注意修改 Namespace readinessProbe: tcpSocket: port: 8070 initialDelaySeconds: 10 periodSeconds: 5 livenessProbe: tcpSocket: port: 8070 initialDelaySeconds: 120 periodSeconds: 15 volumeMounts: - name: volume-configmap-apollo-portal-server mountPath: /apollo-portal-server/config/application-github.properties subPath: application-github.properties - name: volume-configmap-apollo-portal-server mountPath: /apollo-portal-server/config/apollo-env.properties subPath: apollo-env.properties volumes: - name: volume-configmap-apollo-portal-server configMap: name: configmap-apollo-portal-server items: - key: application-github.properties path: application-github.properties - key: apollo-env.properties path: apollo-env.properties(2)、kubernetes 部署 Apollo Portal
- -n:指定应用在 Kubernetes 中部署的 Namespace
$ kubectl apply -f service-apollo-portal-server.yaml -n mydlqclub十、访问 Apollo Portal 界面
本人 Kubernetes 集群地址为 192.168.2.11,并且刚刚创建 Portal 时候配置的 Service 设置的 NodePort 端口为 30011,所以这里输入地址:http://192.168.2.11:30011 访问 Apollo Portal 界面,进入 Apollo 配置中心。
默认用户名密码:
- 用户名:apollo
- 密 码:admin


