Skip to content

K8s方式部署 ​

前言 ​

提供一个 Kubernetes 集群,可参考:kubeadm 部署 Kubernetes

一、部署架构 ​

二、最终资源与目录 ​

所有资源位于 pig 命名空间。

资源名称Service 端口外部 NodePort
pig-mysql-stspig-mysql:3306自动分配,查询 Service
pig-redis-stspig-redis:6379自动分配,查询 Service
pig-register8848 / 9848 / 1808030848 / 31848 / 30080
pig-gateway9999无
pig-auth3000无
pig-upms-biz4000无
pig-monitor5001无
pig-codegen5003无
pig-quartz5007无
pig-ui80临时:30081

三、部署前提与 MySQL、Redis ​

已运行的集群以检查为主,无需重复初始化数据库。

bash
kubectl get nodes -o wide
kubectl get sc nfs-dynamic
kubectl -n pig get pods,svc,pvc

当前 MySQL、Redis 密码均为 root。MySQL、Redis 的 Service selector 已分别指向 name: pig-mysql-sts、name: pig-redis-sts。

3.1 StatefulSet 部署 MySQL ​

文件:pig-k8s/mysql/mysql-sts.yaml

yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: pig-mysql-sts
  namespace: pig
  labels:
    name: pig-mysql-sts
spec:
  replicas: 1
  selector:
    matchLabels:
      name: pig-mysql-sts
  template:
    metadata:
      labels:
        name: pig-mysql-sts
    spec:
      # 给 mysqld 足够时间完成首次初始化和优雅关闭,尤其是使用 NFS 存储时。
      terminationGracePeriodSeconds: 120
      imagePullSecrets:
        - name: pig-harbor-secret
      securityContext:
        fsGroup: 999
      containers:
        - name: pig-mysql-sts
          image: docker.cnb.cool/sre-demo/k8s-demo/mysql:8.0.38_amd64
          imagePullPolicy: IfNotPresent
          args:
            - --max-connections=1000
            - --disable_log_bin
            - --lower_case_table_names=1
            - --character-set-server=utf8mb4
            - --collation-server=utf8mb4_general_ci
            - --log_timestamps=SYSTEM
            - --max-user-connections=500
            - --wait-timeout=300
            - --interactive-timeout=300
            - --max-allowed-packet=256M
            - --innodb-buffer-pool-size=2G
            - --table_open_cache=16384
            - --table_definition_cache=16384
            - --innodb_redo_log_capacity=1G
            - --join_buffer_size=512K
            - --innodb_log_buffer_size=16M
          ports:
            - name: mysql
              containerPort: 3306
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: "root"
          resources:
            limits:
              cpu: "4"
              memory: 8Gi
            requests:
              cpu: "2"
              memory: 4Gi
          # 首次创建 InnoDB 数据文件和 1GiB redo log 在 NFS 上可能较慢。
          # startupProbe 成功前,kubelet 不会执行 readinessProbe 和 livenessProbe。
          startupProbe:
            tcpSocket:
              port: mysql
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 60
          readinessProbe:
            tcpSocket:
              port: mysql
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 6
          livenessProbe:
            tcpSocket:
              port: mysql
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 6
          volumeMounts:
            - name: data-50g-pvc
              mountPath: /var/lib/mysql
  volumeClaimTemplates:
    - metadata:
        name: data-50g-pvc
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: nfs-dynamic
        resources:
          requests:
            storage: 50Gi

文件:pig-k8s/mysql/mysql-svc.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-mysql
  namespace: pig
  labels:
    name: pig-mysql
spec:
  type: ClusterIP
  ports:
    - port: 3306
      targetPort: 3306
      protocol: TCP
      name: mysql
  selector:
    name: pig-mysql-sts

文件:pig-k8s/mysql/mysql-svc-nodeport.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-mysql-nodeport
  namespace: pig
  labels:
    name: pig-mysql-nodeport
spec:
  type: NodePort
  ports:
    - port: 3306
      targetPort: 3306
      protocol: TCP
      name: mysql
  selector:
    name: pig-mysql-sts

3.2 StatefulSet 部署 Redis ​

文件:pig-k8s/redis/redis-sts.yaml

yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: pig-redis-sts
  namespace: pig
  labels:
    name: pig-redis-sts
spec:
  replicas: 1
  selector:
    matchLabels:
      name: pig-redis-sts
  template:
    metadata:
      labels:
        name: pig-redis-sts
    spec:
      imagePullSecrets:
        - name: pig-harbor-secret
      containers:
        - name: pig-redis-sts
          #image: redis:8.10.1
          image: docker.cnb.cool/sre-demo/k8s-demo/redis:8.10.1_amd64
          imagePullPolicy: IfNotPresent
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP
          env:
            - name: TZ
              value: "Asia/Shanghai"
          command:
            - redis-server
            - --requirepass
            - "root"
            - --port
            - "6379"
            - --appendonly
            - "yes"
            - --dbfilename
            - "dump.rdb"
            - --dir
            - /data
            - --timeout
            - "0"
            - --save
            - "900 1"
            - --save
            - "300 10"
            - --save
            - "60 10000"
          resources:
            limits:
              cpu: "1"
              memory: 4096Mi
            requests:
              cpu: "500m"
              memory: 2048Mi
          readinessProbe:
            initialDelaySeconds: 10
            periodSeconds: 5
            timeoutSeconds: 3
            exec:
              command:
                - redis-cli
                - -h
                - 127.0.0.1
                - -p
                - "6379"
                - -a
                - "root"
                - ping
          livenessProbe:
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            exec:
              command:
                - redis-cli
                - -h
                - 127.0.0.1
                - -p
                - "6379"
                - -a
                - "root"
                - ping
          volumeMounts:
            - name: data-10g-pvc
              mountPath: /data
            - name: localtime
              mountPath: /etc/localtime
              readOnly: true
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
            type: File
  volumeClaimTemplates:
    - metadata:
        name: data-10g-pvc
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: nfs-dynamic
        resources:
          requests:
            storage: 10Gi

文件:pig-k8s/redis/redis-svc.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-redis
  namespace: pig
  labels:
    name: pig-redis
spec:
  type: ClusterIP
  ports:
    - port: 6379
      targetPort: 6379
      protocol: TCP
      name: redis
  selector:
    name: pig-redis-sts

文件:pig-k8s/redis/redis-svc-nodeport.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-redis-nodeport
  namespace: pig
  labels:
    name: pig-redis-nodeport
spec:
  type: NodePort
  ports:
    - port: 6379
      targetPort: 6379
      protocol: TCP
      name: redis
  selector:
    name: pig-redis-sts

四、构建并推送 Pig 镜像 ​

以下操作在包含 pig 和 pig-ui 的项目根目录执行。

4.1 设置镜像仓库变量 ​

bash
export PIG_REGISTRY='docker.cnb.cool/sre-demo/k8s-demo'
export PIG_VERSION='4.1.0'

docker login docker.cnb.cool

使用具有该 CNB 仓库推送权限的凭据登录。PIG_VERSION 是镜像标签,统一为 4.1.0,源码版本仍为 4.1.0。

4.2 修改 Nacos 数据库连接信息 ​

编译之前,找到后端代码根目录下的 pig-register/src/main/resources/application.properties,将数据库配置修改为:

properties
db.url.0=jdbc:mysql://${MYSQL_HOST:pig-mysql}:${MYSQL_PORT:3306}/${MYSQL_DB:pig_config}?characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=GMT%2B8&nullCatalogMeansCurrent=true&allowPublicKeyRetrieval=true
db.user=root
db.password=root

配置说明:

  • pig-mysql 是当前 Kubernetes 中 MySQL 的 Service 名(与 Docker Compose 服务名一致)。
  • ${MYSQL_HOST:pig-mysql} 表示优先读取 MYSQL_HOST,未设置时使用 pig-mysql。
  • Nacos 使用 pig_config 配置库,业务服务使用 pig 库,两者不要混用。
  • 当前 MySQL 账号和密码均为 root;如果实际数据库密码已修改,需要同步调整此文件;业务服务的数据库密码在 Nacos 配置中维护。

最终 Nacos Deployment 从 pig-common-env 读取 MYSQL_HOST、MYSQL_PORT,并设置 MYSQL_DB=pig_config;数据库账号和密码仍来自此文件,没有额外的运行时密码 Secret。修改完成后再编译 JAR、构建和推送镜像;单独修改源码文件不会更新已经运行的容器。

4.3 编译后端 JAR ​

后端源码使用 Maven cloud Profile。该 Profile 会把环境标识编译为 dev,并写入 Nacos 默认账号 nacos/nacos。

bash
cd pig
mvn clean install -Pcloud -DskipTests
cd ..

确认主要 JAR 已生成:

bash
ls -lh \
  pig/pig-register/target/pig-register.jar \
  pig/pig-gateway/target/pig-gateway.jar \
  pig/pig-auth/target/pig-auth.jar \
  pig/pig-upms/pig-upms-biz/target/pig-upms-biz.jar \
  pig/pig-visual/pig-monitor/target/pig-monitor.jar \
  pig/pig-visual/pig-codegen/target/pig-codegen.jar \
  pig/pig-visual/pig-quartz/target/pig-quartz.jar

4.4 构建后端镜像 ​

每个 Dockerfile 都从所在模块的 target/ 复制 JAR,因此 Docker build context 不能写成项目根目录。

bash
docker build -t "${PIG_REGISTRY}/pig-register:${PIG_VERSION}" \
  pig/pig-register

docker build -t "${PIG_REGISTRY}/pig-gateway:${PIG_VERSION}" \
  pig/pig-gateway

docker build -t "${PIG_REGISTRY}/pig-auth:${PIG_VERSION}" \
  pig/pig-auth

docker build -t "${PIG_REGISTRY}/pig-upms-biz:${PIG_VERSION}" \
  pig/pig-upms/pig-upms-biz

docker build -t "${PIG_REGISTRY}/pig-monitor:${PIG_VERSION}" \
  pig/pig-visual/pig-monitor

docker build -t "${PIG_REGISTRY}/pig-codegen:${PIG_VERSION}" \
  pig/pig-visual/pig-codegen

docker build -t "${PIG_REGISTRY}/pig-quartz:${PIG_VERSION}" \
  pig/pig-visual/pig-quartz

4.5 构建前端镜像 ​

当前前端项目没有 lock 文件,因此使用 npm install,不能直接使用 npm ci。Node.js 需要满足 package.json 中的 >=20.19.0。

bash
cd pig-ui
node --version
npm install
npm run build:docker
cd ..

docker build -t "${PIG_REGISTRY}/pig-ui:${PIG_VERSION}" \
  pig-ui/docker

npm run build:docker 会把产物生成到 pig-ui/docker/dist/,正好与前端 Dockerfile 的复制路径一致。

4.6 推送全部镜像 ​

bash
for image in \
  pig-register \
  pig-gateway \
  pig-auth \
  pig-upms-biz \
  pig-monitor \
  pig-codegen \
  pig-quartz \
  pig-ui
do
  docker push "${PIG_REGISTRY}/${image}:${PIG_VERSION}"
done

五、初始化数据库(仅首次部署) ​

当前目录已包含初始化脚本:

  • pig-k8s/db/pig_config.sql:Nacos 配置数据库。
  • pig-k8s/db/pig.sql:Pig 业务数据库。

WARNING

脚本含删表重建操作。已经运行成功的环境不要重复导入;下面命令只用于新的空数据库。

bash
kubectl -n pig exec -i pig-mysql-sts-0 -- \
  sh -c 'exec mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' \
  < pig-k8s/db/pig_config.sql

kubectl -n pig exec -i pig-mysql-sts-0 -- \
  sh -c 'exec mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' \
  < pig-k8s/db/pig.sql

检查:

bash
kubectl -n pig exec pig-mysql-sts-0 -- \
  sh -c 'mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "SHOW DATABASES"' | \
  grep -E '^pig$|^pig_config$'

六、命名空间、镜像凭据与公共配置 ​

6.1 命名空间与镜像拉取凭据 ​

创建 pig 命名空间:

bash
kubectl create namespace pig

创建镜像拉取凭证:

bash
kubectl -n pig create secret docker-registry pig-cnb-secret \
  --docker-server=docker.cnb.cool \
  --docker-username='cnb' \
  --docker-password='XXXXXX'

6.2 公共 ConfigMap ​

文件:pig-k8s/configmap/pig-common-env.yaml

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: pig-common-env
  namespace: pig
data:
  TZ: Asia/Shanghai
  NACOS_HOST: pig-register
  NACOS_PORT: "8848"
  MYSQL_HOST: pig-mysql
  MYSQL_PORT: "3306"
  REDIS_HOST: pig-redis
  REDIS_PORT: "6379"
  SPRING_DATA_REDIS_HOST: pig-redis
  SPRING_DATA_REDIS_PORT: "6379"

七、部署 Nacos 注册与配置中心 ​

7.1 Service 与 Deployment ​

文件:pig-k8s/be/svc/pig-register-svc.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-register
  namespace: pig
spec:
  type: ClusterIP
  selector:
    app: pig-register
  ports:
    - name: nacos-http
      port: 8848
      targetPort: nacos-http
    - name: nacos-grpc
      port: 9848
      targetPort: nacos-grpc
    - name: console
      port: 18080
      targetPort: console

文件:pig-k8s/be/svc/pig-register-svc-nodeport.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-register-nodeport
  namespace: pig
spec:
  type: NodePort
  selector:
    app: pig-register
  ports:
    - name: nacos-http
      port: 8848
      targetPort: nacos-http
      nodePort: 30848
    - name: nacos-grpc
      port: 9848
      targetPort: nacos-grpc
      nodePort: 31848
    - name: console
      port: 18080
      targetPort: console
      nodePort: 30080

文件:pig-k8s/be/deployment/pig-register-deployment.yaml

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pig-register
  namespace: pig
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: pig-register
  template:
    metadata:
      labels:
        app: pig-register
    spec:
      imagePullSecrets:
        - name: pig-cnb-secret
      containers:
        - name: pig-register
          image: docker.cnb.cool/sre-demo/k8s-demo/pig-register:4.1.0
          imagePullPolicy: Always
          envFrom:
            - configMapRef:
                name: pig-common-env
          env:
            - name: MYSQL_DB
              value: pig_config
            - name: JAVA_OPTS
              value: >-
                -Xms512m -Xmx1024m
                -Djava.security.egd=file:/dev/./urandom
          ports:
            - name: nacos-http
              containerPort: 8848
            - name: nacos-grpc
              containerPort: 9848
            - name: console
              containerPort: 18080
          startupProbe:
            tcpSocket:
              port: nacos-http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 60
          readinessProbe:
            tcpSocket:
              port: nacos-http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 6
          livenessProbe:
            tcpSocket:
              port: nacos-http
            periodSeconds: 15
            timeoutSeconds: 3
            failureThreshold: 6
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: "2"
              memory: 2Gi
bash
kubectl apply -f pig-k8s/be/svc/pig-register-svc.yaml
kubectl apply -f pig-k8s/be/svc/pig-register-svc-nodeport.yaml
kubectl apply -f pig-k8s/be/deployment/pig-register-deployment.yaml
kubectl -n pig rollout status deployment/pig-register --timeout=600s

控制台:http://172.22.33.101:30080/。集群内部使用 pig-register:8848。默认 Nacos 账号 nacos/nacos。

7.2 配置业务服务使用的 MySQL 和 Redis ​

Nacos 自身使用镜像内的 application.properties 连接 pig_config 库;业务服务启动后从 Nacos 读取配置。启动业务 Deployment 前,先登录 Nacos 控制台,在当前客户端使用的命名空间和 DEFAULT_GROUP 中修改并发布以下配置。

配置文件 Data Id修改配置信息
application-dev.ymlRedis 配置
pig-upms-biz-dev.yml数据库链接信息
pig-codegen-dev.yml数据库链接信息
pig-quartz-dev.yml数据库链接信息
  1. 编辑 application-dev.yml,将现有的 spring.data.redis 段改为下面的内容,将 Redis 数据源的地址、账号、密码改为实际值,保留其他配置:
yaml
spring:
  data:
    redis:
      host: pig-redis
      port: 6379
      password: "root"
  1. 分别编辑 pig-upms-biz-dev.yml、pig-codegen-dev.yml、pig-quartz-dev.yml,将 MySQL 数据源的地址、账号、密码改为实际值,保留其他配置:
yaml
spring:
  datasource:
    type: com.alibaba.druid.pool.DruidDataSource
    druid:
      driver-class-name: com.mysql.cj.jdbc.Driver
      username: ${MYSQL_USER:root}
      password: ${MYSQL_PWD:root}
      url: jdbc:mysql://${MYSQL_HOST:pig-mysql}:${MYSQL_PORT:3306}/${MYSQL_DB:pig}?characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=GMT%2B8&allowMultiQueries=true&allowPublicKeyRetrieval=true

八、部署后端服务 ​

服务名称端口集群内 svc 通讯地址
pig-gateway9999pig-gateway
pig-auth3000pig-auth
pig-upms-biz4000pig-upms-biz
pig-monitor5001pig-monitor
pig-codegen5003pig-codegen
pig-quartz5007pig-quartz

8.1 pig-gateway ​

文件:pig-k8s/be/deployment/pig-gateway-deployment.yaml

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pig-gateway
  namespace: pig
  labels:
    name: pig-gateway
    type: be
    app: pig
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      name: pig-gateway
  template:
    metadata:
      labels:
        name: pig-gateway
        type: be
        app: pig
    spec:
      imagePullSecrets:
        - name: pig-cnb-secret
      containers:
        - name: pig-gateway
          image: docker.cnb.cool/sre-demo/k8s-demo/pig-gateway:4.1.0
          imagePullPolicy: Always
          envFrom:
            - configMapRef:
                name: pig-common-env
          ports:
            - name: http
              containerPort: 9999
          startupProbe:
            tcpSocket:
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 36
          readinessProbe:
            tcpSocket:
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 6
          livenessProbe:
            tcpSocket:
              port: http
            periodSeconds: 15
            timeoutSeconds: 3
            failureThreshold: 6
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi

文件:pig-k8s/be/svc/pig-gateway-svc.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-gateway
  namespace: pig
  labels:
    name: pig-gateway
    type: be
    app: pig
spec:
  selector:
    name: pig-gateway
  ports:
    - name: http
      port: 9999
      targetPort: http
bash
kubectl apply -f pig-k8s/be/svc/pig-gateway-svc.yaml
kubectl apply -f pig-k8s/be/deployment/pig-gateway-deployment.yaml
kubectl -n pig rollout status deployment/pig-gateway --timeout=600s

8.2 pig-auth ​

文件:pig-k8s/be/deployment/pig-auth-deployment.yaml

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pig-auth
  namespace: pig
  labels:
    name: pig-auth
    type: be
    app: pig
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      name: pig-auth
  template:
    metadata:
      labels:
        name: pig-auth
        type: be
        app: pig
    spec:
      imagePullSecrets:
        - name: pig-cnb-secret
      containers:
        - name: pig-auth
          image: docker.cnb.cool/sre-demo/k8s-demo/pig-auth:4.1.0
          imagePullPolicy: Always
          envFrom:
            - configMapRef:
                name: pig-common-env
          ports:
            - name: http
              containerPort: 3000
          startupProbe:
            tcpSocket:
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 36
          readinessProbe:
            tcpSocket:
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 6
          livenessProbe:
            tcpSocket:
              port: http
            periodSeconds: 15
            timeoutSeconds: 3
            failureThreshold: 6
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi

文件:pig-k8s/be/svc/pig-auth-svc.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-auth
  namespace: pig
  labels:
    name: pig-auth
    type: be
    app: pig
spec:
  selector:
    name: pig-auth
  ports:
    - name: http
      port: 3000
      targetPort: http
bash
kubectl apply -f pig-k8s/be/svc/pig-auth-svc.yaml
kubectl apply -f pig-k8s/be/deployment/pig-auth-deployment.yaml
kubectl -n pig rollout status deployment/pig-auth --timeout=600s

8.3 pig-upms-biz ​

此服务需要持久化上传下载文件,先定义 pig-upload-pvc(nfs-dynamic、10Gi、ReadWriteMany),再定义 Deployment。上传路径通过 FILE_LOCAL_BASEPATH 指定为 /data/files,与 PVC 挂载位置一致。

文件:pig-k8s/be/deployment/pig-upms-biz-deployment.yaml

yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pig-upload-pvc
  namespace: pig
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: nfs-dynamic
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pig-upms-biz
  namespace: pig
  labels:
    name: pig-upms-biz
    type: be
    app: pig
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      name: pig-upms-biz
  template:
    metadata:
      labels:
        name: pig-upms-biz
        type: be
        app: pig
    spec:
      imagePullSecrets:
        - name: pig-cnb-secret
      containers:
        - name: pig-upms-biz
          image: docker.cnb.cool/sre-demo/k8s-demo/pig-upms-biz:4.1.0
          imagePullPolicy: Always
          envFrom:
            - configMapRef:
                name: pig-common-env
          env:
            - name: FILE_LOCAL_BASEPATH
              value: /data/files
          ports:
            - name: http
              containerPort: 4000
          startupProbe:
            tcpSocket:
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 36
          readinessProbe:
            tcpSocket:
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 6
          livenessProbe:
            tcpSocket:
              port: http
            periodSeconds: 15
            timeoutSeconds: 3
            failureThreshold: 6
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: "2"
              memory: 2Gi
          volumeMounts:
            - name: upload-data
              mountPath: /data/files
      volumes:
        - name: upload-data
          persistentVolumeClaim:
            # 引用本文件中定义的动态 PVC。
            claimName: pig-upload-pvc

文件:pig-k8s/be/svc/pig-upms-biz-svc.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-upms-biz
  namespace: pig
  labels:
    name: pig-upms-biz
    type: be
    app: pig
spec:
  selector:
    name: pig-upms-biz
  ports:
    - name: http
      port: 4000
      targetPort: http
bash
kubectl apply -f pig-k8s/be/svc/pig-upms-biz-svc.yaml
kubectl apply -f pig-k8s/be/deployment/pig-upms-biz-deployment.yaml
kubectl -n pig rollout status deployment/pig-upms-biz --timeout=600s

8.4 pig-monitor ​

文件:pig-k8s/be/deployment/pig-monitor-deployment.yaml

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pig-monitor
  namespace: pig
  labels:
    name: pig-monitor
    type: be
    app: pig
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      name: pig-monitor
  template:
    metadata:
      labels:
        name: pig-monitor
        type: be
        app: pig
    spec:
      imagePullSecrets:
        - name: pig-cnb-secret
      containers:
        - name: pig-monitor
          image: docker.cnb.cool/sre-demo/k8s-demo/pig-monitor:4.1.0
          imagePullPolicy: Always
          envFrom:
            - configMapRef:
                name: pig-common-env
          ports:
            - name: http
              containerPort: 5001
          startupProbe:
            tcpSocket:
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 36
          readinessProbe:
            tcpSocket:
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 6
          livenessProbe:
            tcpSocket:
              port: http
            periodSeconds: 15
            timeoutSeconds: 3
            failureThreshold: 6
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi

文件:pig-k8s/be/svc/pig-monitor-svc.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-monitor
  namespace: pig
  labels:
    name: pig-monitor
    type: be
    app: pig
spec:
  selector:
    name: pig-monitor
  ports:
    - name: http
      port: 5001
      targetPort: http
bash
kubectl apply -f pig-k8s/be/svc/pig-monitor-svc.yaml
kubectl apply -f pig-k8s/be/deployment/pig-monitor-deployment.yaml
kubectl -n pig rollout status deployment/pig-monitor --timeout=600s

8.5 pig-codegen ​

文件:pig-k8s/be/deployment/pig-codegen-deployment.yaml

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pig-codegen
  namespace: pig
  labels:
    name: pig-codegen
    type: be
    app: pig
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      name: pig-codegen
  template:
    metadata:
      labels:
        name: pig-codegen
        type: be
        app: pig
    spec:
      imagePullSecrets:
        - name: pig-cnb-secret
      containers:
        - name: pig-codegen
          image: docker.cnb.cool/sre-demo/k8s-demo/pig-codegen:4.1.0
          imagePullPolicy: Always
          envFrom:
            - configMapRef:
                name: pig-common-env
          ports:
            - name: http
              containerPort: 5003
          startupProbe:
            tcpSocket:
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 36
          readinessProbe:
            tcpSocket:
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 6
          livenessProbe:
            tcpSocket:
              port: http
            periodSeconds: 15
            timeoutSeconds: 3
            failureThreshold: 6
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi

文件:pig-k8s/be/svc/pig-codegen-svc.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-codegen
  namespace: pig
  labels:
    name: pig-codegen
    type: be
    app: pig
spec:
  selector:
    name: pig-codegen
  ports:
    - name: http
      port: 5003
      targetPort: http
bash
kubectl apply -f pig-k8s/be/svc/pig-codegen-svc.yaml
kubectl apply -f pig-k8s/be/deployment/pig-codegen-deployment.yaml
kubectl -n pig rollout status deployment/pig-codegen --timeout=600s

8.6 pig-quartz ​

文件:pig-k8s/be/deployment/pig-quartz-deployment.yaml

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pig-quartz
  namespace: pig
  labels:
    name: pig-quartz
    type: be
    app: pig
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      name: pig-quartz
  template:
    metadata:
      labels:
        name: pig-quartz
        type: be
        app: pig
    spec:
      imagePullSecrets:
        - name: pig-cnb-secret
      containers:
        - name: pig-quartz
          image: docker.cnb.cool/sre-demo/k8s-demo/pig-quartz:4.1.0
          imagePullPolicy: Always
          envFrom:
            - configMapRef:
                name: pig-common-env
          ports:
            - name: http
              containerPort: 5007
          startupProbe:
            tcpSocket:
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 36
          readinessProbe:
            tcpSocket:
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 6
          livenessProbe:
            tcpSocket:
              port: http
            periodSeconds: 15
            timeoutSeconds: 3
            failureThreshold: 6
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi

文件:pig-k8s/be/svc/pig-quartz-svc.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-quartz
  namespace: pig
  labels:
    name: pig-quartz
    type: be
    app: pig
spec:
  selector:
    name: pig-quartz
  ports:
    - name: http
      port: 5007
      targetPort: http
bash
kubectl apply -f pig-k8s/be/svc/pig-quartz-svc.yaml
kubectl apply -f pig-k8s/be/deployment/pig-quartz-deployment.yaml
kubectl -n pig rollout status deployment/pig-quartz --timeout=600s

九、前端部署与 NodePort 验证 ​

前端镜像自带静态文件和 Nginx 配置,无 env 注入、无 Nginx ConfigMap 挂载。Nginx 通过 pig-gateway:9999 代理 API(后端转发地址在 Dockerfile 阶段硬编码写死)。

文件:pig-k8s/fe/deployment/pig-ui-deployment.yaml

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pig-ui
  namespace: pig
  labels:
    name: pig-ui
    type: fe
    app: pig
spec:
  replicas: 2
  selector:
    matchLabels:
      name: pig-ui
  template:
    metadata:
      labels:
        name: pig-ui
        type: fe
        app: pig
    spec:
      imagePullSecrets:
        - name: pig-cnb-secret
      containers:
        - name: pig-ui
          image: docker.cnb.cool/sre-demo/k8s-demo/pig-ui:4.1.0
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 6
          livenessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 15
            timeoutSeconds: 3
            failureThreshold: 6
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 256Mi

文件:pig-k8s/fe/svc/pig-ui-svc.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: pig-ui
  namespace: pig
  labels:
    name: pig-ui
    type: fe
    app: pig
spec:
  type: ClusterIP
  selector:
    name: pig-ui
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: v1
kind: Service
metadata:
  name: pig-ui-nodeport
  namespace: pig
  labels:
    name: pig-ui-nodeport
    type: fe
    app: pig
spec:
  type: NodePort
  selector:
    name: pig-ui
  ports:
    - name: http
      port: 80
      targetPort: http
      nodePort: 30081
bash
kubectl apply -f pig-k8s/fe/svc/pig-ui-svc.yaml
kubectl apply -f pig-k8s/fe/deployment/pig-ui-deployment.yaml
kubectl -n pig rollout status deployment/pig-ui --timeout=300s
kubectl -n pig get svc pig-ui pig-ui-nodeport

当前已确认通过 NodePort 访问 Pig 成功。访问示例:

  • http://172.22.33.101:30081/
  • http://172.22.33.102:30081/

NodePort 是独立的直接访问入口;最终域名访问与 HTTPS 验证见第十章。前端 Nginx 配置以镜像内容为准:

bash
kubectl -n pig exec deployment/pig-ui -- nginx -T
kubectl -n pig logs deployment/pig-ui --tail=100

十、接入 Traefik 与 HTTPS ​

Traefik 配置可参考:Kubernetes 网络 Traefik

本章记录已完成的 Traefik 与 HTTPS 接入。最终链路为:

text
pig.srebro.cn → HAProxy 172.22.33.99:443
             → Traefik → pig-ui:80 → 前端 Nginx → pig-gateway:9999

10.1 检查现有入口 ​

bash
kubectl get ingressclass traefik
kubectl -n traefik get pods,svc

HAProxy 转发到现有 Traefik 的 HTTP/HTTPS NodePort,端口以实际部署配置为准。域名已解析到 172.22.33.99,TLS Secret 和 Ingress 已按以下步骤完成配置。

10.2 创建 TLS Secret ​

证书需覆盖 pig.srebro.cn 或 *.srebro.cn,Secret 必须在 pig 命名空间。按实际证书路径执行:

bash
kubectl -n pig create secret tls srebro.cn-tls \
  --cert=/root/pig-k8s/traefik/cert/srebro.cn.pem \
  --key=/root/pig-k8s/traefik/cert/srebro.cn.key \
  --dry-run=client -o yaml | kubectl apply -f -

10.3 配置 Ingress ​

服务器上的 Ingress 文件:/root/pig-k8s/traefik/ingress/pig-ui-ingress.yaml。

yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pig-ui
  namespace: pig
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  ingressClassName: traefik
  tls:
    - hosts:
        - pig.srebro.cn
      secretName: srebro.cn-tls
  rules:
    - host: pig.srebro.cn
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: pig-ui
                port:
                  number: 80

应用 Ingress 并查看结果:

bash
kubectl apply -f /root/pig-k8s/traefik/ingress/pig-ui-ingress.yaml
kubectl get ingress -n pig

示例输出:

text
NAME     CLASS     HOSTS           ADDRESS        PORTS     AGE
pig-ui   traefik   pig.srebro.cn   172.22.33.99   80, 443   19s

10.4 配置本地 DNS 解析 ​

在本地 /etc/hosts 增加:

text
172.22.33.99    pig.srebro.cn

10.5 访问测试 ​

浏览器访问:https://pig.srebro.cn

最近更新