Gitlab Helm Chart升级及部署

1,242次阅读
没有评论
内容 隐藏
1 背景
2 前期准备
3 部署阶段
3.1 拉取 helm 资源
3.2 修改 Chart 配置
3.2.1 Global values 字段
3.2.1.1 edition
3.2.1.2 hosts
3.2.1.3 ingress
3.2.1.4 initialRootPassword
3.2.1.5 pgsql
3.2.1.6 redis
3.2.1.7 minio
3.2.1.8 appConfig
3.2.1.9 ldap
3.2.1.10 smtp
3.2.1.11 email
3.2.1.12 time_zone
3.2.1.13 upgradeCheck
3.2.1.14 其他
3.2.2 Subchart 变更
3.2.2.1 Gitaly chart
3.2.2.2 toolbox chart
3.2.2.3 minio chart
3.3 正式部署
3.3.1 拉起 helm
3.3.2 配置 svc/ingress
3.4 数据迁移
3.5 Gitlab 升级

背景

Gitlab helm chart 是 gitlab 官方推荐的，在 kubernetes 中运行的方案，升级上面也有相对应的升级方案，所以从安全方面及维护方面都还是推荐用 helm chart 部署

前期准备

gitlab 持久化数据分为数据库: pgsql、redis，还有 gitlab 的仓库.

  • 创建 storageclass，用于保存 Gitlab 的仓库
  • 创建 PostgreSQL，建议最好用外置的，非 k8s 自建，并建立用户: gitlab, 建立数据库: gitlabhq_production
  • 创建 Redis，建议最好用外置的，非 k8s 自建
  • 生产环境数据的迁移，需要有一份某天的备份数据，例: 1709747060_2024_03_06_14.1.5_gitlab_backup.tar
  • 创建所需下面组件需要的 secrets:
# 创建初始密码
kubectl create secret generic -n gitlab release-gitlab-initial-root-password --from-literal=password=<gitlab password>
# pgsql 密码
kubectl create secret -n gitlab generic gitlab-postgres --from-literal=psql-password=<pgsql password>
# redis
kubectl create secret -n gitlab generic gitlab-redis --from-literal=redis-password=<redis password>
# ldap 用户
kubectl create secret -n gitlab generic gitlab-robot --from-literal=ldap-password=<ldap user password>
# smtp
kubectl create secret -n gitlab generic gitlab-smtp --from-literal=smtp-password=<smtp password>

部署阶段

如果有看 前面的文章 ，可以得知当前在用的 gitlab 是14.1.5 版本，所以需要先将 gitlab-14.1.5 helm chart 服务拉起，下面我将拿部署在阿里云示例，其他国内公有云也同理，海外云特别是 aws，gitlab 官网也有相应的支持，会更简单方便

拉取 helm 资源

# 查看 helm 对应 gitlab 版本, 也可看 version mapping
helm repo add gitlab https://charts.gitlab.io/
helm repo update
helm search repo gitlab/gitlab -l
helm fetch gitlab/gitlab --version=5.1.5
# 这里拉取 helm 下来的原因，是因为 gitlab 子 chart values 需要变更

修改 Chart 配置

这里就只关注关键字段，有很多其他配置请自行到 官网 里面去对照修改

Global values 字段

edition
  edition: ce
hosts

  hosts:
    # 填写域名，即对外用
    domain: gitlab.example.com
    hostSuffix: 
    https: false
    # 可使用指定静态 IP
    # externalIP: 10.6.200.60
    # 拉取代码时 ssh 展示的域名
    ssh: gitlab.example.com
    gitlab: {}
    minio: {}
    registry: {}
    tls: {}
    smartcard: {}
    kas: {}
    pages: {}
ingress
# 不建议使用自带的 ingress
  ingress:
    configureCertmanager: false
    provider: nginx
    annotations: {}
    enabled: false
    tls: {}
    #   enabled: true
    #   secretName:
    path: /
    pathType: Prefix
initialRootPassword
  initialRootPassword:
    secret: release-gitlab-initial-root-password
    key: password
pgsql
  psql:
    connectTimeout:
    keepalives:
    keepalivesIdle:
    keepalivesInterval:
    keepalivesCount:
    tcpUserTimeout:
    password:
      useSecret: true
      # pgsql 的密码
      secret: gitlab-postgres
      key: psql-password
      # file:
    host: <pgsql host>
    port: 5432
    username: gitlab
    database: gitlabhq_production
    applicationName:
    preparedStatements: false
redis
  redis:
    password:
      enabled: true
      # redis 密码
      secret: gitlab-redis
      key: redis-password
    host: <redis host>
    port: 6379
minio
# global/values.yaml
  minio:
    enabled: true
    credentials: {}
appConfig
# 可以参考 gitlab.rb 信息进行对照修改，我用的默认
ldap
    ldap:
      # prevent the use of LDAP for sign-in via web.
      preventSignin: false
      ## 'main' is the GitLab 'provider ID' of this LDAP server
      servers: 
        main:
          label: 'LDAP'
          host: '<ldap address>'
          port: 389
          uid: 'sAMAccountName'
          bind_dn: 'gitlab-robot'
          base: 'OU=<>,OU=<>,DC=<>,DC=<>'
          user_filter: '(memberOf=CN=<>,OU=<>,OU=<>,DC=<>,DC=<>)'
          password:
            secret: gitlab-robot
            key: ldap-password
          encryption: 'plain'
      ## See documentation for complete example of a configured LDAP server
smtp
  smtp:
    enabled: true
    address: <smtp address>
    port: 587
    user_name: "<username>"
    ## https://docs.gitlab.com/charts/installation/secrets#smtp-password
    password:
      secret: gitlab-smtp
      key: smtp-password
    # domain:
    authentication: "login"
    starttls_auto: false
    openssl_verify_mode: "none"
    pool: false
email
  email:
    from: 'devops@exmaple.com' # gitlab 企业邮箱
    display_name: GitLab
    reply_to: ''subject_suffix:''
    smime:
      enabled: false
      secretName: ""keyName:"tls.key"certName:"tls.crt"
time_zone
  ## Timezone for containers.
  time_zone: Asia/Shanghai
upgradeCheck
upgradeCheck:
  enabled: true #观察是否打开了，升级检查
  image: {}
    # repository:
    # tag:
    # pullPolicy: IfNotPresent
    # pullSecrets: []
  securityContext:
    # in alpine/debian/busybox based images, this is `nobody:nogroup`
    runAsUser: 65534
    fsGroup: 65534
  tolerations: []
  annotations: {}
  configMapAnnotations: {}
  resources:
    requests:
      cpu: 50m
  priorityClassName: ""
其他
nginx-ingress:
# 关闭即可，不需要使用
  enabled: false
 ...
prometheus:
  install: false
...
redis:
  install: false
...
postgresql:
  install: false
...
gitlab-runner:
  install: false
...
gitlab-zoekt:
  install: false
....

Subchart 变更

Gitaly chart
#Gitaly chart
# https://docs.gitlab.com/charts/charts/gitlab/gitaly/index.html
...
persistence:
  enabled: true
  ## git repositories Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: "new-gitlab-storage"
  accessMode: ReadWriteOnce
  size: 200Gi
...
toolbox chart
# toolbox chart(task-runner) chart
# https://docs.gitlab.com/charts/charts/gitlab/toolbox/
# 需要关注的是 备份及 toolbox 迁移时需要存储空间来做临时存储
#（若做升级期间，可以不用 enable 存储）...
backups:
  cron:
    enabled: true
    concurrencyPolicy: Replace
    failedJobsHistoryLimit: 1
    schedule: "0 1 * * *"
    successfulJobsHistoryLimit: 5
    extraArgs: ""
    resources:
      # limits:
      #  cpu: 1
      #  memory: 2G
      requests:
        cpu: 50m
        memory: 350M
    persistence:
      enabled: true
      ## task-runner temporarily Persistent Volume Storage Class
      ## If defined, storageClassName: <storageClass>
      ## If set to "-", storageClassName: "", which disables dynamic provisioning
      ## If undefined (the default) or set to null, no storageClassName spec is
      ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
      ##   GKE, AWS & OpenStack)
      ##
      storageClass: "new-gitlab-storage"
      accessMode: ReadWriteOnce
      size: 200Gi
      subPath: ""
      ## if volumeName is set, use this existing PersistentVolume
      # volumeName:
      matchLabels: {}
      matchExpressions: []
...
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  enabled: true
  ## task-runner temporarily Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: "new-gitlab-storage"
  accessMode: ReadWriteOnce
  size: 200Gi
...
minio chart
# 首先还是重新指定 storageClass
## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  enabled: true
  ## minio data Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: "new-gitlab-storage"
  accessMode: ReadWriteOnce
  size: 200Gi

正式部署

拉起 helm

# 建议使用 gitlab 仓库来保存 helm chart 资源及版本
# 进入到 helm-gitlab 仓库
# 查看 gitlab 升级准备文档内容，提前把对应 secret 的密码创建出来
# 创建初始密码
kubectl create secret generic -n gitlab release-gitlab-initial-root-password --from-literal=password=XXXX
# pgsql 密码
kubectl create secret -n gitlab generic gitlab-postgres --from-literal=psql-password=XXXXX
# redis
kubectl create secret -n gitlab generic gitlab-redis --from-literal=redis-password=XXXXXXX
# ldap 用户
kubectl create secret -n gitlab generic gitlab-robot --from-literal=ldap-password=XXXXXXXXX
# smtp
kubectl create secret -n gitlab generic gitlab-smtp --from-literal=smtp-password=XXXXXXXXXXXX
# 先查看有哪些版本，然后拉取 helm repo, 下面举例 5.1.5
helm search repo gitlab/gitlab -l
helm fetch gitlab/gitlab --version=5.1.5
tar -xvf gitlab-5.1.5.tgz
mv gitlab gitlab-5.1.5 && chown -R <mac 用户 > gitlab-5.1.5
# 按照上文 helm 字段，修改 global values 及 chart values
cd gitlab-5.1.5
helm install gitlab <gitlab-5.1.5 全路径 > -f values.yaml -n gitlab
# 例子：helm upgrade gitlab /Users/lilin/Documents/work/gitlab/deploy-templates-public/helm-gitlab/gitlab-5.1.5 -f values.yaml -ngitlab
# 正常情况下有如下服务
# pod 服务
➜  gitlab-5.1.5 kubectl get pods -ngitlab
NAME                                             READY   STATUS      RESTARTS   AGE
gitlab-certmanager-6dc844cd9f-pjpjf              1/1     Running     0          17h
gitlab-certmanager-cainjector-757d585f64-9h7qt   1/1     Running     0          17h
gitlab-certmanager-webhook-8555ffb88-ssk9p       1/1     Running     0          17h
gitlab-gitaly-0                                  1/1     Running     0          47m
gitlab-gitlab-exporter-64d7cdcf47-n22t7          1/1     Running     0          47m
gitlab-gitlab-shell-5f9f4c9479-c857v             1/1     Running     0          47m
gitlab-gitlab-shell-5f9f4c9479-v92sm             1/1     Running     0          42m
gitlab-kas-5c7d85c5f4-9fc5f                      1/1     Running     0          46m
gitlab-kas-5c7d85c5f4-tgwck                      1/1     Running     0          47m
gitlab-migrations-13-gwc4p                       0/1     Completed   0          47m
gitlab-minio-697f4ffd5f-6qm64                    1/1     Running     0          47m
gitlab-minio-create-buckets-13-ppcrk             0/1     Completed   0          47m
gitlab-registry-7fbc44dd-ndscs                   1/1     Running     0          44m
gitlab-registry-7fbc44dd-rbchk                   1/1     Running     0          47m
gitlab-sidekiq-all-in-1-v2-669c4d9965-f7fhn      1/1     Running     0          47m
gitlab-toolbox-5bb69cdc97-qc7kr                  1/1     Running     0          46m
gitlab-webservice-default-678dcd6-8mbp5          2/2     Running     0          47m
gitlab-webservice-default-678dcd6-kghsw          2/2     Running     0          40m
# service 服务
kubectl get svc -ngitlab
NAME                         TYPE           CLUSTER-IP        EXTERNAL-IP   PORT(S)                               AGE
gitlab-certmanager           ClusterIP      192.168.145.118   <none>        9402/TCP                              43h
gitlab-certmanager-webhook   ClusterIP      192.168.140.31    <none>        443/TCP                               43h
gitlab-gitaly                ClusterIP      None              <none>        8075/TCP,9236/TCP                     3d18h
gitlab-gitlab-exporter       ClusterIP      192.168.211.91    <none>        9168/TCP                              3d18h
gitlab-gitlab-shell          ClusterIP      192.168.87.80     <none>        22/TCP                                3d18h
gitlab-kas                   ClusterIP      192.168.152.92    <none>        8150/TCP,8153/TCP,8154/TCP,8151/TCP   25h
gitlab-minio-svc             ClusterIP      192.168.212.213   <none>        9000/TCP                              3d18h
gitlab-registry              ClusterIP      192.168.172.127   <none>        5000/TCP                              3d18h
gitlab-webservice-default    ClusterIP      192.168.225.185   <none>        8080/TCP,8181/TCP,8083/TCP            3d18h
# pvc 服务
kubectl get pvc -ngitlab
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
gitlab-minio                Bound    nas-xxxxxxx-xxxx-xxxx-xxxx-xxxx   200Gi      RWO            new-gitlab-storage   3d18h
repo-data-gitlab-gitaly-0   Bound    nas-xxxxxx-xxxx-xxxx-xxxx-xxxx   200Gi      RWO            new-gitlab-storage   3d20h

# 无误后执行下面流程

配置 svc/ingress

# 配置 loadbalancer 
# https 和 ssh 端口
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    # 付费类型 PayBySpec 按规格计费；PayByCLCU 按使用量计费
    # 具体说明: https://help.aliyun.com/document_detail/181517.html
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type: "insert"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie-timeout: "1800"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: "PayBySpec"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec: "slb.s1.small"
    service.beta.kubernetes.io/alicloud-loadbalancer-address-type: "intranet"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "vsw-xxxxx"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name: "i-4-public-new-gitlab"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip: "10.6.200.60"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
  name: i-4-public-new-gitlab
  namespace: gitlab
spec:
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ports:
    - name: gitlab-https
      port: 443
      protocol: TCP
      targetPort: 8181
      nodePort: 20080
  type: LoadBalancer
  selector:
    app: webservice
    release: gitlab

---
apiVersion: v1
kind: Service
metadata:
  name: i-4-public-new-gitlab-ssh
  namespace: gitlab
spec:
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ports:
    - name: gitlab-ssh
      port: 22
      protocol: TCP
      targetPort: 2222
      nodePort: 20022
  type: NodePort
  selector:
    app: gitlab-shell
    release: gitlab
#ingress
# minio 的 ingress 配置
# 注意，如果 gitlab 的备份文件很大的话，需要调整 ingress
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minio-ingress
  namespace: gitlab
  annotations:
    kubernetes.io/ingress.class: public-internal-nginx
spec:
  rules:
  - host: minio.example.com
    http:
      paths:
      - backend:
          service: 
            name: gitlab-minio-svc
            port:
              number: 9000
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - minio.example.com
    # 这里用的是泛域名的证书，也需要创建 secrets
    secretName: example-com-2024-07

数据迁移

1. 将前期准备内的备份包拷贝进 minio 服务的 backups 目录内
2. 执行 s3cmd la
  2024-02-22 02:52  63996774400  s3://gitlab-backups/1708451228_2024_02_20_14.1.5_gitlab_backup.tar
  2024-02-22 02:36       296960  s3://gitlab-backups/1708569369_2024_02_22_14.1.5_gitlab_backup.tar
正常应该执行，但是每次都读取不出来这个 tar 包：backup-utility --restore -f s3://gitlab-backups/1708451228_2024_02_20_14.1.5_gitlab_backup.tar
所以转换方式，先下载：s3cmd get s3://gitlab-backups/1708451228_2024_02_20_14.1.5_gitlab_backup.tar
然后再从本地 restore
backup-utility --restore -f file:///srv/gitlab/tmp/gitlab-backups/1708451228_2024_02_20_14.1.5_gitlab_backup.tar

Gitlab 升级

helm fetch gitlab/gitlab --version=5.5.4
拉取下来后将 5.1.5 的 values 值同步修改到 5.5.4 中：1. global-values
 后面三个最主要是存储类型修改，storageclass 改为 new-gitlab-storage：2. gitlab-minio-values
    3. gitlab-toolbox-values（可以不启用持久化，只在做数据迁移时需要）4. gitlab-gitaly-values
helm upgrade gitlab -n gitlab /Users/lilin/Documents/work/gitlab/deploy-templates-public/helm-gitlab/5.5.4 -f values.yaml

升级路线：5.1.5 --> 5.5.4 --> 5.9.5 --> 5.10.5 --> 6.0.0 --> 6.11.13 --> 7.0.0 --> 7.3.7

注意：升级 5.9.5 有错误需要单独处理, 官网文档有说明
  pgsql 上，推荐使用的 gitlab 默认用户: gitlab，特别是升级到 6.11.13 的时候，内置 migration 会去调用
正文完
kubernetes
2024-03-12
 1
Helm-Chart模版
Prometheus-配置文件
Kubernetes-service资源
12. Kubernetes-第一个容器化应用
评论（没有评论）
验证码
Ansible
elasticsearch
Gitlab
GO语言
helm
Jenkins
MongoDB
MySQL
Nacos
Prometheus
Python
Vue
其他
容器化(container)
编程学习
运维组件
-「」