K8S容器环境下GitLab-CI和GItLabRunner部署记录
⼀、简单介绍
GitLab-CI
GitLab CI/CD是GitLab的⼀部分，⽀持从计划到部署具有出⾊的⽤户体验。CI/CD是开源GitLab社区版和专有GitLab企业版的⼀部分。可以根据需要添加任意数量的计算节点，每个构建可以拆分为多个作业，这些作业可以在多台计算机上并⾏运⾏。
GitLab-CI轻量级，不需要复杂的安装⼿段。配置简单，与gitlab可直接适配。实时构建⽇志⼗分清晰，UI交互体验很好。使⽤ YAML 进⾏配置，任何⼈都可以很⽅便的使⽤。GitLabCI 有助于DevOps⼈员，例如敏捷开发中，开发与运维是同⼀个⼈，最便捷的开发⽅式。
在⼤多数情况，构建项⽬都会占⽤⼤量的系统资源，如果让gitlab本⾝来运⾏构建任务的话，显然Gitlab的性能会⼤幅度下降。GitLab-CI最⼤的作⽤就是管理各个项⽬的构建状态。因此，运⾏构建任务这种浪费资源的事情交给⼀个独⽴的Gitlab Runner来做就会好很多，更重要的是Gitlab Runner 可以安装到不同的机器上，甚⾄是我们本机，这样完全就不会影响Gitlab本⾝了。
从GitLab8.0开始，GitLab-CI就已经集成在GitLab中，我们只需要在项⽬中添加⼀个.gitlab-ci.yaml⽂件，
然后运⾏⼀个Runner，即可进⾏持续集成。
GitLab-CI：集成、开源、⽆缝、可扩展、更快的结果、针对交付进⾏了优化：
GItLab Runner
Gitlab Runner是⼀个开源项⽬，⽤于运⾏您的作业并将结果发送给gitlab。它与Gitlab CI结合使⽤，gitlab ci是Gitlab随附的⽤于协调作⽤的开源持续集成服务。
Gitlab Runner是⽤Go编写的，可以作为⼀个⼆进制⽂件运⾏，不需要特定于语⾔的要求
它皆在GNU/Linux，MacOS和Windows操作系统上运⾏。另外注意：如果要使⽤Docker，Gitlab Runner要求Docker ⾄少是v1.13.0版本才可以。
Kubernetes Gitlab CICD 演⽰图：
⼆、基于Kubernetes Gitlab CICD 容器化部署记录
Gitlab官⽅提供了Helm的⽅式在Kubernetes集中来快速安装，但是在使⽤的过程中发现Helm提供的Chart包中有很多其他额外的配置。所以这⾥我采⽤K8S⾃定义的⽅式来安装。
Gitlab主要涉及3个应⽤：Redis、Postgresql、Gitlab核⼼程序。
本案例中使⽤的Gitlab-ce镜像部署，镜像中的Gitlab版本是13.7.4。
本案例中使⽤NFS作为持久化存储⽅式。除此之外，还可以选择HostPath本地持久化存储、NAS云端持久化存储、Ceph分布式持久化存储等。
注意：本⽰例部署所涉及到的image镜像均导⼊到Harbor私有私仓（172.16.60.230）。
1）使⽤NFS作为持久化存储
在NFS服务器端（172.16.60.238）创建Redis、Postgresql、Gitlab核⼼程序容器的持久化挂载⽬录
[root@k8s-harbor01 ~]# mkdir -p /data/storage/k8s/gitlab/{postgresql,redis,gitlab}
[root@k8s-harbor01 ~]# ll /data/storage/k8s/gitlab/
total 0
drwxr-xr-x 2 root root 6 Mar 25 14:03 gitlab
drwxr-xr-x 2 root root 6 Mar 25 14:03 postgresql
drwxr-xr-x 2 root root 6 Mar 25 14:03 redis
2）部署Gitlab
可以先创建⼀个命名空间
[root@k8s-master01 gitlab]# kubectl create ns kube-ops
[root@k8s-master01 gitlab]# kubectl get ns|grep kube-ops
kube-ops              Active  7d18h
配置三个核⼼程序的容器化部署的yaml⽂件
[root@k8s-master01 gitlab]# pwd
/opt/k8s/k8s_project/gitlab
[root@k8s-master01 gitlab]# ll
total 12
-
rw-r--r-- 1 root root 1629 Mar 25 14:05 gitlab-postgresql.yaml
-rw-r--r-- 1 root root 1207 Mar 25 14:05 gitlab-redis.yaml
-rw-r--r-- 1 root root 2691 Mar 25 14:05 gitlab.yaml
gitlab-postgresql.yaml ⽂件内容：
[root@k8s-master01 gitlab]# cat gitlab-postgresql.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgresql
namespace: kube-ops
labels:
name: postgresql
spec:
replicas: 1
selector:
matchLabels:
name: postgresql
template:
metadata:
name: postgresql
labels:
name: postgresql
spec:
containers:
- name: postgresql
image: 172.16.60.230/gitlab/postgresql:v1
imagePullPolicy: IfNotPresent
env:
- name: DB_USER
value: gitlab
- name: DB_PASS
value: passw0rd
- name: DB_NAME
value: gitlab_production
- name: DB_EXTENSION
value: pg_trgm
ports:
- name: postgres
containerPort: 5432
volumeMounts:
- mountPath: /var/lib/postgresql
- pg_isready
- -h
- localhost
-
-U
- postgres
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
exec:
command:
- pg_isready
- -h
- localhost
- -U
-
postgres
initialDelaySeconds: 5
timeoutSeconds: 1
volumes:
- name: data
nfs:
server: 172.16.60.238
path: /data/storage/k8s/gitlab/postgresql          readOnly: false
---
apiVersion: v1
kind: Service
metadata:
name: postgresql
namespace: kube-ops
labels:
name: postgresql
spec:
ports:
- name: postgres
port: 5432
targetPort: postgres
selector:
name: postgresql
gitlab-redis.yaml ⽂件内容：
[root@k8s-master01 gitlab]# cat gitlab-redis.yaml apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
namespace: kube-ops
labels:
name: redis
spec:
replicas: 1
selector:
matchLabels:
name: redis
template:
metadata:
name: redis
labels:
name: redis
spec:
containers:
- name: redis
image: 172.16.60.230/gitlab/redis:latest
imagePullPolicy: IfNotPresent
ports:
- name: redis
containerPort: 6379
volumeMounts:
- mountPath: /var/lib/redis
name: data
livenessProbe:
exec:
command:
-
redis-cli
- ping
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
exec:
command:
- redis-cli
- ping
initialDelaySeconds: 5
timeoutSeconds: 1
volumes:
- name: data
nfs:
server: 172.16.60.238
path: /data/storage/k8s/gitlab/redis
readOnly: false
---
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: kube-ops
labels:
name: redis
spec:
ports:
- name: redis
port: 6379
targetPort: redis
selector:
name: redis
gitlab.yaml ⽂件内容：
今天的极致任务：
把所有的⾯试题收集完成
全⼒补到Python
[root@k8s-master01 gitlab]# cat gitlab.yaml
namespace: kube-ops
labels:
name: gitlab
spec:
replicas: 1
selector:
matchLabels:
name: gitlab
template:
metadata:
name: gitlab
labels:
name: gitlab
spec:
containers:
- name: gitlab
image: 172.16.60.230/gitlab/gitlab-ce:latest
imagePullPolicy: IfNotPresent
env:
- name: TZ
value: Asia/Shanghai
- name: GITLAB_TIMEZONE
value: Beijing
- name: GITLAB_SECRETS_DB_KEY_BASE
value: long-and-random-alpha-numeric-string
- name: GITLAB_SECRETS_SECRET_KEY_BASE
value: long-and-random-alpha-numeric-string
- name: GITLAB_SECRETS_OTP_KEY_BASE
value: long-and-random-alpha-numeric-string
-
name: GITLAB_ROOT_PASSWORD
value: admin321
- name: GITLAB_ROOT_EMAIL
value: 1025337607@qq
- name: GITLAB_HOST
value: 0.0.0.0:30004
- name: GITLAB_PORT
value: "80"
- name: GITLAB_SSH_PORT
value: "22"
- name: GITLAB_NOTIFY_ON_BROKEN_BUILDS
value: "true"
- name: GITLAB_NOTIFY_PUSHER
value: "false"
- name: GITLAB_BACKUP_SCHEDULE
value: daily
- name: GITLAB_BACKUP_TIME
value: 01:00
- name: DB_TYPE
value: postgres
- name: DB_HOST
value: postgresql
-
name: DB_PORT
value: "5432"
- name: DB_USER
value: gitlab
- name: DB_PASS
value: passw0rd
- name: DB_NAME
value: gitlab_production
- name: REDIS_HOST
value: redis
- name: REDIS_PORT
value: "6379"
ports:
- name: http
containerPort: 80
- name: ssh
containerPort: 22
volumeMounts:
- mountPath: /home/git/data
name: data
livenessProbe:
httpGet:
path: /
port: 80nodeselector
initialDelaySeconds: 180
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 1
volumes:
-
name: data
nfs:
server: 172.16.60.238
path: /data/storage/k8s/gitlab/gitlab
readOnly: false
---
apiVersion: v1
kind: Service
metadata:
name: gitlab
namespace: kube-ops
labels:
name: gitlab
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: http
nodePort: 30004
- name: ssh
port: 22
targetPort: ssh
selector:
name: gitlab
创建并启动gitlab相关容器进程：
[root@k8s-master01 gitlab]# ll
total 12
-rw-r--r-- 1 root root 1629 Mar 25 14:05 gitlab-postgresql.yaml -rw-r--r-- 1 root root 1207 Mar 25 14:05 gitlab-redis.yaml
-rw-r--r-- 1 root root 2691 Mar 25 14:05 gitlab.yaml
[root@k8s-master01 gitlab]# kubectl apply -f .
deployment.apps/postgresql created
service/postgresql created
deployment.apps/redis created
service/redis created
deployment.apps/gitlab created
service/gitlab created
稍微等⼀会⼉（由于程序启动顺序原因，pod可能会出现重启次数，不过最终都会启动成功），
查看pod状态：
[root@k8s-master01 gitlab]# kubectl get pods -n kube-ops
NAME                          READY  STATUS    RESTARTS  AGE
gitlab-5b887894d5-ntxzj      1/1    Running  1          38m
postgresql-57bf98cdf8-7mdh9  1/1    Running  1          38m
redis-56769dc6b6-c4rnq        1/1    Running  0          38m
查看svc：
[root@k8s-master01 gitlab]# kubectl get svc -n kube-ops
NAME        TYPE        CLUSTER-IP    EXTERNAL-IP  PORT(S)                    AGE
gitlab      NodePort    10.254.48.72  <none>        80:30004/TCP,22:32280/TCP  14m
postgresql  ClusterIP  10.254.88.39  <none>        5432/TCP                    14m
redis        ClusterIP  10.254.198.0  <none>        6379/TCP
3）访问 Gitlab
这⾥采⽤NodePort的⽅式，通过 任意node节点ip:30004 地址访问Gitlab
Gitlab登录⽤户名：root，密码：admin321
Gitlab登录密码可以在yaml⽂件⾥修改
这⾥容器化部署后的Gitlab版本是13.7.4
4）创建演⽰项⽬
接下来顺便创建⼀个项⽬，⽤于演⽰：
接下来在服务器上git clone，进⾏代码提交演⽰：
git clone地址是
地址中的gitlab-5b887894d5-ntxzj是pod名称，在容器外部访问不了，需要修改为对应的nodeport地址，故git clone地址可以是：在其中⼀个node节点上进⾏代码提交演⽰：
[root@k8s-node02 mnt]# mkdir /mnt/haha
[root@k8s-node02 mnt]# cd /mnt/haha
[root@k8s-node02 haha]# git config --ail "1025337607@qq"
[root@k8s-node02 haha]# git config --global user.name "Administrator"
[root@k8s-node02 haha]#
[root@k8s-node02 haha]# git clone 172.16.60.234:30004/root/kevin-test.git
Cloning into 'kevin-test'...
Username for '172.16.60.234:30004': root            #输⼊账号root
Password for 'root@172.16.60.234:30004':            #输⼊账号root的密码
warning: You appear to have cloned an empty repository.
[root@k8s-node02 haha]# ls
kevin-test
[root@k8s-node02 haha]# cd kevin-test/
[root@k8s-node02 kevin-test]# ll
total 0
[root@k8s-node02 kevin-test]#
[root@k8s-node02 kevin-test]# touch test.md
[root@k8s-node02 kevin-test]# echo "come on" > test.md
[root@k8s-node02 kevin-test]# git add test.md
[root@k8s-node02 kevin-test]# git commit -m "add test.md"
[master (root-commit) 8ccda29] add test.md
1 file changed, 1 insertion(+)
create mode 100644 test.md
[root@k8s-node02 kevin-test]# git commit -m "add test.md"
[master (root-commit) 8ccda29] add test.md
1 file changed, 1 insertion(+)
create mode 100644 test.md
[root@k8s-node02 kevin-test]# git push -u origin master
Username for '172.16.60.234:30004': root
Password for 'root@172.16.60.234:30004':
Counting objects: 3, done.
Writing objects: 100% (3/3), 216 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To 172.16.60.234:30004/root/kevin-test.git
* [new branch]      master -> master
Branch master set up to track remote branch master from origin.
如何解决 "每次输⼊⽤户名和密码" 的问题？
在代码⽬录.git/config⽂件内[remote "origin"]的url的gitlab域名前添加gitlab注册时的"⽤户名：密码@"
[root@k8s-node02 kevin-test]# cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = root:admin321@172.16.60.234:30004/root/kevin-test.git
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
接着再次尝试提交内容，就不需要⼿动输⼊⽤户名和密码了：
[root@k8s-node02 kevin-test]# git pull
Already up-to-date.
[root@k8s-node02 kevin-test]# echo "this is gitlab test" > test.md
[root@k8s-node02 kevin-test]# git add test.md
[root@k8s-node02 kevin-test]# git commit -m "modified test.md"
[master f2fbb27] modified test.md
1 file changed, 1 insertion(+), 1 deletion(-)
[root@k8s-node02 kevin-test]# git push -u origin master
Counting objects: 12, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (10/10), 845 bytes | 0 bytes/s, done.
Total 10 (delta 1), reused 0 (delta 0)
To root:admin321@172.16.60.234:30004/root/kevin-test.git
fe40316..f2fbb27  master -> master
Branch master set up to track remote branch master from origin.
代码上传后，gitlab上展⽰效果如下：
进⼊该项⽬下，左侧栏CICD⾥有三种⽅式：Pipelines、Jobs、Schedules
5）Gitlab Runner 安装和注册
gitlab runner⽀持多种⽅式安装，我这⾥就采取在k8s中安装。
官⽅⽂档地址:
打开gitlab，如下图所⽰，左边代表runner状态，右边是配置runner信息。
注意右边栏的token信息，后⾯注册runner的时候会⽤到：
接下来进⾏配置gitlab runner资源清单 (runner-configmap.yaml)
[root@k8s-master01 gitlab]# cat runner-configmap.yaml
apiVersion: v1
data:
REGISTER_NON_INTERACTIVE: "true"
REGISTER_LOCKED: "false"
METRICS_SERVER: "0.0.0.0:9100"
CI_SERVER_URL: "gitlab.kube-ops.svc.cluster.local/ci"
RUNNER_REQUEST_CONCURRENCY: "4"
RUNNER_EXECUTOR: "kubernetes"
KUBERNETES_NAMESPACE: "kube-ops"
KUBERNETES_PRIVILEGED: "true"
KUBERNETES_CPU_LIMIT: "1"
KUBERNETES_CPU_REQUEST: "500m"
KUBERNETES_MEMORY_LIMIT: "1Gi"
KUBERNETES_SERVICE_CPU_LIMIT: "1"
KUBERNETES_SERVICE_MEMORY_LIMIT: "1Gi"
KUBERNETES_HELPER_CPU_LIMIT: "500m"
KUBERNETES_HELPER_MEMORY_LIMIT: "100Mi"
KUBERNETES_PULL_POLICY: "if-not-present"
KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10"
KUBERNETES_POLL_INTERVAL: "5"
KUBERNETES_POLL_TIMEOUT: "360"
kind: ConfigMap
metadata:
labels:
app: gitlab-ci-runner
name: gitlab-ci-runner-cm
namespace: kube-ops
需要注意：
CI_SERVER_URL 这个地址是gitlab的地址，如果gitlab在宿主机直接写宿主机的ip即可，容器是格式为：svc名称.命名空间.svc.cluster.local (如果都按照我的⽂档来进⾏安装不需要修改别的配置了)。
如果定义的gitlab域名并不是通过外⽹DNS解析，⽽是通过/etc/hosts进⾏映射，那么我们需要在Runner的Pod中去添加对应的hosts，需要通过--pre-clone-script参数来指定⼀段脚本来添加hosts信息，也就是在ConfigMap中添加环境变量RUNNER_PRE_CLONE_SCRIPT的值：
本案例，这⾥gitlab地址我是使⽤node节点的ip+port⽅式。如果使⽤gitlab域名⽅式，且不是外⽹DNS解析，⽐如域名地址是gitlab.kevin 则需要在上⾯的ConfigMap中添加环境变量RUNNER_PRE_CLONE_SCRIPT的值: RUNNER_PRE_CLONE_SCRIPT = "echo ' git.i4t' >> /etc/hosts" 其中 为node节点ip地址
另外记住：在ConfigMap添加新选项后，需要删除Gitlab ci Runner Pod
因为这⾥我是使⽤envFrom来注⼊上⾯的这些环境变量⽽不是直接使⽤env(envfrom 通过将环境变量放置到ConfigMaps或Secrets来帮助减⼩清单⽂件)
如果我们想添加其他选项，那么可以在等到后⾯的gitlab-ci-runner的Pod容器启动成功后，登录gitlab-ci-runner的pod容器内部运⾏gitlab-ci-multi-runner register --help 命令来查看所有可使⽤的选项，只需要为配置的标志添加env变量即可：
gitlab-runner@gitlab-ci-runner-0:/$ gitlab-ci-multi-runner register --help
[...]
--kubernetes-cpu-limit value                          The CPU allocation given to build containers (default: "1") [$KUBERNETES_CPU_LIMIT]
--kubernetes-memory-limit value                      The amount of memory allocated to build containers (default: "4Gi") [$KUBERNETES_MEMORY_LIMIT]
--kubernetes-service-cpu-limit value                  The CPU allocation given to build service containers (default: "1") [$KUBERNETES_SERVICE_CPU_LIMIT]
--kubernetes-service-memory-limit value              The amount of memory allocated to build service containers (default: "1Gi") [$KUBERNETES_SERVICE_MEMORY_LIMIT]
--kubernetes-helper-cpu-limit value                  The CPU allocation given to build helper containers (default: "500m") [$KUBERNETES_HELPER_CPU_LIMIT]
--kubernetes-helper-memory-limit value                The amount of memory allocated to build helper containers (default: "3Gi") [$KUBERNETES_HELPER_MEMORY_LIMIT]
-
-kubernetes-cpu-request value                        The CPU allocation requested for build containers [$KUBERNETES_CPU_REQUEST]
...
--pre-clone-script value                              Runner-specific command script executed before code is pulled [$RUNNER_PRE_CLONE_SCRIPT]
[...]
创建资源清单的configmap
[root@k8s-master01 gitlab]# kubectl apply -f runner-configmap.yaml
configmap/gitlab-ci-runner-cm created
[root@k8s-master01 gitlab]# kubectl get configmaps  -n kube-ops
NAME                  DATA  AGE
gitlab-ci-runner-cm  19    4s
可通过下⾯命令来查看此configmap内容：
[root@k8s-master01 gitlab]# kubectl describe cm gitlab-ci-runner-cm -n kube-ops
此时，还需要配置⼀个⽤于注册、运⾏和取消gitlab ci runner的⼩脚本。只有当Pod正常通过K8S (TERM信号)的终⽌流程时，才会触发注销注册。如果强⾏终⽌Pod（SIGKILL

版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。