CI = continuous integration

CD = continuous delivery and continuous deployment

Gitlab安装

略

Gitlab Runner Install

:zap: 查看centos版本

[root@gitlab-runner ~]# cat /etc/redhat-release 
CentOS Linux release 7.8.2003 (Core)

https://docs.gitlab.com/runner/install/index.html

gitlab-runner的安装方式很多：

  • In a container（docker or k8s）.
  • By downloading a binary manually.
  • By using a repository for rpm/deb packages.

:zap: 这里使用第三种方式, 通过官方仓库：
https://docs.gitlab.com/runner/install/linux-repository.html

添加官方仓库

# For RHEL/CentOS/Fedora
[root@gitlab-runner ~]# curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh | sudo bash

查看包信息

[root@gitlab-runner ~]# yum info gitlab-runner
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Available Packages
Name        : gitlab-runner
Arch        : x86_64
Version     : 13.11.0
Release     : 1
Size        : 135 M
Repo        : runner_gitlab-runner/x86_64
Summary     : GitLab Runner
URL         : https://gitlab.com/gitlab-org/gitlab-runner
License     : MIT
Description : GitLab Runner

yum安装

# 安装最新版本
# For RHEL/CentOS/Fedora
sudo yum install gitlab-runner

# 安装指定版本
# for RPM based systems
yum list gitlab-runner --showduplicates | sort -r
sudo yum install gitlab-runner-10.0.0-1

配置用户

略

此处使用root用户

查看是否安装成功

[root@gitlab-runner ~]# gitlab-runner
Runtime platform                                    arch=amd64 os=linux pid=61474 revision=7f7a4bb0 version=13.11.0
NAME:
   gitlab-runner - a GitLab Runner

USAGE:
   gitlab-runner [global options] command [command options] [arguments...]

VERSION:
   13.11.0 (7f7a4bb0)

AUTHOR:
   GitLab Inc. <support@gitlab.com>

COMMANDS:
     exec                  execute a build locally
     list                  List all configured runners
     run                   run multi runner service
     register              register a new runner
     install               install service
     uninstall             uninstall service
     start                 start service
     stop                  stop service
     restart               restart service
     status                get status of a service
     run-single            start single runner
     unregister            unregister specific runner
     verify                verify all registered runners
     artifacts-downloader  download and extract build artifacts (internal)
     artifacts-uploader    create and upload build artifacts (internal)
     cache-archiver        create and upload cache artifacts (internal)
     cache-extractor       download and extract cache artifacts (internal)
     cache-init            changed permissions for cache paths (internal)
     health-check          check health for a specific address
     read-logs             reads job logs from a file, used by kubernetes executor (internal)
     help, h               Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --cpuprofile value           write cpu profile to file [$CPU_PROFILE]
   --debug                      debug mode [$DEBUG]
   --log-format value           Choose log format (options: runner, text, json) [$LOG_FORMAT]
   --log-level value, -l value  Log level (options: debug, info, warn, error, fatal, panic) [$LOG_LEVEL]
   --help, -h                   show help
   --version, -v                print the version

更新启动/停止等

# 更新runner
# For RHEL/CentOS/Fedora
sudo yum update
sudo yum install gitlab-runner

# 将Runner作为服务安装并启动它

gitlab-runner install
gitlab-runner start

# stop and restart
gitlab-runner stop
gitlab-runner restart

gitlab-runner的注册

gitlab-runner register

依次输入gitlab的地址和token
runner的描述/tag/执行器等即可

:zap: 原理：

:one:输入url、token、tag等信息，注册时会向GitLab发送一个请求。
GitLab接收到请求后会返回一个token给runner，runner之后和GitLab的请求通讯都会带上这个token。

:two: 注册成功后runner会向GitLab轮询请求，查看是否要执行任务。

:three: 查询到job后构建执行

查看和修改配置文件

[root@gitlab-runner ~]# cat /etc/gitlab-runner/config.toml 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800


[[runners]]
  name = "gitlab-runner-test"
  url = "http://git.oowin.cn/"
  token = "ec0d2ed02df3e75a99565faa1b6dbc"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache","/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0

volumes = ["/cache"] 修改为
volumes = ["/cache","/var/run/docker.sock:/var/run/docker.sock"]

实现runner与宿主机的数据挂载

gitlab runner 的原理

https://blog.oonne.com/site/blog?id=34

.gitlab-ci.yml

:100:官方文档：http://git.oowin.cn/help/ci/yaml/README.md

.gitlab-ci.yml 位于gitlab仓库的根目录，一个.gitlab-ci.yml 相当定义了一个 作业流水线/管道 pipline。

:zap: 1个pipeline包含不同的stage执行阶段、每个stage包含不同的具体job脚本任务。

一个简单的.gitlab-ci.yml 配置文件如下：

# stages 关键字定义阶段
stages:
  - build
  - test
  - deploy
  
# 定义缓存
cache:
  key: ${CI_BUILD_REF_NAME}
  paths:
    - node_modules/
    - .tmp/

# 不同阶段下的job，job的名称可以任意，但是必须包含 script 关键字
# 如下定义了四个job，分别为job1/job2/job3/job4，执行不同阶段的任务
job 1:
  stage: build
  script: make build dependencies

job 2:
  stage: build
  script: make build artifacts

job 3:
  stage: test
  script: make test

job 4:
  stage: deploy
  # script 必须参数，执行shell脚本，可以是一个数组
  script:
  - uname -a
  - make deploy

Jobs

定义在顶层，不限数量，必须包含关键字 script 的语句, job的命令不是是保留关键字

job1:
  script: "execute-script-for-job1"

job2:
  script: "execute-script-for-job2"

:::tip

job的命令，可以执行一个shell命令或shell脚本（test.sh）

a command can execute code directly (./configure;make;make install)
or run a script (test.sh) in the repository.

:::

:zap:Jobs 中可以使用的参数如下：

pages

githlab的一个特殊的job。用于见gitlab仓库中public 目录下的静态内容部署到服务器。

如：

image: alpine:latest

pages:
  stage: deploy
  script:
  - echo 'Nothing to do...'
  artifacts:
    paths:
    - public
  only:
  - master
  tags:
  - pages

stages

定义jobs可以使用的阶段。

:one: 同一阶段的jobs 是并行执行的。

:two: 下一阶段的作业在上一阶段的作业成功完成之后执行。

:100:例如：

stages:
  - build
  - test
  - deploy

job 1:
  stage: build
  script: make build dependencies

job 2:
  stage: build
  script: make build artifacts

job 3:
  stage: test
  script: make test

job 4:
  stage: deploy
  script: make deploy
  • First, all jobs of build are executed in parallel.
  • If all jobs of build succeed, the test jobs are executed in parallel.
  • If all jobs of test succeed, the deploy jobs are executed in parallel.
  • If all jobs of deploy succeed, the commit is marked as passed.
  • If any of the previous jobs fails, the commit is marked as failed and no jobs of further stage are executed.

There are also two edge cases worth mentioning:

  • If no stages are defined in .gitlab-ci.yml, then the build,test and deploy are allowed to be used as job's stage by default.
  • If a job doesn't specify a stage, the job is assigned the test stage.

如果没有指定，默认是test阶段。

only/except

only 和 except 用于在创建作业时对作业的限制策略。

  • only 定义了哪些分支或标签(branches and tags)的作业会运行
  • except 定义了哪些分支或标签(branches and tags)的作业不会运行

variables

定义变量。

# 全局变量
variables:
  TEST_VAR: "All jobs can use this variable's value"
# 作业级的局部变量
# 当 variables 关键字使用在作业层级时，它会覆盖全局变量或预定义变量。
job1:
  variables:
    TEST_VAR_JOB: "Only job1 can use this variable's value"
  script:
    - echo "$TEST_VAR" and "$TEST_VAR_JOB"

gitlab总预设了很多变量，通常以CI开头，如：CI_BUILDS_DIR，表示执行构建的顶级目录。

https://docs.gitlab.com/ee/ci/variables/predefined_variables.html

tags

tags 关键字用于指定 GitLab Runner 运行器使用哪一个运行器来执行作业。

通常，可以在不同平台定不同的runner，执行不同的job：

windows job:
  stage:
    - build
  tags:
    - windows
  script:
    - echo Hello, %USERNAME%!

osx job:
  stage:
    - build
  tags:
    - osx
  script:
    - echo "Hello, $USER!"

when

when 关键字用于实现在作业失败时或发生故障时运行的作业

:zap: 可以使用如下值：

:one:on_successm -默认, 只有前面的阶段的所有作业都成功时才执行job

:two:on_failure - 前面阶段的作业有一个失败即执行此job

:three:always - 总是执行此job

:four:manual - 需要手动点击执行的job

:five: delayed - Gitlab11.4引入，延迟执行，需要配合start_in关键字

stages:
  - build
  - cleanup_build
  - test
  - deploy
  - cleanup

build_job:
  stage: build
  script:
    - make build

# 如果build_job构建失败，则执行此清理工作
cleanup_build_job:
  stage: cleanup_build
  script:
    - cleanup build when failed
  when: on_failure

test_job:
  stage: test
  script:
    - make test

# pending 状态的job，需要手动点击执行
deploy_job:
  stage: deploy
  script:
    - make deploy
  when: manual

# 总是执行的job
cleanup_job:
  stage: cleanup
  script:
    - cleanup after jobs
  when: always

cache

cache 缓存机制，可以在全局设置或者每个作业中设置。

cache 缓存机制用于指定一系列的文件或文件夹在不同的流水线或作业之间共享数据，仅能使用项目工作空间( project workspace )中的路径作为缓存的路径。

:zap: cache:paths : 指定要缓存的文件或目录，支持通配符。

rspec:
  script: test
  cache:
    paths:
      - binaries/*.apk
      - .config

artifacts

artifacts 用于指定在作业成功、失败、或者一直等状态下时，一系列的文件或文件夹附加到作业中。

artifacts is used to specify a list of files and directories which should be attached to the job after success.

  • 作业完成后，工件被发送到GitLab，可以在GitLab Web界面下载。

  • 默认情况下，只有成功的作业才会生成工件。

  • 并不是所有的 executor 执行器都支持.

# 创建一个压缩包命名为当前job名
job:
  artifacts:
    name: "$CI_JOB_NAME"
    
# 创建一个压缩包，命名为分支或者标签名，并且只包含未追踪的文件
job:
   artifacts:
     name: "$CI_COMMIT_REF_NAME"
     untracked: true

# 创建一个压缩包，命名为“job名_分支名”
job:
  artifacts:
    name: "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
    untracked: true

environment

GitLab 8.9 开始支持

environment是用于定义一个job(作业)部署到某个具名的环境，如果environment被指定，但是没有叫该名的environment存在，一个新的environment将会被自动创建。

# deploy_to_production作业将会执行部署操作部署到production环境
# 如果job做完了，gitlab将会在merge request页面或者environments或者deployments页面创建一个按钮，按钮指向https://prod.example.com
deploy_to_production:
  stage: deploy
  script: git push production HEAD:master
  environment:
    name: production
    url: https://prod.example.com

实战1：同一服务器上部署vue app

目标：

  • 初始化一个vue项目，提交到gitlab仓库

  • 配置仓库的cicd，即和gitlab-runner服务器关联

  • 当push代码到gitlab仓库时，gitlab-runner服务器会根据gitlab-ci.yml的job进行构建

  • 并通过docker的方式部署在gitlab-runner 的服务器上

    真实环境下，如果是部署多个项目 , 通常 gitlab-runner 是一台独立服务器，只执行构建和部署任务，

    每个项目应该部署到不同的另一台服务器上或另一台服务器的容器里面

:one:安装 gitlab and runner

略

:two: 注册runner

略

:three: 创建vue 项目，并关联的gitlab仓库

如图：项目根目录有 .gitlab-ci.yml 和 Dockerfile 两个文件

下面的关键是编写这两个文件，实现自动部署

:four: 编写 配置文件

# Dockerfile
# 构建node 镜像
FROM node:lts-alpine as build-stage
# 设置工作目录为容器中的 /app
WORKDIR /app
COPY package*.json ./
# 安装依赖，并打包到/appp/dist 目录
RUN npm install
COPY . .
RUN npm run build


# 构建nginx 镜像
FROM nginx:stable-alpine as production-stage
# 将上述打包的文件，复制到 nginx 目录
COPY --from=build-stage /app/dist /usr/share/nginx/html
# 设置容器的ngix端口为80
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Dockerfile 文件通过docker 执行具体的构建任务，并定制了一个nginx的镜像环境

具体的CICD，由gitlab-ci 触发，内容如下：

# gitlab-ci.yml
image: docker
services:
  - docker:dind
stages:
  - deploy
step-deploy-prod:
  stage: deploy
  script:
 	# 基于Dockerfile 文件构建一个 app/vue-demo 的镜像
    - docker build -t app/vue-demo .
    # 基于上述镜像，创建容器
    - docker run -d -p 8080:80 --rm --name gitlab-cicd-demo app/vue-demo

然后，我们修改仓库的代码，push后，就会触发gitlab cicd:

:five: 查看结果

点击上述已通过的流水线任务，具体执行如下：

Running with gitlab-runner 13.11.0 (7f7a4bb0)
  on gitlab-runner-test ec0d2ed0
Preparing the "docker" executor
Using Docker executor with image docker ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:dc8c389414c80f3c6510d3690cd03c29fc99d66f58955f138248499a34186bfa for docker:dind with digest docker@sha256:87ed8e3a7b251eef42c2e4251f95ae3c5f8c4c0a64900f19cc532d0a42aa7107 ...
Waiting for services to be up and running...

*** WARNING: Service runner-ec0d2ed0-project-184-concurrent-0-2c6e5c119efec0dd-docker-0 probably didn't start properly.

Health check error:
service "runner-ec0d2ed0-project-184-concurrent-0-2c6e5c119efec0dd-docker-0-wait-for-service" timeout

Health check container logs:


Service container logs:
2021-05-14T01:08:47.063178493Z Generating RSA private key, 4096 bit long modulus (2 primes)
2021-05-14T01:08:47.798669527Z ....................................................................................................................................................................++++
2021-05-14T01:08:48.262807270Z ....................................................................................................++++
2021-05-14T01:08:48.263328819Z e is 65537 (0x010001)
2021-05-14T01:08:48.278777695Z Generating RSA private key, 4096 bit long modulus (2 primes)
2021-05-14T01:08:49.253241639Z .................................................................................................................................................................................................................................++++
2021-05-14T01:08:49.333034825Z .................++++
2021-05-14T01:08:49.333553748Z e is 65537 (0x010001)
2021-05-14T01:08:49.363669965Z Signature ok
2021-05-14T01:08:49.363688135Z subject=CN = docker:dind server
2021-05-14T01:08:49.363882824Z Getting CA Private Key
2021-05-14T01:08:49.376452873Z /certs/server/cert.pem: OK
2021-05-14T01:08:49.379969420Z Generating RSA private key, 4096 bit long modulus (2 primes)
2021-05-14T01:08:49.576082919Z ............................................++++
2021-05-14T01:08:49.661092292Z ..................++++
2021-05-14T01:08:49.661559207Z e is 65537 (0x010001)
2021-05-14T01:08:49.688047531Z Signature ok
2021-05-14T01:08:49.688067306Z subject=CN = docker:dind client
2021-05-14T01:08:49.688266798Z Getting CA Private Key
2021-05-14T01:08:49.700744958Z /certs/client/cert.pem: OK
2021-05-14T01:08:49.704095161Z mount: permission denied (are you root?)
2021-05-14T01:08:49.704169957Z Could not mount /sys/kernel/security.
2021-05-14T01:08:49.704175923Z AppArmor detection and --privileged mode might break.
2021-05-14T01:08:49.705438429Z mount: permission denied (are you root?)

*********

Pulling docker image docker ...
Using docker image sha256:d2979b152a7d43f040c7aef88c4c83de4e545227622b1045adf6fe409293f803 for docker with digest docker@sha256:ad50b8d78b41dc52f42ab123ce0e3f48c54437ed70ecc2a44c99e889924c8e56 ...
Preparing environment
Running on runner-ec0d2ed0-project-184-concurrent-0 via gitlab-runner...
Getting source from Git repository
Fetching changes...
Reinitialized existing Git repository in /builds/gitlab-test/gitlab-runner-test/.git/
Checking out 78d045f9 as master...

Skipping Git submodules setup
Executing "step_script" stage of the job script
Using docker image sha256:d2979b152a7d43f040c7aef88c4c83de4e545227622b1045adf6fe409293f803 for docker with digest docker@sha256:ad50b8d78b41dc52f42ab123ce0e3f48c54437ed70ecc2a44c99e889924c8e56 ...
$ docker build -t app/vue-demo .
Sending build context to Docker daemon  122.4kB

Step 1/10 : FROM node:lts-alpine as build-stage
lts-alpine: Pulling from library/node
ddad3d7c1e96: Pulling fs layer
f845e0f7d73a: Pulling fs layer
47d471c4d820: Pulling fs layer
1a88008f9c83: Pulling fs layer
1a88008f9c83: Waiting
47d471c4d820: Verifying Checksum
47d471c4d820: Download complete
ddad3d7c1e96: Verifying Checksum
ddad3d7c1e96: Download complete
ddad3d7c1e96: Pull complete
1a88008f9c83: Download complete
f845e0f7d73a: Verifying Checksum
f845e0f7d73a: Download complete
f845e0f7d73a: Pull complete
47d471c4d820: Pull complete
1a88008f9c83: Pull complete
Digest: sha256:3689ad4435a413342ccc352170ad0f77433b41173af7fe4c0076f0c9792642cb
Status: Downloaded newer image for node:lts-alpine
 ---> 8e69714aa82b
Step 2/10 : WORKDIR /app
 ---> Running in b508b8d51100
Removing intermediate container b508b8d51100
 ---> 84f76ecf6c67
Step 3/10 : COPY package*.json ./
 ---> b0f92be4bb98
Step 4/10 : RUN npm install
 ---> Running in 36500d6c97b4

> esbuild@0.11.20 postinstall /app/node_modules/esbuild
> node install.js

npm WARN gitlab-cicd-demo@0.0.0 No description
npm WARN gitlab-cicd-demo@0.0.0 No repository field.
npm WARN gitlab-cicd-demo@0.0.0 No license field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.3.2 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 55 packages from 76 contributors and audited 56 packages in 6.022s

3 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

Removing intermediate container 36500d6c97b4
 ---> a72b78663769
Step 5/10 : COPY . .
 ---> 4655fb38129e
Step 6/10 : RUN npm run build
 ---> Running in dbf5bbee62d7

> gitlab-cicd-demo@0.0.0 build /app
> vite build

vite v2.3.2 building for production...
transforming...
✓ 12 modules transformed.
rendering chunks...
dist/assets/logo.03d6d6da.png    6.69kb
dist/index.html                  0.47kb
dist/assets/index.9dcff51c.css   0.20kb / brotli: 0.12kb
dist/assets/index.954eb2af.js    0.84kb / brotli: 0.45kb
dist/assets/vendor.642d02b4.js   42.44kb / brotli: 15.34kb
Removing intermediate container dbf5bbee62d7
 ---> 875590f17e10
Step 7/10 : FROM nginx:stable-alpine as production-stage
stable-alpine: Pulling from library/nginx
540db60ca938: Already exists
3b88e9e9a17d: Pulling fs layer
3019f265923a: Pulling fs layer
fd39c3601fe1: Pulling fs layer
4d0e0e4dee17: Pulling fs layer
f81ca69df58f: Pulling fs layer
4d0e0e4dee17: Waiting
f81ca69df58f: Waiting
3019f265923a: Verifying Checksum
3019f265923a: Download complete
fd39c3601fe1: Download complete
3b88e9e9a17d: Verifying Checksum
3b88e9e9a17d: Download complete
f81ca69df58f: Verifying Checksum
f81ca69df58f: Download complete
3b88e9e9a17d: Pull complete
3019f265923a: Pull complete
4d0e0e4dee17: Verifying Checksum
4d0e0e4dee17: Download complete
fd39c3601fe1: Pull complete
4d0e0e4dee17: Pull complete
f81ca69df58f: Pull complete
Digest: sha256:e015192ec74937149dce3aa1feb8af016b7cce3a2896246b623cfd55c14939a6
Status: Downloaded newer image for nginx:stable-alpine
 ---> 3b715e351972
Step 8/10 : COPY --from=build-stage /app/dist /usr/share/nginx/html
 ---> 6e3b6550f7b8
Step 9/10 : EXPOSE 80
 ---> Running in a3cafd80a196
Removing intermediate container a3cafd80a196
 ---> cb4d46919ce3
Step 10/10 : CMD ["nginx", "-g", "daemon off;"]
 ---> Running in 24a4fa7fe8e8
Removing intermediate container 24a4fa7fe8e8
 ---> 8d5d4c7f95ad
Successfully built 8d5d4c7f95ad
Successfully tagged app/vue-demo:latest
$ docker run -d -p 8080:80 --rm --name gitlab-cicd-demo app/vue-demo
4bf79e391cfa2e26bd5d41ff82241b573346307516670bb2631097e68958945c
Job succeeded

可以看到完整的执行指令。

然后在gitlab-runner的服务器上可以查看已经运行的容器：

[root@gitlab-runner ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
4bf79e391cfa        app/vue-demo        "/docker-entrypoint.…"   42 minutes ago      Up 42 minutes       0.0.0.0:8080->80/tcp   gitlab-cicd-demo

浏览器访问如下：