一、基于Docker配置构建资源池

1.1 整体架构

Alt Image Text

1.2 Docker安装Jenkins Slave节点(静态)

首先我们在Jenkins的节点管理中,添加节点。输入节点的名称和类型。

配置节点信息:自定义目录 启动方式: Launch agent by connecting it to the master

  • docker-agent
  • # of executors: 5
  • /home/vagrant/workspace
  • Labels: docker-agent

Alt Image Text

Alt Image Text

获取JNLP方式运行slave所需要的秘钥信息。

秘钥信息获取方式,点击下载slave-agent.jnlp,打开文件,及获取秘钥信息

<jnlp codebase="http://192.168.33.11:8080/computer/docker-agent/" spec="1.0+"><information><title>Agent for docker-agent</title><vendor>Jenkins project</vendor><homepage href="https://jenkins-ci.org/"></homepage></information><security><all-permissions></all-permissions></security><resources><j2se version="1.8+"></j2se><jar href="http://192.168.33.11:8080/jnlpJars/remoting.jar"></jar></resources><application-desc main-class="hudson.remoting.jnlp.Main"><argument>857f47abd55cb5c13567f8cc8c8ddfb77c5af04260efc2436b50a8b896b522e6</argument><argument>docker-agent</argument><argument>-workDir</argument><argument>/home/vagrant/workspace</argument><argument>-internalDir</argument><argument>remoting</argument><argument>-url</argument><argument>http://192.168.33.11:8080/</argument></application-desc></jnlp>
  • 857f47abd55cb5c13567f8cc8c8ddfb77c5af04260efc2436b50a8b896b522e6

获取jnlp slavedocker镜像: https://hub.docker.com/r/jenkins/inbound-agent/

docker pull jenkins/inbound-agent

docker run --init -itd jenkins/inbound-agent -url http://192.168.33.11:8080 857f47abd55cb5c13567f8cc8c8ddfb77c5af04260efc2436b50a8b896b522e6 docker-agent

启动slave测试,出现以下日志表示成功连接。这个部分容易出现问题,原因很可能是因为网络权限导致的。

Aug 19, 2020 12:48:11 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: docker-agent
Aug 19, 2020 12:48:11 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Aug 19, 2020 12:48:11 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.3
Aug 19, 2020 12:48:11 PM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
Aug 19, 2020 12:48:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://192.168.33.11:8080]
Aug 19, 2020 12:48:11 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Aug 19, 2020 12:48:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: 192.168.33.11
  Agent port:    50000
  Identity:      50:9b:15:1c:3b:3f:ab:bf:86:0b:82:fb:56:56:e5:f6
Aug 19, 2020 12:48:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 19, 2020 12:48:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 192.168.33.11:50000
Aug 19, 2020 12:48:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Aug 19, 2020 12:48:12 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 50:9b:15:1c:3b:3f:ab:bf:86:0b:82:fb:56:56:e5:f6
Aug 19, 2020 12:48:12 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

查看效果

Alt Image Text

pipeline {
    agent { node { label "docker-agent" }}
    stages {
        stage('Test') {
            steps {
                sleep(20)
                echo "Hello world!"

            }
        }
    }
}

console output

Running on docker-agent in /home/vagrant/workspace/workspace/docker-agent2
...

Alt Image Text

1.3 基于Docker配置Jenkins Slave节点(动态)

docker插件: https://plugins.jenkins.io/docker-plugin/

项目地址: https://github.com/jenkinsci/docker-plugin

对与Jenkins动态slave的配置,其实就是Jenkins调用Docker的接口完成的。我们需要开启Docker远程访问。

docker 开启API远程访问 (mac) 参考文档: https://juejin.im/entry/5bdf04b06fb9a049e41223f1

docker pull bobrik/socat
docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 2376:2375 bobrik/socat TCP4-LISTEN:2375,fork,reuseaddr UNIX-CONNECT:/var/run/docker.sock

如果你的docker环境是使用的centos系统,可以做如下配置。编辑/usr/lib/systemd/system/docker.service

$ sudo vim /usr/lib/systemd/system/docker.service

...
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://var/run/docker.sock
sudo systemctl daemon-reload
sudo systemctl restart docker

当我们把上面的配置完成后,可以通过Curl命令进行基本的测试API。出现一下信息表示成功开启。

$ curl -XGET http://127.0.0.1:2375/version
{"Platform":{"Name":"Docker Engine - Community"},"Components":[{"Name":"Engine","Version":"19.03.12","Details":{"ApiVersion":"1.40","Arch":"amd64","BuildTime":"2020-06-22T15:45:28.000000000+00:00","Experimental":"false","GitCommit":"48a66213fe","GoVersion":"go1.13.10","KernelVersion":"3.10.0-957.12.2.el7.x86_64","MinAPIVersion":"1.12","Os":"linux"}},{"Name":"containerd","Version":"1.2.13","Details":{"GitCommit":"7ad184331fa3e55e52b890ea95e65ba581ae3429"}},{"Name":"runc","Version":"1.0.0-rc10","Details":{"GitCommit":"dc9208a3303feef5b3839f4323d9beb36df0a9dd"}},{"Name":"docker-init","Version":"0.18.0","Details":{"GitCommit":"fec3683"}}],"Version":"19.03.12","ApiVersion":"1.40","MinAPIVersion":"1.12","GitCommit":"48a66213fe","GoVersion":"go1.13.10","Os":"linux","Arch":"amd64","KernelVersion":"3.10.0-957.12.2.el7.x86_64","BuildTime":"2020-06-22T15:45:28.000000000+00:00"}

Jenkins配置

我们需要安装插件 Docker plugin

1.3.1 方式1: 启动镜像进行构建(无需连接master)

dockerNode(dockerHost: 'tcp://192.168.33.11:2375', image: 'jenkins/inbound-agent') {
    // some block
    sh "sleep 50"
}

Alt Image Text

[Pipeline] Start of Pipeline
[Pipeline] dockerNode
Launching new docker node based on jenkins/inbound-agent
Pulling from jenkins/inbound-agent
Digest: sha256:9392c06eaef92cebe60deae0581541e8ef19197533cd628d792374c4a32199c9
Status: Image is up to date for jenkins/inbound-agent:latest
Waiting for node to be online ...
Node docker-0001vtsr0hv0n is online.
[Pipeline] {
[Pipeline] sh
+ sleep 50
[Pipeline] }
Terminating docker node ...
Disconnected computer for node 'docker-0001vtsr0hv0n'.
Removed Node for node 'docker-0001vtsr0hv0n'.

1.3.2 方式2: 使用CLoud

Manage Jenkins -> Manage Nodes and Clouds -> Configure Clouds

Docker Cloud details

  • Docker Host URI Docker主机信息(需要开启Docker配置)
    • tcp://192.168.33.11:2375
    • unix:///var/run/docker.sock
  • Connection Timeout 连接超时时间
  • Read Timeout 读操作超时时间 (调大些,容易出现超时的情况)
  • Enabled 是否启用?默认否
  • Error Duration 错误的持续时间 默认300 5分钟

  • Container Cap 容器数量 负值或零,或2147483647都意味着“无限制” ,默认值100。

    • 10

Alt Image Text

Docker Agent templates

  • Labels 节点标签: jenkins-docker-agent
  • Enabled 是否启动 默认否
  • Name 节点名称 jenkins-docker-agent
  • Docker Image 镜像标签: jenkins/inbound-agent
  • Remote File System Root 远程文件系统根目录 /home/jenkins
  • 用法 自定义指定项目运行
  • Connect method 连接方式
  • JNLP 推荐
  • User 运行用户 : jenkins
  • Jenkins URL jenkins地址 http://192.168.33.11:8080
  • Idle timeout 空闲时间多少秒后杀死slave
  • SSH
  • Docker Container
  • Pull strategy 镜像下载策略
  • Pull timeout 镜像下载超时时间 单位秒
  • Instance Capacity 实例数量

Alt Image Text

测试

node("jenkins-docker-agent"){
     sh "sleep 30"
    echo "Hello world!"
}

Alt Image Text

[Pipeline] node
Running on jenkins-docker-agent-0001w25yyyc8a on docker in /home/jenkins/workspace/dokcer-agent3
[Pipeline] {
[Pipeline] sh

二、基于Docker的pipeline流水线

2.1 容器中编译流水线作业

2.1.1 准备工作

配置JenkinsMaster挂载Docker

docker run -d -p 8080:8080 -p 50000:50000 --env=JAVA_OPTS=-Djenkins.install.runSetupWizard=false -v /var/lib/jenkins:/var/jenkins_home  -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker jenkins:v20200816

解决权限问题/以root用户运行

docker exec -it 005a7714ecc9 bash
jenkins@005a7714ecc9:/$ docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json: dial unix /var/run/docker.sock: connect: permission denied
$ docker exec -it -u root 005a7714ecc9 bash

root@005a7714ecc9:/# docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                                              NAMES
005a7714ecc9        jenkins:v20200816       "/sbin/tini -- /usr/…"   4 minutes ago       Up 4 minutes        0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp   hopeful_pascal
384516ab8579        jenkins/inbound-agent   "jenkins-agent -url …"   12 minutes ago      Up 12 minutes                                                          blissful_nash

root@005a7714ecc9:/# usermod -aG root jenkins
root@005a7714ecc9:/# exit
$ docker restart 005a7714ecc9
$ id jenkins
uid=1000(jenkins) gid=1000(jenkins) groups=1000(jenkins),0(root)

我的问题

我在主机上创建了dockergroup, 根据这个文章 https://docs.docker.com/engine/install/linux-postinstall/

sudo groupadd docker
sudo usermod -aG docker $USER

$ cat /etc/group
...
docker:x:993:vagrant

$ docker exec -it -u root 005a7714ecc9 bash
$ groupadd -g 993 docker-test

root@005a7714ecc9:/# usermod -aG docker-test jenkins
root@005a7714ecc9:/# id jenkins
uid=1000(jenkins) gid=1000(jenkins) groups=1000(jenkins),0(root),1001(docker),993(docker-test)

root@005a7714ecc9:/# cat /etc/group
...
docker-test:x:993:jenkins

$ docker exec -it 005a7714ecc9 bash
jenkins@005a7714ecc9:/$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                              NAMES
005a7714ecc9        jenkins:v20200816   "/sbin/tini -- /usr/…"   5 hours ago         Up 3 hours          0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp   hopeful_pascal

jenkins should have permission now.

Getting “Permission Denied” error when pulling a docker image in Jenkins docker container on Mac

2.1.2 测试流水线

pipeline {
    agent {
        docker { 
            image 'maven:3.6.3-jdk-8' 
            args '-v $HOME/.m2:/root/.m2'
        }
    }
    stages {
        stage('Build') {
            steps {
                sh 'mvn -v'
            }
        }
    }
}

Console Output

Running on Jenkins in /var/jenkins_home/workspace/docker-pipeline1
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker inspect -f . maven:3.6.3-jdk-8
.
[Pipeline] withDockerContainer
Jenkins seems to be running inside container 005a7714ecc980f673c3083030ff79a82687681de55d9cd9875625a5aa0520f3
$ docker run -t -d -u 1000:1000 -v $HOME/.m2:/root/.m2 -w /var/jenkins_home/workspace/docker-pipeline1 --volumes-from 005a7714ecc980f673c3083030ff79a82687681de55d9cd9875625a5aa0520f3 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** maven:3.6.3-jdk-8 cat
$ docker top c8aec18483a36a66c120975c09b667ba0292a7abebbe3b4409a5bbc1d23469ae -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] sh
+ mvn -v
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/share/maven
Java version: 1.8.0_265, vendor: Oracle Corporation, runtime: /usr/local/openjdk-8/jre
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-957.12.2.el7.x86_64", arch: "amd64", family: "unix"
[Pipeline] sleep
Sleeping for 30 sec
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
$ docker stop --time=1 c8aec18483a36a66c120975c09b667ba0292a7abebbe3b4409a5bbc1d23469ae
$ docker rm -f c8aec18483a36a66c120975c09b667ba0292a7abebbe3b4409a5bbc1d23469ae
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

2.1.3 前后端未分离项目

对于代码库中既包含前端项目又包含后端项目的配置。可以启动多个容器。

pipeline {
    agent none
    stages {
        stage('ServiceBuild') {
            agent {
                docker { 
                    image 'maven:3.6.3-jdk-8' 
                    args '-v $HOME/.m2:/root/.m2'
                }
            }
            steps {
                sh 'mvn -v  && sleep 15'
            }
        }

        stage('WebBuild') {
            agent {
                docker { 
                    image 'node:7-alpine' 
                    args '-v $HOME/.npm:/root/.npm'
                }
            }
            steps {
                sh 'node -v  && sleep 15'
            }
        }
    }
}

Console output

Running on Jenkins in /var/jenkins_home/workspace/docker-pipeline2
[Pipeline] {
[Pipeline] isUnix (hide)
[Pipeline] sh
+ docker inspect -f . maven:3.6.3-jdk-8
.
[Pipeline] withDockerContainer
Jenkins seems to be running inside container 005a7714ecc980f673c3083030ff79a82687681de55d9cd9875625a5aa0520f3
$ docker run -t -d -u 1000:1000 -v $HOME/.m2:/root/.m2 -w /var/jenkins_home/workspace/docker-pipeline2 --volumes-from 005a7714ecc980f673c3083030ff79a82687681de55d9cd9875625a5aa0520f3 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** maven:3.6.3-jdk-8 cat
$ docker top 9f15b56640e3d65660876a4f854918103f7f099150dbb5f99dff43ace1296392 -eo pid,comm
[Pipeline] {
[Pipeline] sh
+ mvn -v
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/share/maven
Java version: 1.8.0_265, vendor: Oracle Corporation, runtime: /usr/local/openjdk-8/jre
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-957.12.2.el7.x86_64", arch: "amd64", family: "unix"
+ sleep 15
[Pipeline] }
$ docker stop --time=1 9f15b56640e3d65660876a4f854918103f7f099150dbb5f99dff43ace1296392
$ docker rm -f 9f15b56640e3d65660876a4f854918103f7f099150dbb5f99dff43ace1296392
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (WebBuild)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/docker-pipeline2
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker inspect -f . node:7-alpine
.
[Pipeline] withDockerContainer
Jenkins seems to be running inside container 005a7714ecc980f673c3083030ff79a82687681de55d9cd9875625a5aa0520f3
$ docker run -t -d -u 1000:1000 -v $HOME/.npm:/root/.npm -w /var/jenkins_home/workspace/docker-pipeline2 --volumes-from 005a7714ecc980f673c3083030ff79a82687681de55d9cd9875625a5aa0520f3 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:7-alpine cat
$ docker top 33be7fc88b0f15f3cd3f947250e3b0d6d57f05096b6bac2bee7006cf0e380680 -eo pid,comm
[Pipeline] {
[Pipeline] sh
+ node -v
v7.10.1
+ sleep 15
[Pipeline] }
$ docker stop --time=1 33be7fc88b0f15f3cd3f947250e3b0d6d57f05096b6bac2bee7006cf0e380680
$ docker rm -f 33be7fc88b0f15f3cd3f947250e3b0d6d57f05096b6bac2bee7006cf0e380680
[Pipeline] // withDockerContainer

2.1.4 前端项目流水线

$ cd /home/vagrant/workspace/workspace/demo-pipeline3

// 安装 vue-cli

$ sudo chown -R 1000:1000 "/home/vagrant/.npm"

$  npm install -g @vue/cli-init
npm WARN deprecated vue-cli@2.9.6: This package has been deprecated in favour of @vue/cli
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated coffee-script@1.12.7: CoffeeScript on NPM has moved to "coffeescript" (no hyphen)
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
+ @vue/cli-init@4.5.4
added 251 packages from 206 contributors in 17.774s

$ vue --version
@vue/cli 4.5.4

$ cd /home/vagrant/workspace/workspace/demo-pipeline3
$ $ vue init webpack demo

? Project name demo
? Project description A Vue.js project
? Author 
? Vue build standalone
? Install vue-router? Yes
? Use ESLint to lint your code? Yes
? Pick an ESLint preset Standard
? Set up unit tests Yes
? Pick a test runner jest
? Setup e2e tests with Nightwatch? No
? Should we run `npm install` for you after the project has been created? (recommended) npm

   vue-cli · Generated "demo".

## Installing project dependencies ...

编写jenkinsfile

pipeline {
   agent {node {label "hostmachine"}}
    stages {
        stage('WebBuild') {
            steps {
                script {
                    docker.image('node:10.19.0-alpine').inside('-u 0:0 -v /var/jenkins_home/.npm:/root/.npm') {

                        sh """
                            id 
                            ls /root/.npm

                            ls /root/ -a
                            npm config set unsafe-perm=true
                            npm config list
                            npm config set cache  /root/.npm
                            #npm config set registry https://registry.npm.taobao.org
                            npm config list
                            ls 
                            cd demo && npm install  --unsafe-perm=true && npm run build  && ls -l dist/ && sleep 15 
                        """
                    }
                }
            }
        }
    }
}

Console output

$ docker run -t -d -u 1000:1000 -u 0:0 -v /var/jenkins_home/.npm:/root/.npm -w /home/vagrant/workspace/workspace/demo-pipeline3 -v /home/vagrant/workspace/workspace/demo-pipeline3:/home/vagrant/workspace/workspace/demo-pipeline3:rw,z -v /home/vagrant/workspace/workspace/demo-pipeline3@tmp:/home/vagrant/workspace/workspace/demo-pipeline3@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:10.19.0-alpine cat
$ docker top ea767dafdc3ba98baf9399ebcde3fd491911d77a0c993f92a8a3b6f44a51fc7a -eo pid,comm
[Pipeline] {
[Pipeline] sh
+ id
uid=0(root) gid=0(root)
+ ls /root/.npm
_cacache
_locks
_logs
anonymous-cli-metrics.json
+ npm config set 'unsafe-perm=true'
+ npm config list
; cli configs
metrics-registry = "https://registry.npmjs.org/"
scope = ""
user-agent = "npm/6.13.4 node/v10.19.0 linux x64 ci/jenkins"

; userconfig /root/.npmrc
unsafe-perm = true

; node bin location = /usr/local/bin/node
; cwd = /home/vagrant/workspace/workspace/demo-pipeline3
; HOME = /root
; "npm config ls -l" to show all defaults.

+ npm config set cache /root/.npm
+ npm config list
; cli configs
metrics-registry = "https://registry.npmjs.org/"
scope = ""
user-agent = "npm/6.13.4 node/v10.19.0 linux x64 ci/jenkins"

; userconfig /root/.npmrc
cache = "/root/.npm"
unsafe-perm = true

; node bin location = /usr/local/bin/node
; cwd = /home/vagrant/workspace/workspace/demo-pipeline3
; HOME = /root
; "npm config ls -l" to show all defaults.

+ ls
demo
package-lock.json
+ cd demo
+ npm install '--unsafe-perm=true'
Cannot contact hostmachine: java.lang.InterruptedException
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.3 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

audited 1734 packages in 10.906s

28 packages are looking for funding
  run `npm fund` for details

found 89 vulnerabilities (73 low, 9 moderate, 7 high)
  run `npm audit fix` to fix them, or `npm audit` for details
+ npm run build

> demo@1.0.0 build /home/vagrant/workspace/workspace/demo-pipeline3/demo
> node build/build.js

Hash: [1mc4e4fdef7b4fbb5efe81[39m[22m
Version: webpack [1m3.12.0[39m[22m
Time: [1m12173[39m[22mms
                                                  [1mAsset[39m[22m       [1mSize[39m[22m  [1mChunks[39m[22m  [1m[39m[22m           [1m[39m[22m[1mChunk Names[39m[22m
               [1m[32mstatic/js/vendor.936b7041a764ab1c3f2c.js[39m[22m     123 kB       [1m0[39m[22m  [1m[32m[emitted][39m[22m  vendor
                  [1m[32mstatic/js/app.b22ce679862c47a75225.js[39m[22m    11.6 kB       [1m1[39m[22m  [1m[32m[emitted][39m[22m  app
             [1m[32mstatic/js/manifest.2ae2e69a05c33dfc65f8.js[39m[22m  857 bytes       [1m2[39m[22m  [1m[32m[emitted][39m[22m  manifest
    [1m[32mstatic/css/app.30790115300ab27614ce176899523b62.css[39m[22m  432 bytes       [1m1[39m[22m  [1m[32m[emitted][39m[22m  app
[1m[32mstatic/css/app.30790115300ab27614ce176899523b62.css.map[39m[22m  797 bytes        [1m[39m[22m  [1m[32m[emitted][39m[22m  
           [1m[32mstatic/js/vendor.936b7041a764ab1c3f2c.js.map[39m[22m     619 kB       [1m0[39m[22m  [1m[32m[emitted][39m[22m  vendor
              [1m[32mstatic/js/app.b22ce679862c47a75225.js.map[39m[22m    22.2 kB       [1m1[39m[22m  [1m[32m[emitted][39m[22m  app
         [1m[32mstatic/js/manifest.2ae2e69a05c33dfc65f8.js.map[39m[22m    4.97 kB       [1m2[39m[22m  [1m[32m[emitted][39m[22m  manifest
                                             [1m[32mindex.html[39m[22m  506 bytes        [1m[39m[22m  [1m[32m[emitted][39m[22m  

  Build complete.

  Tip: built files are meant to be served over an HTTP server.
  Opening index.html over file:// won't work.

+ ls -l dist/
total 4
-rw-r--r--    1 root     root           506 Aug 20 01:34 index.html
drwxr-xr-x    4 root     root            27 Aug 20 01:34 static
+ sleep 15
[Pipeline] }
$ docker stop --time=1 ea767dafdc3ba98baf9399ebcde3fd491911d77a0c993f92a8a3b6f44a51fc7a
$ docker rm -f ea767dafdc3ba98baf9399ebcde3fd491911d77a0c993f92a8a3b6f44a51fc7a

2.1.5 FAQ

npm构建权限问题:使用root用户构建。设置容器运行用户 -u 0:0

三、构建应用镜像到镜像仓库管理

3.1 构建应用镜像到镜像仓库管理

将应用构建成镜像、将镜像上传到镜像仓库非常简单。通过命令就能解决。镜像仓库巨大爆满如何解决? 我们需要在开始使用前就应该设置好镜像的构建策略。(每个公司中管理不一样,具体可根据分支策略决定)。

3.2 分支开发策略

  • 主干分支 master
  • 特性分支 f1 f2
  • 版本分支 release

你会发现:

  • 特性分支需要构建发布到特性环境验证。
  • 版本分支需要构建发布到UAT/STAG/PROD环境验证。

考虑到:

  • 特性分支产生的镜像会很多,而且并不重要可以定时清理掉。
  • 版本分支产生相对较少(考虑到版本修复情况),每个版本只有一个镜像。(上线发布完成后清除掉其他)

Alt Image Text

3.3 镜像管理规范

3.3.1 命名规范

  • 仓库类型
    • snapshot : 开发版本仓库
    • release : 生产正式版本仓库
  • 仓库命名
    • snapshot : 业务/项目名称-snapshot demo-snapshot
    • release : 业务/项目名称-release demo-release
  • 镜像命名
    • (业务/项目名称)/应用名称/标签
    • DEV : demo-snapshot/demo-devops-service:branch_commitid
    • PRD: demo-release/demo-devops-service:version_commitid
  • 标签命名
    • 分支名_提交ID
    • 版本号_提交ID

提交ID的意义: 减少重复构建

3.3.2 镜像清理策略

Alt Image Text

随着镜像越来越多,频繁更新导致Harbor镜像仓库容量很快爆满。

  • snapshot仓库: 每天晚上定时清理前一天的镜像(注意风险)
  • release仓库: 版本发布完成后,清除版本其他镜像

  • 1.1.1_xxxxxxxxxx11

  • 1.1.1_xxxxxxxxxx22

3.4 构建应用镜像

3.4.1 编写应用Dockerfile

$ cd demo
$ pwd
/home/vagrant/workspace/workspace/demo-pipeline3/demo

$ touch Dockerfile
$ vim Dockerfile

FROM nginx:latest

COPY dist /usr/share/nginx/html

3.4.2 配置流水线构建镜像

docker build -t demo-web-app:1.1.1 .

3.4.3 上传镜像

docker push demo-web-app:1.1.1

Jenkinsfile

pipeline {
    agent {node {label "hostmachine"}}
    stages {
        stage('WebBuild') {
            steps {
                script {
                    docker.image('node:10.19.0-alpine').inside('-u 0:0 -v /var/jenkins_home/.npm:/root/.npm') {

                        sh """
                            id 
                            ls /root/.npm

                            ls /root/ -a
                            npm config set unsafe-perm=true
                            npm config list
                            npm config set cache  /root/.npm
                            #npm config set registry https://registry.npm.taobao.org
                            npm config list
                            ls 
                            cd demo && npm install  --unsafe-perm=true && npm run build  && ls -l dist/ && sleep 15 
                        """
                    }
                }
            }
        }

      stage("BuildImage"){
        steps {
          script{

            sh """
                #构建镜像
                cd demo
                docker build -t demo/demo-web-app:1.1.1_xxxxxxxx1 .

                #docker push demo/demo-web-app:1.1.1_xxxxxxxx1

            """

          }
        }
      }
    }
}

console output

+ cd demo
+ docker build -t demo/demo-web-app:1.1.1_xxxxxxxx1 .
Sending build context to Docker daemon    203MB
..
Successfully built 3149ee3602a7
Successfully tagged demo/demo-web-app:1.1.1_xxxxxxxx1
 $ docker images
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
demo/demo-web-app               1.1.1_xxxxxxxx1     3149ee3602a7        6 minutes ago       133MB

运行镜像

docker run -itd -p 8088:80 --name nginx-server demo/demo-web-app:1.1.1_xxxxxxxx1
http://192.168.33.11:8088/#/

Alt Image Text

3.5 镜像清理(Harbor) 注意风险 扩展研究

获取标签

  • https://registry.demo.com/api/repositories/${registryName}/${serviceName}/tags
  • 方式: GET

删除标签

  • https://registry.demo.com/api/repositories/${registryName}/${serviceName}/tags/${tag}
  • 方式: DELETE
#!groovy
@Library('jenkinslib@master') _

def tools = new org.devops.tools()

String registryName = "${env.registryName}"
String serviceName = "${env.serviceName}"
String tagName = "${env.tagName}"
def harborProjects = []

currentBuild.description = "Trigger by ${serviceName} ${tagName}"

pipeline {
agent { node { label "build"} }

stages{

stage("GetHarborTags"){
    steps{
        timeout(time:5, unit:"MINUTES"){
            script{
                tools.PrintMes("获取Harbor仓库中的项目信息","green")
                println(serviceName)

                try {
                    response = httpRequest authentication: 'harbor-admin,
                                           url: "https://registry.demo.com/api/repositories/${registryName}/${serviceName}/tags",
                                           ignoreSslErrors: true
                    //println(response.content)
                    response = readJSON text: """${response.content}"""

                } catch(e){
                    response = ['name':'']
                    println(e)
                    println("Harbor镜像不存在此标签!")

                }

                /*println(tagName)
                for (tagname in response){
                    //println(response)
                    harborProjects << tagname['name']
                }

                println(harborProjects)*/

            }
        }
    }
}

stage("DeleteHarborTags"){
    steps{
        timeout(time:20, unit:"MINUTES"){
            script{
                tools.PrintMes("总共找到 ${harborProjects.size()} 个标签","green")
                sumImageNum = harborProjects.size()
                for (tag in harborProjects){
                    sumImageNum -= 1

                    tools.PrintMes(" ${sumImageNum}  Delete Tags ---> ${registryName} --> ${serviceName} --> ${tag} ","green")

                    httpRequest httpMode: 'DELETE',
                               authentication: 'c016027e-0573-4246-93cf-f4a55b08a86a',
                               url: "https://registry.demo.com/api/repositories/${registryName}/${serviceName}/tags/${tag}",
                               ignoreSslErrors: true

                    sleep 1
                }
            }
        }
    }
}
}

post {
always{
    script{
        cleanWs notFailBuild: true 
    }
}
}
}

四、使用Groovy代码初始化Docker配置

4.1 解析官方提供的groovy代码

插件地址: https://plugins.jenkins.io/docker-plugin/

  • https://github.com/jenkinsci/docker-plugin/blob/master/docs/attachments/docker-plugin-configuration-script.groovy
import com.nirima.jenkins.plugins.docker.DockerCloud
import com.nirima.jenkins.plugins.docker.DockerTemplate
import com.nirima.jenkins.plugins.docker.DockerTemplateBase
import com.nirima.jenkins.plugins.docker.launcher.AttachedDockerComputerLauncher
import io.jenkins.docker.connector.DockerComputerAttachConnector
import jenkins.model.Jenkins

src/main/java/com/nirima/jenkins/plugins/docker目录可以看到这些源码文件。

Alt Image Text

src/main/java/com/nirima/jenkins/plugins/docker/launcher 目录下可以看多支持的启动类型。

Alt Image Text

src/main/java/io/jenkins/docker/connector 目录下定义的是每个启动类型对应的连接器。

Alt Image Text

dockerTemplateBaseParameters定义容器的模板所需要的基本参数,实例化新的对象dockerTemplateBase

Alt Image Text

//Docker容器的配置参数
def dockerTemplateBaseParameters = [
  bindAllPorts:       false,
  bindPorts:          '',
  cpuShares:          null,
  dnsString:          '',
  dockerCommand:      '',
  environmentsString: '',
  extraHostsString:   '',
  hostname:           '',
  image:              'jenkins/inbound-agent:latest',
  macAddress:         '',
  memoryLimit:        null,
  memorySwap:         null,
  network:            '',
  privileged:         false,
  pullCredentialsId:  '',
  sharedMemorySize:   null,
  tty:                true,
  volumesFromString:  '',
  volumesString:      ''
]

//Docker容器的配置参数
def DockerTemplateParameters = [
  instanceCapStr: '4',    //实例数量
  labelString:    'docker.local.jenkins.slave',    //节点标签
  remoteFs:       ''     //根目录
]

//src/main/java/com/nirima/jenkins/plugins/docker/DockerTemplateBase.java
//实例化参数
DockerTemplateBase dockerTemplateBase = new DockerTemplateBase(
  dockerTemplateBaseParameters.image,
  dockerTemplateBaseParameters.pullCredentialsId,
  dockerTemplateBaseParameters.dnsString,
  dockerTemplateBaseParameters.network,
  dockerTemplateBaseParameters.dockerCommand,
  dockerTemplateBaseParameters.volumesString,
  dockerTemplateBaseParameters.volumesFromString,
  dockerTemplateBaseParameters.environmentsString,
  dockerTemplateBaseParameters.hostname,
  dockerTemplateBaseParameters.memoryLimit,
  dockerTemplateBaseParameters.memorySwap,
  dockerTemplateBaseParameters.cpuShares,
  dockerTemplateBaseParameters.sharedMemorySize,
  dockerTemplateBaseParameters.bindPorts,
  dockerTemplateBaseParameters.bindAllPorts,
  dockerTemplateBaseParameters.privileged,
  dockerTemplateBaseParameters.tty,
  dockerTemplateBaseParameters.macAddress,
  dockerTemplateBaseParameters.extraHostsString
)

根据Docker容器的配置参数,创建一个DockerComputerAttachConnector类型的容器模板。

//创建容器模板
// src/main/java/com/nirima/jenkins/plugins/docker/DockerTemplate.java
DockerTemplate dockerTemplate = new DockerTemplate(
  dockerTemplateBase,
  new DockerComputerAttachConnector(),
  DockerTemplateParameters.labelString,
  DockerTemplateParameters.remoteFs,
  DockerTemplateParameters.instanceCapStr
)

声明DockerCLoud的配置信息:连接超时时间、容器实例数量、认证ID、主机名、读超时时间、服务远程地址、Docker版本信息。

//Cloud配置信息
def dockerCloudParameters = [
  connectTimeout:   3,     
  containerCapStr:  '4',
  credentialsId:    '',
  dockerHostname:   '',
  name:             'docker.local',
  readTimeout:      60,
  serverUrl:        'unix:///var/run/docker.sock',
  version:          ''
]

将Docker容器的配置模板加入到DockerCloud模板中,通过Jenkins.clouds添加的jenkins配置。

//配置DockerCloud
// src/main/java/com/nirima/jenkins/plugins/docker/DockerCloud.java
DockerCloud dockerCloud = new DockerCloud(
  dockerCloudParameters.name,
  [dockerTemplate],   //docker容器模板
  dockerCloudParameters.serverUrl,
  dockerCloudParameters.containerCapStr,
  dockerCloudParameters.connectTimeout,
  dockerCloudParameters.readTimeout,
  dockerCloudParameters.credentialsId,
  dockerCloudParameters.version,
  dockerCloudParameters.dockerHostname
)

// get Jenkins instance
Jenkins jenkins = Jenkins.getInstance()

// add cloud configuration to Jenkins
jenkins.clouds.add(dockerCloud)

// save current Jenkins state to disk
jenkins.save()

复制写好的代码,在Jenkins脚本命令行运行。

import com.nirima.jenkins.plugins.docker.DockerCloud
import com.nirima.jenkins.plugins.docker.DockerTemplate
import com.nirima.jenkins.plugins.docker.DockerTemplateBase
import com.nirima.jenkins.plugins.docker.launcher.DockerComputerJNLPLauncher
import io.jenkins.docker.connector.DockerComputerJNLPConnector
import jenkins.model.Jenkins

// parameters
def dockerTemplateBaseParameters = [
  bindAllPorts:       false,
  bindPorts:          '',
  cpuPeriod:          null,
  cpuQuota:           null,
  cpuShares:          null,
  dnsString:          '',
  dockerCommand:      '',
  environmentsString: '',
  extraHostsString:   '',
  hostname:           '',
  image:              'jenkins/inbound-agent:latest',
  macAddress:         '',
  memoryLimit:        null,
  memorySwap:         null,
  network:            '',
  privileged:         false,
  pullCredentialsId:  '',
  sharedMemorySize:   null,
  tty:                true,
  volumesFromString:  '',
  volumesString:      ''
]

def DockerTemplateParameters = [
  instanceCapStr: '4',
  labelString:    'docker.local.jenkins.agent',
  remoteFs:       ''
]

def dockerCloudParameters = [
  connectTimeout:   3,
  containerCapStr:  '4',
  credentialsId:    '',
  dockerHostname:   '',
  name:             'docker.local',
  readTimeout:      60,
  serverUrl:        'unix:///var/run/docker.sock',
  version:          ''
]

// https://github.com/jenkinsci/docker-plugin/blob/docker-plugin-1.1.2/src/main/java/com/nirima/jenkins/plugins/docker/DockerTemplateBase.java
DockerTemplateBase dockerTemplateBase = new DockerTemplateBase(
  dockerTemplateBaseParameters.image,
  dockerTemplateBaseParameters.pullCredentialsId,
  dockerTemplateBaseParameters.dnsString,
  dockerTemplateBaseParameters.network,
  dockerTemplateBaseParameters.dockerCommand,
  dockerTemplateBaseParameters.volumesString,
  dockerTemplateBaseParameters.volumesFromString,
  dockerTemplateBaseParameters.environmentsString,
  dockerTemplateBaseParameters.hostname,
  dockerTemplateBaseParameters.memoryLimit,
  dockerTemplateBaseParameters.memorySwap,
  dockerTemplateBaseParameters.cpuPeriod,
  dockerTemplateBaseParameters.cpuQuota,
  dockerTemplateBaseParameters.cpuShares,
  dockerTemplateBaseParameters.sharedMemorySize,
  dockerTemplateBaseParameters.bindPorts,
  dockerTemplateBaseParameters.bindAllPorts,
  dockerTemplateBaseParameters.privileged,
  dockerTemplateBaseParameters.tty,
  dockerTemplateBaseParameters.macAddress,
  dockerTemplateBaseParameters.extraHostsString
)

// https://github.com/jenkinsci/docker-plugin/blob/docker-plugin-1.1.2/src/main/java/com/nirima/jenkins/plugins/docker/DockerTemplate.java
DockerTemplate dockerTemplate = new DockerTemplate(
  dockerTemplateBase,
  new DockerComputerAttachConnector(),
  DockerTemplateParameters.labelString,
  DockerTemplateParameters.remoteFs,
  DockerTemplateParameters.instanceCapStr
)

// https://github.com/jenkinsci/docker-plugin/blob/docker-plugin-1.1.2/src/main/java/com/nirima/jenkins/plugins/docker/DockerCloud.java
DockerCloud dockerCloud = new DockerCloud(
  dockerCloudParameters.name,
  [dockerTemplate],
  dockerCloudParameters.serverUrl,
  dockerCloudParameters.containerCapStr,
  dockerCloudParameters.connectTimeout,
  dockerCloudParameters.readTimeout,
  dockerCloudParameters.credentialsId,
  dockerCloudParameters.version,
  dockerCloudParameters.dockerHostname
)

// get Jenkins instance
Jenkins jenkins = Jenkins.getInstance()

// add cloud configuration to Jenkins
jenkins.clouds.add(dockerCloud)

// save current Jenkins state to disk
jenkins.save()

Alt Image Text

4.2 实例: 添加一个JNLP类型的Docker Cloud配置

不同点1: 需要引入DockerComputerJNLPLauncherDockerComputerJNLPConnectorhudson.slaves.JNLPLauncher

import com.nirima.jenkins.plugins.docker.DockerCloud
import com.nirima.jenkins.plugins.docker.DockerTemplate
import com.nirima.jenkins.plugins.docker.DockerTemplateBase
//import com.nirima.jenkins.plugins.docker.launcher.AttachedDockerComputerLauncher
import com.nirima.jenkins.plugins.docker.launcher.DockerComputerJNLPLauncher
//import io.jenkins.docker.connector.DockerComputerAttachConnector
import io.jenkins.docker.connector.DockerComputerJNLPConnector
import jenkins.model.Jenkins
import hudson.slaves.JNLPLauncher;  

不同点2: JNLP类型的启动方式需要添加两个变量,分别是Jenkinsuser和Jenkins服务器地址。

// src/main/java/com/nirima/jenkins/plugins/docker/DockerTemplate.java
DockerTemplate dockerTemplate = new DockerTemplate(
  dockerTemplateBase,
  //new DockerComputerAttachConnector(),
  new DockerComputerJNLPConnector(new JNLPLauncher(null, null)).withUser("jenkins")
                        .withJenkinsUrl("http://123.123.123:8080"),
  DockerTemplateParameters.labelString,
  DockerTemplateParameters.remoteFs,
  DockerTemplateParameters.instanceCapStr
)

参考测试文件,获取JNLP方式的容器模板创建方式。

final DockerTemplate template = new DockerTemplate(
                new DockerTemplateBase(JNLP_SLAVE_IMAGE_IMAGENAME),
                new DockerComputerJNLPConnector(new JNLPLauncher(null, null)).withUser(COMMON_IMAGE_USERNAME)
                        .withJenkinsUrl(uri.toString()),
                        LABEL, COMMON_IMAGE_HOMEDIR, INSTANCE_CAP
        );

完整源码

import com.nirima.jenkins.plugins.docker.DockerCloud
import com.nirima.jenkins.plugins.docker.DockerTemplate
import com.nirima.jenkins.plugins.docker.DockerTemplateBase
//import com.nirima.jenkins.plugins.docker.launcher.AttachedDockerComputerLauncher
import com.nirima.jenkins.plugins.docker.launcher.DockerComputerJNLPLauncher
//import io.jenkins.docker.connector.DockerComputerAttachConnector
import io.jenkins.docker.connector.DockerComputerJNLPConnector
import jenkins.model.Jenkins
import hudson.slaves.JNLPLauncher;
// parameters

//Docker Agent 基本参数
def dockerTemplateBaseParameters = [
  bindAllPorts:       false,
  bindPorts:          '',
  cpuShares:          null,
  dnsString:          '',
  dockerCommand:      '',
  environmentsString: '',
  extraHostsString:   '',
  hostname:           '',
  image:              'jenkins/inbound-agent:latest',   //镜像
  macAddress:         '',
  memoryLimit:        null,
  memorySwap:         null,
  network:            '',
  privileged:         false,
  pullCredentialsId:  '',
  sharedMemorySize:   null,
  tty:                true,
  volumesFromString:  '',
  volumesString:      ''
]

//Docker Agnet 参数
def DockerTemplateParameters = [
  instanceCapStr: '10',    //实例数
  labelString:    'jenkins-agent-test',   //节点标签
  remoteFs:       '/home/jenkins'      //主目录

]

def dockerCloudParameters = [
  connectTimeout:   60,       //链接超时
  containerCapStr:  '10',     //实例数量
  credentialsId:    '',
  dockerHostname:   '',
  name:             'docker.local.test',     
  readTimeout:      60,
  serverUrl:        'tcp://192.168.33.11:2375',
  version:          ''
]

// https://github.com/jenkinsci/docker-plugin/blob/docker-plugin-1.1.2/src/main/java/com/nirima/jenkins/plugins/docker/DockerTemplateBase.java
DockerTemplateBase dockerTemplateBase = new DockerTemplateBase(
  dockerTemplateBaseParameters.image,
  dockerTemplateBaseParameters.pullCredentialsId,
  dockerTemplateBaseParameters.dnsString,
  dockerTemplateBaseParameters.network,
  dockerTemplateBaseParameters.dockerCommand,
  dockerTemplateBaseParameters.volumesString,
  dockerTemplateBaseParameters.volumesFromString,
  dockerTemplateBaseParameters.environmentsString,
  dockerTemplateBaseParameters.hostname,
  dockerTemplateBaseParameters.memoryLimit,
  dockerTemplateBaseParameters.memorySwap,
  dockerTemplateBaseParameters.cpuPeriod,
  dockerTemplateBaseParameters.cpuQuota,
  dockerTemplateBaseParameters.cpuShares,
  dockerTemplateBaseParameters.sharedMemorySize,
  dockerTemplateBaseParameters.bindPorts,
  dockerTemplateBaseParameters.bindAllPorts,
  dockerTemplateBaseParameters.privileged,
  dockerTemplateBaseParameters.tty,
  dockerTemplateBaseParameters.macAddress,
  dockerTemplateBaseParameters.extraHostsString
)

// https://github.com/jenkinsci/docker-plugin/blob/docker-plugin-1.1.2/src/main/java/com/nirima/jenkins/plugins/docker/DockerTemplate.java
DockerTemplate dockerTemplate = new DockerTemplate(
  dockerTemplateBase,
  //new DockerComputerAttachConnector(),
  new DockerComputerJNLPConnector(new JNLPLauncher(null, null)).withUser("jenkins")
                        .withJenkinsUrl("http://192.168.33.11:8080"),
  DockerTemplateParameters.labelString,
  DockerTemplateParameters.remoteFs,
  DockerTemplateParameters.instanceCapStr
)

// https://github.com/jenkinsci/docker-plugin/blob/docker-plugin-1.1.2/src/main/java/com/nirima/jenkins/plugins/docker/DockerCloud.java
DockerCloud dockerCloud = new DockerCloud(
  dockerCloudParameters.name,
  [dockerTemplate],
  dockerCloudParameters.serverUrl,
  dockerCloudParameters.containerCapStr,
  dockerCloudParameters.connectTimeout,
  dockerCloudParameters.readTimeout,
  dockerCloudParameters.credentialsId,
  dockerCloudParameters.version,
  dockerCloudParameters.dockerHostname
)

// get Jenkins instance
Jenkins jenkins = Jenkins.getInstance()

// add cloud configuration to Jenkins
jenkins.clouds.add(dockerCloud)

// save current Jenkins state to disk
jenkins.save()

Alt Image Text