分享几种用 dokcer 方式部署的 jenkins 中如何使用 docker 的解决方案
解决方案一:挂载 docker.sock,直接使用 docker 命令
docker-compose 中增加下面的挂载项
...
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /usr/bin/docker:/usr/bin/docker
- /etc/docker:/etc/docker
给/var/run/docker.sock 增加权限,否则可能报下面的错
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/auth": dial unix /var/run/docker.sock: connect: permission denied
方式一:在 docker.service 中添加权限修改命令 ExecStartPost=/usr/bin/chmod 666 /var/run/docker.sock
...
[Service]
...
ExecStart=/usr/bin/dockerd
ExecStartPost=/usr/bin/chmod 666 /var/run/docker.sock
ExecReload=/bin/kill -s HUP $MAINPID
...
方式二:给容器内的 jenkins 用户增加 id=128 的组权限
启动 jenkins 容器之后进去可以看到 jenkins 的 id 为 128:
$ cat /etc/group | grep docker
[sudo] password for fly:
docker:x:128:fly
# 先移除旧容器
sudo docker rm -f jenkins
# 重新创建容器,启动参数添加:--group-add=128
sudo docker run -d -p 8080:8080 -p 50000:50000 \
-v /data2/jenkins/jenkins_home:/var/jenkins_home \
-v /etc/localtime:/etc/localtime:ro \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /etc/docker:/etc/docker \
-v /usr/bin/docker:/usr/bin/docker \
--restart=always \
--group-add=128 \
--name jenkins \
jenkins/jenkins:latest
方式三:使用 root 用户启动 jenkins
version: "3"
services:
jenkins:
...
user: root
privileged: true
3. jenkinsfile 中直接调用 docker
pipeline {
...
stages {
...
stage(' docker build and push') {
steps {
sh '''
docker build -f ./.cicd/Dockerfile -t ${HARBOR_URL}/${HARBOR_REPO}/${PROJECT_NAME}:${IMAGE_TAG}
docker login -u ${HARBOR_USER} -p ${HARBOR_PASSWORD} ${HARBOR_URL}
docker push ${HARBOR_URL}/${HARBOR_REPO}/${PROJECT_NAME}:${IMAGE_TAG}
docker rmi ${HARBOR_URL}/${HARBOR_REPO}/${PROJECT_NAME}:${IMAGE_TAG}
'''
}
}
...
}
}
解决方案二:调用远程 docker 服务
找一台机器安装好 docker 环境的服务器,可以是本机也可以是其他机器,开启 Remote API 访问端口,这样就能被远程调用了
$ cat /etc/docker/daemon.json
{
...
"hosts": [
"tcp://0.0.0.0:2375",
"unix:///var/run/docker.sock"
]
}
$ systemctl restart docker
新版可能是需要修改 service 文件:修改 ExecStart 行,增加内容 -H tcp://0.0.0.0:2375
$ cat /usr/lib/systemd/system/docker.service
...
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
# 注意需要增加-H tcp://0.0.0.0:2375,原来的-H 不能去掉
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H fd:// --containerd=/run/containerd/containerd.sock
#ExecStartPost=/usr/bin/chmod 666 /var/run/docker.sock
...
$ systemctl daemon-reload
$ systemctl restart docker
测试是否可用
$ docker -H tcp://127.0.0.1:2375 version
Client: Docker Engine - Community
Version: 26.1.2
API version: 1.45
Go version: go1.21.10
Git commit: 211e74b
Built: Wed May 8 14:01:00 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 26.1.2
API version: 1.45 (minimum version 1.24)
Go version: go1.21.10
Git commit: ef1912d
Built: Wed May 8 13:59:54 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.32
GitCommit: 8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
jenkinsfile 中引用,所有命令前需要加上-H tcp://xx.xx.xx.xx:2375
pipeline {
...
stages {
...
stage('docker build&push'){
steps {
sh '''
docker -H tcp://192.168.31.79:2375 build -f ./.cicd/Dockerfile -t ${HARBOR_URL}/${HARBOR_REPO}/${PROJECT_NAME}:${IMAGE_TAG}
docker -H tcp://192.168.31.79:2375 login -u ${HARBOR_USER} -p ${HARBOR_PASSWORD} ${HARBOR_URL}
docker -H tcp://192.168.31.79:2375 push ${HARBOR_URL}/${HARBOR_REPO}/${PROJECT_NAME}:${IMAGE_TAG}
docker -H tcp://192.168.31.79:2375 rmi ${HARBOR_URL}/${HARBOR_REPO}/${PROJECT_NAME}:${IMAGE_TAG}
'''
}
}
...
}
}
解决方案三:使用 docker/k8s 插件加 buildah
另外还可以使用 buildah 来构建镜像,因为它不依赖 docker 等容器运行时,支持 dockerfile,build 出来的镜像也支持 docker 和其他镜像运行时使用。这种方式其实不能说是解决 docker-in-docker 的一种方式,因为本身 jenkins 就需要 docker 环境了,这种适合 jenkins 本身就在宿主机运行又不想使用 docker 的场景。
项目的 Dockerfile 中使用多阶段构建写法,示例
FROM node:20.16.0-alpine3.20 AS builder
USER root
ARG ENV_VAR
WORKDIR /src
COPY . .
RUN node --version \
&& npm -v \
&& cat /etc/os-release \
&& rm -f yarn.lock\
&& rm -f package-lock.json \
&& npm cache clean --force \
&& npm cache verify \
&& npm config get proxy \
&& npm config get https-proxy \
&& npm config rm proxy \
&& npm config rm https-proxy \
&& unset http_proxy \
&& unset https_proxy \
&& unset ftp_proxy \
&& unset no_proxy \
&& npm config set cache "~/.npm" \
&& npm config set prefix "~/.npm" \
&& npm config set strict-ssl false \
&& cat ~/.npmrc \
&& npm config set registry https://registry.npmmirror.com \
&& npm install --unsafe-perm=true --loglevel=verbose --allow-root \
&& npm run build:${ENV_VAR}
FROM nginx:1.24.0-alpine AS final
USER root
ADD .cicd/ca.crt /etc/pki/ca-trust/source/anchors/
RUN update-ca-trust
COPY .cicd/nginx.conf /etc/nginx/nginx.conf
WORKDIR /data/frontend
COPY --from=builder /src/dist .
EXPOSE 8081
CMD ["nginx", "-g", "daemon off;"]
在 jenkins master 上的 jenkins slave pod 中添加 buildah 的 container
kubernetes
1)安装 kubernetes 插件
2)在 cloud 中添加 buildah 的 container template
docker
1) 把/var/run/docker.sock /usr/bin/docker 挂载到 jenkins 容器中
version: "3"
services:
jenkins:
...
volumes:
- $PWD/jenkins/data:/var/jenkins_home
- $PWD/jenkins/.ssh:/var/jenkins_home/.ssh
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/var/run/docker.sock
- /usr/bin/docker:/usr/bin/docker
- /etc/docker:/etc/docker
...
2)安装 docker 插件
在 jenkins pipeline 中使用 buildah 构建镜像,只需要指定多阶段构建的 Dockerfile 即可,不用再增加而外的编译打包等 step。
k8s
pipeline {
agent {label 'jenkins-slave'}
...
stages {
stage('拉取代码') {
steps {
container('git'){
script {
checkout scmGit(branches: [[name: "${git_branch}"]], extensions: [], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_url}" ]])
}
}
}
}
stage('build&push image'){
steps {
container('buildah'){
sh '''
pwd
ls
buildah --storage-driver=vfs bud --format=oci --tls-verify=false --no-cache -f ./.cicd/Dockerfile -t ${REGISTRY_URL}/${PROJECT_NAME}/${IMAGE_NAME}:${IMAGE_TAG} .
buildah images
buildah --storage-driver=vfs push --tls-verify=false ${REGISTRY_URL}/${PROJECT_NAME}/${IMAGE_NAME}:${IMAGE_TAG} docker://${REGISTRY_URL}/${PROJECT_NAME}/${IMAGE_NAME}:${IMAGE_TAG}
'''
}
}
}
...
}
}
docker
pipeline {
agent any
...
stages {
stage('拉取代码') {
//注意:这里不加 args '--entrypoint=\'\'' 会报错:Alternatively you can force image entrypoint to be disabled by adding option `--entrypoint=''`.
agent {
docker {
image 'bitnami/git:2.43.2'
args "--privileged"
args '--entrypoint=\'\''
}
}
steps {
script {
//检出代码
checkout scmGit(branches: [[name: "${GIT_BRANCH_OR_TAG}"]], extensions: [], userRemoteConfigs: [[credentialsId: "${GIT_AUTH}", url: "${GIT_URL}" ]])
//组装 IMAGE_TAG
sh '''
BRANCH_NAME=$(cd ${WORKSPACE} && git name-rev --name-only HEAD|awk -F"/" '{print $NF}'|tr 'A-Z' 'a-z')
DATE_TIME=$(date '+%Y%m%d%H%M%S'|sed 's/[[:space:]]//g')
COMMIT_ID=$(git log -n 1 --pretty=format:'%h'|sed 's/[[:space:]]//g')
export IMAGE_TAG=${BRANCH_NAME}-${DATE_TIME}-${COMMIT_ID}
echo ${IMAGE_TAG} > ${WORKSPACE}/imagetag.txt
'''
}
}
}
stage('build&push image'){
agent {
docker {
image 'quay.io/buildah/stable:v1.35.4'
args '--privileged'
}
}
steps {
script {
withCredentials([usernamePassword(credentialsId: "${env.HARBOR_CREDENTIALS_ID}", usernameVariable: 'HARBOR_USER', passwordVariable: 'HARBOR_PASS')]) {
sh '''
pwd
ls -la
export IMAGE_TAG=$(cat ${WORKSPACE}/imagetag.txt)
echo ${IMAGE_TAG}
buildah login -u ${HARBOR_USER} -p ${HARBOR_PASS} ${DOCKER_REGISTRY}
buildah --storage-driver=vfs build --build-arg ENV_VAR=prod --format=oci --tls-verify=false --no-cache -f ${DOCKERFILE_PATH} -t ${DOCKER_REGISTRY}/${PROJECT_NAME}/${IMAGE_NAME}:${IMAGE_TAG} .
buildah images
buildah --storage-driver=vfs push --tls-verify=false ${DOCKER_REGISTRY}/${PROJECT_NAME}/${IMAGE_NAME}:${IMAGE_TAG}
buildah rmi ${DOCKER_REGISTRY}/${PROJECT_NAME}/${IMAGE_NAME}:${IMAGE_TAG}
'''
}
}
}
}
...
}
...
}