基于 Kubernetes 1.17.16 搭建 Tekton 动态构建环境
前言
动态构建是 DevOps 中至关重要的一环，它可以显著提高团队的工作效率，减少重复劳动，节省时间和成本。
上次我们聊到了如何使用 Kubernetes 和 Jenkins 实现动态构建(基于 Kubernetes 1.17.16 搭建 Jenkins 2.253 动态构建环境)。
这次则聚焦于利用 Kubernetes 和 Tekton 来实现同样的目标。
在我的工作环境中，由于公司现有的系统设置，Jenkins 和 Tekton 被配置为协同合作的方式。
Kubernetes 是一个强大且灵活的容器编排管理系统，宛如一位精通调度的管家，可以动态管理容器资源。
Tekton 是一种现代化的开源持续集成与持续交付（CI/CD）框架，专为在 Kubernetes 环境中执行构建、测试和发布任务而设计。
结合 Kubernetes 和 Tekton，我们可以构建出更加动态化的 CI/CD 流水线，以更好地适应复杂的业务需求。
本教程将演示如何通过 Kubernetes 提供的容器资源，在其上动态运行 Tekton Task，以实现自动化动态构建。
接下来，我们将逐步完成以下步骤：
- 部署
Jenkins：用于管理构建流水线的触发与调度。请注意，NFS服务器的部署步骤在此省略，只需确保集群内部能够访问即可。本教程基于公司内部环境的特定配置，建议读者根据自己的实际情况进行调整，不建议直接复用。 - 部署
Tekton：包括Tekton Pipelines和Tekton Dashboard的安装和配置。 - 配置
Tekton Task：定义具体的构建和发布任务。 - 修改
Jenkinsfile：使其能够调用Tekton进行任务执行。 - 测试构建流程：进行一个简单的构建流程测试，以验证动态构建链条的有效性。
通过这些步骤，您将看到如何利用 Jenkins 和 Tekton 协作，在 Kubernetes 环境中实现灵活高效的动态构建流水线。
部署 Jenkins
NFS 服务器的部署省略，只需要集群内能访问即可。
值得注意的是，我这一套是基于我们公司具体情况定制化的，请勿直接使用，可以拿来参考参考。
部署 k8s-nfs 客户端
先做 RBAC 和认证准备，命名为 01-nfs-rbac.yaml。
# 创建一个名为nfs-kube-ops的ServiceAccount在kube-ops命名空间中。
# 这会被NFS客户端使用。
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-kube-ops
namespace: kube-ops
---
# 定义一个ClusterRole，授予NFS客户端需要的权限，包括创建/删除PersistentVolumes，更新PersistentVolumeClaims等。
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-kube-ops-runner
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "watch", "list"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get","list","patch","watch","create","delete"]
---
# 绑定ServiceAccount到ClusterRole。
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-kube-ops
subjects:
- kind: ServiceAccount
name: nfs-kube-ops
namespace: kube-ops
roleRef:
kind: ClusterRole
name: nfs-kube-ops-runner
apiGroup: rbac.authorization.k8s.io
---
# 定义一个Role，授予修改Endpoints的权限，用于Leader选举。
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-kube-ops
namespace: kube-ops
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
# 绑定Role到ServiceAccount。
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-kube-ops
namespace: kube-ops
subjects:
- kind: ServiceAccount
name: nfs-kube-ops
namespace: kube-ops
roleRef:
kind: Role
name: leader-locking-nfs-kube-ops
apiGroup: rbac.authorization.k8s.io
创建 StorageClass 对象来管理存储资源，分别为 Jenkins Home 和 Jenkins Repo，命名为 02-jenkins-home-class.yaml 和 03-jenkins-repo-class.yaml，也可以合并一起。
# 02-jenkins-home-class.yaml
# nfs-jenkins-home对应的是nfs deployment中的PROVISIONER_NAME变量名
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-jenkins-home
provisioner: nfs-jenkins-home
allowVolumeExpansion: true
parameters:
archiveOnDelete: "true"
# 03-jenkins-repo-class.yaml
# nfs-jenkins-repo对应的是nfs deployment中的PROVISIONER_NAME变量名
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-jenkins-repo
provisioner: nfs-jenkins-repo
allowVolumeExpansion: true
parameters:
archiveOnDelete: "true"
动态卷供应器（Provisioner）用于实现存储卷的动态创建和管理。定义 Jenkins Home 和 Mvn Repo 的 Deployment 文件，命名为 04-jenkins-home-deployment.yaml 和 05-jenkins-repo-deployment.yaml。
为什么需要 NFS 动态卷供应器？
动态卷供应器(Provisioner)之所以需要被创建和部署，是因为它实现了存储卷的创建和管理逻辑。
在 Kubernetes 中，StorageClass 只是描述存储配置的一个资源对象，本身不具备动态创建存储卷的能力。
简单来说，StorageClass 只是声明存储配置和类信息，而 NFS 动态卷供应器实现了实际的存储操作和供应逻辑。
# 04-jenkins-home-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-jenkins-home
labels:
app: nfs-jenkins-home
namespace: kube-ops
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-jenkins-home
template:
metadata:
labels:
app: nfs-jenkins-home
spec:
serviceAccountName: nfs-kube-ops
containers:
- name: nfs-jenkins-home
image: dyrnq/nfs-subdir-external-provisioner:v4.0.2
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
# 这里的nfs-jenkins-home对应的是nfs StorageClass中的nfs-jenkins-home
- name: PROVISIONER_NAME
value: nfs-jenkins-home
- name: NFS_SERVER
value: 10.200.0.144
- name: NFS_PATH
value: /data/k8s_jenkins/jenkins
volumes:
- name: nfs-client-root
nfs:
server: 10.200.0.144
path: /data/k8s_jenkins/jenkins
# 05-jenkins-repo-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-jenkins-repo
labels:
app: nfs-jenkins-repo
namespace: kube-ops
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-jenkins-repo
template:
metadata:
labels:
app: nfs-jenkins-repo
spec:
serviceAccountName: nfs-kube-ops
containers:
- name: nfs-jenkins-repo
image: dyrnq/nfs-subdir-external-provisioner:v4.0.2
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
# 这里的nfs-jenkins-repo对应的是nfs StorageClass中的nfs-jenkins-repo
- name: PROVISIONER_NAME
value: nfs-jenkins-repo
- name: NFS_SERVER
value: 10.200.0.144
- name: NFS_PATH
value: /data/k8s_jenkins/repo
volumes:
- name: nfs-client-root
nfs:
server: 10.200.0.144
path: /data/k8s_jenkins/repo
部署 Jenkins 节点
创建集群管理员用户(也可适当限制下权限)，命名为 06-jenkins-account.yaml。
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: jenkins-default-view-kube-ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: view
subjects:
- kind: ServiceAccount
name: default
namespace: kube-ops
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: jenkins-master
name: jenkins-admin
namespace: kube-ops
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins-admin
labels:
k8s-app: jenkins-master
subjects:
- kind: ServiceAccount
name: jenkins-admin
namespace: kube-ops
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
创建 Jenkins 的 PVC 卷，命名为 07-jenkins-home-pvc.yaml 和 08-jenkins-repo-pvc.yaml。
# 07-jenkins-home-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nfs-jenkins-home
namespace: kube-ops
annotations:
nfs.io/storage-path: "nfs-jenkins-home"
spec:
storageClassName: nfs-jenkins-home
accessModes:
- ReadWriteMany
resources:
requests:
storage: 500Gi
# 08-jenkins-repo-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nfs-jenkins-repo
namespace: kube-ops
annotations:
nfs.io/storage-path: "nfs-jenkins-repo"
spec:
storageClassName: nfs-jenkins-repo
accessModes:
- ReadWriteMany
resources:
requests:
storage: 500Gi
根据实际情况构建 Jenkins 镜像，以下是我的 Dockerfile，仅供参考。
我这里构建的镜像名为：10.200.0.143:80/wenwo/devops/aw_k8s_jenkins:v3。
Jenkins 镜像其实可以更简单，我这里装那么多玩意只是为了更通用，方便之后扩展。
FROM jenkins/jenkins:2.346.3-2-lts-jdk8
LABEL maintainer="runfa.li"
# 切换到 root 用户
USER root
SHELL ["/bin/bash", "-c"]
# 复制必要的文件
COPY sources.list .
COPY maven.tar.gz .
COPY jdk8.tar.gz .
COPY kubectl .
# 更新源、安装必要的包、安装 Maven 和 JDK8，清理缓存
RUN mv -f sources.list /etc/apt/sources.list && \
echo 'Acquire::http::Pipeline-Depth "0";' > /etc/apt/apt.conf.d/99nopipelining && \
apt-get update && \
apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
gnupg \
zip \
unzip \
gconf-service \
libxext6 \
libxfixes3 \
libxi6 \
libxrandr2 \
libxrender1 \
libcairo2 \
libcups2 \
libdbus-1-3 \
libexpat1 \
libfontconfig1 \
libgcc1 \
libgconf-2-4 \
libgdk-pixbuf2.0-0 \
libglib2.0-0 \
libgtk-3-0 \
libnspr4 \
libpango-1.0-0 \
libpangocairo-1.0-0 \
libstdc++6 \
libx11-6 \
libx11-xcb1 \
libxcb1 \
libxcomposite1 \
libxcursor1 \
libxdamage1 \
libxss1 \
libxtst6 \
libappindicator1 \
libnss3 \
libasound2 \
libatk1.0-0 \
libc6 \
fonts-liberation \
lsb-release \
xdg-utils \
wget \
git \
jq && \
# 安装 Docker CE
install -m 0755 -d /etc/apt/keyrings && \
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/debian/gpg | \
gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/debian \
$(. /etc/os-release && echo \"$VERSION_CODENAME\") stable" \
| tee /etc/apt/sources.list.d/docker.list > /dev/null && \
apt-get update && \
apt-get install -y docker-ce && \
# 安装 Maven
tar -xvf maven.tar.gz && \
mkdir -p /data && \
mv maven /data && \
rm -f maven.tar.gz && \
chmod 755 -R /data/maven/bin/ && \
ln -sf /data/maven/bin/mvn /usr/bin/mvn && \
# 安装 JDK8
tar -xvf jdk8.tar.gz && \
mv jdk8 /usr/local && \
rm -f jdk8.tar.gz && \
chmod 755 -R /usr/local/jdk8/bin/ && \
# 配置kubectl
mv kubectl /usr/bin/kubectl && \
chmod 755 /usr/bin/kubectl && \
# 清理 APT 缓存和临时文件
apt-get clean && \
rm -rf /var/lib/apt/lists/*
# 设置环境变量
ENV JAVA_HOME="/usr/local/jdk8"
ENV JRE_HOME="$JAVA_HOME/jre"
ENV MAVEN_HOME="/data/maven"
ENV CLASSPATH=".:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib"
ENV PATH="$JAVA_HOME/bin:$JRE_HOME/bin:$MAVEN_HOME/bin:/usr/local/bin:$PATH"
# 设置代理环境变量（仅在构建时）
ARG http_proxy
ARG https_proxy
# 安装 n 版本管理器、Node.js 版本，配置 cnpm，设置默认 Node.js 版本
RUN curl -L https://raw.githubusercontent.com/tj/n/master/bin/n -o /usr/local/bin/n && \
chmod +x /usr/local/bin/n && \
# 安装 Node.js 版本
n 14.18.0 && \
n 16.14.2 && \
n 16.20.2 && \
n 18.16.0 && \
# 切换到每个版本，安装 cnpm
for version in 14.18.0 16.14.2 16.20.2 18.16.0; do \
n $version && \
echo "使用 Node.js 版本: $(node -v)" && \
npm -v && \
npm config set registry http://mirrors.cloud.tencent.com/npm/ && \
npm install -g cnpm && \
echo "cnpm 版本: $(cnpm -v)"; \
done && \
# 清除代理环境变量
unset http_proxy https_proxy && \
# 设置默认的 Node.js 版本（可根据需要修改）
n 14.18.0
# 验证默认的 Node.js 版本
RUN npm install -g pnpm@6.23.2 && node -v && npm -v
编写 Jenkins Deployment 的 yaml 文件，命名为 09-jenkins-deployment.yaml。
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins-master
namespace: kube-ops
spec:
selector:
matchLabels:
app: jenkins-master
template:
metadata:
labels:
app: jenkins-master
spec:
imagePullSecrets:
- name: aw-registrykey
terminationGracePeriodSeconds: 10
serviceAccount: jenkins-admin
containers:
- name: jenkins-master
image: 10.200.0.143:80/wenwo/devops/aw_k8s_jenkins:v3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
resources:
limits:
memory: 16Gi
requests:
memory: 1Gi
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
volumeMounts:
- name: jenkinshome
subPath: jenkins-master
mountPath: /var/jenkins_home
- name: jenkinsrepo
subPath: jenkins-master
mountPath: /data/repo
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: TZ
value: Hongkong
- name: JAVA_OPTS
value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
securityContext:
fsGroup: 1000
volumes:
- name: jenkinshome
persistentVolumeClaim:
claimName: nfs-jenkins-home
- name: jenkinsrepo
persistentVolumeClaim:
claimName: nfs-jenkins-repo
编写 Jenkins Services 的 yaml 文件，命名为 10-jenkins-services.yaml。
apiVersion: v1
kind: Service
metadata:
name: jenkins-master
namespace: kube-ops
labels:
app: jenkins-master
spec:
selector:
app: jenkins-master
# 这里的作用类似端口映射，这是个公司内网IP地址
# 这样配置之后公司内网可以直接访问http://10.200.1.146打开jenkins
externalIPs:
- 10.200.1.146
ports:
- name: web
port: 80
targetPort: 8080
- name: agent
port: 50000
targetPort: 50000
按顺序执行 yaml 文件
kubectl apply -f 01-nfs-rbac.yaml
kubectl apply -f 02-jenkins-home-class.yaml
kubectl apply -f 03-jenkins-repo-class.yaml
kubectl apply -f 04-jenkins-home-deployment.yaml
kubectl apply -f 05-jenkins-repo-deployment.yaml
kubectl apply -f 06-jenkins-account.yaml
kubectl apply -f 07-jenkins-home-pvc.yaml
kubectl apply -f 08-jenkins-repo-pvc.yaml
kubectl apply -f 09-jenkins-deployment.yaml
kubectl apply -f 10-jenkins-services.yaml
安装需要的插件(比如 Git，Kubernetes，Pipeline 等)
我这里因为是需要把老 Jenkins 迁移到 K8S，所以是直接把相关目录还原到了 NFS 机器上的 /data/k8s_jenkins/jenkins，记得还原前需要先停止 Jenkins Deployment。
部分截图展示
部署 Tekton
部署 Tekton
创建命名空间 yaml 文件，命名为：01-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: tekton-pipelines
labels:
app.kubernetes.io/instance: default
app.kubernetes.io/part-of: tekton-pipelines
准备 Tekton 的 yaml 文件，命名为：02-tekton-pipelines.yaml
因内容过多，此处不展示，我这使用的是 v0.23.0 版本。
下载地址为：[[https://github.com/tektoncd/pipeline/releases/download/v0.23.0/release.yaml]]
由于每个 TaskRun 启动时都会使用 02-tekton-pipelines.yaml 中配置的 Base 镜像进行初始化，而默认镜像环境较为简单，因此我们做了一些调整。
如果你也对镜像进行了调整，请记得修改 02-tekton-pipelines.yaml 中的 base 镜像名称。
Base 镜像 Dockerfile 如下：
FROM alpine:latest
USER root
# 使用清华大学的 alpine 镜像源
RUN sed -i 's#https\?://dl-cdn.alpinelinux.org/alpine#https://mirrors4.tuna.tsinghua.edu.cn/alpine#g' /etc/apk/repositories && \
apk update && \
# 安装必要的工具
apk add --no-cache bash git curl jq libxml2-utils ca-certificates wget make vim busybox-extras ttf-dejavu fontconfig && \
# 清理缓存
rm -rf /var/cache/apk/*
准备 Tekton Dashboard 的 yaml 文件，命名为：03-tekton-dashboard.yaml
因内容过多，此处不展示，我这使用的是 v0.21.0 版本。
下载地址为：[[https://github.com/tektoncd/dashboard/releases/download/v0.21.0/tekton-dashboard-release.yaml]]
准备 Tekton Dashboard Ingress 的 yaml 文件，命名为：04-ingress.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: tekton-pipelines
namespace: tekton-pipelines
spec:
entryPoints:
- web
routes:
- kind: Rule
match: host("tekton.wenwo.cn")
services:
- name: tekton-dashboard
port: 9097
按顺序执行 yaml 文件
kubectl apply -f 01-namespace.yaml
kubectl apply -f 02-tekton-pipelines.yaml
kubectl apply -f 03-tekton-dashboard.yaml
kubectl apply -f 04-ingress.yaml
准备任务执行镜像
为适应公司构建环境，我这里自定义了合适的 TaskRun 镜像。
命名为：10.200.0.143:80/wenwo/devops/tekton-pipeline/aw-alpine:v1
Dockerfile 如下：
FROM alpine:latest
USER root
COPY jdk8.tar.gz ./
COPY maven.tar.gz ./
COPY daemon.json ./
RUN sed -i 's#https\?://dl-cdn.alpinelinux.org/alpine#https://mirrors4.tuna.tsinghua.edu.cn/alpine#g' /etc/apk/repositories && \
apk update && \
apk add --no-cache bash git curl jq libxml2-utils docker openrc ca-certificates wget bash make vim busybox-extras ttf-dejavu fontconfig && \
mkdir -p /etc/docker/ && \
mv daemon.json /etc/docker/daemon.json && \
tar -xvf jdk8.tar.gz && \
mv jdk8 /usr/local/ && \
rm -f jdk8.tar.gz && \
chmod 755 -R /usr/local/jdk8/bin/ && \
tar -xvf maven.tar.gz && \
mkdir -p /data && \
mv maven /data/ && \
rm -f maven.tar.gz && \
chmod 755 -R /data/maven/bin/ && \
ln -sf /data/maven/bin/mvn /usr/bin/mvn && \
rc-update add docker default && \
mkdir -p /opt/glibc && \
cd /opt/glibc && \
wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.30-r0/glibc-2.30-r0.apk && \
apk add glibc-2.30-r0.apk --allow-untrusted --force-overwrite && \
rm -rf *.apk && \
rm -rf /var/cache/apk/*
ENV JAVA_HOME=/usr/local/jdk8
ENV MAVEN_HOME=/data/maven
ENV PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH
RUN java -version && mvn -version
准备 Task 文件
根据公司实际情况，我这里的 Task 内容仅供参考，文件命名为：java-task.yaml。
具体如下：
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: java-task
namespace: tekton-pipelines
spec:
params:
- name: BRANCH
type: string
- name: GIT_BRANCH
type: string
- name: GIT_URL_CLEAN
type: string
- name: REGISTRY
type: string
- name: BUILD_ID
type: string
- name: JOB_NAME
type: string
- name: BUILD_CMD
type: string
- name: SONAR_CMD
type: string
- name: VERSION
type: string
- name: COMMITSHA1_10
type: string
- name: BUILD_URL
type: string
- name: GROUPID
type: string
- name: ARTIFACTID
type: string
- name: SONARSERVER
type: string
- name: IMAGENAME
type: string
steps:
- name: pull-code
image: 10.200.0.143:80/wenwo/devops/tekton-pipeline/aw-alpine:v1
env:
- name: GIT_URL_CLEAN
value: "$(params.GIT_URL_CLEAN)"
- name: GIT_BRANCH
value: "$(params.GIT_BRANCH)"
script: |
echo "从 ${GIT_URL_CLEAN} 拉取 ${GIT_BRANCH} 分支..."
set +x
GIT_USERNAME=$(cat /var/secrets/git-credentials/username)
GIT_PASSWORD=$(cat /var/secrets/git-credentials/password)
git clone http://${GIT_USERNAME}:$(echo ${GIT_PASSWORD} | sed 's/@/%40/')@$(echo ${GIT_URL_CLEAN} | sed 's#http://##') -b ${GIT_BRANCH} /workspace/source
set -x
echo "${GIT_URL_CLEAN} 拉取 ${GIT_BRANCH} 分支完成！"
volumeMounts:
- name: git-credentials-volume
mountPath: /var/secrets/git-credentials
- name: shared-volume
mountPath: /workspace/source
- name: build-tasks
image: 10.200.0.143:80/wenwo/devops/tekton-pipeline/aw-alpine:v1
workingDir: /workspace/source
env:
- name: REGISTRY
value: "$(params.REGISTRY)"
- name: BUILD_ID
value: "$(params.BUILD_ID)"
- name: JOB_NAME
value: "$(params.JOB_NAME)"
- name: GIT_BRANCH
value: "$(params.GIT_BRANCH)"
- name: BUILD_CMD
value: "$(params.BUILD_CMD)"
- name: SONAR_CMD
value: "$(params.SONAR_CMD)"
- name: BUILD_URL
value: "$(params.BUILD_URL)"
script: |
echo "开始构建项目..."
set +x
DOCKER_USERNAME=$(cat /var/secrets/aw-credentials/username)
DOCKER_PASSWORD=$(cat /var/secrets/aw-credentials/password)
echo ${DOCKER_PASSWORD} | docker login ${REGISTRY} -u ${DOCKER_USERNAME} --password-stdin
set -x
curl -i -X POST -H "Content-type: application/json" -d "{\"buildid\":\"${BUILD_ID}\",\"jenkinsjobname\":\"${JOB_NAME}\",\"buildurl\":\"${BUILD_URL}\",\"branchname\":\"${GIT_BRANCH}\",\"images\":\"\",\"status\":\"构建中\",\"steps\":\"start\"}" http://10.200.1.172:8080/project/jenkins/buildInfo
${BUILD_CMD}
echo "项目构建完成！"
echo "开始代码扫描..."
${SONAR_CMD}
echo "代码扫描完成！"
volumeMounts:
- name: nfs-volume
mountPath: /data/repo
- name: aw-credentials-volume
mountPath: /var/secrets/aw-credentials
- name: shared-volume
mountPath: /workspace/source
- name: docker-sock
mountPath: /var/run/docker.sock
- name: quality-inspection
image: 10.200.0.143:80/wenwo/devops/tekton-pipeline/aw-alpine:v1
workingDir: /workspace/source
env:
- name: GROUPID
value: "$(params.GROUPID)"
- name: ARTIFACTID
value: "$(params.ARTIFACTID)"
- name: SONARSERVER
value: "$(params.SONARSERVER)"
script: |
if [ -z "$SONARSERVER" ]; then
echo "该项目不做质量检测！"
exit 0
fi
sleep 10
echo "开始质量检测..."
response=$(curl -s -k -X GET "${SONARSERVER}/project_branches/list?project=${GROUPID}:${ARTIFACTID}")
status=$(echo $response | jq -r '.branches[0].status.qualityGateStatus')
echo "质量检测状态为：$status"
if [ "$status" != "OK" ]; then
echo "质量检测不通过！"
exit 1
fi
echo "质量检测通过！"
volumeMounts:
- name: shared-volume
mountPath: /workspace/source
- name: push-docker
image: 10.200.0.143:80/wenwo/devops/tekton-pipeline/aw-alpine:v1
workingDir: /workspace/source
env:
- name: REGISTRY
value: "$(params.REGISTRY)"
- name: BUILD_ID
value: "$(params.BUILD_ID)"
- name: GIT_BRANCH
value: "$(params.GIT_BRANCH)"
- name: JOB_NAME
value: "$(params.JOB_NAME)"
- name: BRANCH
value: "$(params.BRANCH)"
- name: VERSION
value: "$(params.VERSION)"
- name: COMMITSHA1_10
value: "$(params.COMMITSHA1_10)"
- name: BUILD_URL
value: "$(params.BUILD_URL)"
- name: IMAGENAME
value: "$(params.IMAGENAME)"
script: |
echo "开始上传镜像..."
set +x
DOCKER_USERNAME=$(cat /var/secrets/aw-credentials/username)
DOCKER_PASSWORD=$(cat /var/secrets/aw-credentials/password)
echo ${DOCKER_PASSWORD} | docker login ${REGISTRY} -u ${DOCKER_USERNAME} --password-stdin
set -x
docker tag ${IMAGENAME} ${REGISTRY}/wenwo/${JOB_NAME}:${VERSION}-${BRANCH}-${BUILD_ID}-${COMMITSHA1_10}
docker push ${REGISTRY}/wenwo/${JOB_NAME}:${VERSION}-${BRANCH}-${BUILD_ID}-${COMMITSHA1_10}
docker rmi ${IMAGENAME} ${REGISTRY}/wenwo/${JOB_NAME}:${VERSION}-${BRANCH}-${BUILD_ID}-${COMMITSHA1_10}
curl -i -X POST -H "Content-type: application/json" -d "{\"buildid\":\"${BUILD_ID}\",\"jenkinsjobname\":\"${JOB_NAME}\",\"buildurl\":\"${BUILD_URL}\",\"branchname\":\"${GIT_BRANCH}\",\"images\":\"${REGISTRY}/wenwo/${JOB_NAME}:${VERSION}-${BRANCH}-${BUILD_ID}-${COMMITSHA1_10}\",\"status\":\"构建成功\",\"steps\":\"end\"}" http://10.200.1.172:8080/project/jenkins/buildInfo
echo "镜像上传成功！"
volumeMounts:
- name: aw-credentials-volume
mountPath: /var/secrets/aw-credentials
- name: shared-volume
mountPath: /workspace/source
- name: docker-sock
mountPath: /var/run/docker.sock
volumes:
- name: nfs-volume
nfs:
server: 10.200.0.144
path: /data/k8s_jenkins/repo
- name: shared-volume
emptyDir:
medium: Memory
- name: git-credentials-volume
secret:
secretName: git-credentials
- name: aw-credentials-volume
secret:
secretName: aw-credentials
- name: docker-sock
hostPath:
path: /var/run/docker.sock
应用 Task 文件
kubectl apply -f java-task.yaml
部分截图展示
项目整改
根据具体情况，对项目做一些调整，如 pom.xml 文件，如 Jenkinsfile。
pom.xml 整改
# 把docker插件的设置部分修改为这样子
<configuration>
<imageName>java/${project.artifactId}:${env}</imageName>
<imageTags>
<imageTag>${env}</imageTag>
</imageTags>
<forceTags>true</forceTags>
</configuration>
# ${project.artifactId}参数是必须的
<artifactId>wenwo-cloud-basic-wechat-spider</artifactId>
Jenkinsfile 整改
- 整改后的
def REGISTRY = "10.200.0.143:80"
def BRANCH = params.BRANCH_NAME
def GIT_URL = scm.userRemoteConfigs
def BUILD_ID = env.BUILD_ID
def JOB_NAME = env.JOB_NAME
def BUILD_URL = env.BUILD_URL
pipeline {
agent any
environment {
KUBE_NAMESPACE = "tekton-pipelines" // Kubernetes 命名空间
JOB_NAME_CLEAN = JOB_NAME.replaceAll('_', '-')
}
stages {
stage('触发 Tekton TaskRun') {
steps {
script {
def POM = readMavenPom file: "pom.xml"
def VERSION = POM.version
def COMMIT = "${env.GIT_COMMIT}"
def COMMITSHA1_10 = sh label: "", returnStdout: true, script: "expr substr '${COMMIT}' 1 10"
COMMITSHA1_10=COMMITSHA1_10.replaceAll("\n","")
def GROUPID = POM.groupId
def ARTIFACTID = POM.artifactId
//def SONARSERVER = "http://10.200.0.144:9000/api"
def SONARSERVER = ""
def GIT_URL_CLEAN = GIT_URL[0].url
dir("${env.WORKSPACE}-${BRANCH}") {
def ENVNAME = BRANCH
if (BRANCH == "master") {
ENVNAME = "prd"
} else if (BRANCH == "develop") {
ENVNAME = "dev"
}
def BUILD_CMD = "mvn -B -U clean package -Dfile.encoding=UTF-8 -Dmaven.test.skip=true -Denv=${ENVNAME}"
def SONAR_CMD = "mvn sonar:sonar -Dsonar.host.url=http://10.200.0.144:9000 -Dsonar.login=1c0bc447ba1ebf80d0dd49e95fd081faaea71611 -Dsonar.exclusions=**/util/*,**/utils/* -Denv=${ENVNAME}"
def IMAGENAME = "java/${ARTIFACTID}:${ENVNAME}"
// 生成 Tekton TaskRun YAML，使用 git-credentials Secret
def taskRunYaml = """
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
name: ${JOB_NAME_CLEAN}-${BUILD_ID}
namespace: ${KUBE_NAMESPACE}
spec:
taskRef:
name: java-task
params:
- name: GIT_BRANCH
value: "${BRANCH}"
- name: BRANCH
value: "${ENVNAME}"
- name: GIT_URL_CLEAN
value: "${GIT_URL_CLEAN}"
- name: REGISTRY
value: "${REGISTRY}"
- name: BUILD_ID
value: "${BUILD_ID}"
- name: JOB_NAME
value: "${JOB_NAME}"
- name: BUILD_CMD
value: "${BUILD_CMD}"
- name: SONAR_CMD
value: "${SONAR_CMD}"
- name: VERSION
value: "${VERSION}"
- name: COMMITSHA1_10
value: "${COMMITSHA1_10}"
- name: BUILD_URL
value: "${BUILD_URL}"
- name: GROUPID
value: "${GROUPID}"
- name: ARTIFACTID
value: "${ARTIFACTID}"
- name: SONARSERVER
value: "${SONARSERVER}"
- name: IMAGENAME
value: "${IMAGENAME}"
podTemplate:
imagePullSecrets:
- name: aw-registrykey
"""
// 将 TaskRun YAML 写入到一个文件中
writeFile file: 'TaskRun.yaml', text: taskRunYaml
// 使用 kubectl 应用 TaskRun
sh 'kubectl apply -f TaskRun.yaml --namespace=${KUBE_NAMESPACE}'
sh 'echo "构建地址为：http://tekton.wenwo.cn/#/namespaces/tekton-pipelines/taskruns/${JOB_NAME_CLEAN}-${BUILD_ID}"'
}
}
}
}
stage('监控 Tekton TaskRun') {
steps {
script {
dir("${env.WORKSPACE}-${BRANCH}") {
def status = ""
while (status != "True" && status != "False") {
// 查询Tekton状态的逻辑
status = sh(script: 'kubectl get TaskRun ${JOB_NAME_CLEAN}-${BUILD_ID} --namespace=tekton-pipelines -o jsonpath="{.status.conditions[0].status}"', returnStdout: true).trim()
echo "当前状态: ${status}"
if (status == "True") {
echo "任务成功"
} else if (status == "False") {
error "任务失败"
} else {
sleep(5) // 每5秒检查一次
}
}
}
}
}
}
stage("清理过期任务") {
steps {
dir("${env.WORKSPACE}-${BRANCH}") {
sh '''
task_runs=$(kubectl get TaskRun -n tekton-pipelines --sort-by=.metadata.creationTimestamp -o json | jq -r '.items | .[].metadata.name' | grep "${JOB_NAME_CLEAN}")
total_count=$(echo "$task_runs" | wc -l)
if [ "$total_count" -gt 10 ]; then
to_delete=$(echo "$task_runs" | head -n -10)
for task_run in $to_delete; do
echo "删除 TaskRun: $task_run"
kubectl delete TaskRun "$task_run" -n tekton-pipelines
done
else
echo "TaskRun 数量未超过 10，当前数量: $total_count"
fi
'''
}
}
}
stage("清理空间") {
steps {
sh "ls -al"
deleteDir()
sh "ls -al"
}
}
}
post {
failure {
sh "curl -i -X POST -H \"'Content-type':'application/json'\" -d \'{\"buildid\":\"${env.BUILD_ID}\",\"jenkinsjobname\":\"${env.JOB_NAME}\",\"buildurl\":\"${env.BUILD_URL}\",\"branchname\":\"${env.GIT_BRANCH}\",\"images\":\"\",\"status\":\"构建失败\",\"steps\":\"end\"}\' http://10.200.1.172:8080/project/jenkins/buildInfo"
}
}
}
- 整改前的
pipeline {
agent any
stages {
stage("编译打包") {
steps {
script {
def projectBranch = env.GIT_BRANCH.split("/")[1]
def envName = projectBranch
if (projectBranch.contains("master")) {
envName = "prd"
} else if (projectBranch == "develop") {
envName = "dev"
}
sh "curl -i -X POST -H \"'Content-type':'application/json'\" -d \'{\"buildid\":\"${env.BUILD_ID}\",\"jenkinsjobname\":\"${env.JOB_NAME}\",\"buildurl\":\"${env.BUILD_URL}\",\"branchname\":\"${env.GIT_BRANCH}\",\"images\":\"\",\"status\":\"构建中\",\"steps\":\"start\"}\' http://10.200.1.172:8080/project/jenkins/buildInfo"
// 修改原有打包编译命令
// 注意：这里需要双引号，否则无法读取到 envName 变量
sh "/data/maven/bin/mvn -B -U clean package -Dfile.encoding=UTF-8 -Dmaven.test.skip=true -Denv=${envName}"
}
}
}
stage("代码扫描"){
steps {
dir(env.WORKSPACE){
script {
def projectBranch = env.GIT_BRANCH.split("/")[1]
def envName = projectBranch
if (projectBranch.contains("master")) {
envName = "prd"
} else if (projectBranch == "develop") {
envName = "dev"
}
sh "/data/maven/bin/mvn sonar:sonar -Dsonar.host.url=http://10.200.0.144:9000 -Dsonar.login=1c0bc447ba1ebf80d0dd49e95fd081faaea71611 -Dsonar.exclusions=**/util/*,**/utils/* -Denv=${envName}" //指定sonar的ip和token
}
}
}
}
stage("质量检测"){
steps {
script{
sh 'sleep 10'
def projectName = env.JOB_NAME
def pom = readMavenPom file: 'pom.xml'
def GroupId = pom.groupId
def ArtifactId = pom.artifactId
def sonarServer = "http://10.200.0.144:9000/api"
def response = httpRequest httpMode: 'GET',
contentType: "APPLICATION_JSON",
consoleLogResponseBody: true,
ignoreSslErrors: true,
requestBody: '',
url: "${sonarServer}/project_branches/list?project=${GroupId}:${ArtifactId}"
def props = readJSON text: response.content
status = props["branches"][0]["status"]["qualityGateStatus"]
echo status
if (status != 'OK'){
sh 'exit 1'
}
}
}
}
stage("上传镜像") {
steps {
script {
def projectBranch = env.GIT_BRANCH.split("/")[1]
def projectName = env.JOB_NAME
def pom = readMavenPom file: 'pom.xml'
def version = pom.version
def contentRegex = "/.{0,10}/"
def commit = "${env.GIT_COMMIT}"
def commitSha1_10 = sh label: "", returnStdout: true, script: "expr substr '${commit}' 1 10"
commitSha1_10=commitSha1_10.replaceAll("\n","")
sh "docker tag java/${projectName}:${version} 10.200.0.143:80/wenwo/${projectName}:${version}-${projectBranch}-${env.BUILD_ID}-${commitSha1_10}"
sh "docker push 10.200.0.143:80/wenwo/${projectName}:${version}-${projectBranch}-${env.BUILD_ID}-${commitSha1_10}"
sh "docker rmi java/${projectName}:${version} java/${projectName}:latest 10.200.0.143:80/wenwo/${projectName}:${version}-${projectBranch}-${env.BUILD_ID}-${commitSha1_10}"
sh "curl -i -X POST -H \"'Content-type':'application/json'\" -d \'{\"buildid\":\"${env.BUILD_ID}\",\"jenkinsjobname\":\"${env.JOB_NAME}\",\"buildurl\":\"${env.BUILD_URL}\",\"branchname\":\"${env.GIT_BRANCH}\",\"images\":\"10.200.0.143:80/wenwo/${projectName}:${version}-${projectBranch}-${env.BUILD_ID}-${commitSha1_10}\",\"status\":\"构建成功\",\"steps\":\"end\"}\' http://10.200.1.172:8080/project/jenkins/buildInfo"
}
}
}
stage("清理空间") {
steps {
sh "ls -al"
deleteDir()
sh "ls -al"
}
}
}
post {
failure {
sh "curl -i -X POST -H \"'Content-type':'application/json'\" -d \'{\"buildid\":\"${env.BUILD_ID}\",\"jenkinsjobname\":\"${env.JOB_NAME}\",\"buildurl\":\"${env.BUILD_URL}\",\"branchname\":\"${env.GIT_BRANCH}\",\"images\":\"\",\"status\":\"构建失败\",\"steps\":\"end\"}\' http://10.200.1.172:8080/project/jenkins/buildInfo"
}
}
}
变动还是挺大的，此处就不详述了。
测试构建
我这里用的是 Pipeline script from SCM 的方式。
点了构建后，看截图。
构建完成后
解释
由截图可以看出，此时当项目点击构建后，会根据项目 Jenkinsfile 配置创建一个 TaskRun Pod，构建完成之后，会只保留前10次的构建结果，这就完成了动态构建了。
后记
如果在部署过程中遇到问题欢迎留言讨论，后面有空我再补充一些调试技巧。
希望本文可以帮助到大家，感谢阅读！
本文链接：
/archives/tekton
版权声明：
本站所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自
Linux 小白鼠！
觉得文章不错，打赏一点吧，1分也是爱~
打赏
微信
支付宝
微信
支付宝