基于 Kubernetes 1.17.16 搭建 Tekton 动态构建环境

前言

动态构建是 DevOps 中至关重要的一环，它可以显著提高团队的工作效率，减少重复劳动，节省时间和成本。
上次我们聊到了如何使用 Kubernetes 和 Jenkins 实现动态构建(基于 Kubernetes 1.17.16 搭建 Jenkins 2.253 动态构建环境)。
这次则聚焦于利用 Kubernetes 和 Tekton 来实现同样的目标。
在我的工作环境中，由于公司现有的系统设置，Jenkins 和 Tekton 被配置为协同合作的方式。

Kubernetes 是一个强大且灵活的容器编排管理系统，宛如一位精通调度的管家，可以动态管理容器资源。
Tekton 是一种现代化的开源持续集成与持续交付（CI/CD）框架，专为在 Kubernetes 环境中执行构建、测试和发布任务而设计。
结合 Kubernetes 和 Tekton，我们可以构建出更加动态化的 CI/CD 流水线，以更好地适应复杂的业务需求。

本教程将演示如何通过 Kubernetes 提供的容器资源，在其上动态运行 Tekton Task，以实现自动化动态构建。

接下来，我们将逐步完成以下步骤：

部署 Jenkins：用于管理构建流水线的触发与调度。请注意，NFS 服务器的部署步骤在此省略，只需确保集群内部能够访问即可。本教程基于公司内部环境的特定配置，建议读者根据自己的实际情况进行调整，不建议直接复用。
部署 Tekton：包括 Tekton Pipelines 和 Tekton Dashboard 的安装和配置。
配置 Tekton Task：定义具体的构建和发布任务。
修改 Jenkinsfile：使其能够调用 Tekton 进行任务执行。
测试构建流程：进行一个简单的构建流程测试，以验证动态构建链条的有效性。

通过这些步骤，您将看到如何利用 Jenkins 和 Tekton 协作，在 Kubernetes 环境中实现灵活高效的动态构建流水线。

部署 `Jenkins`

NFS 服务器的部署省略，只需要集群内能访问即可。
值得注意的是，我这一套是基于我们公司具体情况定制化的，请勿直接使用，可以拿来参考参考。

部署 `k8s-nfs` 客户端

先做 RBAC 和认证准备，命名为 01-nfs-rbac.yaml。

# 创建一个名为nfs-kube-ops的ServiceAccount在kube-ops命名空间中。
# 这会被NFS客户端使用。
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-kube-ops
  namespace: kube-ops
---
# 定义一个ClusterRole，授予NFS客户端需要的权限，包括创建/删除PersistentVolumes，更新PersistentVolumeClaims等。
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-kube-ops-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get", "watch", "list"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get","list","patch","watch","create","delete"]
---
# 绑定ServiceAccount到ClusterRole。
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-kube-ops
subjects:
  - kind: ServiceAccount
    name: nfs-kube-ops
    namespace: kube-ops
roleRef:
  kind: ClusterRole
  name: nfs-kube-ops-runner
  apiGroup: rbac.authorization.k8s.io
---
# 定义一个Role，授予修改Endpoints的权限，用于Leader选举。
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-kube-ops
  namespace: kube-ops
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
# 绑定Role到ServiceAccount。
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-kube-ops
  namespace: kube-ops
subjects:
  - kind: ServiceAccount
    name: nfs-kube-ops
    namespace: kube-ops
roleRef:
  kind: Role
  name: leader-locking-nfs-kube-ops
  apiGroup: rbac.authorization.k8s.io

创建 StorageClass 对象来管理存储资源，分别为 Jenkins Home 和 Jenkins Repo，命名为 02-jenkins-home-class.yaml 和 03-jenkins-repo-class.yaml，也可以合并一起。

# 02-jenkins-home-class.yaml
# nfs-jenkins-home对应的是nfs deployment中的PROVISIONER_NAME变量名
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-jenkins-home
provisioner: nfs-jenkins-home
allowVolumeExpansion: true
parameters:
  archiveOnDelete: "true"

# 03-jenkins-repo-class.yaml
# nfs-jenkins-repo对应的是nfs deployment中的PROVISIONER_NAME变量名
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-jenkins-repo
provisioner: nfs-jenkins-repo
allowVolumeExpansion: true
parameters:
  archiveOnDelete: "true"

动态卷供应器（Provisioner）用于实现存储卷的动态创建和管理。定义 Jenkins Home 和 Mvn Repo 的 Deployment 文件，命名为 04-jenkins-home-deployment.yaml 和 05-jenkins-repo-deployment.yaml。

为什么需要 NFS 动态卷供应器？
动态卷供应器(Provisioner)之所以需要被创建和部署，是因为它实现了存储卷的创建和管理逻辑。
在 Kubernetes 中，StorageClass 只是描述存储配置的一个资源对象，本身不具备动态创建存储卷的能力。
简单来说，StorageClass 只是声明存储配置和类信息，而 NFS 动态卷供应器实现了实际的存储操作和供应逻辑。

# 04-jenkins-home-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-jenkins-home
  labels:
    app: nfs-jenkins-home
  namespace: kube-ops
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-jenkins-home
  template:
    metadata:
      labels:
        app: nfs-jenkins-home
    spec:
      serviceAccountName: nfs-kube-ops
      containers:
        - name: nfs-jenkins-home
          image: dyrnq/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            # 这里的nfs-jenkins-home对应的是nfs StorageClass中的nfs-jenkins-home
            - name: PROVISIONER_NAME
              value: nfs-jenkins-home
            - name: NFS_SERVER
              value: 10.200.0.144
            - name: NFS_PATH
              value: /data/k8s_jenkins/jenkins
      volumes:
        - name: nfs-client-root
          nfs:
            server: 10.200.0.144
            path: /data/k8s_jenkins/jenkins

# 05-jenkins-repo-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-jenkins-repo
  labels:
    app: nfs-jenkins-repo
  namespace: kube-ops
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-jenkins-repo
  template:
    metadata:
      labels:
        app: nfs-jenkins-repo
    spec:
      serviceAccountName: nfs-kube-ops
      containers:
        - name: nfs-jenkins-repo
          image: dyrnq/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            # 这里的nfs-jenkins-repo对应的是nfs StorageClass中的nfs-jenkins-repo
            - name: PROVISIONER_NAME
              value: nfs-jenkins-repo
            - name: NFS_SERVER
              value: 10.200.0.144
            - name: NFS_PATH
              value: /data/k8s_jenkins/repo
      volumes:
        - name: nfs-client-root
          nfs:
            server: 10.200.0.144
            path: /data/k8s_jenkins/repo

部署 `Jenkins` 节点

创建集群管理员用户(也可适当限制下权限)，命名为 06-jenkins-account.yaml。

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins-default-view-kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: ServiceAccount
    name: default
    namespace: kube-ops

---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: jenkins-master
  name: jenkins-admin
  namespace: kube-ops

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins-admin
  labels:
    k8s-app: jenkins-master
subjects:
  - kind: ServiceAccount
    name: jenkins-admin
    namespace: kube-ops
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

创建 Jenkins 的 PVC 卷，命名为 07-jenkins-home-pvc.yaml 和 08-jenkins-repo-pvc.yaml。

# 07-jenkins-home-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-jenkins-home
  namespace: kube-ops
  annotations:
    nfs.io/storage-path: "nfs-jenkins-home"
spec:
  storageClassName: nfs-jenkins-home
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Gi

# 08-jenkins-repo-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-jenkins-repo
  namespace: kube-ops
  annotations:
    nfs.io/storage-path: "nfs-jenkins-repo"
spec:
  storageClassName: nfs-jenkins-repo
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Gi

根据实际情况构建 Jenkins 镜像，以下是我的 Dockerfile，仅供参考。
我这里构建的镜像名为：10.200.0.143:80/wenwo/devops/aw_k8s_jenkins:v3。
Jenkins 镜像其实可以更简单，我这里装那么多玩意只是为了更通用，方便之后扩展。

FROM jenkins/jenkins:2.346.3-2-lts-jdk8
LABEL maintainer="runfa.li"

# 切换到 root 用户
USER root
SHELL ["/bin/bash", "-c"]

# 复制必要的文件
COPY sources.list .
COPY maven.tar.gz .
COPY jdk8.tar.gz .
COPY kubectl .

# 更新源、安装必要的包、安装 Maven 和 JDK8，清理缓存
RUN mv -f sources.list /etc/apt/sources.list && \
    echo 'Acquire::http::Pipeline-Depth "0";' > /etc/apt/apt.conf.d/99nopipelining && \
    apt-get update && \
    apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    zip \
    unzip \
    gconf-service \
    libxext6 \
    libxfixes3 \
    libxi6 \
    libxrandr2 \
    libxrender1 \
    libcairo2 \
    libcups2 \
    libdbus-1-3 \
    libexpat1 \
    libfontconfig1 \
    libgcc1 \
    libgconf-2-4 \
    libgdk-pixbuf2.0-0 \
    libglib2.0-0 \
    libgtk-3-0 \
    libnspr4 \
    libpango-1.0-0 \
    libpangocairo-1.0-0 \
    libstdc++6 \
    libx11-6 \
    libx11-xcb1 \
    libxcb1 \
    libxcomposite1 \
    libxcursor1 \
    libxdamage1 \
    libxss1 \
    libxtst6 \
    libappindicator1 \
    libnss3 \
    libasound2 \
    libatk1.0-0 \
    libc6 \
    fonts-liberation \
    lsb-release \
    xdg-utils \
    wget \
    git \
    jq && \
    # 安装 Docker CE
    install -m 0755 -d /etc/apt/keyrings && \
    curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/debian/gpg | \
    gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
    https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/debian \
    $(. /etc/os-release && echo \"$VERSION_CODENAME\") stable" \
    | tee /etc/apt/sources.list.d/docker.list > /dev/null && \
    apt-get update && \
    apt-get install -y docker-ce && \
    # 安装 Maven
    tar -xvf maven.tar.gz && \
    mkdir -p /data && \
    mv maven /data && \
    rm -f maven.tar.gz && \
    chmod 755 -R /data/maven/bin/ && \
    ln -sf /data/maven/bin/mvn /usr/bin/mvn && \
    # 安装 JDK8
    tar -xvf jdk8.tar.gz && \
    mv jdk8 /usr/local && \
    rm -f jdk8.tar.gz && \
    chmod 755 -R /usr/local/jdk8/bin/ && \
    # 配置kubectl
    mv kubectl /usr/bin/kubectl && \
    chmod 755 /usr/bin/kubectl && \
    # 清理 APT 缓存和临时文件
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# 设置环境变量
ENV JAVA_HOME="/usr/local/jdk8"
ENV JRE_HOME="$JAVA_HOME/jre"
ENV MAVEN_HOME="/data/maven"
ENV CLASSPATH=".:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib"
ENV PATH="$JAVA_HOME/bin:$JRE_HOME/bin:$MAVEN_HOME/bin:/usr/local/bin:$PATH"

# 设置代理环境变量（仅在构建时）
ARG http_proxy
ARG https_proxy

# 安装 n 版本管理器、Node.js 版本，配置 cnpm，设置默认 Node.js 版本
RUN curl -L https://raw.githubusercontent.com/tj/n/master/bin/n -o /usr/local/bin/n && \
    chmod +x /usr/local/bin/n && \
    # 安装 Node.js 版本
    n 14.18.0 && \
    n 16.14.2 && \
    n 16.20.2 && \
    n 18.16.0 && \
    # 切换到每个版本，安装 cnpm
    for version in 14.18.0 16.14.2 16.20.2 18.16.0; do \
    n $version && \
    echo "使用 Node.js 版本: $(node -v)" && \
    npm -v && \
    npm config set registry http://mirrors.cloud.tencent.com/npm/ && \
    npm install -g cnpm && \
    echo "cnpm 版本: $(cnpm -v)"; \
    done && \
    # 清除代理环境变量
    unset http_proxy https_proxy && \
    # 设置默认的 Node.js 版本（可根据需要修改）
    n 14.18.0

# 验证默认的 Node.js 版本
RUN npm install -g [email protected] && node -v && npm -v

编写 Jenkins Deployment 的 yaml 文件，命名为 09-jenkins-deployment.yaml。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-master
  namespace: kube-ops
spec:
  selector:
    matchLabels:
      app: jenkins-master
  template:
    metadata:
      labels:
        app: jenkins-master
    spec:
      imagePullSecrets:
        - name: aw-registrykey
      terminationGracePeriodSeconds: 10
      serviceAccount: jenkins-admin
      containers:
        - name: jenkins-master
          image: 10.200.0.143:80/wenwo/devops/aw_k8s_jenkins:v3
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
              name: web
              protocol: TCP
            - containerPort: 50000
              name: agent
              protocol: TCP
          resources:
            limits:
              memory: 16Gi
            requests:
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          volumeMounts:
            - name: jenkinshome
              subPath: jenkins-master
              mountPath: /var/jenkins_home
            - name: jenkinsrepo
              subPath: jenkins-master
              mountPath: /data/repo
          env:
            - name: LIMITS_MEMORY
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory
                  divisor: 1Mi
            - name: TZ
              value: Hongkong
            - name: JAVA_OPTS
              value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
      securityContext:
        fsGroup: 1000
      volumes:
        - name: jenkinshome
          persistentVolumeClaim:
            claimName: nfs-jenkins-home
        - name: jenkinsrepo
          persistentVolumeClaim:
            claimName: nfs-jenkins-repo

编写 Jenkins Services 的 yaml 文件，命名为 10-jenkins-services.yaml。

apiVersion: v1
kind: Service
metadata:
  name: jenkins-master
  namespace: kube-ops
  labels:
    app: jenkins-master
spec:
  selector:
    app: jenkins-master
  # 这里的作用类似端口映射，这是个公司内网IP地址
  # 这样配置之后公司内网可以直接访问http://10.200.1.146打开jenkins
  externalIPs:
    - 10.200.1.146
  ports:
    - name: web
      port: 80
      targetPort: 8080
    - name: agent
      port: 50000
      targetPort: 50000

按顺序执行 yaml 文件

kubectl apply -f 01-nfs-rbac.yaml
kubectl apply -f 02-jenkins-home-class.yaml
kubectl apply -f 03-jenkins-repo-class.yaml
kubectl apply -f 04-jenkins-home-deployment.yaml
kubectl apply -f 05-jenkins-repo-deployment.yaml
kubectl apply -f 06-jenkins-account.yaml
kubectl apply -f 07-jenkins-home-pvc.yaml
kubectl apply -f 08-jenkins-repo-pvc.yaml
kubectl apply -f 09-jenkins-deployment.yaml
kubectl apply -f 10-jenkins-services.yaml

安装需要的插件(比如 Git，Kubernetes，Pipeline 等)
我这里因为是需要把老 Jenkins 迁移到 K8S，所以是直接把相关目录还原到了 NFS 机器上的 /data/k8s_jenkins/jenkins，记得还原前需要先停止 Jenkins Deployment。

部分截图展示

部署 `Tekton`

创建命名空间 yaml 文件，命名为：01-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: tekton-pipelines
  labels:
    app.kubernetes.io/instance: default
    app.kubernetes.io/part-of: tekton-pipelines

准备 Tekton 的 yaml 文件，命名为：02-tekton-pipelines.yaml
因内容过多，此处不展示，我这使用的是 v0.23.0 版本。
下载地址为：[[https://github.com/tektoncd/pipeline/releases/download/v0.23.0/release.yaml]]

由于每个 TaskRun 启动时都会使用 02-tekton-pipelines.yaml 中配置的 Base 镜像进行初始化，而默认镜像环境较为简单，因此我们做了一些调整。
如果你也对镜像进行了调整，请记得修改 02-tekton-pipelines.yaml 中的 base 镜像名称。

Base 镜像 Dockerfile 如下：

FROM alpine:latest

USER root

# 使用清华大学的 alpine 镜像源
RUN sed -i 's#https\?://dl-cdn.alpinelinux.org/alpine#https://mirrors4.tuna.tsinghua.edu.cn/alpine#g' /etc/apk/repositories && \
    apk update && \
    # 安装必要的工具
    apk add --no-cache bash git curl jq libxml2-utils ca-certificates wget make vim busybox-extras ttf-dejavu fontconfig && \
    # 清理缓存
    rm -rf /var/cache/apk/*

准备 Tekton Dashboard 的 yaml 文件，命名为：03-tekton-dashboard.yaml
因内容过多，此处不展示，我这使用的是 v0.21.0 版本。
下载地址为：[[https://github.com/tektoncd/dashboard/releases/download/v0.21.0/tekton-dashboard-release.yaml]]

准备 Tekton Dashboard Ingress 的 yaml 文件，命名为：04-ingress.yaml

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: tekton-pipelines
  namespace: tekton-pipelines
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: host("tekton.wenwo.cn")
      services:
        - name: tekton-dashboard
          port: 9097

按顺序执行 yaml 文件

kubectl apply -f 01-namespace.yaml
kubectl apply -f 02-tekton-pipelines.yaml
kubectl apply -f 03-tekton-dashboard.yaml
kubectl apply -f 04-ingress.yaml

准备任务执行镜像

为适应公司构建环境，我这里自定义了合适的 TaskRun 镜像。
命名为：10.200.0.143:80/wenwo/devops/tekton-pipeline/aw-alpine:v1
Dockerfile 如下：

FROM alpine:latest

USER root
COPY jdk8.tar.gz ./
COPY maven.tar.gz ./
COPY daemon.json ./

RUN sed -i 's#https\?://dl-cdn.alpinelinux.org/alpine#https://mirrors4.tuna.tsinghua.edu.cn/alpine#g' /etc/apk/repositories && \
    apk update && \
    apk add --no-cache bash git curl jq libxml2-utils docker openrc ca-certificates wget bash make vim busybox-extras ttf-dejavu fontconfig && \
    mkdir -p /etc/docker/ && \
    mv daemon.json /etc/docker/daemon.json && \
    tar -xvf jdk8.tar.gz && \
    mv jdk8 /usr/local/ && \
    rm -f jdk8.tar.gz && \
    chmod 755 -R /usr/local/jdk8/bin/ && \
    tar -xvf maven.tar.gz && \
    mkdir -p /data && \
    mv maven /data/ && \
    rm -f maven.tar.gz && \
    chmod 755 -R /data/maven/bin/ && \
    ln -sf /data/maven/bin/mvn /usr/bin/mvn && \
    rc-update add docker default && \
    mkdir -p /opt/glibc && \
    cd /opt/glibc && \
    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.30-r0/glibc-2.30-r0.apk && \
    apk add glibc-2.30-r0.apk --allow-untrusted --force-overwrite && \
    rm -rf *.apk && \
    rm -rf /var/cache/apk/*

ENV JAVA_HOME=/usr/local/jdk8
ENV MAVEN_HOME=/data/maven
ENV PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH

RUN java -version && mvn -version

准备 `Task` 文件

根据公司实际情况，我这里的 Task 内容仅供参考，文件命名为：java-task.yaml。
具体如下：

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: java-task
  namespace: tekton-pipelines
spec:
  params:
    - name: BRANCH
      type: string
    - name: GIT_BRANCH
      type: string
    - name: GIT_URL_CLEAN
      type: string
    - name: REGISTRY
      type: string
    - name: BUILD_ID
      type: string
    - name: JOB_NAME
      type: string
    - name: BUILD_CMD
      type: string
    - name: SONAR_CMD
      type: string
    - name: VERSION
      type: string
    - name: COMMITSHA1_10
      type: string
    - name: BUILD_URL
      type: string
    - name: GROUPID
      type: string
    - name: ARTIFACTID
      type: string
    - name: SONARSERVER
      type: string
    - name: IMAGENAME
      type: string
  steps:
    - name: pull-code
      image: 10.200.0.143:80/wenwo/devops/tekton-pipeline/aw-alpine:v1
      env:
        - name: GIT_URL_CLEAN
          value: "$(params.GIT_URL_CLEAN)"
        - name: GIT_BRANCH
          value: "$(params.GIT_BRANCH)"
      script: |
        echo "从 ${GIT_URL_CLEAN} 拉取 ${GIT_BRANCH} 分支..."
        set +x
        GIT_USERNAME=$(cat /var/secrets/git-credentials/username)
        GIT_PASSWORD=$(cat /var/secrets/git-credentials/password)
        git clone http://${GIT_USERNAME}:$(echo ${GIT_PASSWORD} | sed 's/@/%40/')@$(echo ${GIT_URL_CLEAN} | sed 's#http://##') -b ${GIT_BRANCH} /workspace/source
        set -x
        echo "${GIT_URL_CLEAN} 拉取 ${GIT_BRANCH} 分支完成！"
      volumeMounts:
        - name: git-credentials-volume
          mountPath: /var/secrets/git-credentials
        - name: shared-volume
          mountPath: /workspace/source

    - name: build-tasks
      image: 10.200.0.143:80/wenwo/devops/tekton-pipeline/aw-alpine:v1
      workingDir: /workspace/source
      env:
        - name: REGISTRY
          value: "$(params.REGISTRY)"
        - name: BUILD_ID
          value: "$(params.BUILD_ID)"
        - name: JOB_NAME
          value: "$(params.JOB_NAME)"
        - name: GIT_BRANCH
          value: "$(params.GIT_BRANCH)"
        - name: BUILD_CMD
          value: "$(params.BUILD_CMD)"
        - name: SONAR_CMD
          value: "$(params.SONAR_CMD)"
        - name: BUILD_URL
          value: "$(params.BUILD_URL)"
      script: |
        echo "开始构建项目..."
        set +x
        DOCKER_USERNAME=$(cat /var/secrets/aw-credentials/username)
        DOCKER_PASSWORD=$(cat /var/secrets/aw-credentials/password)
        echo ${DOCKER_PASSWORD} | docker login ${REGISTRY} -u ${DOCKER_USERNAME} --password-stdin
        set -x
        curl -i -X POST -H "Content-type: application/json" -d "{\"buildid\":\"${BUILD_ID}\",\"jenkinsjobname\":\"${JOB_NAME}\",\"buildurl\":\"${BUILD_URL}\",\"branchname\":\"${GIT_BRANCH}\",\"images\":\"\",\"status\":\"构建中\",\"steps\":\"start\"}" http://10.200.1.172:8080/project/jenkins/buildInfo
        ${BUILD_CMD}
        echo "项目构建完成！"
        echo "开始代码扫描..."
        ${SONAR_CMD}
        echo "代码扫描完成！"
      volumeMounts:
        - name: nfs-volume
          mountPath: /data/repo
        - name: aw-credentials-volume
          mountPath: /var/secrets/aw-credentials
        - name: shared-volume
          mountPath: /workspace/source
        - name: docker-sock
          mountPath: /var/run/docker.sock

    - name: quality-inspection
      image: 10.200.0.143:80/wenwo/devops/tekton-pipeline/aw-alpine:v1
      workingDir: /workspace/source
      env:
        - name: GROUPID
          value: "$(params.GROUPID)"
        - name: ARTIFACTID
          value: "$(params.ARTIFACTID)"
        - name: SONARSERVER
          value: "$(params.SONARSERVER)"
      script: |
        if [ -z "$SONARSERVER" ]; then
          echo "该项目不做质量检测！"
          exit 0
        fi 
        sleep 10
        echo "开始质量检测..."
        response=$(curl -s -k -X GET "${SONARSERVER}/project_branches/list?project=${GROUPID}:${ARTIFACTID}")
        status=$(echo $response | jq -r '.branches[0].status.qualityGateStatus')
        echo "质量检测状态为：$status"
        if [ "$status" != "OK" ]; then
            echo "质量检测不通过！"
            exit 1
        fi
        echo "质量检测通过！"
      volumeMounts:
        - name: shared-volume
          mountPath: /workspace/source

    - name: push-docker
      image: 10.200.0.143:80/wenwo/devops/tekton-pipeline/aw-alpine:v1
      workingDir: /workspace/source
      env:
        - name: REGISTRY
          value: "$(params.REGISTRY)"
        - name: BUILD_ID
          value: "$(params.BUILD_ID)"
        - name: GIT_BRANCH
          value: "$(params.GIT_BRANCH)"
        - name: JOB_NAME
          value: "$(params.JOB_NAME)"
        - name: BRANCH
          value: "$(params.BRANCH)"
        - name: VERSION
          value: "$(params.VERSION)"
        - name: COMMITSHA1_10
          value: "$(params.COMMITSHA1_10)"
        - name: BUILD_URL
          value: "$(params.BUILD_URL)"
        - name: IMAGENAME
          value: "$(params.IMAGENAME)"
      script: |
        echo "开始上传镜像..."
        set +x
        DOCKER_USERNAME=$(cat /var/secrets/aw-credentials/username)
        DOCKER_PASSWORD=$(cat /var/secrets/aw-credentials/password)
        echo ${DOCKER_PASSWORD} | docker login ${REGISTRY} -u ${DOCKER_USERNAME} --password-stdin
        set -x
        docker tag ${IMAGENAME} ${REGISTRY}/wenwo/${JOB_NAME}:${VERSION}-${BRANCH}-${BUILD_ID}-${COMMITSHA1_10}
        docker push ${REGISTRY}/wenwo/${JOB_NAME}:${VERSION}-${BRANCH}-${BUILD_ID}-${COMMITSHA1_10}
        docker rmi ${IMAGENAME} ${REGISTRY}/wenwo/${JOB_NAME}:${VERSION}-${BRANCH}-${BUILD_ID}-${COMMITSHA1_10}
        curl -i -X POST -H "Content-type: application/json" -d "{\"buildid\":\"${BUILD_ID}\",\"jenkinsjobname\":\"${JOB_NAME}\",\"buildurl\":\"${BUILD_URL}\",\"branchname\":\"${GIT_BRANCH}\",\"images\":\"${REGISTRY}/wenwo/${JOB_NAME}:${VERSION}-${BRANCH}-${BUILD_ID}-${COMMITSHA1_10}\",\"status\":\"构建成功\",\"steps\":\"end\"}" http://10.200.1.172:8080/project/jenkins/buildInfo
        echo "镜像上传成功！"
      volumeMounts:
        - name: aw-credentials-volume
          mountPath: /var/secrets/aw-credentials
        - name: shared-volume
          mountPath: /workspace/source
        - name: docker-sock
          mountPath: /var/run/docker.sock

  volumes:
    - name: nfs-volume
      nfs:
        server: 10.200.0.144
        path: /data/k8s_jenkins/repo
    - name: shared-volume
      emptyDir:
        medium: Memory
    - name: git-credentials-volume
      secret:
        secretName: git-credentials
    - name: aw-credentials-volume
      secret:
        secretName: aw-credentials
    - name: docker-sock
      hostPath:
        path: /var/run/docker.sock

应用 Task 文件

kubectl apply -f java-task.yaml

部分截图展示

项目整改

根据具体情况，对项目做一些调整，如 pom.xml 文件，如 Jenkinsfile。

`pom.xml` 整改

# 把docker插件的设置部分修改为这样子
<configuration>
    <imageName>java/${project.artifactId}:${env}</imageName>
    <imageTags>
        <imageTag>${env}</imageTag>
    </imageTags>
    <forceTags>true</forceTags>
</configuration>

# ${project.artifactId}参数是必须的
<artifactId>wenwo-cloud-basic-wechat-spider</artifactId>

`Jenkinsfile` 整改

整改后的

def REGISTRY = "10.200.0.143:80"
def BRANCH = params.BRANCH_NAME
def GIT_URL = scm.userRemoteConfigs
def BUILD_ID = env.BUILD_ID
def JOB_NAME = env.JOB_NAME
def BUILD_URL = env.BUILD_URL

pipeline {
    agent any
    environment {
        KUBE_NAMESPACE = "tekton-pipelines"  // Kubernetes 命名空间
        JOB_NAME_CLEAN = JOB_NAME.replaceAll('_', '-')
    }

    stages {
        stage('触发 Tekton TaskRun') {
            steps {
                script {
                    def POM = readMavenPom file: "pom.xml"
                    def VERSION = POM.version
                    def COMMIT = "${env.GIT_COMMIT}"
                    def COMMITSHA1_10 = sh label: "", returnStdout: true, script: "expr substr '${COMMIT}' 1 10"
                    COMMITSHA1_10=COMMITSHA1_10.replaceAll("\n","")
                    def GROUPID = POM.groupId
                    def ARTIFACTID = POM.artifactId
                    //def SONARSERVER = "http://10.200.0.144:9000/api"
                    def SONARSERVER = ""
                    def GIT_URL_CLEAN = GIT_URL[0].url
                    dir("${env.WORKSPACE}-${BRANCH}") {
                        def ENVNAME = BRANCH
                        if (BRANCH == "master") {
                            ENVNAME = "prd"
                        } else if (BRANCH == "develop") {
                            ENVNAME = "dev"
                        }
                        def BUILD_CMD = "mvn -B -U clean package -Dfile.encoding=UTF-8 -Dmaven.test.skip=true -Denv=${ENVNAME}"
                        def SONAR_CMD = "mvn sonar:sonar -Dsonar.host.url=http://10.200.0.144:9000 -Dsonar.login=1c0bc447ba1ebf80d0dd49e95fd081faaea71611 -Dsonar.exclusions=**/util/*,**/utils/* -Denv=${ENVNAME}"
                        def IMAGENAME = "java/${ARTIFACTID}:${ENVNAME}"
                        // 生成 Tekton TaskRun YAML，使用 git-credentials Secret
                        def taskRunYaml = """
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: ${JOB_NAME_CLEAN}-${BUILD_ID}
  namespace: ${KUBE_NAMESPACE}
spec:
  taskRef:
    name: java-task
  params:
    - name: GIT_BRANCH
      value: "${BRANCH}"
    - name: BRANCH
      value: "${ENVNAME}"
    - name: GIT_URL_CLEAN
      value: "${GIT_URL_CLEAN}"
    - name: REGISTRY
      value: "${REGISTRY}"
    - name: BUILD_ID
      value: "${BUILD_ID}"
    - name: JOB_NAME
      value: "${JOB_NAME}"
    - name: BUILD_CMD
      value: "${BUILD_CMD}"
    - name: SONAR_CMD
      value: "${SONAR_CMD}"
    - name: VERSION
      value: "${VERSION}"
    - name: COMMITSHA1_10
      value: "${COMMITSHA1_10}"
    - name: BUILD_URL
      value: "${BUILD_URL}"
    - name: GROUPID
      value: "${GROUPID}"
    - name: ARTIFACTID
      value: "${ARTIFACTID}"
    - name: SONARSERVER
      value: "${SONARSERVER}"
    - name: IMAGENAME
      value: "${IMAGENAME}"
  podTemplate:
    imagePullSecrets:
      - name: aw-registrykey
                        """

                        // 将 TaskRun YAML 写入到一个文件中
                        writeFile file: 'TaskRun.yaml', text: taskRunYaml
                        // 使用 kubectl 应用 TaskRun
                        sh 'kubectl apply -f TaskRun.yaml --namespace=${KUBE_NAMESPACE}'
                        sh 'echo "构建地址为：http://tekton.wenwo.cn/#/namespaces/tekton-pipelines/taskruns/${JOB_NAME_CLEAN}-${BUILD_ID}"'
                    }
                }
            }
        }

        stage('监控 Tekton TaskRun') {
            steps {
                script {
                    dir("${env.WORKSPACE}-${BRANCH}") {
                        def status = ""
                        while (status != "True" && status != "False") {
                            // 查询Tekton状态的逻辑
                            status = sh(script: 'kubectl get TaskRun ${JOB_NAME_CLEAN}-${BUILD_ID} --namespace=tekton-pipelines -o jsonpath="{.status.conditions[0].status}"', returnStdout: true).trim()
                            echo "当前状态: ${status}"
                            if (status == "True") {
                                echo "任务成功"
                            } else if (status == "False") {
                                error "任务失败"
                            } else {
                                sleep(5) // 每5秒检查一次
                            }
                        }
                    }
                }
            }
        }

        stage("清理过期任务") {
            steps {
                dir("${env.WORKSPACE}-${BRANCH}") {
                    sh '''
                        task_runs=$(kubectl get TaskRun -n tekton-pipelines --sort-by=.metadata.creationTimestamp -o json | jq -r '.items | .[].metadata.name' | grep "${JOB_NAME_CLEAN}")
                        total_count=$(echo "$task_runs" | wc -l)

                        if [ "$total_count" -gt 10 ]; then
                            to_delete=$(echo "$task_runs" | head -n -10)
                            for task_run in $to_delete; do
                                echo "删除 TaskRun: $task_run"
                                kubectl delete TaskRun "$task_run" -n tekton-pipelines
                            done
                        else
                            echo "TaskRun 数量未超过 10，当前数量: $total_count"
                        fi
                    '''
                }
            }
        }

        stage("清理空间") {
            steps {
                sh "ls -al"
                deleteDir()
                sh "ls -al"
            }
        }
    }

    post {
        failure {
            sh "curl -i -X POST -H \"'Content-type':'application/json'\" -d \'{\"buildid\":\"${env.BUILD_ID}\",\"jenkinsjobname\":\"${env.JOB_NAME}\",\"buildurl\":\"${env.BUILD_URL}\",\"branchname\":\"${env.GIT_BRANCH}\",\"images\":\"\",\"status\":\"构建失败\",\"steps\":\"end\"}\' http://10.200.1.172:8080/project/jenkins/buildInfo"
        }
    }
}

整改前的

pipeline {
    agent any
    stages {
      stage("编译打包") {
          steps {
              script {
                  def projectBranch = env.GIT_BRANCH.split("/")[1]
                  def envName = projectBranch
                  if (projectBranch.contains("master")) {
                      envName = "prd"
                  } else if (projectBranch == "develop") {
                      envName = "dev"
                  }
                  sh "curl -i -X POST -H \"'Content-type':'application/json'\" -d \'{\"buildid\":\"${env.BUILD_ID}\",\"jenkinsjobname\":\"${env.JOB_NAME}\",\"buildurl\":\"${env.BUILD_URL}\",\"branchname\":\"${env.GIT_BRANCH}\",\"images\":\"\",\"status\":\"构建中\",\"steps\":\"start\"}\' http://10.200.1.172:8080/project/jenkins/buildInfo"
                  // 修改原有打包编译命令
                  // 注意：这里需要双引号，否则无法读取到 envName 变量
                  sh "/data/maven/bin/mvn -B -U clean package -Dfile.encoding=UTF-8 -Dmaven.test.skip=true -Denv=${envName}"
              }
          }
      }


        stage("代码扫描"){
                    steps {
                        dir(env.WORKSPACE){
                             script {
                                def projectBranch = env.GIT_BRANCH.split("/")[1]
                                def envName = projectBranch
                                if (projectBranch.contains("master")) {
                                   envName = "prd"
                                } else if (projectBranch == "develop") {
                                   envName = "dev"
                                }
                                sh "/data/maven/bin/mvn sonar:sonar -Dsonar.host.url=http://10.200.0.144:9000 -Dsonar.login=1c0bc447ba1ebf80d0dd49e95fd081faaea71611 -Dsonar.exclusions=**/util/*,**/utils/* -Denv=${envName}"    //指定sonar的ip和token
                            }
                        }
                    }
                }

        stage("质量检测"){
            steps {
                script{
                    sh 'sleep 10'
                    def projectName = env.JOB_NAME
                    def pom = readMavenPom file: 'pom.xml'
                    def GroupId = pom.groupId
                    def ArtifactId = pom.artifactId
                    def sonarServer = "http://10.200.0.144:9000/api"
                    def response = httpRequest httpMode: 'GET',
                        contentType: "APPLICATION_JSON",
                        consoleLogResponseBody: true,
                        ignoreSslErrors: true,
                        requestBody: '',
                        url: "${sonarServer}/project_branches/list?project=${GroupId}:${ArtifactId}"
                    def props = readJSON text: response.content
                    status = props["branches"][0]["status"]["qualityGateStatus"]
                    echo status
                    if (status != 'OK'){
                        sh 'exit 1'
                    }
                }
            }
        }

        stage("上传镜像") {
            steps {
                script {
                    def projectBranch = env.GIT_BRANCH.split("/")[1]
                    def projectName = env.JOB_NAME
                    def pom = readMavenPom file: 'pom.xml'
                    def version = pom.version
                    def contentRegex = "/.{0,10}/"
                    def commit = "${env.GIT_COMMIT}"
                    def commitSha1_10 = sh label: "", returnStdout: true, script: "expr substr '${commit}' 1 10"
                    commitSha1_10=commitSha1_10.replaceAll("\n","")
                    sh "docker tag java/${projectName}:${version} 10.200.0.143:80/wenwo/${projectName}:${version}-${projectBranch}-${env.BUILD_ID}-${commitSha1_10}"
                    sh "docker push 10.200.0.143:80/wenwo/${projectName}:${version}-${projectBranch}-${env.BUILD_ID}-${commitSha1_10}"
                    sh "docker rmi java/${projectName}:${version} java/${projectName}:latest 10.200.0.143:80/wenwo/${projectName}:${version}-${projectBranch}-${env.BUILD_ID}-${commitSha1_10}"
                    sh "curl -i -X POST -H \"'Content-type':'application/json'\" -d \'{\"buildid\":\"${env.BUILD_ID}\",\"jenkinsjobname\":\"${env.JOB_NAME}\",\"buildurl\":\"${env.BUILD_URL}\",\"branchname\":\"${env.GIT_BRANCH}\",\"images\":\"10.200.0.143:80/wenwo/${projectName}:${version}-${projectBranch}-${env.BUILD_ID}-${commitSha1_10}\",\"status\":\"构建成功\",\"steps\":\"end\"}\' http://10.200.1.172:8080/project/jenkins/buildInfo"

                }
            }
        }

        stage("清理空间") {
            steps {
                sh "ls -al"
                deleteDir()
                sh "ls -al"
            }
        }
    }
     post {
            failure {
                sh "curl -i -X POST -H \"'Content-type':'application/json'\" -d \'{\"buildid\":\"${env.BUILD_ID}\",\"jenkinsjobname\":\"${env.JOB_NAME}\",\"buildurl\":\"${env.BUILD_URL}\",\"branchname\":\"${env.GIT_BRANCH}\",\"images\":\"\",\"status\":\"构建失败\",\"steps\":\"end\"}\' http://10.200.1.172:8080/project/jenkins/buildInfo"
            }
        }
}

变动还是挺大的，此处就不详述了。

测试构建

我这里用的是 Pipeline script from SCM 的方式。
点了构建后，看截图。

构建完成后

解释

由截图可以看出，此时当项目点击构建后，会根据项目 Jenkinsfile 配置创建一个 TaskRun Pod，构建完成之后，会只保留前10次的构建结果，这就完成了动态构建了。

后记

如果在部署过程中遇到问题欢迎留言讨论，后面有空我再补充一些调试技巧。
希望本文可以帮助到大家，感谢阅读！

前言

部署 Jenkins

部署 k8s-nfs 客户端

部署 Jenkins 节点

部分截图展示

部署 Tekton

部署 Tekton

准备任务执行镜像

准备 Task 文件

部分截图展示

项目整改

pom.xml 整改

Jenkinsfile 整改

测试构建

解释

后记

部署 `Jenkins`

部署 `k8s-nfs` 客户端

部署 `Jenkins` 节点

部署 `Tekton`

部署 `Tekton`

准备 `Task` 文件

`pom.xml` 整改

`Jenkinsfile` 整改