Skip to content

About

Enterprise Jenkins CI/CD & ArgoCD GitOps for OpenShift 4.20+ with OpenTelemetry | Paired with nubenetes/jenkins-git-parameter-global-vars

github.com/nubenetes/jenkins-git-parameter-global-vars

Topics

angularargocdcicddevopsgit-parametergitopsgrafanahelmiacjava21jcascjenkinsjob-dslmulti-clusteropenshiftopentelemetryprometheusskopeospring-boottrivy

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository
BranchesTags

Repository files navigation

Jenkins Git Parameter & Multi-Cluster GitOps IaC Platform on OpenShift 4.20+

Important

⚠️ AI Generation & Model Review Notice

  • Initial Generation: Originally generated by Gemini 3.7 Flash with Antigravity.
  • Review & Production Hardening: Audited, remediated, and verified for enterprise readiness using Gemini 3.8 Flash with Antigravity.
  • Testing Status: This repository serves as an architectural reference blueprint, educational design pattern, and foundational boilerplate. Platform engineers must review and validate all cluster endpoints, secrets, and TLS certificates prior to live production deployment.

Generated by Reviewed & Hardened by


OpenShift 4.20+ Kubernetes 1.31+ Jenkins LTS ArgoCD Helm 3

JCasC Job DSL Git Parameter OpenTelemetry Prometheus Grafana Java 21 Spring Boot 3 Angular 18 Skopeo Trivy SCC restricted-v2 License PRs Welcome Maintenance

Important

🔗 Centralized Architecture & Linked Repositories

This platform operates in a tightly integrated multi-repository GitOps ecosystem:

  • 🌐 Centralized Governance & Configuration SSOT: nubenetes/jenkins-git-parameter-global-vars — Master repository containing cluster topologies, environment overlays, application inventory, and dynamic gitParameter dropdown targets.
  • 🚀 CI/CD Platform Orchestrator (This Repository): nubenetes/jenkins-git-parameter — Infrastructure-as-Code, JCasC, Job DSL, Jenkinsfiles, OpenTelemetry, Grafana 13.2.0, and ArgoCD 3.5 manifests.
  • ☕ Workload Reference Microservice: nubenetes/jhipster-microservice — Java 21 / Spring Boot 3 cloud-native microservice with OpenTelemetry and Prometheus integration.

📑 Table of Contents

Executive Summary & Architecture Overview

This repository provides an enterprise-grade, end-to-end Infrastructure as Code (IaC) blueprint and GitOps orchestration engine hosted on Red Hat OpenShift Container Platform (OCP) 4.20+.

The solution addresses a core enterprise challenge: how to build, govern, promote, and release multi-repository cloud-native microservices across 3 distinct OpenShift clusters (DEV, STAGING, PROD) while leveraging dynamic Git Parameter selection, Declarative Pipelines as Code, ArgoCD 3.5 GitOps synchronization, and unified OpenTelemetry observability.

🗺️ Click to expand: End-to-End Multi-Cluster Platform Topology Diagram 
flowchart TB
    subgraph DeveloperWorkspace["1. Developer & External Systems"]
        Dev["👩‍💻 Developer / Ops"]
        Backstage["🏛️ Backstage IDP<br/>(Software Catalog)"]
        ITSM["🎫 Jira Service Mgmt<br/>& ServiceNow CMDB"]
    end

    subgraph OCP_DEV["OpenShift Cluster 1: DEV (Control Plane & Workloads)"]
        direction TB
        subgraph JenkinsPlatform["Jenkins Controller (JCasC & Job DSL)"]
            Master["Jenkins Controller<br/>(2.492.2 LTS)"]
            Seed["00-Seed-Job<br/>Orchestrator"]
            CIJob["01-CI-Build-Pipeline<br/>(Git Param: App Repo)"]
            CDJob["02-CD-Release-Orchestrator<br/>(Git Param: Global Vars)"]
        end

        subgraph Agents["Ephemeral Kubernetes Agent Pods"]
            MavenAgent["maven-jdk21 Agent<br/>(Build & Test)"]
            GitOpsAgent["argocd-gitops Agent<br/>(Skopeo & Argo CLI)"]
        end

        subgraph OCPDevRegistry["OCP DEV Internal Registry"]
            DevReg["image-registry:5000<br/>nubenetes-dev-apps"]
        end

        subgraph ArgoCDMaster["ArgoCD 3.5 Control Plane"]
            ArgoServer["ArgoCD Server &<br/>ApplicationSets"]
        end

        subgraph ObservabilityStack["Full-Stack Observability"]
            OTel["OpenTelemetry<br/>Collector (OTLP)"]
            Prom["Prometheus<br/>Server"]
            Grafana["Grafana<br/>Dashboards"]
        end

        DevApps["DEV Workloads<br/>(nubenetes-dev-apps)"]
    end

    subgraph OCP_STG["OpenShift Cluster 2: STAGING (UAT)"]
        StgReg["OCP Staging Registry"]
        StgApps["Staging Workloads<br/>(nubenetes-staging-apps)"]
    end

    subgraph OCP_PRD["OpenShift Cluster 3: PROD (High Availability)"]
        PrdReg["OCP Prod Registry"]
        PrdApps["Production Workloads<br/>(nubenetes-prod-apps)"]
    end

    Dev -->|Selects Branch or Tag| CIJob
    Dev -->|Selects Config Version| CDJob
    Backstage -->|REST API Trigger| CDJob
    ITSM -->|Webhook Dispatch| CDJob

    CIJob -->|Launches| MavenAgent
    MavenAgent -->|Pushes Image| DevReg
    CIJob -->|Triggers Downstream| CDJob

    CDJob -->|Launches| GitOpsAgent
    GitOpsAgent -->|Skopeo Copy| StgReg
    GitOpsAgent -->|Skopeo Copy| PrdReg
    GitOpsAgent -->|Sync & Health Check| ArgoServer

    ArgoServer -->|GitOps Deploy| DevApps
    ArgoServer -->|GitOps Deploy| StgApps
    ArgoServer -->|Sync Waves Deploy| PrdApps

    Master -.->|OTLP Traces| OTel
    OTel -.->|Forward Spans| Grafana
    Master -.->|Metrics Export| Prom
    Prom -.->|Metrics Scrape| Grafana
Loading

📋 Architectural Breakdown & Workflow Steps:

  • Developer & External Ingestion: Developers, Backstage IDP, and Jira/ServiceNow trigger parameterized builds specifying application and global configuration revisions.
  • Master Controller (OpenShift Cluster 1): Jenkins JCasC boots the master Seed Job and materializes isolated CI and CD pipelines via Job DSL.
  • Pipeline 01 (CI Build): Ephemeral maven-jdk21 agents compile source code, execute tests, push container images to the internal DEV registry, and trigger downstream CD.
  • Pipeline 02 (CD Release Orchestration): Ephemeral gitops-tools agents promote images across registries using Skopeo and execute argoAppSync to synchronize DEV, STAGING, and PROD clusters.
  • Observability & Tracing: OpenTelemetry collector exports build spans to Grafana, while Prometheus exports metrics for real-time monitoring.

💡 Architectural Summary & Conclusion: This topology decouples build creation from release promotion while keeping release orchestration parameterized inside Jenkins. By isolating CI agents from CD promotion agents and automating ArgoCD synchronizations, engineering teams gain multi-cluster delivery with centralized auditing.

In-Depth Architectural Analysis & Design Decisions

1. Multi-Repository Git Parameter Investigation: Two Architectural Patterns

The Core Problem: Why Standard Pipelines Fail with Multiple Git Repositories

In cloud-native architectures, applications separate source code (app-repo) from centralized environmental configuration, cluster definitions, and Helm values (jenkins-git-parameter-global-vars).

When engineering teams attempt to introduce multiple gitParameter dropdowns into a single Jenkins Pipeline, they almost universally encounter a hard blocker where GitSCM appears to only support one repository, or secondary dropdowns fail with No GIT repository configured in SCM configuration.

⚠️ Click to expand: Jenkins SCM Lifecycle & Pre-Execution Blindspot Diagram 
flowchart TB
    subgraph UI_Phase["1. Pre-Execution Phase (Master)"]
        direction TB
        User["👤 User opens<br/>Build UI Form"]
        Master["⚙️ Master reads<br/>Job XML SCM"]
        GitParam["🔍 git-parameter<br/>queries remote refs"]
        Dropdown["📋 Renders Dropdown<br/>for Primary Repo"]

        User --> Master --> GitParam --> Dropdown
    end

    subgraph Runtime_Phase["2. Runtime Execution Phase (Agent)"]
        direction TB
        AllocAgent["☸️ Ephemeral Agent<br/>Pod Allocated"]
        RunStage["📦 Pipeline Stage:<br/>checkout(repo-2)"]

        AllocAgent --> RunStage
    end

    Gap["⚠️ SCM Blindspot:<br/>Dynamic stage checkouts<br/>run during build and are<br/>invisible at UI render"]

    Dropdown -.-> Gap
    Gap -.-> RunStage
Loading

📋 Why Multi-Repo Pipelines Fail at Parameter Render Time:

  • Pre-Execution Master Phase: The Jenkins Master renders gitParameter dropdowns by querying only repositories statically declared in the Jenkins Job XML.
  • Runtime Agent Execution: Secondary checkouts executed dynamically inside pipeline stages run on ephemeral agents long after the parameter form has been submitted.
  • The Blindspot: The Master cannot discover secondary stage checkouts ahead of time, causing secondary dropdowns to fail or duplicate primary repository branches.

💡 Architectural Summary & Conclusion: Understanding the pre-execution vs. runtime paradox is essential when designing multi-repository pipelines in Jenkins. Pattern 1 (Multi-Remote SCM) solves this statically in Job DSL, while Pattern 2 (Decoupled Hand-off) eliminates the conflict by separating pipelines completely.

Root Cause Analysis: The Three Jenkins Architectural Constraints
  1. The Pre-Execution vs. Runtime Lifecycle Paradox: Jenkins calculates and renders job parameters in the user's browser before allocating an agent, before checking out the source tree, and before executing any Jenkinsfile stage. When git-parameter inspects the job, it only knows about Git repositories defined statically in the Jenkins Master Job XML configuration.
  2. The SCM Binding Constraint (Dynamic Checkouts are Invisible): If a Jenkinsfile clones a secondary repository dynamically inside a stage (steps { checkout([$class: 'GitSCM', userRemoteConfigs: [[url: '...']]]) }), the Jenkins Master cannot discover or query this secondary repository at parameter render time. Consequently, the second gitParameter dropdown fails or mistakenly duplicates the branches of the primary repository.
  3. Pipeline (cpsScm) GUI Limitations & Deprecation of Multiple SCMs Plugin: In legacy Jenkins Freestyle jobs, the community used the Multiple SCMs Plugin. However, this plugin was never ported or supported in Declarative Pipelines (cpsScm). The standard Jenkins Pipeline Web UI only allows configuring a single SCM Git block with one repository URL.

To solve this challenge, this repository implements and delivers BOTH architectural patterns as production-ready code:

Master Seed Job & JCasC Automation Architecture

The entire platform is self-bootstrapping and self-healing using Jenkins Configuration as Code (JCasC) combined with the Job DSL Plugin:

⚙️ Click to expand: Seed Job & Job DSL Pipeline Provisioning Flow Diagram 
flowchart TB
    subgraph JCasC_Phase ["1. Controller Bootstrap (JCasC)"]
        direction TB
        JCasC["jenkins-jcasc.yaml<br/>(Master Config)"] -->|"Registers"| SeedJob["00-Seed-Job<br/>(Platform Seed)"]
    end

    subgraph SCM_Phase ["2. SCM Synchronization"]
        direction TB
        GitRepo["Git Repository<br/>(Platform Code)"]
        SeedJob -->|"Polls H/15 * * * *"| GitRepo
        SeedJob -->|"Evaluates"| JobDSLFiles["Target Scripts<br/>(jobdsl/*.groovy)"]
    end

    subgraph Materialization ["3. Pipeline Materialization"]
        direction TB
        JobDSLFiles -->|"seed-job.groovy"| Folders["Creates Folders:<br/>• CI Pipelines<br/>• CD Orchestrators<br/>• Maintenance"]
        JobDSLFiles -->|"pipelines-ci.groovy"| CIPipelines["Generates CI:<br/>• jhipster-ci<br/>• angular-ci"]
        JobDSLFiles -->|"pipelines-cd.groovy"| CDPipelines["Generates CD:<br/>• release-orchestrator<br/>• hotfix-deploy"]
    end

    Folders -.->|"Parent"| CIPipelines
    Folders -.->|"Parent"| CDPipelines
Loading

📋 Pipeline Provisioning Chain:

  • JCasC Bootstrap: Master controller starts up and executes JCasC to register 00-Seed-Job-Platform-Orchestrator.
  • Dynamic SCM Polling: The Seed Job polls this repository every 15 minutes for changes in jobdsl/*.groovy.
  • Separation of Concerns: seed-job.groovy creates folder taxonomies, pipelines-ci.groovy materializes CI jobs, and pipelines-cd.groovy provisions release orchestrators.
  • Automated Lifecycle Pruning: removedJobAction('DELETE') automatically purges decommissioned pipelines without touching master configuration.

💡 Architectural Summary & Conclusion: Combining JCasC with Job DSL treats the entire Jenkins pipeline hierarchy as version-controlled code. Teams can onboard new microservices or modify pipeline definitions via Git commits without manual controller restarts or UI drift.

1. Why jobdsl/seed-job.groovy Only Declares Folders (Separation of Concerns)

In enterprise Jenkins-as-Code implementations, we deliberately apply the Separation of Concerns (SoC) principle:

  • Taxonomy & Hierarchy Initialization: jobdsl/seed-job.groovy acts as the root taxonomy initializer. It creates and configures the top-level folder containers (01-CI-Build-Pipelines, 02-CD-Release-Orchestrators, and 03-Platform-Maintenance) along with their descriptions and metadata.
  • Decoupled Pipeline Generation Logic: The application CI pipelines (jobdsl/pipelines-ci.groovy) and CD release orchestrators (jobdsl/pipelines-cd.groovy) contain iterative business logic (looping over the apps inventory, configuring multi-remote refspecs, Git Parameter dropdowns, log rotation, and OpenTelemetry spans). Keeping folder declarations separate prevents monolithic scripts and enables modular additions of new business domains.
2. Managing Folders in Job DSL vs. Static JCasC

While folders can technically be defined directly in jenkins-jcasc.yaml, managing them through Job DSL is the recommended SRE practice for multi-tenant platforms:

Architectural Metric Static JCasC Management (jenkins-jcasc.yaml) Dynamic Job DSL Management (jobdsl/seed-job.groovy)
Creation Lifecycle Executed strictly upon Jenkins Controller startup or manual JCasC reload. Evaluated continuously on every Seed Job execution (automated SCM polling or manual trigger).
Orphaned Item Pruning Does not automatically purge folders or child jobs if deleted from configuration. Automated Pruning: removedJobAction('DELETE') and removedViewAction('DELETE') cleanly destroy removed jobs/folders.
Configuration Ownership Belongs to Controller System Configuration (Platform Admin scope). Belongs to Version-Controlled GitOps Code (Developer / Application Domain scope).
Zero-Downtime Evolution Renaming or restructuring folders requires reloading master JCasC. Restructuring folders is applied dynamically via Git commit without touching master configuration.
3. Step-by-Step Technical Chain & Execution Flow
  1. JCasC Master Bootstrap:
    When the Jenkins container starts, jcasc/jenkins-jcasc.yaml configures the master Seed Job: 
    jobs:
  - script: |
      job('00-Seed-Job-Platform-Orchestrator') {
        description('Master Seed Job provisioning CI/CD Pipelines as Code.')
        scm {
          git {
            remote { url('https://github.com/nubenetes/jenkins-git-parameter.git') }
            branch('*/main')
          }
        }
        steps {
          jobDsl {
            targets('jobdsl/seed-job.groovy\njobdsl/pipelines-ci.groovy\njobdsl/pipelines-cd.groovy')
            removedJobAction('DELETE')
            removedViewAction('DELETE')
            lookupStrategy('JENKINS_ROOT')
          }
        }
        triggers { scm('H/15 * * * *') }
      }
  2. lookupStrategy('JENKINS_ROOT') Context Resolution:
    By setting lookupStrategy('JENKINS_ROOT'), Job DSL resolves all relative job paths from the root of Jenkins. This allows pipelines-ci.groovy to target "01-CI-Build-Pipelines/${app.name}-ci-build" and nest jobs into the folders previously created by seed-job.groovy.
  3. Automated Drift Correction via SCM Polling:
    The trigger scm('H/15 * * * *') polls this repository every 15 minutes. When an engineer adds a new microservice to the def apps = [...] inventory in pipelines-ci.groovy and pushes to main, the Seed Job automatically generates the new CI/CD pipelines in Jenkins with zero manual UI configuration.

Pattern 1: Dual Git Parameter Dropdowns in a Single Pipeline (Multi-Remote SCM)

Implemented in: jobdsl/pipelines-ci.groovy (*-ci-dual-dropdown) and jenkinsfiles/ci/Jenkinsfile.app-java-maven-dual-dropdown

Job DSL Multi-Remote SCM Implementation

Rather than attempting to define multiple scm blocks (which Jenkins rejects), Job DSL configures a single GitSCM definition containing multiple named Git remote blocks with dedicated refspecs. The Git Parameter plugin can then target each remote independently using useRepository matching on the remote identifier or URL regex:

// Job DSL Implementation: Multi-Remote SCM with 2 Dynamic Dropdowns
pipelineJob("01-CI-Build-Pipelines/jhipster-microservice-ci-dual-dropdown") {
    parameters {
        // Dropdown 1: Application Repository (Branches & Tags)
        gitParameterDefinition {
            name('APP_GIT_REVISION')
            type('PT_BRANCH_TAG')
            defaultValue('main')
            useRepository('origin-app') // Targets Remote 1
            sortMode('DESCENDING_SMART')
        }

        // Dropdown 2: Global Variables Repository (Branches & Tags)
        gitParameterDefinition {
            name('GLOBAL_VARS_REVISION')
            type('PT_BRANCH_TAG')
            defaultValue('main')
            useRepository('origin-vars') // Targets Remote 2
            sortMode('DESCENDING_SMART')
        }
    }

    definition {
        cpsScm {
            scm {
                git {
                    remote {
                        name('origin-app')
                        url('https://github.com/nubenetes/jhipster-microservice.git')
                        refspec('+refs/heads/*:refs/remotes/origin-app/* +refs/tags/*:refs/remotes/origin-app/tags/*')
                    }
                    remote {
                        name('origin-vars')
                        url('https://github.com/nubenetes/jenkins-git-parameter-global-vars.git')
                        refspec('+refs/heads/*:refs/remotes/origin-vars/* +refs/tags/*:refs/remotes/origin-vars/tags/*')
                    }
                    branch('origin-app/main')
                }
            }
            scriptPath('jenkinsfiles/ci/Jenkinsfile.app-java-maven-dual-dropdown')
        }
    }
}
Hidden Pitfalls, Failure Modes & Operational Drawbacks

While Pattern 1 successfully renders two dropdowns on a single screen, enterprise teams should be aware of several critical technical caveats:

Pitfall / Caveat Technical Root Cause Operational Impact
Branch & Tag Namespace Collisions Both repositories frequently have identical branch/tag names (e.g. main, release-1.0). Without strict custom refspecs (refs/remotes/origin-app/* vs refs/remotes/origin-vars/*), Git checkout becomes non-deterministic and can check out the wrong commit.
Misleading Workspace State Jenkins' initial cpsScm checkout only materializes one branch into the workspace directory. Even if the user selected two branches, the files of the second repository are not present in the workspace. The Jenkinsfile must still execute an explicit secondary checkout into a subfolder.
Jenkins Master Performance Overhead On every page load of "Build with Parameters", the Master executes synchronous git ls-remote network calls to both Git repositories. If either repository has high latency, rate limits, or network degradation, the Jenkins UI blocks, freezes, or times out.
useRepository Regex Fragility useRepository relies on regular expression matching against UserRemoteConfig repository URLs. Known bugs (e.g. JENKINS-75780) can cause dropdowns to cross-contaminate or fail if repositories share common domains or credentials.

Pattern 2: Decoupled Two-Pipeline Architecture (CI $\rightarrow$ CD Hand-off) [RECOMMENDED]

Implemented in: jobdsl/pipelines-ci.groovy (*-ci-build), jenkinsfiles/ci/Jenkinsfile.app-java-maven, jobdsl/pipelines-cd.groovy (multi-cluster-release-orchestrator), and jenkinsfiles/cd/Jenkinsfile.release-orchestrator

Separates artifact creation from release promotion. Pipeline 01 focuses on building the application code with its own Git Parameter dropdown, while Pipeline 02 is bound to jenkins-git-parameter-global-vars with its own Git Parameter dropdown for multi-cluster release orchestration.

Why Decoupling is the Enterprise Industry Standard

Decoupling CI from CD solves the limitations of Pattern 1 cleanly by aligning with core cloud-native architecture principles:

  1. Single Responsibility Principle (SRP):
    • Pipeline 01 (CI) is strictly responsible for code quality, unit testing, static analysis, and packaging container images.
    • Pipeline 02 (CD) is strictly responsible for environment state, configuration management, deployment orchestration, and GitOps synchronization.
  2. Artifact Immutability ("Build Once, Deploy Anywhere"):
    • In Pattern 1, redeploying or promoting to another environment forces a recompilation of the application source code.
    • In Pattern 2, the container image is built once in CI, tagged immutably (e.g. 2.1.0-42-a1b2c3d), and promoted across DEV $\rightarrow$ STAGING $\rightarrow$ PROD by updating GitOps manifests without rebuilding.
  3. Zero SCM Hacks & Zero Namespace Collision Risk:
    • Each pipeline uses standard, isolated SCM bindings. No multi-remote refspec gymnastics or fragile regex matching.
  4. Universal REST API & ITSM Interoperability:
    • Pipeline 02 can be triggered downstream by Pipeline 01, but can also be invoked directly by Jira Service Management Forms, ServiceNow Change Requests, or Backstage Developer Portals via standard parameterized REST API calls.
1. Pipeline 01: CI Build Pipeline (App Code & Downstream Hand-off)

A. Job DSL Implementation (jobdsl/pipelines-ci.groovy): Configures a dedicated gitParameterDefinition targeting the application repository, alongside parameters to configure the downstream release trigger:

// Job DSL Implementation: Decoupled CI Build Pipeline with Application Git Parameter
pipelineJob("01-CI-Build-Pipelines/jhipster-microservice-ci-build") {
    description('''
    <b>[PATTERN 2: Decoupled CI Build Pipeline - Recommended]</b><br/>
    Cloud-Native Java 21 / Spring Boot Microservice<br/>
    • <b>App Revision</b>: Dynamic Git Parameter Dropdown on https://github.com/nubenetes/jhipster-microservice.git<br/>
    • <b>Global Vars</b>: Passed via parameter to downstream 02-CD-Release-Orchestrator.
    '''.stripIndent())

    parameters {
        // Dropdown: Application Repository (Branches & Tags)
        gitParameterDefinition {
            name('APP_GIT_REVISION')
            type('PT_BRANCH_TAG')
            defaultValue('main')
            description('Select Application Git Branch/Tag from https://github.com/nubenetes/jhipster-microservice.git')
            branchFilter('.*')
            tagFilter('*')
            sortMode('DESCENDING_SMART')
            selectedValue('DEFAULT')
            useRepository('https://github.com/nubenetes/jhipster-microservice.git')
            quickFilterEnabled(true)
        }

        booleanParam('TRIGGER_CD_RELEASE', true, 'Automatically trigger downstream CD Release Orchestrator.')
        stringParam('GLOBAL_VARS_BRANCH', 'main', 'Global Variables branch to pass to downstream CD Release Orchestrator.')
        choiceParam('TARGET_ENVIRONMENT', ['dev', 'staging', 'prod'], 'Initial deployment target environment for downstream CD release.')
    }

    definition {
        cpsScm {
            scm {
                git {
                    remote {
                        url('https://github.com/nubenetes/jenkins-git-parameter.git')
                    }
                    branch('*/main')
                }
            }
            scriptPath('jenkinsfiles/ci/Jenkinsfile.app-java-maven')
            lightweight(true)
        }
    }
}

B. Declarative Jenkinsfile CI Stage (jenkinsfiles/ci/Jenkinsfile.app-java-maven): Builds the container image, tags it with an immutable build identifier, and triggers Pipeline 02 passing the configuration branch and artifact metadata:

// Jenkinsfile Stage: Compile, Package Image, and Trigger Downstream CD Orchestrator
stage('Checkout Application Code') {
    steps {
        script {
            checkout([
                $class: 'GitSCM',
                branches: [[name: "${params.APP_GIT_REVISION}"]],
                userRemoteConfigs: [[url: env.APP_REPO_URL]]
            ])
            env.GIT_COMMIT_SHORT = sh(script: 'git rev-parse --short HEAD', returnStdout: true).trim()
            env.CALCULATED_TAG   = "2.1.0-${BUILD_NUMBER}-${env.GIT_COMMIT_SHORT}"
        }
    }
}

// ... Compile, Test, Package, and Push Image stages ...

stage('Trigger CD Release Orchestrator') {
    when {
        expression { return params.TRIGGER_CD_RELEASE == true }
    }
    steps {
        echo "===> Triggering Downstream CD Release Orchestration Pipeline..."
        build job: '02-CD-Release-Orchestrators/multi-cluster-release-orchestrator',
            parameters: [
                string(name: 'GLOBAL_VARS_REVISION', value: params.GLOBAL_VARS_BRANCH ?: 'main'),
                string(name: 'APP_NAME', value: env.APP_NAME),
                string(name: 'IMAGE_TAG', value: env.CALCULATED_TAG),
                string(name: 'TARGET_ENVIRONMENT', value: params.TARGET_ENVIRONMENT ?: 'dev'),
                booleanParam(name: 'AUTO_PROMOTE_TO_STAGING', value: true),
                booleanParam(name: 'REQUIRE_PROD_APPROVAL', value: true),
                string(name: 'TRIGGERED_BY', value: 'CI_PIPELINE'),
                string(name: 'CHANGE_REQUEST_ID', value: "CI-AUTO-${BUILD_NUMBER}")
            ],
            wait: false
    }
}
2. Pipeline 02: CD Release Orchestrator (Global Config Dropdown & GitOps Promotion)

A. Job DSL Implementation (jobdsl/pipelines-cd.groovy): Configures a dedicated gitParameterDefinition targeting the centralized jenkins-git-parameter-global-vars configuration repository, exposing promotion flags and token-based REST dispatching:

// Job DSL Implementation: Multi-Cluster CD Release Orchestrator with Global Vars Dropdown
pipelineJob("02-CD-Release-Orchestrators/multi-cluster-release-orchestrator") {
    description('''
    🚀 <b>Enterprise Multi-Cluster Release & Promotion Orchestrator</b><br/>
    Orchestrates promotion across 3 OpenShift clusters (DEV -> STAGING -> PROD) with ArgoCD 3.5.<br/>
    • Parameterized with <b>Git Parameter</b> on the Global Variables repository.<br/>
    • Can be triggered manually, by upstream CI build pipelines, or externally via REST API.
    '''.stripIndent())

    authenticationToken('RELEASE_DISPATCH_TOKEN_2026')

    parameters {
        // Dropdown: Global Variables & Multi-Cluster Environment Configuration Repo
        gitParameterDefinition {
            name('GLOBAL_VARS_REVISION')
            type('PT_BRANCH_TAG')
            defaultValue('main')
            description('Select the Git Branch or Tag from Global Configuration (jenkins-git-parameter-global-vars)')
            branchFilter('.*')
            tagFilter('*')
            sortMode('DESCENDING_SMART')
            selectedValue('DEFAULT')
            useRepository('https://github.com/nubenetes/jenkins-git-parameter-global-vars.git')
            quickFilterEnabled(true)
        }

        choiceParam('APP_NAME', ['jhipster-microservice', 'angular-frontend', 'all-apps'], 'Application to release and promote.')
        stringParam('IMAGE_TAG', 'latest', 'Container Image Tag to deploy and promote across OpenShift clusters.')
        choiceParam('TARGET_ENVIRONMENT', ['dev', 'staging', 'prod', 'full-promotion-chain'], 'Target environment or full promotional chain.')
        booleanParam('AUTO_PROMOTE_TO_STAGING', true, 'Automatically promote image and deploy to OCP STAGING cluster if DEV tests pass.')
        booleanParam('REQUIRE_PROD_APPROVAL', true, 'Require interactive human gate approval before deploying to OCP PROD cluster.')
        choiceParam('TRIGGERED_BY', ['MANUAL', 'CI_PIPELINE', 'BACKSTAGE_IDP', 'SERVICENOW_ITSM', 'JIRA_CMDB'], 'Caller identity for audit and trace metadata.')
        stringParam('CHANGE_REQUEST_ID', 'CHG-DEFAULT-0000', 'ITSM / Jira Change Request ID for production compliance.')
    }

    definition {
        cpsScm {
            scm {
                git {
                    remote {
                        url('https://github.com/nubenetes/jenkins-git-parameter.git')
                    }
                    branch('*/main')
                }
            }
            scriptPath('jenkinsfiles/cd/Jenkinsfile.release-orchestrator')
            lightweight(true)
        }
    }
}

B. Declarative Jenkinsfile CD Stages (jenkinsfiles/cd/Jenkinsfile.release-orchestrator): Checks out the selected global variables revision dynamically, updates GitOps environment manifests, triggers ArgoCD syncs, and promotes immutable images across OpenShift clusters:

// Jenkinsfile Stage: Dynamic Global Config Checkout & Multi-Cluster Promotion Flow
stage('Initialize & Resolve Global Configuration') {
    steps {
        script {
            echo "===> Checking out Global Vars Repository at revision: ${params.GLOBAL_VARS_REVISION}"
            dir('jenkins-git-parameter-global-vars') {
                checkout([
                    $class: 'GitSCM',
                    branches: [[name: "${params.GLOBAL_VARS_REVISION}"]],
                    userRemoteConfigs: [[url: 'https://github.com/nubenetes/jenkins-git-parameter-global-vars.git']]
                ])
            }
        }
    }
}

stage('Deploy to OCP DEV Cluster') {
    when {
        expression { return params.TARGET_ENVIRONMENT in ['dev', 'full-promotion-chain', 'staging', 'prod'] }
    }
    steps {
        script {
            // Update GitOps manifests in global vars repo & Sync ArgoCD DEV App
            gitopsCommit(envName: 'dev', appName: params.APP_NAME, imageTag: params.IMAGE_TAG, configDir: 'jenkins-git-parameter-global-vars')
            argoAppSync(appName: "${params.APP_NAME}-dev", targetRevision: params.GLOBAL_VARS_REVISION, server: env.ARGOCD_SERVER)
        }
    }
}

stage('Promote to OCP STAGING Cluster') {
    when {
        expression { return params.AUTO_PROMOTE_TO_STAGING == true && params.TARGET_ENVIRONMENT in ['staging', 'full-promotion-chain', 'prod'] }
    }
    steps {
        script {
            // Promote immutable image across cluster registries via Skopeo
            skopeoPromote(
                sourceImage: "${env.DEV_REGISTRY}/${env.DEV_NAMESPACE}/${params.APP_NAME}:${params.IMAGE_TAG}",
                destImage: "${env.STAGING_REGISTRY}/${env.STAGING_NAMESPACE}/${params.APP_NAME}:${params.IMAGE_TAG}"
            )
            gitopsCommit(envName: 'staging', appName: params.APP_NAME, imageTag: params.IMAGE_TAG, configDir: 'jenkins-git-parameter-global-vars')
            argoAppSync(appName: "${params.APP_NAME}-staging", targetRevision: params.GLOBAL_VARS_REVISION, server: env.ARGOCD_SERVER)
        }
    }
}

stage('Promote to OCP PROD Cluster') {
    when {
        expression { return params.TARGET_ENVIRONMENT in ['prod', 'full-promotion-chain'] }
    }
    steps {
        script {
            if (params.REQUIRE_PROD_APPROVAL) {
                input message: "Approve deployment of ${params.APP_NAME}:${params.IMAGE_TAG} to OCP PRODUCTION?", submitter: 'release-managers,platform-admins'
            }
            skopeoPromote(
                sourceImage: "${env.STAGING_REGISTRY}/${env.STAGING_NAMESPACE}/${params.APP_NAME}:${params.IMAGE_TAG}",
                destImage: "${env.PROD_REGISTRY}/${env.PROD_NAMESPACE}/${params.APP_NAME}:${params.IMAGE_TAG}"
            )
            gitopsCommit(envName: 'prod', appName: params.APP_NAME, imageTag: params.IMAGE_TAG, configDir: 'jenkins-git-parameter-global-vars')
            argoAppSync(appName: "${params.APP_NAME}-prod", targetRevision: params.GLOBAL_VARS_REVISION, server: env.ARGOCD_SERVER)
        }
    }
}
3. End-to-End Hand-off Sequence Diagram
🔄 Click to expand: End-to-End CI/CD Hand-off Sequence Diagram 
sequenceDiagram
    autonumber
    actor Dev as Developer / Operator
    participant CI as "01-CI-Build-Pipeline (App Repo)"
    participant DevReg as "OCP DEV Registry"
    participant CD as "02-CD-Release-Orchestrator (Global Vars)"
    participant Skopeo as "Skopeo Image Promotion"
    participant StgReg as "OCP STAGING Registry"
    participant Argo as "ArgoCD 3.5 GitOps Engine"
    participant PrdReg as "OCP PROD Registry"

    alt Trigger Path A: Developer CI Dispatch
        Dev->>CI: Trigger Build with Git Parameter (Select App Tag/Branch)
        CI->>CI: Compile Java 21 / Run Unit Tests / Sonar / Trivy
        CI->>DevReg: Build & Push Container Image (e.g. 2.1.0-42-a1b2c3d)
        CI->>CD: Trigger Downstream with (IMAGE_TAG, GLOBAL_VARS_REVISION='main')
    else Trigger Path B: External API / Backstage / ServiceNow / Jira
        Dev->>CD: Direct REST API / UI Trigger with (GLOBAL_VARS_REVISION, IMAGE_TAG)
    end

    CD->>CD: Checkout jenkins-git-parameter-global-vars @ GLOBAL_VARS_REVISION
    CD->>Argo: Update DEV manifests & Sync ArgoCD DEV App
    Argo->>Argo: Validate Health on OCP DEV Cluster

    opt Auto Promote to STAGING
        CD->>Skopeo: Copy Image: OCP DEV Registry -> OCP STAGING Registry
        CD->>Argo: Update STAGING manifests & Sync ArgoCD STAGING App
        Argo->>Argo: Validate Health & Run Integration Suite
    end

    opt Production Promotion Gate
        CD->>Dev: Interactive Human Approval Gate (Submitter: release-managers)
        Dev-->>CD: Approval Granted
        CD->>Skopeo: Copy Image: OCP STAGING Registry -> OCP PROD Registry
        CD->>Argo: Update PROD manifests & Sync with Sync Waves
        Argo->>Argo: Verify Canary / Blue-Green Zero-Downtime Rollout
    end
Loading

📋 Step-by-Step CI/CD Hand-off Sequence:

  • 1. CI Pipeline Trigger: Developer launches Pipeline 01 (jhipster-microservice-ci-build) with APP_GIT_REVISION.
  • 2. Build & Immutability: Jenkins compiles Java 21 code, executes tests, and pushes image jhipster-microservice:2.1.0-42.
  • 3. Downstream Hand-off: Pipeline 01 triggers Pipeline 02 (multi-cluster-release-orchestrator), passing IMAGE_TAG and environment targets.
  • 4. Multi-Cluster Promotion: Pipeline 02 copies the image via Skopeo from DEV to STAGING/PROD registries and commits the new tag to jenkins-git-parameter-global-vars.
  • 5. GitOps Synchronization: Pipeline 02 invokes argoAppSync, verifying HTTP 200 health on target OpenShift clusters.

💡 Architectural Summary & Conclusion: Pattern 2 achieves strict 'Build Once, Deploy Anywhere' artifact immutability. Re-deploying or promoting across environments never recompiles application code, eliminating environment-specific build drift.

3. Automated Downstream Hand-off & Environment Targeting Flow

A foundational capability of Pattern 2 is the automated downstream linkage between Pipeline 01 (CI Build) and Pipeline 02 (CD Release Orchestrator). When an engineer or external system launches Pipeline 01, they can choose whether and where the resulting immutable artifact should be deployed:

🔄 Click to expand: Downstream Hand-off & Environment Targeting Flow Diagram 
flowchart LR
    subgraph CIStage["1. Pipeline 01: CI Build"]
        direction TB
        Dev1["👩‍💻 User"] -->|"Selects Branch & Env"| CI["🏗️ 01-CI-Build<br/>• Compiles Artifact<br/>• Signs Image<br/>• Pushes to DEV"]
        CI -->|"Trigger CD"| Dispatch["⚡ build job:<br/>02-CD-Orchestrator<br/>Passes: IMAGE_TAG"]
    end

    subgraph CDStage["2. Pipeline 02: CD Orchestrator"]
        direction TB
        Dispatch --> CD["🚀 02-CD-Orchestrator<br/>• Resolves vars<br/>• Routes by Env"]
        
        CD -->|"env == 'dev'"| DevTarget["☸️ OpenShift DEV<br/>• gitopsCommit(dev)<br/>• argoAppSync(dev)"]
        CD -->|"env == 'staging'"| StgTarget["☸️ OpenShift STAGING<br/>• skopeoPromote<br/>• argoAppSync(stg)"]
        CD -->|"env == 'prod'"| PrdTarget["☸️ OpenShift PROD<br/>• Approval Gate<br/>• skopeoPromote<br/>• argoAppSync(prd)"]
    end
Loading

📋 Downstream Parameter Passing & Execution Logic:

  • Configured in Job DSL (jobdsl/pipelines-ci.groovy):
    • TRIGGER_CD_RELEASE (boolean, default: true): Automatically dispatches downstream release.
    • GLOBAL_VARS_BRANCH (string, default: 'main'): Global configuration branch passed to Pipeline 02.
    • TARGET_ENVIRONMENT (choice: dev, staging, prod, full-promotion-chain): Initial target environment.
  • Trigger Stage in Pipeline 01 (Jenkinsfile.app-java-maven): 
    stage('Trigger CD Release Orchestrator') {
    when { expression { return params.TRIGGER_CD_RELEASE == true } }
    steps {
        build job: '02-CD-Release-Orchestrators/multi-cluster-release-orchestrator',
            parameters: [
                string(name: 'GLOBAL_VARS_REVISION', value: params.GLOBAL_VARS_BRANCH ?: 'main'),
                string(name: 'APP_NAME', value: env.APP_NAME),
                string(name: 'IMAGE_TAG', value: env.CALCULATED_TAG),
                string(name: 'TARGET_ENVIRONMENT', value: params.TARGET_ENVIRONMENT ?: 'dev'),
                booleanParam(name: 'AUTO_PROMOTE_TO_STAGING', value: true),
                booleanParam(name: 'REQUIRE_PROD_APPROVAL', value: true),
                string(name: 'TRIGGERED_BY', value: 'CI_PIPELINE'),
                string(name: 'CHANGE_REQUEST_ID', value: "CI-AUTO-${BUILD_NUMBER}")
            ],
            wait: false
    }
}
  • Environment-Specific Routing in Pipeline 02 (Jenkinsfile.release-orchestrator):
    • Target dev: Updates dev overlays and triggers argoAppSync on OpenShift DEV.
    • Target staging: Promotes container image via Skopeo from DEV to STAGING registry, updates manifests, and synchronizes OpenShift STAGING.
    • Target prod: Promotes image to PROD registry, pauses at the interactive Production Approval Gate (input step), and deploys with ArgoCD sync waves on OpenShift PROD.

💡 Architectural Summary & Conclusion: This automated linkage guarantees 'Build Once, Deploy Anywhere' artifact immutability. Pipeline 02 never recompiles source code—it promotes the exact, cryptographically verified container image digest created by Pipeline 01.

When to Use Each Pattern: Environment & Persona Decision Matrix

Dimension Pattern 1: Dual Dropdown (Multi-Remote SCM) Pattern 2: Decoupled CI $\rightarrow$ CD Orchestration
Primary Use Case Developer Workspaces & Sandboxes Enterprise Multi-Cluster Production
Target Environments dev, feature branches, ephemeral PR preview clusters dev $\rightarrow$ staging (UAT) $\rightarrow$ prod
User Persona Individual Software Engineers & Frontend Developers Release Managers, QA Leads, Platform Teams, ITSM
UI Experience Single "Build with Parameters" form with 2 live dropdowns Dedicated CI form for code; Dedicated CD form for release
Jenkins Master Overhead Fetches remote refs from both repos on every page render SCM queries are isolated to the specific pipeline
Branch Collision Risk Requires strict refspec prefixes (origin-app/* vs origin-vars/*) Zero risk; completely isolated Git repositories
External Integration Complex; external callers must know build & compilation parameters Simple REST API: Callers only supply IMAGE_TAG & GLOBAL_VARS_REVISION
Promotion & Rollbacks Rebuilds application code on every run Promotes existing immutable images without recompiling
Governance & SoD Code committers deploy directly to target cluster Enforces Segregation of Duties via Jira Forms / CAB

Coexistence Strategy: Combining Pattern 1 (Inner-Loop) & Pattern 2 (Outer-Loop) in Development

Can Pattern 1 and Pattern 2 coexist? Yes! Coexistence is the recommended industry standard for separating Inner-Loop (rapid developer experimentation) from Outer-Loop (governed, automated CI/CD release lifecycle).

🧪 Click to expand: Inner-Loop (Pattern 1) vs Outer-Loop (Pattern 2) Coexistence Diagram 
flowchart TD
    subgraph InnerLoop["🧪 Inner-Loop (Pattern 1: Dual-Dropdown Pipeline)"]
        Dev1["👨‍💻 Feature Developer"]
        DualJob["01-CI-*-ci-dual-dropdown<br/>(Dropdown 1: App | Dropdown 2: Config)"]
        DevSandbox["☸️ Ephemeral Dev Sandbox / PR Preview<br/>(e.g. nubenetes-dev-apps / pr-42)"]
        
        Dev1 -->|Ad-Hoc Experimentation| DualJob
        DualJob -->|Direct 1-Click Deploy| DevSandbox
    end

    subgraph OuterLoop["🚀 Outer-Loop (Pattern 2: Decoupled CI ➔ CD Orchestration)"]
        GitWebhook["⚡ Git Push / PR Merge Event"]
        CIJob["01-CI-*-ci-build<br/>(Compiles & Tests App Code)"]
        DevRegistry["📦 OCP DEV Image Registry<br/>(Immutable Tag: 2.1.0-42-sha)"]
        CDJob["02-CD-Release-Orchestrator<br/>(ArgoCD 3.5 Multi-Cluster Promotion)"]
        SharedClusters["☸️ Shared Environments<br/>(Shared DEV ➔ STAGING ➔ PROD)"]

        GitWebhook -->|Automated Trigger| CIJob
        CIJob -->|Publishes Image| DevRegistry
        CIJob -->|Hands off to CD| CDJob
        CDJob -->|GitOps Promotion| SharedClusters
    end
Loading

📋 Dual-Pattern Coexistence Mechanics:

  • Inner-Loop (Pattern 1: Dual-Dropdown): Optimized for active feature development, sandbox testing, and rapid integration experimentation using a single unified pipeline.
  • Outer-Loop (Pattern 2: Decoupled Hand-off): Standardized for official release promotions, UAT gates, and high-governance production rollouts.
  • Shared Infrastructure: Both patterns share the same container registries, Helm charts, and ArgoCD application synchronizers.

💡 Architectural Summary & Conclusion: Providing both patterns side-by-side offers enterprise platform teams maximum flexibility: rapid inner-loop velocity for developers during active feature testing, paired with rigorous, auditable release orchestration for production deployments.

1. Pattern 1 for Inner-Loop: Developer Sandboxes & Feature Branch Pairing
  • Scenario: An engineer is developing a feature requiring simultaneous code and environment configuration changes (e.g. a new microservice endpoint requiring a new database connection string or secret in jenkins-git-parameter-global-vars).
  • Workflow:
    1. The developer pushes feature/order-v2 to the application repository.
    2. The developer pushes feature/order-config-v2 to jenkins-git-parameter-global-vars.
    3. The developer runs jhipster-microservice-ci-dual-dropdown (Pattern 1), selecting both branches from the two live dropdowns.
    4. The pipeline compiles and deploys the paired code and configuration directly into an isolated DEV preview namespace for instant verification.
  • Benefit: Fast, self-contained iteration with zero impact on release promotion pipelines or shared environments.
2. Pattern 2 for Outer-Loop: Automated Shared DEV Integration & Multi-Cluster Promotion
  • Scenario: The feature branch is merged into main, or an automated scheduled CI build is triggered.
  • Workflow:
    1. A GitHub Webhook automatically triggers jhipster-microservice-ci-build (Pattern 2).
    2. It compiles, unit-tests, scans vulnerabilities (Trivy), builds an immutable container image (2.1.0-42-a1b2c3d), and pushes it to the OCP DEV registry.
    3. It triggers 02-CD-Release-Orchestrator, which syncs ArgoCD 3.5 in the shared DEV cluster, executes integration tests, and prepares the image for promotion to STAGING and PROD.
  • Benefit: 100% auditability, immutable artifact promotion via Skopeo, and strict GitOps compliance across shared environments.
Coexistence Architecture in This Repository
Pipeline in Jenkins UI Pattern Best Suited For Target OpenShift Scope
*-ci-dual-dropdown Pattern 1 (Dual Dropdowns) Manual Ad-hoc Developer Testing & Feature Pairing Developer Sandbox / Preview Namespaces
*-ci-build Pattern 2 (CI Build) Automated Git Webhooks, PR Scans & Trunk Integration Pushes immutable image to OCP DEV Registry
multi-cluster-release-orchestrator Pattern 2 (CD Release) Release Promotions, Staging UAT & Production Releases Deploys across DEV $\rightarrow$ STAGING $\rightarrow$ PROD with ArgoCD 3.5

2. Job DSL + Declarative Pipelines vs. Monolithic Shared Libraries

Dimension Monolithic Shared Library Antipattern (@Library('...') _) Modern Architecture: Job DSL + Declarative Templates + Granular Steps
Pipeline Visibility Opaque black box; stages hidden inside Groovy scripts Modern Stage View, Pipeline Graph View, and OpenTelemetry trace spans natively rendered
Jenkins Replay ❌ Broken: Replay cannot modify Shared Library code ✅ 100% Functional: Developers can replay and edit pipeline stages in UI
Job Provisioning Manual UI creation or complex custom scripts Automated Seed Job (seed-job.groovy) reconciles all jobs from Git
Syntax Validation Runtime-only Groovy failures Declarative linter validates syntax at definition time
Step Reusability Tight coupling; copy-pasting scripted logic Granular custom steps (argoAppSync, skopeoPromote, gitopsCommit)

Note

Blue Ocean Deprecation & Modern UX Architecture: Blue Ocean has been officially deprecated by the Jenkins project due to maintenance overhead, heavy resource footprints, and limited compatibility with modern Declarative features. This platform adopts the modern recommended Jenkins UX stack:

  1. Pipeline Graph View Plugin (pipeline-graph-view): Lightweight, responsive pipeline visualization directly embedded in the core Jenkins interface.
  2. Pipeline Stage View (pipeline-stage-view): Fast tabular build and stage matrix.
  3. OpenTelemetry + Grafana: Deep distributed trace analysis with sub-millisecond stage breakdown and span correlations.

Decision: We use Job DSL to generate Declarative Pipeline jobs pointing to standard declarative Jenkinsfile definitions, complemented by a lightweight Jenkins Shared Library containing modular, reusable steps.

3. OpenShift 4.20+ Security & Ephemeral Kubernetes Agents

  1. Security Context Constraints (SCC):
    • Jenkins Controller runs under restricted-v2 / nonroot SCC without requiring root privileges.
    • UID and GID are pinned to 1000:1000.
    • Dynamic ephemeral agents run inside dedicated non-privileged containers.
  2. Zero Permanent Executors on Controller:
    • numExecutors: 0 configured in JCasC. The master controller is exclusively a coordinator.
    • All builds run inside short-lived agent pods (maven-jdk21, node-angular, skopeo-trivy, argocd-gitops) created on-demand and terminated immediately upon stage completion.
  3. OpenShift Route with Edge TLS:
    • Secure external ingress via OpenShift HAProxy Route with automatic HTTP-to-HTTPS redirect and TLS termination.

4. ArgoCD 3.5 Integration & Multi-Cluster Promotion

  1. Multi-Cluster Topology:
    • Cluster 1 (ocp-dev): Hosts Jenkins Master, ArgoCD 3.5 Controller, OpenTelemetry Collector, and DEV workloads.
    • Cluster 2 (ocp-staging): External target cluster registered in ArgoCD via Kubernetes cluster secrets (cluster-ocp-staging).
    • Cluster 3 (ocp-prod): Production cluster managed remotely by ArgoCD (cluster-ocp-prod).
  2. ApplicationSets & Sync Waves:
    • ApplicationSet generators dynamically provision application targets per cluster.
    • ArgoCD sync waves enforce strict ordering: Database migrations (Wave 1) $\rightarrow$ Backend Microservices (Wave 2) $\rightarrow$ Frontend SPA & Routes (Wave 3).
  3. Skopeo Cross-Cluster Image Copy:
    • Images are copied directly between cluster internal registries (image-registry.openshift-image-registry.svc:5000) without spinning up a Docker daemon or intermediate worker storage.

5. Full-Stack Observability: OpenTelemetry, Prometheus & Grafana 13.2.0 (ArgoCD GitOps Managed)

This platform implements a unified, end-to-end observability mesh covering both Platform Operations (Jenkins Controller, Ephemeral Agent Pods, ArgoCD GitOps Engine, OpenShift Cluster Metrics) and Application Workloads (Java 21 / Spring Boot Microservices, Angular SPAs).

Grafana 13.2.0 is deployed and continuously reconciled via ArgoCD GitOps (argocd-apps/apps/dev/grafana-observability.yaml), with pinned image tags and automated drift management.

By leveraging the W3C Distributed Tracing standard, every CI build, container promotion, GitOps sync, and live HTTP user request is linked through a single correlated traceparent.

End-to-End Multi-Layer Observability Architecture

📊 Click to expand: Full-Stack Observability & Trace Correlation Architecture Diagram 
flowchart TB
    subgraph Layer1["1. CI/CD Platform & GitOps Engine"]
        direction TB
        Jenkins["⚙️ Jenkins Controller<br/>(OpenTelemetry Plugin)"]
        Agents["☸️ Ephemeral Agent Pods<br/>(Maven / Skopeo / Cosign)"]
        ArgoCD["🐙 ArgoCD 3.5 Controller<br/>(ApplicationSet Engine)"]
    end

    subgraph Layer2["2. Observability & Telemetry Processing"]
        direction TB
        Collector["🔭 OpenTelemetry Collector<br/>(OTLP gRPC:4317 / HTTP:4318)"]
        Prom["🔥 Prometheus Server<br/>(Metrics Scrape & Storage)"]
    end

    subgraph Layer3["3. Deployed Microservices & Web Apps"]
        direction TB
        Angular["🅰️ Angular 18 Web App<br/>(@opentelemetry/sdk-trace-web)"]
        SpringBoot["☕ Spring Boot 3 Microservice<br/>(Micrometer Tracing + OTLP)"]
    end

    subgraph Layer4["4. Unified Grafana 13.2.0 Correlation"]
        direction TB
        Grafana["📊 Grafana 13.2.0 Dashboards<br/>(Trace-to-Log & Exemplars)"]
    end

    Jenkins -->|"OTLP Traces"| Collector
    Agents -->|"Span Context (W3C)"| Collector
    ArgoCD -->|"Prometheus Metrics"| Prom
    Collector -->|"Export Metrics"| Prom
    Angular -->|"HTTP + W3C Traceparent"| SpringBoot
    SpringBoot -->|"OTLP Traces & Exemplars"| Collector
    SpringBoot -->|"/actuator/prometheus"| Prom
    Prom -->|"Metrics with Exemplars"| Grafana
    Collector -->|"Correlated Traces"| Grafana
Loading

📋 Observability & Telemetry Data Flow:

  • Pipeline Telemetry: OpenTelemetry Jenkins Plugin traces pipeline stages (checkout, build, test, deploy) with W3C trace IDs.
  • Metric Export: Prometheus plugin exports live controller queue metrics, build execution durations, and executor utilization.
  • Unified Dashboards: Grafana 13.2.0 consolidates OTLP distributed traces, Prometheus metrics, and Loki pipeline logs in a single pane of glass.

💡 Architectural Summary & Conclusion: End-to-end telemetry gives platform engineers deep visibility into build bottlenecks, test suite degradation, and deployment latencies across all OpenShift clusters.

The 3 Pillars of Observability with Full Correlation

Pillar Platform Telemetry (Jenkins & ArgoCD) Workload Telemetry (Spring Boot & Angular) End-to-End Correlation Link
Traces (OpenTelemetry) Pipeline Root Spans, Stage Spans, Step Spans, Image Push & ArgoCD Sync Spans Incoming HTTP Controller Spans, Database Queries, Downstream REST calls W3C traceparent Header: Extracted in CI $\rightarrow$ passed to GitOps commit $\rightarrow$ queried in live app logs.
Metrics (Prometheus) jenkins_runs_total, jenkins_executor_queue_time_bucket, argocd_app_sync_total http_server_requests_seconds_bucket, jvm_memory_used_bytes, jvm_gc_pause_seconds Prometheus Exemplars: High latency spikes in Grafana graph link directly to the specific OTel trace ID.
Logs (Structured) Jenkins Controller & Agent console logs formatted with build metadata Spring Boot Logback JSON logs with %X{trace_id} and %X{span_id} Trace-to-Logs Derived Fields: 1-click jump from a Grafana trace span to the exact container log lines.

1. W3C Trace Context Propagation (Pipeline $\rightarrow$ GitOps $\rightarrow$ Live App)

The platform maintains unbroken distributed trace continuity using standard W3C Trace Context (traceparent):

  1. Pipeline Execution: The Jenkins OpenTelemetry plugin generates a root trace_id (e.g. 4bf92f3577b34da6a3ce929d0e0e4736).
  2. Shared Library Step (otelLogEvent.groovy): Emits span annotations with build number, git commit SHA, and target cluster, exporting CURRENT_TRACEPARENT to the agent environment: 
    otelLogEvent(
    name: 'cd.orchestration.initialized',
    app: params.APP_NAME,
    imageTag: params.IMAGE_TAG,
    gitCommit: env.GIT_COMMIT_SHORT,
    caller: params.TRIGGERED_BY
)
  3. Application Runtime Tracing: Spring Boot 3 uses micrometer-tracing-bridge-otel to ingest the trace context from incoming requests and forwards spans to the OpenTelemetry Collector: 
    management:
  tracing:
    sampling:
      probability: 1.0
    propagation:
      type: W3C
  otlp:
    tracing:
      endpoint: http://otel-collector.observability.svc.cluster.local:4318/v1/traces
  metrics:
    distribution:
      percentiles-histogram:
        http.server.requests: true
  4. Client-Side SPA Tracing: Angular Single Page Apps instrumented with @opentelemetry/sdk-trace-web attach the traceparent HTTP header to all AJAX/Fetch calls, connecting user button clicks in the browser to backend Java transactions.

2. Prometheus Exemplars & Traces-to-Logs Correlation in Grafana 13.2.0

Grafana 13.2.0 is pre-configured with Prometheus Exemplars enabled in helm/observability/grafana-values.yaml:

  • When Prometheus collects http_server_requests_seconds_bucket latency metrics, Micrometer attaches the active trace_id as an exemplar.
  • In Grafana 13.2.0, hovering over a latency spike displays a blue exemplar dot; clicking it opens the exact OpenTelemetry trace span in the side-by-side Trace View.

3. Pre-configured Enterprise Grafana Dashboards

The repository ships with 3 production-grade, pre-provisioned Grafana Dashboards:

Dashboard UID Dashboard Name Scope & Visualized Metrics
jenkins-performance-otel Jenkins Platform & Ephemeral Agent APM Stage durations, p95 queue latency, success/failure rate, dynamic Kubernetes agent pod concurrency, OTel trace spans.
argocd-gitops-sync ArgoCD 3.5 Multi-Cluster GitOps Sync Application health status (Healthy/Degraded), sync phases (Synced/OutOfSync), sync waves progression, drift detection.
workload-apm-correlation Workload APM & CI/CD GitOps Correlation Request throughput (req/sec), p50/p90/p95/p99 latency percentiles with OTel Exemplars, HTTP 5xx error budget, JVM heap & GC pause metrics, correlated CI/CD deployment annotations.

6. Centralized Configuration Architecture: config/environments.env vs jenkins-git-parameter-global-vars

An enterprise-grade cloud-native platform maintains a strict separation between Platform Bootstrap (Day-1 IaC) and Dynamic Workload / Cluster Inventory Governance (Day-2 GitOps SSOT).

Lifecycle & Integration Flow

🏛️ Click to expand: Platform IaC vs Centralized Governance Integration Flow Diagram 
flowchart TB
    subgraph Day1["1. Platform Bootstrap & IaC (jenkins-git-parameter)"]
        direction TB
        EnvFile["📄 config/environments.env<br/>• GLOBAL_VARS_REPO_URL<br/>• OCP_CLUSTER_DEV/STG/PRD<br/>• OCP_APPS_DOMAIN"]
        DeployScript["⚙️ deploy.sh / Makefile"]
        JCasC["📦 JCasC (jenkins-jcasc.yaml)<br/>• Injects env.GLOBAL_VARS_REPO"]

        EnvFile --> DeployScript
        DeployScript --> JCasC
    end

    subgraph Day2["2. Centralized Governance SSOT (jenkins-git-parameter-global-vars)"]
        direction TB
        ClustersDir["📁 clusters/<br/>• ocp-dev-cluster.yaml<br/>• ocp-staging-cluster.yaml<br/>• ocp-prod-cluster.yaml"]
        EnvsDir["📁 environments/<br/>• dev.yaml<br/>• staging.yaml<br/>• prod.yaml"]
        AppInventory["📁 apps/<br/>• applications-inventory.yaml"]
    end

    subgraph Runtime["3. Jenkins CI/CD & ArgoCD Runtime Execution"]
        direction TB
        JobDSL["📜 Job DSL (pipelines-cd.groovy)<br/>• gitParameter queries GLOBAL_VARS_REPO"]
        UIForm["📋 UI Dropdown: GLOBAL_VARS_REVISION<br/>• Real-time git ls-remote resolution"]
        PipelineStage["🚀 Jenkinsfile.release-orchestrator<br/>• Dynamic checkout of selected branch/tag<br/>• Parses cluster YAMLs & triggers ArgoCD"]

        JCasC --> JobDSL
        JobDSL --> UIForm
        UIForm --> PipelineStage
    end

    Day2 -.->|"Queried dynamically in UI"| UIForm
    Day2 -.->|"Checked out at runtime"| PipelineStage
Loading

📋 Separation of Platform IaC vs. Centralized Governance:

  • Day 1 Platform IaC (jenkins-git-parameter): Provisions Jenkins controller, JCasC, Job DSL scripts, and cluster-wide Helm charts.
  • Day 2 Governance (jenkins-git-parameter-global-vars): Holds environment variables, OpenShift cluster inventories, Helm values, and Vault secret references.
  • Pipeline Synchronization: CD Release Orchestrators query global configuration to dynamically inject environment parameters at runtime.

💡 Architectural Summary & Conclusion: Decoupling platform infrastructure code from environmental configuration allows central platform teams to enforce governance policies across microservices without modifying core Jenkins automation code.

Parameter Binding & Schema Mapping

Dimension config/environments.env (Platform IaC) jenkins-git-parameter-global-vars (Workload SSOT)
Primary Role Day-0 / Day-1 Platform deployment, Helm version pinning, namespace provisioning. Day-2 Workload governance, cluster inventory, environment parameter overlays.
Consuming Actors Platform Engineers, Bash scripts (deploy.sh, destroy.sh), Makefile. Developers, Release Managers, Jenkins gitParameter UI, CD Orchestrators.
Modification Frequency Infrequent (Cluster upgrades, Helm chart updates, new platform tools). High (Daily feature branches, environment config tweaks, new microservices).
Failure Blast Radius High (Platform-wide provisioning disruption if misconfigured). Isolated (Targeted to a specific application or promotional environment).
Git Branching Policy Trunk-based (main), changes deployed via infrastructure maintenance windows. Multi-branch & Semantic Tags (main, feature/*, env/staging-override, v*.*.*-release).

Why Separation of Concerns Protects Production

  1. Zero-Downtime Governance Changes: Adding a new cluster target (clusters/ocp-dr-cluster.yaml), updating microservice memory limits (environments/prod.yaml), or adding a new application to apps/applications-inventory.yaml requires zero changes and zero restarts on Jenkins, ArgoCD, or JCasC.
  2. Immutable Platform, Mutable Workloads: The Jenkins controller and JCasC configuration remain static and secure. Operators interact with jenkins-git-parameter-global-vars via standard Pull Requests, enabling review gates, branch protections, and compliance audits before a configuration is released.

Repository Structure

jenkins-git-parameter/
├── README.md                                # Architectural analysis, documentation, and references
├── Makefile                                 # Platform lifecycle commands (deploy, destroy, lint, status)
├── deploy.sh                                # 1-Click deployment automation script
├── destroy.sh                               # 1-Click decommissioning script
├── reinstall.sh                             # Clean reinstall orchestration
├── config/
│   ├── clusters.yaml                        # Multi-cluster topology definition (DEV, STAGE, PROD)
│   └── environments.env                     # Pinned version matrix and environment parameters
├── helm/
│   ├── jenkins/
│   │   ├── Chart.yaml                       # Jenkins Helm chart wrapper
│   │   ├── values.yaml                      # Pinned base values & plugin list
│   │   ├── values-openshift.yaml            # OpenShift 4.20+ overrides (SCC, Routes, JCasC mounts)
│   │   └── plugins.txt                      # Pinned plugin manifest with exact version numbers
│   ├── argocd/
│   │   └── values-argocd-3.5.yaml           # ArgoCD 3.5 Helm values & ApplicationSet controller
│   └── observability/
│       ├── otel-collector-values.yaml       # OpenTelemetry Collector configuration
│       ├── grafana-values.yaml              # Grafana datasources & dashboard providers
│       └── prometheus-values.yaml           # Prometheus scrape configurations
├── jcasc/
│   ├── jenkins-jcasc.yaml                   # Jenkins Configuration as Code (Security, OTel, Seed Job)
│   ├── pod-templates.yaml                   # Ephemeral Kubernetes Agent pod templates
│   └── github-app-credentials.yaml          # Automated short-lived GitHub App authentication
├── jobdsl/
│   ├── seed-job.groovy                      # Master Seed Job provisioning all pipelines
│   ├── pipelines-ci.groovy                  # Job DSL for Application CI pipelines (Patterns 1 & 2)
│   └── pipelines-cd.groovy                  # Job DSL for CD Release Orchestrators (Global Vars Git Parameter)
├── jenkinsfiles/
│   ├── ci/
│   │   ├── Jenkinsfile.app-java-maven                 # Pattern 2: Declarative CI pipeline for Java 21
│   │   ├── Jenkinsfile.app-angular                    # Pattern 2: Declarative CI pipeline for Angular 18+
│   │   ├── Jenkinsfile.app-java-maven-dual-dropdown   # Pattern 1: Dual-Dropdown Multi-Remote SCM pipeline
│   │   └── Jenkinsfile.app-angular-dual-dropdown      # Pattern 1: Dual-Dropdown Angular pipeline
│   └── cd/
│       ├── Jenkinsfile.release-orchestrator           # Pattern 2: Multi-Cluster Release & Promotion pipeline
│       └── Jenkinsfile.hotfix-deploy                  # Fast-track emergency hotfix pipeline
├── shared-library/
│   ├── vars/
│   │   ├── argoAppSync.groovy               # Step: Trigger & wait for ArgoCD 3.5 sync
│   │   ├── skopeoPromote.groovy             # Step: Promote container image between cluster registries
│   │   ├── gitopsCommit.groovy              # Step: Update GitOps environment repository
│   │   ├── otelLogEvent.groovy              # Step: Annotate OpenTelemetry spans
│   │   ├── cosignSign.groovy                # Step: Cosign container image signing & SLSA 3 attestation
│   │   └── sbomGenerate.groovy              # Step: Syft SBOM generation & Cosign attestation
│   └── src/com/nubenetes/gitops/
│       └── ClusterEnvironment.groovy       # Strongly typed Groovy cluster model
├── security/
│   ├── external-secrets-operator/
│   │   ├── cluster-secret-store-vault.yaml  # Vault HashiCorp ClusterSecretStore
│   │   └── external-secret-template.yaml    # ExternalSecret custom resource definition
│   └── openshift-image-signature-policy.yaml# OpenShift MachineConfig signature verification policy
├── argocd-apps/
│   ├── root-app-of-apps.yaml                # ArgoCD Root App-of-Apps pattern
│   ├── applicationset-clusters.yaml         # Multi-Cluster ApplicationSet generator
│   ├── applicationset-pull-request-preview.yaml # Dynamic ephemeral PR preview generator
│   └── apps/
│       ├── dev/jhipster-microservice.yaml
│       ├── dev/grafana-observability.yaml       # ArgoCD GitOps Application for Grafana 13.2.0
│       ├── staging/jhipster-microservice.yaml
│       └── prod/jhipster-microservice.yaml
├── sample-apps/
│   ├── jhipster-microservice/               # PoC Java 21 / Spring Boot 3 microservice
│   │   ├── Dockerfile                       # Multi-stage container build
│   │   ├── pom.xml
│   │   ├── src/...
│   │   ├── k8s/                             # Kustomize base & overlays (dev, staging, prod)
│   │   └── rollout/                         # Argo Rollouts progressive delivery manifests
│   │       ├── rollout-canary-jhipster.yaml
│   │       ├── analysis-template-prometheus.yaml
│   │       └── services-active-canary.yaml
│   └── nubenetes-global-vars/               # Centralized Global Configuration template reference
│       ├── apps/applications-inventory.yaml
│       ├── clusters/
│       │   ├── ocp-dev-cluster.yaml
│       │   ├── ocp-staging-cluster.yaml
│       │   └── ocp-prod-cluster.yaml
│       └── environments/
│           ├── dev.yaml
│           ├── staging.yaml
│           └── prod.yaml
├── observability/
│   └── dashboards/
│       ├── jenkins-performance-otel.json    # Grafana Dashboard: Pipeline & OTel Spans
│       ├── argocd-gitops-sync.json          # Grafana Dashboard: ArgoCD GitOps Sync & Health
│       └── app-full-stack-correlation.json  # Grafana Dashboard: Workload APM & CI/CD Correlation
└── scripts/
    ├── ocp-setup-scc.sh                     # Configure OpenShift SCC & RBAC
    ├── setup-argocd-clusters.sh             # Register STAGING and PROD clusters in ArgoCD
    └── generate-tokens.sh                   # Generate auth secrets and tokens

Quick Start: 1-Click Automated Deployment

Prerequisites

  • OpenShift 4.20+ CLI (oc or kubectl) logged in with cluster administrator permissions (cluster-admin).
  • Helm 3.14+.
  • Bash shell.

1. Deploy the Complete Platform

To automatically provision all namespaces, security context constraints, JCasC ConfigMaps, OpenTelemetry Collector, Prometheus, Grafana, ArgoCD 3.5, and Jenkins Controller with the Seed Job:

git clone https://github.com/nubenetes/jenkins-git-parameter.git
cd jenkins-git-parameter
chmod +x deploy.sh
./deploy.sh

Or using Make:

make deploy

2. Verify Platform Endpoints

After deployment completes, the script outputs the operational routes:

Service URL / Route Credentials
Jenkins Master https://jenkins-jenkins.apps.ocp-dev.nubenetes.internal admin / admin123!
ArgoCD 3.5 UI https://argocd-server.apps.ocp-dev.nubenetes.internal admin / (retrieved via secret)
Grafana 13.2.0 Dashboards http://grafana.observability.svc.cluster.local:3000 admin / admin123!
OTel Collector http://otel-collector.observability.svc.cluster.local:4317 OTLP gRPC endpoint

3. Triggering Pipelines & GLOBAL_VARS_REVISION Selection

This platform provides flexible ways to select and propagate the GLOBAL_VARS_REVISION (branch, tag, or commit of the centralized jenkins-git-parameter-global-vars repository) across both CI and CD pipelines.

Scenario A: Triggering from Pipeline 01 (01-CI-Build-Pipelines)

When starting from the application CI build pipeline, developers select their application revision and specify which global configuration branch should be targeted downstream.

  1. Navigate to Jenkins UI $\rightarrow$ 01-CI-Build-Pipelines $\rightarrow$ jhipster-microservice-ci-build.
  2. Click "Build with Parameters" in the sidebar.
  3. Configure the build parameters:
    • APP_GIT_REVISION (Git Parameter Dropdown): Select the application Git branch, tag, or PR from jhipster-microservice (e.g. main, feature/payment-v2, v2.1.0).
    • TRIGGER_CD_RELEASE (Checkbox, default true): Auto-triggers Pipeline 02 upon successful build and vulnerability scan.
    • GLOBAL_VARS_BRANCH (Text input, default main): Enter the branch, tag, or commit of jenkins-git-parameter-global-vars to use (e.g. main, release-2026.1, staging-env-v3).
    • TARGET_ENVIRONMENT (Choice Dropdown): Select initial deployment environment (dev, staging, prod).
  4. Click Build.

How Parameter Propagation Works: Pipeline 01 compiles the code, builds the container image with a unique SHA-based tag (e.g. 2.1.0-42-a1b2c3d), pushes it to the OCP DEV registry, and automatically passes GLOBAL_VARS_BRANCH as GLOBAL_VARS_REVISION to Pipeline 02:

build job: '02-CD-Release-Orchestrators/multi-cluster-release-orchestrator',
    parameters: [
        string(name: 'GLOBAL_VARS_REVISION', value: params.GLOBAL_VARS_BRANCH ?: 'main'),
        string(name: 'APP_NAME', value: env.APP_NAME),
        string(name: 'IMAGE_TAG', value: env.CALCULATED_TAG),
        string(name: 'TARGET_ENVIRONMENT', value: params.TARGET_ENVIRONMENT ?: 'dev'),
        booleanParam(name: 'AUTO_PROMOTE_TO_STAGING', value: true),
        booleanParam(name: 'REQUIRE_PROD_APPROVAL', value: true),
        string(name: 'TRIGGERED_BY', value: 'CI_PIPELINE'),
        string(name: 'CHANGE_REQUEST_ID', value: "CI-AUTO-${BUILD_NUMBER}")
    ],
    wait: false

Scenario B: Triggering Directly from Pipeline 02 (02-CD-Release-Orchestrators)

When operators, release engineers, or DevOps teams want to deploy or promote an existing immutable container image against a specific configuration branch:

  1. Navigate to Jenkins UI $\rightarrow$ 02-CD-Release-Orchestrators $\rightarrow$ multi-cluster-release-orchestrator.
  2. Click "Build with Parameters".
  3. Configure the parameters:
    • GLOBAL_VARS_REVISION (Interactive Git Parameter Dropdown with live search):
      • The Jenkins Git Parameter plugin dynamically queries https://github.com/nubenetes/jenkins-git-parameter-global-vars.git in real-time.
      • You can scroll or type to search all remote branches and tags (e.g. main, v2.1.0-release, env/staging-hotfix).
    • APP_NAME (Choice Dropdown): Select jhipster-microservice or angular-frontend.
    • IMAGE_TAG (Text input): Enter the pre-built image tag to deploy/promote (e.g. 2.1.0-42-a1b2c3d).
    • TARGET_ENVIRONMENT (Choice): Select dev, staging, prod, or full-promotion-chain.
    • AUTO_PROMOTE_TO_STAGING / REQUIRE_PROD_APPROVAL: Toggle automated stage gates.
  4. Click Build.

How Configuration Checkout Works: In the first stage (Initialize & Resolve Global Configuration), Pipeline 02 checks out the global variables repository at the exact branch/tag you selected:

dir('jenkins-git-parameter-global-vars') {
    checkout([
        $class: 'GitSCM',
        branches: [[name: "${params.GLOBAL_VARS_REVISION}"]],
        userRemoteConfigs: [[url: 'https://github.com/nubenetes/jenkins-git-parameter-global-vars.git']]
    ])
}

Scenario C: Triggering via Jira Forms, CMDB (Assets) & ITSM / Backstage IDP

In enterprise environments, production deployments and cross-cluster promotions are rarely triggered manually by individual developers clicking buttons in a CI tool. Instead, organizations enforce structured, auditable IT Service Management (ITSM) workflows using Jira Service Management (JSM) Forms, Jira Assets / CMDB, ServiceNow, or Backstage Developer Portals.

🎫 Click to expand: Jira ITSM & Backstage IDP Self-Service Architecture Diagram 
flowchart TB
    subgraph SelfServicePortals["1. Governed Self-Service & Developer Portals"]
        direction LR
        subgraph JiraTrack["ITSM & Governance Track"]
            JiraUser["👨‍💻 Release Engineer"]
            JiraForm["📝 Jira Service Mgmt Form<br/>(App, Version & Env)"]
            CMDB[("🗄️ Jira CMDB / Assets<br/>(Apps, Clusters, CAB)")]
            CAB["🛡️ Change Advisory Board<br/>(CAB Approval Gate)"]
            JiraWebhook["⚡ Jira Webhook Trigger"]
        end

        subgraph BackstageTrack["Developer Portal Track"]
            DevUser["👩‍💻 Application Developer"]
            BackstageUI["🏛️ Backstage IDP Portal<br/>(Software Template)"]
            BackstageCatalog[("📚 Backstage Catalog<br/>(Components & APIs)")]
            BackstagePlugin["⚡ Backstage Plugin<br/>(API Dispatch)"]
        end
    end

    subgraph CI_CD_Layer["2. CI/CD Release Orchestration Engine"]
        JenkinsCD["🚀 02-CD-Release-Orchestrator<br/>(Git Param: Global Vars)"]
        GlobalVars[("📦 Global Variables Repo<br/>(Environment Configs)")]
    end

    subgraph DeploymentObservabilityLayer["3. Multi-Cluster GitOps & Telemetry"]
        Argo["🐙 ArgoCD 3.5 Controller<br/>(Multi-Cluster Sync Waves)"]
        OCP["☸️ OpenShift 4.20+ Clusters<br/>(DEV ➔ STAGING ➔ PROD)"]
        OTel["📊 OpenTelemetry & Grafana<br/>(Traces, Metrics & Logs)"]
    end

    JiraUser -->|Submit Release Form| JiraForm
    JiraForm <-->|Query CMDB CIs| CMDB
    JiraForm -->|Request CAB Approval| CAB
    CAB -->|Approve Change| JiraWebhook
    JiraWebhook -->|POST /buildWithParameters| JenkinsCD

    DevUser -->|Trigger Self-Service| BackstageUI
    BackstageUI <-->|Fetch Catalog Metadata| BackstageCatalog
    BackstageUI -->|Dispatch API Call| BackstagePlugin
    BackstagePlugin -->|POST /buildWithParameters| JenkinsCD

    JenkinsCD -->|Clone Selected Revision| GlobalVars
    JenkinsCD -->|Promote & Sync GitOps| Argo
    Argo -->|Apply Manifests| OCP
    JenkinsCD -.->|Emit W3C Trace Spans| OTel
    JenkinsCD -.->|Close Jira Ticket| JiraForm
    JenkinsCD -.->|Update Status in UI| BackstageUI
Loading

📋 Governed Self-Service Integration Steps:

  • Backstage IDP Scaffolding: Developers select software templates in Backstage, which triggers Jenkins Pipeline 02 via REST API.
  • ITSM Governance Gate: Jira Service Management / ServiceNow Change Requests invoke Pipeline 02 upon formal CAB approval.
  • Downstream Deployment: Pipeline 02 executes Skopeo image promotion and ArgoCD synchronization across target clusters.

💡 Architectural Summary & Conclusion: Parameterized CD pipelines seamlessly integrate into enterprise developer portals and ITSM tools, transforming Jenkins into a governed execution engine for self-service operations.

Why Jira Forms + CMDB is the Industry Standard for Cloud-Native Releases

  1. Centralized Configuration Management Database (CMDB) as Single Source of Truth:

    • Modern IT organizations manage hundreds of microservices. The CMDB maintains a single source of truth for Application Metadata, Business Criticality (Tier 1 vs Tier 3), Regulatory Scope (PCI-DSS, HIPAA, SOX, GDPR), Data Classification, and Assigned Service Owners.
    • Connecting Jira Forms to CMDB Assets ensures developers only select validated, registered applications, compliant target clusters, and authorized environment variables.

  2. Schema-Driven, Governed Self-Service via Jira Forms:

    • Rather than providing direct cluster admin or Jenkins master access to developers, engineers fill out an intuitive Jira Release Request Form.
    • When the user selects an application (e.g. jhipster-microservice), Jira's schema-driven form dynamically queries the CMDB to auto-populate:
      • The approved GitOps configuration repository (jenkins-git-parameter-global-vars).
      • The target OpenShift cluster topology (ocp-dev, ocp-staging, ocp-prod).
      • The required approval gates (e.g., automated QA lead sign-off for STAGING, automated CAB approval for standard PROD changes, or VP approval for emergency hotfixes).

  3. Segregation of Duties (SoD) & Regulatory Audit Compliance:

    • Regulated industries require strict Segregation of Duties (the engineer who writes the code cannot be the sole approver of the production release).
    • Jira Forms + CMDB enforce immutable change records where the Jira Ticket ID (CHG-2026-98124) tracks:
      • Exact Git commit SHA and Container Image Digest.
      • Selected Global Configuration branch/tag (GLOBAL_VARS_REVISION).
      • Test execution evidence, security scan (Trivy/SonarQube) results, and CAB approval timestamps.

  4. Automated Webhook Dispatch to Jenkins with Parameter Injection:

    • When the Jira Change Request reaches the Approved state, a Jira Automation Rule fires an HTTP POST webhook to Jenkins' buildWithParameters endpoint, injecting the form inputs directly:
# Jira Automation / ServiceNow REST API Dispatch Payload
curl -X POST "https://jenkins-jenkins.apps.ocp-dev.nubenetes.internal/job/02-CD-Release-Orchestrators/job/multi-cluster-release-orchestrator/buildWithParameters" \
  --user "ci-service:service123!" \
  --data-urlencode "GLOBAL_VARS_REVISION=v2.1.0-release" \
  --data-urlencode "APP_NAME=jhipster-microservice" \
  --data-urlencode "IMAGE_TAG=2.1.0-42-a1b2c3d" \
  --data-urlencode "TARGET_ENVIRONMENT=full-promotion-chain" \
  --data-urlencode "AUTO_PROMOTE_TO_STAGING=true" \
  --data-urlencode "REQUIRE_PROD_APPROVAL=true" \
  --data-urlencode "TRIGGERED_BY=JIRA_FORMS_CMDB" \
  --data-urlencode "CHANGE_REQUEST_ID=CHG-2026-98124"
  1. Closed-Loop Feedback & Automated Ticket Closure:
    • During pipeline execution, Jenkins publishes progress comments to the Jira issue via the Jira REST API.
    • Upon successful multi-cluster deployment and ArgoCD health verification, Jenkins updates the Jira issue with the Skopeo image promotion digest, OpenShift Route URL, and OpenTelemetry trace link, automatically transitioning the Jira ticket to Closed / Resolved.

Parameter Selection Matrix

Trigger Entrypoint Parameter Name UI Element Type Target Repository Default Value Propagation Logic
Pipeline 01 (CI) APP_GIT_REVISION Git Parameter Dropdown Application Repo main Used to build source code
Pipeline 01 (CI) GLOBAL_VARS_BRANCH Text Field with Default jenkins-git-parameter-global-vars main Forwarded to Pipeline 02 as GLOBAL_VARS_REVISION
Pipeline 02 (CD) GLOBAL_VARS_REVISION Git Parameter Dropdown jenkins-git-parameter-global-vars main Cloned directly to drive GitOps manifests
External API / ITSM GLOBAL_VARS_REVISION HTTP POST Parameter jenkins-git-parameter-global-vars Explicit String Injected into Pipeline 02 runtime

🛡️ Enterprise Production Hardening & Advanced Best Practices

To transition this platform into a mission-critical, regulated enterprise production environment, we have implemented 5 advanced architectural capabilities:

┌───────────────────────────────────────────────────────────────────────────────────┐
│                    ENTERPRISE ARCHITECTURE ADVANCED CAPABILITIES                  │
├───────────────────────────────┬───────────────────────────────────────────────────┤
│ 1. Supply Chain Security      │ Cosign Keyless Signing, Syft SBOM & SLSA Level 3  │
│ 2. Zero-Trust Secrets         │ External Secrets Operator (ESO) + HashiCorp Vault │
│ 3. Dynamic PR Preview Envs    │ ArgoCD 3.5 ApplicationSet Pull Request Generator  │
│ 4. Progressive Delivery       │ Argo Rollouts (Canary + Prometheus Metric Auto)   │
│ 5. Automated Identity         │ GitHub App Short-Lived Token Rotation via JCasC   │
└───────────────────────────────┴───────────────────────────────────────────────────┘

1. Supply Chain Security: Cosign Image Signing & SBOM Attestation (SLSA 3)

In multi-cluster promotion (ocp-dev $\rightarrow$ ocp-staging $\rightarrow$ ocp-prod), promoting raw container images without cryptographic signatures creates a software supply chain vulnerability.

🔏 Click to expand: Cosign Image Signing & SLSA Level 3 Attestation Flow Diagram 
flowchart LR
    CI["🏗️ CI Build (Maven/Node)"]
    Trivy["🛡️ Trivy Scan & Syft SBOM"]
    Cosign["🔏 Cosign Signature & SLSA Attestation"]
    Registry["📦 OCP DEV Registry"]
    Skopeo["🚀 Skopeo Cross-Cluster Promotion"]
    ClusterPolicy["🛡️ OpenShift Image Signature Policy"]
    ProdEnv["☸️ OCP Production Workload"]

    CI --> Trivy
    Trivy --> Cosign
    Cosign -->|Signed Digest & SBOM| Registry
    Registry -->|Promote Digest + Sig| Skopeo
    Skopeo --> ClusterPolicy
    ClusterPolicy -->|Signature Verified Valid| ProdEnv
Loading

📋 Supply Chain Security & Signing Chain:

  • Artifact Creation: Maven build packages container image.
  • Cosign Signing: Ephemeral agent signs the container digest using private keys stored in HashiCorp Vault.
  • Attestation Generation: Generates and attaches CycloneDX SBOM and in-toto provenance metadata to the OpenShift registry.
  • Admission Verification: OpenShift Image Signature Policy enforces signature verification before pod startup.

💡 Architectural Summary & Conclusion: Enforcing SLSA Level 3 supply chain security guarantees that only cryptographically verified container images built by authorized Jenkins pipelines can run in production.

Implemented Components:

2. Zero-Trust Secret Management: External Secrets Operator (ESO) + HashiCorp Vault

Plaintext database credentials, API tokens, and private keys should never be committed into Git repositories.

🔐 Click to expand: External Secrets Operator & Vault Synchronization Flow Diagram 
flowchart LR
    Vault[("🔐 HashiCorp Vault<br/>(Raw Secrets & Passwords)")]
    GlobalVars[("📦 Global Vars Repo<br/>(ExternalSecret Custom Resource)")]
    ESO["⚡ External Secrets Operator<br/>(OpenShift Cluster Controller)"]
    K8sSecret["🔑 Native Kubernetes Secret<br/>(Decrypted in Memory)"]
    Pod["☸️ Application Pod"]

    GlobalVars -->|GitOps Sync Manifest| ESO
    ESO <-->|Kubernetes SA Token Auth| Vault
    ESO -->|Creates / Rotates Secret| K8sSecret
    K8sSecret -->|Mounted as Env / Volume| Pod
Loading

📋 Secret Management & Dynamic Injection:

  • Central Secret Storage: HashiCorp Vault stores raw database passwords and API tokens securely.
  • External Secrets Operator (ESO): Synchronizes secrets from Vault into native Kubernetes Secret resources based on ExternalSecret custom resources.
  • Pod Injection: Workload pods consume secrets as environment variables or volume mounts without hardcoding credentials in Git.

💡 Architectural Summary & Conclusion: ESO with HashiCorp Vault eliminates hardcoded secrets across all Git repositories, ensuring centralized secret rotation and dynamic access control.

Implemented Components:

3. Dynamic Ephemeral PR Preview Environments with ArgoCD ApplicationSet

Supercharges Pattern 1 (Inner-Loop) by automatically provisioning isolated preview namespaces for active GitHub Pull Requests, and deleting them upon PR merge/close.

⚡ Click to expand: Ephemeral PR Preview Environment Provisioning Flow Diagram 
flowchart LR
    Dev["👨‍💻 Developer"]
    PR["📝 GitHub Pull Request #42"]
    AppSet["⚡ ArgoCD ApplicationSet<br/>(Pull Request Generator)"]
    PreviewNS["☸️ Ephemeral Preview Namespace<br/>(pr-42-preview)"]
    Route["🌐 Preview Route URL<br/>pr-42-jhipster.apps.ocp-dev..."]

    Dev -->|Opens PR| PR
    PR -->|Webhook Notification| AppSet
    AppSet -->|Provisions Namespace & Deploy| PreviewNS
    PreviewNS -->|Generates Endpoint| Route
    PR -.->|PR Merged or Closed| AppSet
    AppSet -.->|Automatically Prunes & Deletes| PreviewNS
Loading

📋 Dynamic Ephemeral Preview Provisioning Flow:

  • PR Webhook: Opening a GitHub PR triggers Jenkins CI to build and push an ephemeral preview container image.
  • ArgoCD Dynamic Sync: ArgoCD provisions an isolated namespace pr-preview-<id> on the DEV cluster.
  • Automated Cleanup: Closing or merging the pull request automatically tears down the namespace and destroys temporary cloud resources.

💡 Architectural Summary & Conclusion: Dynamic preview environments enable automated visual and integration testing for every pull request, drastically accelerating review velocity before merging to main.

Implemented Component:

4. Progressive Delivery with Argo Rollouts (Metric-Driven Canary)

Replaces risky all-at-once rolling updates in STAGING and PROD with automated canary analysis that continuously measures Prometheus / OpenTelemetry telemetry.

🐣 Click to expand: Argo Rollouts Canary Traffic Splitting & Metric Analysis Diagram 
flowchart LR
    Ingress["🌐 OpenShift Route Traffic"]
    Canary["🐣 Canary Pods (10% - 25% - 50%)<br/>v2.1.0 (New Image)"]
    Stable["🛡️ Stable Pods (90% - 75% - 50%)<br/>v2.0.8 (Current Image)"]
    Prometheus["📊 Prometheus / OTel Metrics<br/>(5xx Rate < 0.1%, p99 < 250ms)"]
    RolloutCtrl["⚡ Argo Rollouts Controller"]

    Ingress -->|Traffic Split| Canary
    Ingress -->|Traffic Split| Stable
    Canary -.->|Emits Telemetry| Prometheus
    Prometheus -.->|Metric Evaluation| RolloutCtrl
    RolloutCtrl -->|Pass: Promote to 100%<br/>Fail: Automated Rollback| Ingress
Loading

📋 Progressive Delivery Execution Steps:

  • Traffic Routing: Argo Rollouts splits OpenShift route traffic, directing 20% to canary pods and 80% to stable pods.
  • Metric SLA Validation: Automated analysis queries Prometheus metrics (error rates and latency thresholds).
  • Automated Promotion / Rollback: If SLAs pass, traffic scales to 100%; if errors occur, instant automated rollback protects live users.

💡 Architectural Summary & Conclusion: Progressive canary delivery with automated metric analysis eliminates deployment risk, ensuring zero-downtime releases and automated safety rollbacks.

Implemented Components:

5. Automated Credential Rotation: GitHub App Authentication via JCasC

Replaces static, long-lived Personal Access Tokens (PATs) with fine-grained GitHub Apps that issue short-lived, self-rotating installation tokens (1-hour lifespan).

Implemented Component:

Strategic Roadmap: Impact vs. Implementation Effort

Recommendation Target Area Business Value Implemented Artifacts
Cosign + SBOM (SLSA 3) Supply Chain Security Cryptographic proof of origin & automated vulnerability tracking cosignSign.groovy, openshift-image-signature-policy.yaml
External Secrets Operator Zero-Trust Secrets Eliminates plaintext secrets from Git repositories cluster-secret-store-vault.yaml, secrets/
GitHub App Auth Identity & Security Short-lived, self-rotating credentials with least privilege github-app-credentials.yaml
ArgoCD PR Generator Developer Experience 1-click preview environments per pull request applicationset-pull-request-preview.yaml
Argo Rollouts (Canary) Production Resiliency Automated metric analysis with zero-downtime rollback rollout-canary-jhipster.yaml, analysis-template-prometheus.yaml

Decommissioning & Reinstallation

Clean Decommission

To cleanly delete all Helm releases, ConfigMaps, and namespaces:

./destroy.sh
# or
make destroy

Full Reinstallation

To perform a complete wipe and redeployment:

./reinstall.sh
# or
make reinstall

Video Walkthroughs & Architecture References (YouTube)

Architectural deep dives, video walkthroughs, and technical shorts for jenkins-git-parameter and multi-cluster GitOps promotion are hosted on the Nubenetes YouTube Channel (@nubenetes).

📂 Full-Length Technical Deep Dives & Explanations
Jenkins Pure GitOps: Decoupling CI from ArgoCD Multi-Cluster CD
  • 🔗 Link: https://www.youtube.com/watch?v=qntcMvzBx4w
  • 🌐 Language: English (Original Audio)
  • ⏱️ Duration: 7:52
  • 🏷️ Domain: CI/CD Decoupling, SCM Blind Spot & Evolution to Pure GitOps
  • 📝 Full Description:

🚀 Architectural breakdown of the SCM pre-execution render paradox when using Jenkins git-parameter plugins across multiple repositories, and how shifting to a pure GitOps model permanently solves it.

📌 Core Discussion Points: • The Pre-Execution Render Paradox: Why UI parameter dropdowns fail before checkout in multi-repo topologies. • Jenkins as a Pure Build Factory: Compiling immutable containers, scanning with Trivy, and cryptographically signing with Cosign (SLSA Level 3). • ArgoCD Multi-Cluster Synchronization: Replacing push deployment scripts with pull-based internal controllers. • Dynamic Tuning without Throttling: Eliminating pod-level CPU limits to prevent Linux CFS bandwidth throttling on multi-threaded runtimes.

📂 Architecture Video Shorts & Guides
1. How Jenkins Git Parameters Protect Production
  • 🔗 Link: https://www.youtube.com/shorts/0Zd7zNSae48
  • 🌐 Language: English (Original Audio)
  • ⏱️ Duration: 1:13
  • 🏷️ Domain: Immutable Container Promotion & Jenkins Parameterization
  • 📝 Full Description:

🚀 Why rebuilding code for production risks sneaking in subtle bugs, and how immutable container promotion with Jenkins Git Parameters prevents it.

📌 Key Concepts: • Strict blocking of recompilation during production release runs. • Dynamic scanning of repository tags with the Jenkins Git Parameter plugin. • Promoting pre-validated container images between dev and prod registries via Skopeo. • Bumping GitOps manifests for bit-for-bit parity.

🔗 Architectural Blueprint (This Repository): https://github.com/nubenetes/jenkins-git-parameter 🔗 Next Architectural Stage (Pure GitOps without Jenkins parameter plugins): https://github.com/nubenetes/jenkins-without-git-parameter

2. The Shift to Pure GitOps Deployments

🚀 Why leading platform engineering teams eliminate manual UI deploy buttons, moving from fragile multi-repo parameter dropdowns to webhook-triggered CI and pull-based ArgoCD synchronization.

3. How Pure GitOps Reverses Deployments

🚀 How GitOps completely inverts traditional deployment architecture: replacing fragile external push scripts with an internal ArgoCD controller pulling state from Git without exposing cluster credentials.

References & Evidence Links

The design patterns and technical configurations implemented in this repository are based on official documentation, security frameworks, and industry standards:

  1. Jenkins Core, IaC & Job DSL:

  2. Red Hat OpenShift 4.20+ Hardening & Networking:

  3. ArgoCD 3.5 & Multi-Cluster GitOps:

  4. Progressive Delivery & Automated Rollouts:

  5. Supply Chain Security & Attestation (SLSA Level 3):

  6. Zero-Trust Secrets & Identity Management:

  7. ITSM, Developer Portals & CMDB Integration:

  8. Observability & Distributed Tracing:

About

Enterprise Jenkins CI/CD & ArgoCD GitOps for OpenShift 4.20+ with OpenTelemetry | Paired with nubenetes/jenkins-git-parameter-global-vars

github.com/nubenetes/jenkins-git-parameter-global-vars

Topics

angularargocdcicddevopsgit-parametergitopsgrafanahelmiacjava21jcascjenkinsjob-dslmulti-clusteropenshiftopentelemetryprometheusskopeospring-boottrivy

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages