From 0a9c9d27685bc30c550c7c3b1682ad07faa7eccd Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:13:24 +0530 Subject: [PATCH 01/15] docs(jenkins): add chapter 1 control-plane and Pipeline internals --- docs/guides/jenkins/q1.md | 109 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 docs/guides/jenkins/q1.md diff --git a/docs/guides/jenkins/q1.md b/docs/guides/jenkins/q1.md new file mode 100644 index 0000000..48a32e4 --- /dev/null +++ b/docs/guides/jenkins/q1.md @@ -0,0 +1,109 @@ +# Jenkins Architecture: Controller, Agents, Queue and Execution Flow + +> **Engineering scenario:** A company operates hundreds of builds across different languages and architectures. Explain what the Jenkins controller owns, what agents execute, and what happens from a webhook to an archived artifact. Where can the entire system fail? + +## Architectural model + +Jenkins is an extensible automation server. Its controller serves the UI and API, handles authentication/authorization, owns job definitions and build records, coordinates Pipeline execution, tracks agent connections, schedules work and integrates plugins. **Executors run work on agents**. Configure zero executors on the built-in controller node in a production design so untrusted build code does not execute alongside its management plane. + +```mermaid +flowchart TD + SCM["SCM webhook / periodic indexing"] --> WEB["Jenkins controller UI/API"] + WEB --> QUEUE["Queue: runnable tasks and resource constraints"] + QUEUE --> PICK{"Eligible online executor?"} + PICK -->|No| WAIT["Queue / dynamic provisioning"] + WAIT --> PICK + PICK -->|Yes| AGENT["Agent JVM and workspace"] + AGENT --> TOOLS["Compiler, tests, container build"] + TOOLS --> STORE["External artifact and test storage"] + AGENT --> LOG["Build log and result events"] + LOG --> HOME[("Controller JENKINS_HOME")] + WEB <--> HOME + ``` + +This is a logical flow; some plugin actions, SCM probes, Groovy orchestration and metadata access execute on the controller. The `agent` directive and `node` step acquire an executor and normally a workspace. A Pipeline does not automatically reserve a single executor for its entire lifetime if it is designed with `agent none` and stage-level agents. + +## Terms that must not be conflated + +| Term | Meaning | Failure signature | +| --- | --- | --- | +| Controller | Coordinates jobs, plugins, users, history and Pipeline graph | UI/API slow, queue/flows stop progressing | +| Node | Registered machine or cloud-provisioned compute | Offline/mismatched labels | +| Agent | Remoting process providing execution on a node | Lost connection or launch failure | +| Executor | Slot for one concurrently allocated task | All busy even while CPU is idle | +| Workspace | Filesystem area for checkout/build on a node | Disk full, cross-job contamination, stale files | +| Queue | Jobs waiting for executor/resources/provisioning | Waiting for next available executor | +| Build record | Console output, artifacts/metadata and status | Storage pressure, slow job listing | + +A label is a scheduling constraint, not a security boundary. Multiple executors on one agent can share filesystem and operating-system identity, so running unrelated untrusted tenants together remains risky even if the jobs have different workspaces. + +### End-to-end sequence + +```mermaid +sequenceDiagram + participant SCM as Source hosting + participant J as Jenkins controller + participant Q as Queue + participant K as Cloud/agent provisioner + participant A as Agent + participant R as Registry + SCM->>J: Webhook (repository/ref update) + J->>J: Validate event / evaluate branch policy + J->>Q: Schedule Pipeline run + Q->>K: Need node with eligible label + K->>A: Provision node or Pod, launch agent + A-->>J: Agent registers and supplies executors + J->>A: Checkout selected revision + A->>A: Build, test and package + A->>R: Publish immutable artifact + A-->>J: Step logs, status and metadata + J->>J: Persist result and notify + ``` + +## Durability and CPS nuance + +Pipeline's Groovy execution model supports persisted program state across some controller restarts. It is **not equivalent** to durable arbitrary agent processes: the step, agent availability, workspace and external commands determine what resumes safely. Pipeline CPS continuation state and build records create controller I/O and CPU demand; doing large data transformations in Groovy on the controller is a common scaling mistake. Offload heavy parsing or dependency operations to agents via `sh` or language-specific executables. + +```groovy +pipeline { + agent none + options { timestamps(); timeout(time: 30, unit: 'MINUTES') } + stages { + stage('Verify') { + agent { label 'linux-small' } + steps { + checkout scm + sh 'make test' + } + } + } +} +``` + +This requires a configured eligible agent and a repository with a real test target. + +## Production incident: agent nodes are online, builds remain queued + +Check **why** an item is queued before adding capacity: labels, node mode, offline cause, executor busy state, throttling/resource locks, or cloud provisioning errors. Increasing controller CPU does not resolve a missing label. Likewise, adding agent Pods is ineffective if the Kubernetes API denies creation or the agent cannot connect to the controller. + +```bash +# Read-only Linux diagnostics on an authorized controller host: +systemctl status jenkins +journalctl -u jenkins --since '15 minutes ago' --no-pager +df -h +df -i +# In the UI: Build Queue -> item -> 'Why is it not running?' +# In Manage Jenkins -> Nodes: labels, online status, executor utilization +``` + +The service commands apply to a systemd deployment, not every container or Helm installation. + +## Trade-offs and platform leadership + +Jenkins is flexible but carries stateful controller, plugin and upgrade responsibilities. Separate trusted release agents from untrusted PR agents; use ephemeral agents where appropriate; define retention and disaster-recovery objectives for the controller. Treat agent capacity and controller throughput as different resource budgets. + +### Spoken explanation + +> “Jenkins has a controller responsible for scheduling, pipeline coordination, auth, plugins and records; the agents execute actual workloads through executors and workspaces. A webhook queues a versioned job, Jenkins matches it to an eligible agent, runs build/test/publish stages, and persists logs and results. I would keep executors off the controller, isolate trust levels across agents, and diagnose queue constraints separately from controller performance. The controller is stateful, so backup, I/O and restart recovery remain essential.” + +**References:** [Using Jenkins agents](https://www.jenkins.io/doc/book/using/using-agents/) · [Managing nodes](https://www.jenkins.io/doc/book/managing/nodes/) · [Scaling Pipelines](https://www.jenkins.io/doc/book/pipeline/scaling-pipeline/) From eb840f8effbe6e7fac351ea604196b89c360b82d Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:13:26 +0530 Subject: [PATCH 02/15] docs(jenkins): add chapter 2 control-plane and Pipeline internals --- docs/guides/jenkins/q2.md | 107 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 docs/guides/jenkins/q2.md diff --git a/docs/guides/jenkins/q2.md b/docs/guides/jenkins/q2.md new file mode 100644 index 0000000..fbceb6f --- /dev/null +++ b/docs/guides/jenkins/q2.md @@ -0,0 +1,107 @@ +# Jenkins Pipeline Syntax, Groovy Execution and Reliable Stage Design + +> **Engineering scenario:** Design a build-and-test Jenkinsfile that is readable, bounded, repeatable and recoverable. Explain how Declarative syntax differs from Scripted Pipeline, and why a Groovy script is not the same as an ordinary shell script. + +## Pipeline concepts from first principles + +A `Jenkinsfile` defines a Pipeline as code. **Declarative Pipeline** offers structured `pipeline { agent; options; environment; stages; post }` syntax with controlled escape to `script { }`. **Scripted Pipeline** exposes a Groovy-based DSL where `node`, `stage` and steps are composed programmatically. Both invoke Jenkins Pipeline steps, not arbitrary Groovy functions running within build containers. + +```mermaid +flowchart TD + DEF["Jenkinsfile from reviewed source"] --> PARSE["Declarative syntax and Pipeline CPS"] + PARSE --> ACQ["Acquire agent / workspace"] + ACQ --> CHECK["Checkout + deterministic validation"] + CHECK --> PAR["Parallel independent test stages"] + PAR --> GATE{"Success?"} + GATE -->|No| POST["Post failure + retention"] + GATE -->|Yes| ART["Create artifact / test evidence"] + ART --> POST + ``` + +### A practical Declarative Jenkinsfile + +```groovy +pipeline { + agent none + options { + timestamps() + timeout(time: 45, unit: 'MINUTES') + disableConcurrentBuilds() + buildDiscarder(logRotator(numToKeepStr: '30')) + } + stages { + stage('Validate') { + agent { label 'linux-small' } + steps { + checkout scm + sh 'set -eu; ./scripts/lint.sh' + stash name: 'verified-source', includes: '**/*', useDefaultExcludes: false + } + } + stage('Test') { + parallel { + stage('Unit') { + agent { label 'linux-small' } + steps { + unstash 'verified-source' + sh './scripts/unit-test.sh' + } + } + stage('Integration') { + agent { label 'linux-large' } + steps { + unstash 'verified-source' + sh './scripts/integration-test.sh' + } + } + } + } + } + post { + always { + echo 'Record result, links and any follow-up actions' + } + failure { + echo 'Notify responsible team by an approved channel' + } + } +} +``` + +**Illustrative only:** real repositories must contain the scripts; each agent needs compatible tools. `stash/unstash` is convenient for **small** file sets, not a substitute for a large artifact repository. For large sources/artifacts, use a registry/object store or re-checkout a fixed SCM revision. The example's broad stash includes can be expensive and should be restricted to what consumers require. + +### Stage options and ordering + +| Directive / step | What it does | Avoid | +| --- | --- | --- | +| `agent none` | No one global executor is reserved | Assuming stages inherit a workspace | +| `when` | Conditional stage execution | Using a PR-supplied parameter as deploy authorization | +| `parallel` / `matrix` | Independent execution variants | Sharing mutable workspace between branches | +| `timeout` | Bounds stage or pipeline time | Treating timeout as proof external side effect failed | +| `retry` | Repeats failed block | Retrying non-idempotent uploads/payment jobs | +| `post` | Cleanup/reporting after stage/pipeline | Swallowing the original error in cleanup | +| `catchError` | Adjusts recorded failure behavior | Marking a required validation failure successful | +| `stash` | Captures files for later steps | Moving gigabytes through controller storage | + +### What CPS changes + +Groovy CPS transforms much Pipeline code into resumable continuations. Local variables surviving pause/resumption generally need serializable values. A Java/Groovy iterator, regex matcher or streaming object held across a step can trigger serialization errors. `@NonCPS` permits some pure transformations without CPS overhead, but **must not call Pipeline steps** such as `sh` or `node`. Keep heavyweight computation on agents and limit controller-side Groovy to orchestration. + +```groovy +@NonCPS +def rankCounts(Map counts) { + counts.entrySet().sort { a, b -> b.value <=> a.value } +} +// Call this pure method with plain data. +// Do not call sh(), node(), or other Pipeline steps inside it. +``` + +### Production failure + +A Pipeline runs an expensive JSON parser in a large Groovy loop, causing high controller CPU while agents remain mostly idle. Moving parsing to a Python/shell command executed on an agent removes controller-side interpreter overhead and produces more testable code. Avoid treating a higher heap as the first and only fix. + +### Spoken explanation + +> “I'd default to Declarative syntax for consistency, define bounded stages, use stage-scoped agents and keep validations isolated from release authority. Parallel stages should share only immutable source revisions, not a mutable workspace. CPS means Pipeline Groovy is an orchestration layer with serialization limits, so I would push real computation into agent processes. Post blocks capture evidence without suppressing failed gates.” + +**References:** [Pipeline syntax](https://www.jenkins.io/doc/book/pipeline/syntax/) · [CPS limitations](https://www.jenkins.io/doc/book/pipeline/cps-method-mismatches/) · [Scaling Pipelines](https://www.jenkins.io/doc/book/pipeline/scaling-pipeline/) From 00c643cb47b2327d7bc18f0009400355bec72cbd Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:14:56 +0530 Subject: [PATCH 03/15] docs(jenkins): add chapter 3 SCM and dynamic agents --- docs/guides/jenkins/q3.md | 97 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 docs/guides/jenkins/q3.md diff --git a/docs/guides/jenkins/q3.md b/docs/guides/jenkins/q3.md new file mode 100644 index 0000000..aca1123 --- /dev/null +++ b/docs/guides/jenkins/q3.md @@ -0,0 +1,97 @@ +# Multibranch Pipelines, Webhooks and Source Trust + +> **Engineering scenario:** Onboard a repository with feature branches and external pull requests. How do Jenkins branch indexing, webhook triggers, credentials and PR trust rules interact, and how do you prevent an unreviewed fork from publishing to production? + +## The three distinct operations + +**Discovery** finds repositories, branches and change requests through a configured SCM source. **Triggering** reacts to a webhook or a scheduled scan and schedules builds. **Checkout** retrieves the code revision or merge result used by the Pipeline. They can fail independently: a webhook can arrive even when a job lacks SCM permission, and a branch can be indexed without a build being authorized to deploy. + +```mermaid +sequenceDiagram + participant S as Git hosting + participant W as Webhook endpoint + participant C as Jenkins SCM source + participant B as Multibranch job + participant A as Isolated agent + S->>W: Signed event about branch/PR + W->>C: Deliver event + C->>S: Query repository, ref and metadata + C->>B: Discover or update branch job + B->>B: Apply trust and build policies + B->>A: Checkout selected commit / merge revision + A->>A: Run permitted PR checks + A-->>B: Test result and commit status + ``` + +Configure webhook authentication/signature verification where the corresponding plugin supports it, maintain least-privileged SCM access, and avoid making scheduled full scans the sole reliability mechanism. Event delivery may be duplicated or delayed. Build exactly the intended SCM revision and show it in the build metadata; moving branch tips should not change an already-approved artifact. + +## Trust boundary: Jenkinsfile versus submitted code + +A fork contributor can modify both source and Jenkinsfile. Plugins providing Multibranch or Organization Folder support implement trust policies that can select a **trusted Jenkinsfile** while testing untrusted code, but the precise behavior depends on SCM source plugin and its version. A trusted Jenkinsfile doesn't make executing attacker-controlled tests safe when its agent has release secrets. + +```mermaid +flowchart TD + EVENT["Branch / change request detected"] --> CLASS{"Source trusted?"} + CLASS -->|Untrusted| CHECK["Restricted, no-secret build"] + CLASS -->|Trusted| MAIN["Normal validation"] + CHECK --> TEST["Build/test with isolated credentials"] + MAIN --> TEST + TEST --> RESULT["Immutable result for specific commit"] + RESULT --> GATE{"Protected branch and authorization?"} + GATE -->|No| END["Report check result only"] + GATE -->|Yes| RELEASE["Separate trusted promotion"] + ``` + +### Example conditions + +```groovy +pipeline { + agent { label 'pr-sandbox' } + stages { + stage('Test source') { + steps { + checkout scm + sh './scripts/test.sh' + } + } + stage('Describe release eligibility') { + when { + allOf { + branch 'main' + not { changeRequest() } + } + } + steps { + echo 'Eligible to request a separately authorized release' + } + } + } +} +``` + +**Important:** `branch 'main'` and `changeRequest()` are not security controls by themselves; anyone who can configure Pipeline steps on an agent holding credentials can abuse those credentials. Release must be a separate constrained identity and policy boundary. + +## Why Organization Folder credentials need special care + +One broad SCM scan credential can unintentionally reach several private repositories. Users able to influence Jenkinsfile content may be able to abuse capabilities exposed by the organization's jobs. Prefer repository-scoped credentials/Apps, explicit folder separation, reviewed SCM source plugin settings and minimal rights to create/manage webhooks. + +## Troubleshooting a missing PR build + +1. Was a webhook received and accepted, or was its signature rejected? +2. Did SCM branch indexing recognize the change request? +3. Does the trusted Jenkinsfile path exist and satisfy branch trust settings? +4. Does Jenkins have permission to query that repository/ref? +5. Was the build filtered by branch discovery, build strategy or duplicate event? +6. Is it scheduled but blocked in the build queue, or actually running? + +Distinguish **no build scheduled** from **build queued**. Check scan logs before debugging executors. + +## Generic incident + +A developer forks a public service repo. A misconfigured job allows forks to access the same registry-push credential as the release branch. An attacker can exfiltrate it from a malicious test step despite console masking. Immediate mitigation is credential revocation and trusted-agent isolation, not a superficial change to the Jenkinsfile. Review every dependent job and token scope. + +### Spoken explanation + +> “I'd separate event delivery, branch discovery and checkout, then review the trust model. Untrusted PRs can run tests only on isolated agents with no release secrets. I'd lock down SCM scanning credentials and treat branch conditions as convenience logic, not authorization. Release promotion needs a protected identity tied to reviewed revisions. When a PR doesn't build, I'd inspect webhook and indexing logs before assuming agent capacity is the problem.” + +**References:** [Multibranch Pipelines](https://www.jenkins.io/doc/book/pipeline/multibranch/) · [Securing organization folders](https://www.jenkins.io/doc/book/security/securing-org-folders-and-multibranch-pipelines/) From b844877d7c9e7e9f1ef0192637978571d173fef8 Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:14:59 +0530 Subject: [PATCH 04/15] docs(jenkins): add chapter 4 SCM and dynamic agents --- docs/guides/jenkins/q4.md | 98 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 docs/guides/jenkins/q4.md diff --git a/docs/guides/jenkins/q4.md b/docs/guides/jenkins/q4.md new file mode 100644 index 0000000..2b79ad2 --- /dev/null +++ b/docs/guides/jenkins/q4.md @@ -0,0 +1,98 @@ +# Kubernetes Agents, Provisioning, Capacity and Workspace Isolation + +> **Engineering scenario:** Jobs queue even though Kubernetes nodes have free CPU. Explain how an ephemeral Jenkins agent is provisioned, why it may never connect, and when to choose static VMs over Pods. + +## Architecture and critical path + +The Jenkins Kubernetes plugin provisions Pods as **inbound agents**. The Pod typically contains a special agent container (commonly named `jnlp`) and optional build containers. The Pod must become schedulable, pull its images, start the remoting process and connect to the Jenkins controller before an executor can perform work. The controller may run outside Kubernetes. + +```mermaid +sequenceDiagram + participant J as Controller + Kubernetes plugin + participant API as Kubernetes API + participant S as Scheduler + participant P as Agent Pod + participant C as Controller remoting endpoint + J->>API: Create Pod from template + API->>S: Pending Pod enters scheduling + S->>P: Bind to node; start containers + P->>C: Inbound agent connection + C-->>J: Online executor + J->>P: Checkout and run steps + P-->>J: Logs and result + J->>API: Delete/reuse Pod per plugin policy + ``` + +Total time-to-executor includes Jenkins queue policy + cloud provisioning + Pod scheduling + image download + startup + connection handshake. Cluster CPU headroom does not help if no Pod can be created due to RBAC, quota, taints, lack of matching nodes, image pull limits or missing network path. + +## Practical Pipeline using a pod template + +```groovy +podTemplate( + containers: [ + containerTemplate(name: 'tools', + image: 'python:3.12-slim', + command: 'sleep', + args: '99d') + ]) { + node(POD_LABEL) { + container('tools') { + checkout scm + sh 'python --version' + sh 'python -m unittest discover -v' + } + } +} +``` + +Requires an installed Kubernetes plugin, configured cloud, inbound agent image (provided by plugin/template), executable `sleep`, and repo tests. It's a **generic example**, not a runnable standalone service. Pin container images to immutable digests in production. + +A more declarative Jenkins Kubernetes agent is also possible using a compatible plugin and `agent { kubernetes { yaml ... } }`. Choose one idiom for consistency, not because one automatically avoids all provisioning or security problems. + +### Workspaces and filesystem + +Containers inside one Pod can share an `emptyDir` workspace. This is temporary and can disappear with the Pod. `stash`/external artifacts must be used for outputs needed by a different stage or node. Avoid PVC reuse between untrusted jobs unless tenant isolation and ownership are designed. A writable Docker socket grants powerful host control; do not mount it into untrusted build Pods. + +## Capacity model + +```text +effective_start_delay = + cloud provisioning + scheduler wait + image pull + agent connect + +sustainable_parallelism <= min( + Jenkins policy/executors, + available eligible agent Pods, + node CPU/memory/ephemeral-storage capacity, + Kubernetes quotas, API provisioning throughput, + downstream test DB/registry/rate limits +) +``` + +Cold image pulls, DNS failures, a controller endpoint unavailable from the cluster, or stalled admission webhook can all cause agents to remain offline while node metrics look healthy. + +## Inspection commands + +```bash +kubectl get pods -n ci -o wide +kubectl get events -n ci --sort-by=.lastTimestamp +kubectl describe pod EXAMPLE_AGENT_POD -n ci +kubectl logs -n ci EXAMPLE_AGENT_POD -c jnlp --tail=100 +kubectl get resourcequota -n ci +kubectl get nodes -L topology.kubernetes.io/zone +``` + +Never dump environment variables from a real agent Pod; they can include temporary authentication material. Use restricted access to CI namespaces because build artifacts and logs may contain sensitive data. + +### Hypothetical outage and recovery + +An unrelated infrastructure change alters the controller URL. Kubernetes agent Pods start successfully but cannot resolve or connect to the remoting endpoint. Queue length grows and Pod images appear healthy. Inspect the agent container's connection error, Jenkins cloud configuration, proxy/TLS/DNS and network policies. Restarting every agent without fixing the endpoint just burns capacity. + +## Platform trade-offs + +Static VMs provide stable caches and specialized hardware but accumulate drift and need patching. Ephemeral Pods improve isolation and clean state but introduce provisioning delays, image-pull cost and dependency on Kubernetes API/control plane. Select separate build pools for untrusted PR code, privileged signing and special architectures, and validate actual cold-start/recovery SLO. + +### Spoken explanation + +> “An agent Pod must pass several stages before Jenkins sees an executor: API creation, scheduling, image pull, remoting connection and agent registration. I'd inspect the reason for Pending Pods or failed inbound connections before increasing replicas. Ephemeral agents help reproducibility and isolation, but they need carefully scoped RBAC, image caching, workspace policy and capacity headroom.” + +**References:** [Jenkins Kubernetes plugin](https://plugins.jenkins.io/kubernetes/) · [Jenkins agents](https://www.jenkins.io/doc/book/using/using-agents/) · [Kubernetes Pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) From 79db2fed271cb97a3cf301c7de29613b1591edf2 Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:16:27 +0530 Subject: [PATCH 05/15] docs(jenkins): add chapter 5 builders and shared libraries --- docs/guides/jenkins/q5.md | 99 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 docs/guides/jenkins/q5.md diff --git a/docs/guides/jenkins/q5.md b/docs/guides/jenkins/q5.md new file mode 100644 index 0000000..b85593c --- /dev/null +++ b/docs/guides/jenkins/q5.md @@ -0,0 +1,99 @@ +# Build Images, Caching, Artifacts and Reproducible Delivery + +> **Engineering scenario:** A service Docker build takes 35 minutes and frequently fails when workers are scaled. Design faster container builds that remain reproducible, secure and traceable to tested source. + +## Distinguish the layers + +A CI pipeline has at least four potential bottlenecks: checkout/download, dependency resolution, compile/test, and image/artifact creation. A Docker cache is not a general build cache. A faster image build that silently reuses stale dependencies or publishes an unreviewed tag is a correctness regression. + +```mermaid +flowchart LR + SCM["Reviewed source + lockfiles"] --> CI["Jenkins agent"] + CI --> TEST["Unit/contract checks"] + TEST --> BUILDER["BuildKit or approved image builder"] + CACHE["Read/write cache scoped by trust"] <--> BUILDER + BUILDER --> IMAGE["Digest-addressed image registry"] + BUILDER --> META["SBOM + provenance metadata"] + IMAGE --> PROM["Separate release promotion"] + META --> PROM + ``` + +## Cache architecture + +| Cache | Typical data | Invalidation / risk | +| --- | --- | --- | +| SCM mirror | Git objects | Revision identity, permissions | +| Package manager | Downloaded dependencies | Lockfile + registry trust | +| Compilation | Object files/test result cache | Compiler flags, architecture, source hash | +| Docker layers/BuildKit | Layer snapshots/build contexts | `COPY` order, build arguments, secret leakage | +| Registry remote cache | Cache manifests and layers | Access policy, retention, trust domain | +| Persistent workspace | Checkout and intermediate files | Stale files and cross-build contamination | + +Cache hits should be observable; benchmark warm and cold builds separately. Avoid one writable shared cache spanning both untrusted pull requests and privileged release builders. A cache exporter can become a registry/storage cost center, so apply retention and cleanup. + +## BuildKit concept with controlled Dockerfile + +```dockerfile +# syntax=docker/dockerfile:1 +FROM python:3.12-slim +WORKDIR /app +COPY requirements.txt . +RUN --mount=type=cache,target=/root/.cache/pip \ + pip install --no-input -r requirements.txt +COPY src/ ./src/ +CMD ["python", "-m", "src.main"] +``` + +For true repeatability pin the base image digest and package hashes, vet index trust and define architecture. Secrets belong in ephemeral BuildKit secret mounts or a trusted identity mechanism, not `ARG` or copied into an image layer. This example is simplified and needs a real project plus a verified `requirements.txt`. + +### Jenkins publish flow + +```groovy +pipeline { + agent { label 'trusted-builder' } + stages { + stage('Verify') { + steps { checkout scm; sh './scripts/test.sh' } + } + stage('Build') { + steps { + sh ''' + set -eu + docker buildx build --platform linux/amd64 \ + --tag registry.example.com/catalog:build-example \ + --load . + ''' + } + } + } +} +``` + +The above intentionally **does not push or authenticate**. A production publish stage must resolve the exact built digest, authenticate using restricted release credentials, push once, attest the output where required, and hand a digest to deployment. `--load` generally loads an image into a local Docker image store and is not a universal solution for multi-platform manifests. Avoid mounting the host Docker daemon into untrusted jobs. + +## Multi-architecture builds + +Architectures such as arm64 and amd64 may differ in dependency wheels, native compilation, test behavior and emulation performance. Buildx can build multi-platform manifests using suitable builders, but a build that succeeds under emulation doesn't establish runtime compatibility on native nodes. Test each target architecture when practical; record per-platform digest and resulting image index digest. + +## Failure scenario: build time regression + +Cache hit rate falls after a dependency manifest moves behind a frequently changed `COPY . .`. The whole dependency layer is invalidated on every source edit. Reorder Dockerfile layers to copy lockfiles first, then application sources, and measure effect. If cold builds remain slow, inspect registry throttling, network throughput, ephemeral disk I/O and CPU before adding more concurrent builders. + +```bash +docker buildx inspect +docker buildx du +docker image inspect IMAGE_ID +df -h +df -i +# On an approved agent: capture wall time and cache hit/miss evidence +``` + +### Artifact identity and retention + +Archived Jenkins artifacts are useful for short-lived test reports, but high-volume binaries usually belong in an external artifact repository or object store with immutability and retention. Releasing `:latest` makes rollback ambiguous. Persist source revision, dependency/lockfile identity, builder reference, artifact digest, test result and promotion approval. + +### Spoken explanation + +> “I'd profile checkout, dependencies, tests and container build separately. Then I'd fix the expensive layer using correct lockfile-based caching and trusted remote cache where justified, preserving immutable inputs. I would isolate build credentials from PR agents, build once, record the digest and promote the tested artifact through a separate authorization boundary. For multi-architecture images I'd validate each target runtime, not just a successful build command.” + +**References:** [Docker Build cache](https://docs.docker.com/build/cache/) · [Buildx](https://docs.docker.com/build/building/multi-platform/) · [Jenkins Pipeline syntax](https://www.jenkins.io/doc/book/pipeline/syntax/) From 43d7a4bafc2c1444f7d053e8cb2eb51f65334708 Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:16:30 +0530 Subject: [PATCH 06/15] docs(jenkins): add chapter 6 builders and shared libraries --- docs/guides/jenkins/q6.md | 91 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 docs/guides/jenkins/q6.md diff --git a/docs/guides/jenkins/q6.md b/docs/guides/jenkins/q6.md new file mode 100644 index 0000000..480a5c8 --- /dev/null +++ b/docs/guides/jenkins/q6.md @@ -0,0 +1,91 @@ +# Jenkins Shared Libraries, Versioning and Organization-Wide Standards + +> **Engineering scenario:** Twenty teams copy 300 lines of Pipeline logic. Propose a Shared Library that standardizes test, build and publish behavior without becoming a hidden cluster-admin backdoor. + +## Why use libraries? + +A Shared Library centralizes reusable Pipeline steps and policies but must retain discoverability and version control. Jenkins recognizes a typical library structure: + +```text +(platform-ci) + vars/ + buildService.groovy + buildService.txt + src/ + org/example/ci/Versioning.groovy + resources/ + org/example/ci/config.json + test/ + ... +``` + +`vars/` exposes global vars/steps, `src/` holds Groovy/Java-like class files and `resources/` contains optional bundled data accessed through `libraryResource`. Exact testing structure is a project convention, not a built-in requirement. + +```mermaid +flowchart TD + REPOS["Team Jenkinsfiles"] --> LIB["@Library pinned version"] + LIB --> API["Shared high-level steps"] + API --> TEST["Validation and testing defaults"] + API --> BUILD["Trusted artifact workflow"] + API --> POLICY["Policy and audit hooks"] + LIB --> PERM{"Trusted library?"} + PERM -->|Yes| RISK["Full Jenkins API access, high impact"] + PERM -->|No| SAFE["Groovy sandbox with approvals"] + ``` + +### Small and reviewable example + +```groovy +// vars/runChecks.groovy +def call(Map opts = [:]) { + String command = opts.command ?: './scripts/test.sh' + sh command +} +``` + +```groovy +// Jenkinsfile, with a preconfigured global library +@Library('platform-ci@v1') _ +pipeline { + agent { label 'linux-small' } + stages { + stage('Test') { + steps { + checkout scm + runChecks(command: './scripts/test.sh') + } + } + } +} +``` + +**Security note:** A library API that accepts arbitrary shell commands can be misused. A stronger golden path constrains allowed operations and validates options. The example demonstrates syntax, not a hardened release library. + +### Trusted versus untrusted libraries + +**Trusted global libraries** can use Jenkins core APIs and escape the Groovy sandbox, so commit access to that library may imply effectively unlimited controller privileges. Restrict repository write access, approval and default-version override. **Folder libraries are untrusted**, useful when different teams need scoped behavior. Do not put private keys or high-impact controller operations in code maintained by untrusted developers. + +Version pinning limits surprise regressions. If every service loads floating `main`, a library push can break all pipelines at once. Prefer controlled versions (tags or commit SHAs) and phased promotion: library test, canary consumers, then broad rollout. A library version is not a container/image identity; record both. + +## Declarative limits and CPS + +Shared libraries can define full Declarative Pipelines through supported `vars/*.groovy` patterns, but avoid nesting a new Declarative `pipeline {}` inside another. Pure Groovy utility methods must obey CPS/serialization rules if called from Pipeline. Keep substantial computation and tests in normal executables on agents. + +### Test strategy + +1. Test pure utility methods with a normal Groovy unit test framework where possible. +2. Mock Jenkins steps using an approved Pipeline testing approach to verify arguments and branching. +3. Run a sandbox Jenkins instance with the actual plugin set for syntax/step compatibility. +4. Canary library updates with a small set of real consumers. +5. Pin or rollback library versions if metrics show breakage. +6. Inspect permission changes when adding new `withCredentials`, shell or controller API access. + +### Incident example + +A change to the default global shared library replaces a `try/finally` around artifact cleanup. Hundreds of jobs start leaving temporary credentials/files. Rolling back a single application Jenkinsfile won't fix it because every job uses the same floating library revision. Re-pin the library to a known-good revision, invalidate vulnerable credentials/agents if needed, and add a regression test and canary policy. + +### Spoken explanation + +> “Shared libraries are useful for a stable high-level pipeline API: test, build, publish and report. I'd keep business-specific commands in repositories, version the library and test it like production code. Trusted libraries are effectively an administrative capability, so their SCM permissions need especially strong controls. I'd roll out changes gradually and keep Pipeline Groovy focused on orchestration, not arbitrary controller computation.” + +**References:** [Shared Libraries](https://www.jenkins.io/doc/book/pipeline/shared-libraries/) · [CPS behavior](https://www.jenkins.io/doc/book/pipeline/cps-method-mismatches/) From de5e15d70b3c23a1bebe655ed2cc98e1d39ad64d Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:18:00 +0530 Subject: [PATCH 07/15] docs(jenkins): add chapter 7 security and lifecycle --- docs/guides/jenkins/q7.md | 87 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 docs/guides/jenkins/q7.md diff --git a/docs/guides/jenkins/q7.md b/docs/guides/jenkins/q7.md new file mode 100644 index 0000000..d7f88d6 --- /dev/null +++ b/docs/guides/jenkins/q7.md @@ -0,0 +1,87 @@ +# Jenkins Security: Identity, Credentials, Agents and Pipeline Trust + +> **Engineering scenario:** A public pull request runs tests on Jenkins, while production release credentials are also configured. Explain the attack paths and the controls required to prevent a contributor from reaching secrets, the controller or another project's workspace. + +## Trust boundaries first + +A developer who can modify a Jenkinsfile can generally execute chosen Pipeline steps subject to that job's permissions. If it can bind a credential, it can often use or exfiltrate that credential despite UI masking. Treat the Jenkins controller, privileged shared libraries, SCM sources, build agents and release credentials as **separate security principals**. + +```mermaid +flowchart TD + USER["Users through SSO / API tokens"] --> AUTH["Authentication and authorization"] + AUTH --> CTRL["Controller + plugin trust"] + SCM["SCM: trusted / untrusted revisions"] --> JOB["Pipeline definitions"] + JOB --> AGENT["Restricted or privileged agent"] + CTRL --> AGENT + AGENT --> CREDS{"Credential scope"} + CREDS -->|PR sandbox| NONE["No release credentials"] + CREDS -->|Approved release| PROD["Short-lived scoped release identity"] + SECRET["Secret storage and rotation"] -.-> CREDS + ``` + +### Controls and misunderstandings + +- Enable a reviewed security realm and authorization strategy (matrix/role-based where installed); grant minimal **Job, Agent, Credential, and Script** permissions. +- Disable executors on the built-in controller, isolate untrusted workloads from controller filesystem/network. +- Use separate agents/pools, Unix identities and Kubernetes namespaces/accounts for different trust levels. Different directories on the same host may not be enough. +- Limit Credentials scope and readers. A Jenkins masked secret is still accessible to pipeline code that can use the credential. +- Keep CSRF protection enabled. Jenkins API token authentication is typically exempt from crumb requirements; username/password scripted POSTs generally need a crumb. Always use TLS. +- Pin plugins and review security advisories. A plugin adds code inside the controller's trust boundary; manage plugin and script-console privileges tightly. +- Protect artifact publishing, signing keys and deployment separately from fork PR validation. + +### Correct handling of an approved secret + +```groovy +pipeline { + agent { label 'trusted-release' } + stages { + stage('Authorized API request') { + steps { + withCredentials([string(credentialsId: 'release-service-token', + variable: 'TOKEN')]) { + sh ''' + set +x + curl --fail --silent --show-error \ + -H "Authorization: Bearer $TOKEN" \ + https://release.example.org/status + ''' + } + } + } + } +} +``` + +This illustrates **credential binding syntax only**. It assumes the specific job/agent is trusted, the endpoint is authorized and credential permissions are appropriately scoped. Prefer ephemeral identity federation when supported, and avoid passing secrets in command arguments if process-list visibility or audit policy makes that unacceptable. For high assurance, use a client capable of credentials from a protected file/descriptor. + +### Groovy interpolation pitfall + +```groovy +// WRONG: Groovy interpolation can expose a secret too early. +// sh "curl -H 'Authorization: Bearer ' ..." +// +// Better: single-quoted Groovy script so the shell expands a +// scoped environment variable only at execution time. +``` + +Masking is best-effort and is not a leak-prevention boundary. `set +x` reduces command echo, but child processes, crash reports, verbose tools and privileged co-tenants can still expose secrets. Never print environments or rely on log redaction to enforce policy. + +## Attack walk-through: untrusted change request + +1. Attacker submits a PR changing the test script. +2. Trusted Jenkinsfile invokes `./scripts/test.sh` from the PR. +3. The script runs with the agent's OS permissions. +4. If privileged credentials are present, it can access them or exfiltrate through network access. +5. Even without a bound secret, shared Docker socket, host mount, insecure kubeconfig or broad cloud instance role can create escalation paths. + +The fix is **agent- and credential-level isolation**, not simply using a trusted Jenkinsfile. + +### Incident response + +If a token was exposed, revoke it and its downstream sessions; inspect job logs, agent filesystem, network egress, artifact registry and cloud audits. Identify all jobs/branches that could read the credential. Rotate keys, remove overbroad grants and replace compromised agents. Avoid publishing logs that themselves reveal the leaked values. + +### Spoken explanation + +> “I'd separate Jenkins authentication, Pipeline execution and release identity. Untrusted source code runs only on isolated agents without registry-push or cloud-deploy credentials, regardless of the Jenkinsfile trust policy. I'd use scoped credential bindings, API tokens and TLS, restrict trusted libraries and admin scripts, and keep plugins current. Masking reduces accidental disclosure but does not prevent a malicious build from extracting a credential it can access.” + +**References:** [Jenkins security](https://www.jenkins.io/doc/book/security/) · [Credential binding](https://www.jenkins.io/doc/pipeline/steps/credentials-binding/) · [Multibranch credential risks](https://www.jenkins.io/doc/book/security/securing-org-folders-and-multibranch-pipelines/) From 325e138e55ea59affe93e3629e800ea815b1f888 Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:18:03 +0530 Subject: [PATCH 08/15] docs(jenkins): add chapter 8 security and lifecycle --- docs/guides/jenkins/q8.md | 94 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 docs/guides/jenkins/q8.md diff --git a/docs/guides/jenkins/q8.md b/docs/guides/jenkins/q8.md new file mode 100644 index 0000000..f49d5f9 --- /dev/null +++ b/docs/guides/jenkins/q8.md @@ -0,0 +1,94 @@ +# Operating the Jenkins Controller: State, Configuration, Backups and Upgrades + +> **Engineering scenario:** The controller must be upgraded without losing job history, credentials, Pipelines or agent registration. Build a rollback and disaster-recovery plan that recognizes the stateful parts of Jenkins. + +## Why Jenkins controller recovery differs from a stateless web service + +The controller stores much of its operational state under `JENKINS_HOME`: global configuration, installed plugins, job/build metadata, Pipeline state, credentials material, and history. Containerizing it doesn't make this state disposable. A load balancer distributing writes across two unrelated controllers is **not** a transparent Jenkins active-active architecture. Jenkins core is not generally operated as multiple interchangeable replicas sharing the same writable home directory. + +```mermaid +flowchart TD + LB["Authenticated HTTPS endpoint"] --> CTRL["One active Jenkins controller"] + CTRL <--> VOL[("Persistent JENKINS_HOME")] + CTRL --> AGENTS["Ephemeral or static agents"] + CTRL --> ART["External artifact registry"] + JCAS["Reviewed JCasC + plugin lock"] --> CTRL + VOL --> BACK["Consistent encrypted backup"] + KEY["Separately protected master key"] -.-> REST["Controlled restore"] + BACK --> REST + JCAS --> REST + REST --> TEST["Isolated recovery drill"] + ``` + +### Configuration as Code (JCasC) + +The Configuration as Code plugin reads YAML for supported controller/plugin configuration, often from `CASC_JENKINS_CONFIG`. It improves reviewability and bootstrap repeatability, but not every plugin/global state is represented by one portable YAML file. JCasC can **supplement**, not replace, a complete `JENKINS_HOME` recovery strategy and plugin compatibility plan. + +```yaml +jenkins: + numExecutors: 0 + systemMessage: "Managed by reviewed configuration." + mode: NORMAL +``` + +This is a minimal example and requires an installed, compatible JCasC plugin. **Do not** commit real secret values in YAML. Use an approved secret source/resolver, protect the repository and validate expansions in a sandbox. + +### Backup invariants + +- Capture a consistent filesystem snapshot or a quiesced copy of required `JENKINS_HOME` content. +- Include versioned plugin inventory and configuration; build history/artifacts according to retention requirements. +- Store encryption key material separately with strong access controls. Jenkins documentation recommends protecting `master.key` independently of routine backups; restores need compatible key material for encrypted data. +- Verify backup consistency, confidentiality, restore completeness and **actual measured RTO/RPO** with a disconnected environment. +- Avoid restoring production credentials into a testing environment that has network access to real production systems. +- A snapshot does not undo secrets leaked before it was taken. + +### Safe upgrade runbook + +1. Inventory Jenkins core/LTS line, Java runtime, plugin dependencies, controller image, storage and integrations. +2. Read changelogs, security advisories and minimum Java requirements for the **selected versions**. Don't assume all LTS versions share requirements. +3. Freeze or drain risky releases. Observe running Pipelines and decide whether to allow completion, interrupt or resume later. +4. Take and verify a consistent backup and independently secured key material. +5. Upgrade in a staging clone with equivalent plugins and sanitized credentials. Check real Pipeline library/job compatibility. +6. Roll canary workloads through the upgraded environment, validate agents, webhooks, SCM, credentials binding and artifact publication. +7. Apply an approved maintenance window to production, observe heap, logs, queue latency, plugin loading and business workflows. +8. Rollback requires restoring compatible persisted state, not blindly downgrading binaries after schema/plugin migrations. + +```mermaid +sequenceDiagram + participant O as Operator + participant J as Controller + participant B as Backup store + participant S as Isolated staging + O->>J: Inspect version/plugins/running jobs + O->>B: Consistent state backup + B-->>O: Verified restore point + O->>S: Restore sanitized state + tested versions + S-->>O: Functional smoke + plugin compatibility + O->>J: Drain and perform authorized upgrade + J-->>O: Validate jobs, logs, agents and queue + ``` + +## Fault scenarios + +- **Disk full:** controller cannot persist logs/state. Stop uncontrolled disk growth, identify large artifacts/workspaces and archive/retire safely; do not delete arbitrary `program.dat`, job history or secret material during an incident. +- **Plugin mismatch:** startup logs show missing classes/failed dependencies. Restore a compatible core/plugin set after reading the full dependency error; forcing the plugin directory by hand can compound corruption. +- **Controller restart during build:** some resumable steps may continue, others fail if their agent/workspace or external step isn't recoverable. Test restart behavior for the actual plugin set. +- **Lost encryption key:** encrypted credentials may become unrecoverable despite intact `JENKINS_HOME` backups. + +### Read-only checks + +```bash +# Systemd deployment only; use pod logs for a container installation. +systemctl status jenkins +journalctl -u jenkins --since '30 min ago' --no-pager +df -h +df -i +free -h +# In Manage Jenkins: plugin inventory, system logs, nodes and warnings +``` + +### Spoken explanation + +> “Jenkins is a stateful controller. I'd keep builds off the controller, version its configuration/plugins, back up home data consistently, secure master key material separately and exercise restores in an isolated environment. Before an LTS upgrade I'd validate Java and plugin compatibility, agent connectivity, Pipeline resume behavior and rollback constraints. One active controller with recoverable state is not the same as a magically active-active Jenkins cluster.” + +**References:** [Jenkins backup and restore](https://www.jenkins.io/doc/book/system-administration/backing-up/) · [JCasC plugin](https://plugins.jenkins.io/configuration-as-code/) · [Jenkins upgrade guide](https://www.jenkins.io/doc/upgrade-guide/) From ceadfcf05b8a9a39890e42885a944eaebdbc5b9b Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:19:52 +0530 Subject: [PATCH 09/15] docs(jenkins): add chapter 9 troubleshooting and platform design --- docs/guides/jenkins/q9.md | 104 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 docs/guides/jenkins/q9.md diff --git a/docs/guides/jenkins/q9.md b/docs/guides/jenkins/q9.md new file mode 100644 index 0000000..dcda213 --- /dev/null +++ b/docs/guides/jenkins/q9.md @@ -0,0 +1,104 @@ +# Jenkins Production Troubleshooting: Queues, Controller Latency and Build Failures + +> **Engineering scenario:** The Jenkins UI intermittently returns 502, controller CPU spikes, agents remain idle and builds are delayed. Design a diagnosis plan that differentiates reverse-proxy, controller JVM, queue, storage and agent failures. + +## Investigate layers, not symptoms + +```mermaid +flowchart TD + SYM["Slow UI, failed builds or queue growth"] --> SCOPE["Is it UI/API, scheduling, or job execution?"] + SCOPE --> PROXY["Proxy, TLS, health, upstream timeouts"] + SCOPE --> JVM["JVM CPU, heap, GC, thread dumps"] + SCOPE --> QUEUE["Build queue / executors / locks"] + SCOPE --> DISK["JENKINS_HOME IOPS, disk, inodes"] + SCOPE --> AGENT["Agent connections, workspace and tools"] + PROXY --> COR["Correlate one timeline"] + JVM --> COR + QUEUE --> COR + DISK --> COR + AGENT --> COR + COR --> MIT["Narrow reversible mitigation and verify"] + ``` + +### Distinguish three forms of slowness + +1. **Queue latency:** a job is accepted but cannot acquire an appropriate executor. Check label, agent eligibility, capacity and scheduling constraints. +2. **Controller latency:** UI/API responsiveness, job graph processing, plugin hooks, Groovy CPS, thread saturation and disk I/O. Even idle agents can coexist with a busy controller. +3. **Step runtime latency:** tests, SCM checkout, registry, cache, dependency servers, database and workspace I/O. Increasing controller CPU does not optimize a slow compiler on an agent. + +### Minimal evidence table + +| Signal | Possible interpretation | Confirm with | +| --- | --- | --- | +| NGINX/ALB 502 | No healthy upstream, controller restart, connection failure | Proxy access/error logs, upstream TCP, controller restart | +| Slow Jenkins pages | JVM GC pauses, plugin overhead, disk I/O, reverse proxy | JVM/GC metrics, thread dumps, I/O latency | +| Build queue grows | No executor/label, cloud launch delay, resource lock | Queue reason + agent cloud logs | +| Executor busy | Long jobs or hung step | Stage timings, agent process/thread inspection | +| Lost agents | Network/remoting/TLS failure or OOM kill | Agent launch logs, Kubernetes events, container exit | +| Disk high | Archived artifacts, build records, snapshots, workspace | Directory sizes, disk/inodes, retention settings | + +### Authoritative commands + +```bash +# Controller host, read-only; use containers/pod exec when hosted there: +uptime +top -H -p "$(pgrep -f 'jenkins.war|jenkins.Main' | head -n1)" +free -h +vmstat 1 5 +iostat -xz 1 5 # Requires sysstat +df -h +df -i +ss -tanp | head -n 30 +journalctl -u jenkins --since '20 min ago' --no-pager + +# JVM diagnostic tools need an authorized runtime and compatible JDK: +jcmd PID VM.version +jcmd PID Thread.print +jcmd PID GC.heap_info +``` + +**Safety:** Thread dumps, logs and heap data may expose secrets, tokens, plugin configuration or paths. Restrict access, redact before sharing and avoid automatically producing large heap dumps on a nearly-full filesystem. `pgrep` can match the wrong process; verify the PID manually. + +### Controller UI 502 investigation + +```mermaid +sequenceDiagram + participant U as Operator + participant LB as Reverse proxy / LB + participant J as Jenkins HTTP/JVM + participant A as Agents + U->>LB: GET /job/example/ + LB->>J: Proxy HTTP request + alt Controller unreachable or failed + LB-->>U: 502/504 depending on proxy + else Controller responds slowly + J->>J: Plugin, UI and filesystem work + J-->>LB: Slow response or timeout + LB-->>U: Response / gateway timeout + end + Note over A: Agent CPU may be unrelated to controller UI latency + ``` + +Check upstream error category (connection refused, timeout, unhealthy target, DNS resolution), controller readiness, restarts/OOM, JVM GC and expensive plugin endpoints. **A 502 is a proxy-visible symptom, not proof of controller CPU exhaustion.** Rollback a recent plugin change only when evidence supports it; restart may temporarily hide the root cause and interrupt resumability. + +## CPS/serialization failure + +If a Pipeline throws `NotSerializableException`, identify nonserializable variables kept alive across Pipeline step suspension. Convert them to plain serializable data, scope them to a pure `@NonCPS` helper, or perform the transformation in an agent-executed language. Increasing heap does not make Java objects serializable. + +## Queue starvation incident + +A busy architecture-label pool handles all builds with two executors while four other nodes have different labels. Adding nodes with the wrong label doesn't solve the problem. Inspect queue reasons and requests/limits. If a build asks for a distributed lock, the lock may be the actual constraint even when executors appear free. + +## Capacity and observability baseline + +Measure build starts, duration per stage, time in queue, agent provision/connection latency, controller CPU/heap/GC pause, queue depth, storage usage, failure classifications, SCM indexing latency and artifact-push errors. Avoid unbounded per-build IDs as Prometheus labels; correlate unique IDs through logs and traces. + +### Incident command and recovery + +Identify blast radius (all jobs, one repo, one agent pool, UI only, release blocked), freeze high-risk release changes, assign controller/agent/network investigations, pick the smallest reversible mitigation, verify user-facing CI journey, and record follow-up owners. Restarts should be planned with knowledge of running durable steps and downstream consequences. + +### Spoken explanation + +> “I'd separate three timelines: how long a build waits for an executor, how long it takes Jenkins to orchestrate and serve requests, and how long steps actually run. A 502 could be proxy routing or a stalled JVM, while idle agents could indicate a missing label or broken cloud provisioning. I'd correlate controller JVM, disk and network with agent logs and queue reasons before resizing anything. Then I'd implement one reversible mitigation and prove both UI and build-journey recovery.” + +**References:** [Scaling Pipelines](https://www.jenkins.io/doc/book/pipeline/scaling-pipeline/) · [System administration](https://www.jenkins.io/doc/book/system-administration/) · [Jenkins troubleshooting](https://www.jenkins.io/doc/book/troubleshooting/) From 05a5a7bf70bf182690573120ded87334644a0160 Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:19:54 +0530 Subject: [PATCH 10/15] docs(jenkins): add chapter 10 troubleshooting and platform design --- docs/guides/jenkins/q10.md | 92 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 docs/guides/jenkins/q10.md diff --git a/docs/guides/jenkins/q10.md b/docs/guides/jenkins/q10.md new file mode 100644 index 0000000..d532650 --- /dev/null +++ b/docs/guides/jenkins/q10.md @@ -0,0 +1,92 @@ +# Jenkins as a Platform: Delivery Architecture, Governance and Reliability + +> **Engineering scenario:** Several product teams need fast pipelines, but a shared Jenkins controller has become a release bottleneck. Design a supportable Jenkins platform with reusable build policies, safe deployment boundaries, ownership, cost allocation and measurable reliability. + +## Platform system design + +A Jenkins installation is an internal platform component, not the entire developer platform. It should deliver trusted source validation, reproducible build artifacts, auditable results and a documented recovery path. Runtime deployment can be performed by a separate GitOps reconciler instead of granting the build controller broad access to production clusters. + +```mermaid +flowchart TD + TEAM["Developer teams"] --> SCM["Protected repositories"] + SCM --> PR["Jenkins multibranch PR checks"] + PR --> AG["Isolated ephemeral agents"] + AG --> TST["Tests, scans and artifact builds"] + TST --> REG["Digest-addressed image/attestation registry"] + REG --> PROM["Approved promotion Git change"] + PROM --> CD["GitOps / constrained deploy system"] + CD --> CLUS["Runtime Kubernetes / cloud"] + LIB["Versioned Jenkins shared libraries"] --> PR + POLICY["CI policy, credentials and audit"] --> PR + OBS["SLOs and incident response"] -.-> PR + OBS -.-> CD + ``` + +### Separate roles + +| Responsibility | Recommended owner/boundary | +| --- | --- | +| Build-from-PR | Untrusted job identity, sandboxed agent and no release credentials | +| Provenance and publish | Trusted build/release identity, immutable digest and policy | +| Production deploy | Restricted CD identity with approvals and audit | +| Pipeline standards | Reviewed Shared Library versions and templates | +| Controller management | Platform operations, backups, plugins, configuration | +| Product validation | Owning service team, contract/user-journey tests | +| SLOs and recovery | Shared, explicit ownership and on-call agreement | + +### Golden paths without hiding behavior + +A default Jenkinsfile template should define source identity, fast tests, dependency/cache policy, artifact publication, timeouts, limited retries and clear failure messages. Service teams need supported extension points for genuine requirements; the shared library should not silently run global-admin Groovy code on arbitrary SCM input. + +Version the library and provide changelogs, migration paths and compatibility tests. Treat an upgrade to the global pipeline API similarly to a production change, because it may affect every service even if the Jenkins controller itself is healthy. + +## Trusted CI versus runtime CD + +```mermaid +sequenceDiagram + participant S as Source owner + participant J as Jenkins CI + participant R as Artifact registry + participant G as Deployment Git + participant D as GitOps controller + participant K as Runtime + S->>J: Merge reviewed revision + J->>J: Test and build once + J->>R: Publish immutable artifact + evidence + J->>G: Propose digest promotion + S->>G: Approve according to environment risk + G-->>D: Observe desired state + D->>K: Reconcile permitted resources + K-->>D: Health and rollout state + ``` + +This split is a **design recommendation**, not a universal requirement. A team may deploy directly from Jenkins with narrowly scoped credentials, but then it must own more controller-to-production risk, action auditing and recovery behavior. The correct choice depends on scale, compliance and existing CD tooling. + +## Progressive delivery and immutable identity + +A successful Jenkins build isn't proof that the application meets a runtime SLO. Promote the digest that passed checks, then evaluate p95/p99 latency, error budgets, user-facing business outcomes and canary/smoke checks. Rolling back a container cannot undo database schema changes or external side effects; use backward-compatible migrations and documented forward-recovery where required. + +### SLO examples + +- **Time to first feedback:** p90 from PR update to actionable test result. +- **CI completion reliability:** good completed eligible builds / total eligible builds; define whether infrastructure aborts count as bad. +- **Provisioning latency:** p95 from eligible queue item to online executor. +- **Release integrity:** proportion of deployments traceable to an approved tested digest. +- **Controller recoverability:** measured restore RTO/RPO in a game day. +- **Secret hygiene:** zero untrusted PR jobs capable of using production credentials. + +Do not equate high deployment count with engineering effectiveness; measure change failure rate, recovery and developer experience as well. + +## Failure design exercise + +An organization-wide shared library upgrade breaks every Pipeline at source checkout due to a plugin compatibility change. Containment: halt new promotions, pin the library to last verified revision, investigate plugin core requirements, isolate failed branches from runnable builds and avoid force-resetting controller state. Prevention: versioned library rollout, canary jobs, compatibility tests against the actual Jenkins/plugin stack, and clear ownership of default versions. + +## Senior design trade-offs + +One large controller simplifies administration but concentrates plugin risk, heap/storage contention and outage impact. Multiple controllers give separate failure/permission domains but increase upgrade burden and require common developer experience, SCM onboarding and build observability. The right unit of isolation may be business criticality, trust level or tenancy, not simply team count. + +### Spoken explanation + +> “I'd define Jenkins as the CI orchestration service: reviewed source in, verified immutable artifacts and evidence out. Untrusted PR checks and trusted publishing get separate agent and identity boundaries, while runtime CD is an independently authorized process where possible. I'd version shared libraries, monitor queue and build SLOs, enforce security policy and rehearse controller restore. The goal is reliable developer feedback and traceable delivery, not just more executors.” + +**References:** [Jenkins Pipeline](https://www.jenkins.io/doc/book/pipeline/) · [Shared Libraries](https://www.jenkins.io/doc/book/pipeline/shared-libraries/) · [Google SRE SLOs](https://sre.google/sre-book/service-level-objectives/) From 09538756f46c12af24d9924a6a146e4eb4853b39 Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:22:23 +0530 Subject: [PATCH 11/15] docs(jenkins): add 47 deep scenario questions and model answers --- docs/guides/jenkins/follow-ups.md | 287 ++++++++++++++++++++++++++++++ 1 file changed, 287 insertions(+) create mode 100644 docs/guides/jenkins/follow-ups.md diff --git a/docs/guides/jenkins/follow-ups.md b/docs/guides/jenkins/follow-ups.md new file mode 100644 index 0000000..dc0778f --- /dev/null +++ b/docs/guides/jenkins/follow-ups.md @@ -0,0 +1,287 @@ +# 47 Advanced Technical FAQs: Jenkins Architecture and Operations + +These scenario questions follow the ten chapters. Try to reason from first principles before reading the model answer. Scenarios are hypothetical and some controls depend on installed plugins. + +## 1. Why should the built-in controller node have zero executors? + +*Topic: Architecture* + +**Model answer:** The controller performs scheduling, authorization, UI/API, plugin orchestration and Pipeline graph processing. Running arbitrary build processes there consumes JVM/host resources and places untrusted code near controller state and credentials. Set built-in executors to zero and use appropriately isolated agents. This does not by itself secure the controller; Script Console, trusted libraries and plugins remain privileged paths. + +## 2. Can Jenkins controllers be made active-active by mounting the same JENKINS_HOME on two Pods? + +*Topic: Architecture* + +**Model answer:** No. Typical Jenkins controllers are not designed for independent active-active writers over the same home directory. Concurrent writes to jobs, credentials, program state and plugins risk corruption, and a load balancer cannot coordinate them. Use one active controller with tested state backups and recovery, or separate independent controllers with independent homes and an explicit federation of workloads. + +## 3. What is the distinction between a node, agent and executor? + +*Topic: Architecture* + +**Model answer:** A node represents registered compute. The agent remoting process connects that compute to the controller and runs assigned tasks. An executor is a concurrent execution slot on the node. A Pod or VM might have one or more executors, but increasing executors without increasing CPU, memory, isolation and dependency headroom may reduce throughput or allow cross-build interference. + +## 4. Why might an idle worker fleet coexist with a growing Jenkins queue? + +*Topic: Architecture* + +**Model answer:** A queue waits for eligible executors, not any idle CPU in the fleet. Label mismatch, node restricted-use mode, concurrency throttling, locks, cloud-provisioning failure, offline agents or blocked dependencies may prevent allocation. Start with the queue item's exact reason and agent eligibility; blindly adding generic workers can worsen cost without resolving the scheduling constraint. + +## 5. Does Pipeline durability mean a test process survives any controller restart? + +*Topic: Architecture* + +**Model answer:** No. Jenkins Pipeline CPS can persist continuation state for supported steps, but the running external process, connected agent, workspace and plugin behavior affect resumption. Steps that aren't durable or disappear with an ephemeral Pod may fail. Test restart behavior of critical pipelines using the actual plugins, and design retried operations to be idempotent. + +## 6. Why use agent none with stage-level agents? + +*Topic: Pipeline* + +**Model answer:** It prevents holding a global executor for stages that do not need one, and permits stage-specific build environments and resource sizing. Each allocated agent may have a distinct workspace, so stages must explicitly re-checkout an immutable revision or transfer bounded artifacts. This can improve scheduling efficiency but adds provisioning overhead when agents are ephemeral. + +## 7. What is the difference between Scripted and Declarative Pipeline? + +*Topic: Pipeline* + +**Model answer:** Declarative constrains structure using pipeline, agent, stages, options and post, making defaults and governance easier. Scripted uses a Groovy DSL for dynamic control flow. Both rely on Jenkins steps and the Pipeline execution engine. Use Declarative for predictable delivery and a small, reviewed scripted escape hatch where dynamic behavior genuinely improves readability. + +## 8. Why is @NonCPS not a universal fix for Pipeline slowness? + +*Topic: Pipeline* + +**Model answer:** A @NonCPS method executes without the CPS transform but must not call CPS-transformed Pipeline steps. It still executes in controller-side Groovy and can consume substantial controller CPU if doing heavyweight work. Use it for small pure transforms where justified. Run large parsing, compilation or file processing on agents using ordinary executables. + +## 9. What causes NotSerializableException around Pipeline steps? + +*Topic: Pipeline* + +**Model answer:** The CPS engine may need to persist a continuation across a pause. Retained objects such as matchers, iterators, open streams or plugin objects may not support serialization. Reduce the state to basic serializable values, scope complex objects to a pure helper or move transformation to an external program. Increasing Java heap doesn't make an object serializable. + +## 10. What happens if a parallel branch fails while other branches continue? + +*Topic: Pipeline* + +**Model answer:** Behavior depends on Declarative/step configuration, including failFast controls, and where error handling is applied. Remaining branches may continue, be interrupted, or finish before a final failure is recorded. Set explicit fail-fast policy and preserve test reports from each branch. Don't hide a required failure with catchError merely to make the overall status green. + +## 11. When should stash and unstash be avoided? + +*Topic: Pipeline* + +**Model answer:** They help move small files between stages/agents but can route archive/compression and storage pressure through the controller or configured external artifact manager. Large multi-gigabyte workspaces can drive controller disk and I/O issues. Use immutable Git revision re-checkout, dedicated artifact storage or a tested external artifact manager for large outputs. + +## 12. Why isn't branch main in a Jenkinsfile a production authorization boundary? + +*Topic: SCM* + +**Model answer:** A Pipeline-controlled condition governs logic but doesn't prevent execution of other code under the job's permissions. A malicious contributor or compromised maintainer could change it if the job trusts their Jenkinsfile. Restrict actual release credentials and deployment rights to separately authorized trusted jobs, protected source and identity checks. + +## 13. How does a webhook differ from Multibranch indexing? + +*Topic: SCM* + +**Model answer:** A webhook is event delivery from the SCM provider. Indexing discovers eligible repositories, branches and change requests according to SCM source configuration. A webhook can be delivered while Jenkins lacks repository access or the branch strategy excludes the ref. Examine event acceptance, indexing logs, discovered branch jobs and queue state separately. + +## 14. Is a trusted Jenkinsfile enough to safely test untrusted PR code? + +*Topic: SCM* + +**Model answer:** No. Even a trusted Jenkinsfile can invoke scripts checked out from a PR, and those scripts may execute arbitrary code under the agent's OS and cloud identity. Run untrusted PR checks on isolated agents with no sensitive credentials, Docker socket or privileged cloud role. Treat Jenkinsfile trust as one control, not the entire boundary. + +## 15. Why can Organization Folder credentials be a hidden privilege escalation? + +*Topic: SCM* + +**Model answer:** A folder's SCM scan credential may access multiple repositories or permissions that a contributor's repository doesn't normally have. If a job under that scope can cause the credential to be used or leaked, a small repo role becomes broader access. Split repositories by trust domain, use fine-grained SCM credentials and audit source plugin behavior. + +## 16. Why can Kubernetes agents remain offline while Pods report Running? + +*Topic: Agents* + +**Model answer:** Running only means containers exist; the inbound remoting agent may fail to connect due to DNS, TLS, wrong Jenkins URL, network policy, token/secret mismatch or incompatible Java/remoting versions. Inspect the special agent container's connection logs, Jenkins cloud logs and endpoint reachability. Pod status alone is not proof of an online Jenkins executor. + +## 17. What is the latency budget for a newly provisioned Kubernetes executor? + +*Topic: Agents* + +**Model answer:** It includes Jenkins queue processing, cloud plugin calls, Kubernetes API admission, Pod scheduling, node provisioning, image pulls, agent JVM startup, network handshake and registration. For known bursts, prewarm eligible node capacity and commonly used images if appropriate. Report provisioning p95 separately from test execution duration. + +## 18. Why is mounting the host Docker socket risky in a build Pod? + +*Topic: Agents* + +**Model answer:** A client with unrestricted access to the daemon can frequently control host containers/filesystems and cross isolation boundaries, making it unsuitable for untrusted PR code. Prefer a carefully isolated builder/service with reviewed permissions, rootless or daemonless options where fit, and restrict publishing credentials. No builder choice removes the need for a threat model. + +## 19. What might be wrong if the Pod cannot be created due to quota? + +*Topic: Agents* + +**Model answer:** Kubernetes ResourceQuota or LimitRange, service account RBAC, Pod Security admission, CNI policy or required labels/taints can reject or block provisioning. Inspect plugin provisioner logs, Kubernetes events, quota usage and the rejected object. Raising controller executors doesn't fix a denied API operation. Change only the correct policy under approved access. + +## 20. How do you separate untrusted builds from privileged signing jobs? + +*Topic: Agents* + +**Model answer:** Use distinct node pools/Pods, namespaces/accounts, cloud identities, network egress and credentials access. The signing pool should accept only approved release artifacts or vetted revisions, not arbitrary PR scripts. Share a digest and attested evidence across the trust boundary instead of an untrusted writable filesystem or cache. + +## 21. Why doesn't a faster Docker layer cache prove a reproducible build? + +*Topic: Caching* + +**Model answer:** A cache can reuse layers from a different builder or trust level, and build output may depend on mutable tags, package indexes, timestamps and environment. Reproducibility requires controlled inputs, dependency hashes, architecture and builder behavior; provenance records what actually built the artifact. Benchmark caches separately from verifying artifact identity. + +## 22. What is the impact of moving COPY . . before package installation? + +*Topic: Caching* + +**Model answer:** If the whole source tree is copied before dependency resolution, any source change often invalidates the following dependency layer. Copy lockfiles/manifests first, install vetted dependencies, then copy frequently changing source. Some build systems also use their own dependency cache. Verify improved time without leaking credentials into cached layers. + +## 23. Is a writable cache shared across fork PRs and trusted release jobs safe? + +*Topic: Caching* + +**Model answer:** Not by default. A malicious PR may poison caches that a privileged build later uses, changing output or executing unreviewed artifacts. Partition by trust level, validate checksums and use read-only cache imports where appropriate. Distinct build contexts are not adequate if the same cache writing identity spans trusted and untrusted workloads. + +## 24. Why should the deployment reference an immutable image digest instead of stable or latest? + +*Topic: Artifacts* + +**Model answer:** Tags can move, so different agents/hosts may pull different content while displaying the same tag. A digest identifies immutable bytes; the promotion record should link that digest to reviewed source, tests and builder. For multi-platform images distinguish manifest index digest from per-architecture manifest digests. Rollback then selects a known tested identity. + +## 25. Can a successful amd64 image build guarantee an arm64 runtime works? + +*Topic: Artifacts* + +**Model answer:** No. Native dependencies, wheels, compiler flags, architecture-specific instructions and performance may differ. An emulated builder may produce a manifest without proving behavior on native arm64. Build for supported target architectures, inspect the manifest and run meaningful tests on native or representative runtime where possible. + +## 26. What makes a trusted global Shared Library unusually powerful? + +*Topic: Libraries* + +**Model answer:** Trusted libraries can invoke Jenkins core/plugin APIs outside the Groovy sandbox, potentially obtaining full controller privileges. Commit rights to that SCM repository can become high-impact access. Restrict writers and version overrides, review changes, test plugin compatibility and avoid exposing unconstrained privileged methods to arbitrary Jenkinsfiles. + +## 27. Why use pinned library tags or commits rather than floating main? + +*Topic: Libraries* + +**Model answer:** A floating default causes a library change to affect many Pipeline executions simultaneously without a change in consuming repos. Pinning a reviewed version improves reproducibility and phased rollout, with documented migration paths. A pinned library still needs a secure SCM identity and compatibility tests; tags must be protected from retargeting. + +## 28. What belongs in vars versus src in a Jenkins Shared Library? + +*Topic: Libraries* + +**Model answer:** vars usually supplies global steps and human-friendly entrypoints, while src contains structured Groovy classes/utilities. Keep vars entrypoints small, validate parameters and avoid burying release policy in surprising side effects. CPS/serialization rules still matter when library code participates in Pipeline execution, and pure utilities should be independently unit-tested. + +## 29. Why is adding a shell command option to a shared step a security decision? + +*Topic: Libraries* + +**Model answer:** If a trusted step receives an arbitrary command string from an untrusted Jenkinsfile, it might execute that text with elevated credentials or on a privileged agent. Define narrow typed options, sanitize/validate inputs and enforce authorization at the agent/credential layer, not through a fragile regex. Test that untrusted callers cannot access privileged behavior. + +## 30. Does Jenkins masking prevent a malicious job from exfiltrating secrets? + +*Topic: Security* + +**Model answer:** No. Masking attempts to redact console output but the Pipeline or child process that receives the credential can intentionally transmit it elsewhere. Restrict which jobs can bind a credential, use isolated agents and egress policy, prefer short-lived least-privileged identities and rotate exposure. Treat log masking as accident reduction only. + +## 31. Why prefer single-quoted Groovy shell text with bound credentials? + +*Topic: Security* + +**Model answer:** Double-quoted Groovy strings interpolate before execution and can place secret values into Jenkins step arguments, metadata or process command lines. Single-quoted Groovy passes the literal dollar-variable to the agent shell for expansion after binding. Also disable shell tracing and avoid unsafe argument exposure. This pattern still isn't a complete security boundary. + +## 32. When is a Jenkins API crumb required? + +*Topic: Security* + +**Model answer:** Scripted state-changing requests with username/password authentication typically need a CSRF crumb and matching session behavior. Requests authenticated using API tokens are exempt from the standard crumb requirement in current documented Jenkins behavior. Don't disable CSRF to resolve integration breakage; use correctly authenticated clients and inspect the effective setup. + +## 33. Does a namespace separate hostile Jenkins tenants securely? + +*Topic: Security* + +**Model answer:** Not by itself. Shared worker nodes, privileged service accounts, a host Docker daemon, plugin/controller access and broad cloud identity can cross namespace boundaries. For genuinely hostile trust levels consider distinct cloud accounts/clusters and build controllers, plus workload admission, network policy and artifact authority boundaries. + +## 34. What should an incident response do after a Jenkins credential leak? + +*Topic: Security* + +**Model answer:** Revoke and rotate affected tokens/keys, isolate compromised agents, inspect Jenkins/job/SCM and cloud audit trails, identify every job and user allowed to bind the credential, and replace artifacts whose integrity cannot be established. Preserve sensitive evidence securely and never paste raw secrets into tickets. Correct the underlying scope and agent isolation afterward. + +## 35. Can JCasC YAML completely replace backing up JENKINS_HOME? + +*Topic: Lifecycle* + +**Model answer:** No. JCasC describes many supported configuration values, but job records, build history, plugin binaries/versions, secrets material and resumable Pipeline state may exist outside that representation. A recovery strategy needs consistent home-state backup, protected key material, versioned configuration and tested restoration. Treat JCasC as configuration governance, not a magical data recovery feature. + +## 36. Why store Jenkins master.key separately from routine backups? + +*Topic: Lifecycle* + +**Model answer:** Encrypted credentials rely on protected controller key material. Keeping every secret and its decryption key together in a widely accessible backup increases compromise impact. Official guidance recommends backing up master.key separately in a highly secured location. Test an isolated restore with correct keys and versions, while ensuring the test environment cannot contact production endpoints. + +## 37. Is downgrading Jenkins core after a failed plugin upgrade always a safe rollback? + +*Topic: Lifecycle* + +**Model answer:** No. Plugins/core may migrate persisted formats or require newer APIs and Java versions. A binary rollback alone can leave incompatible controller state. Use a verified backup from before migration, a compatible plugin inventory and an exercised recovery runbook. Validate restore and restart in a sandbox before the production maintenance window. + +## 38. Which metric proves Jenkins backup quality? + +*Topic: Lifecycle* + +**Model answer:** A successful backup job only proves files were written. Quality requires a restore into an isolated environment, functional job/auth/agent tests, intact build records and credentials, and measured RPO/RTO against objectives. Ensure backups are access controlled, encrypted and regularly checked for corruption. + +## 39. Does a 502 response prove Jenkins CPU is saturated? + +*Topic: Troubleshooting* + +**Model answer:** No. A reverse proxy may return 502 for connection refusal, TLS/backend resolution issues, no healthy target or premature connection closure; some timeouts produce 504 depending on the proxy. Correlate proxy error class, controller process uptime, JVM/GC pauses, disk and network metrics. Avoid making a resize decision based on status code alone. + +## 40. How can Jenkins have high CPU even when agents are idle? + +*Topic: Troubleshooting* + +**Model answer:** Controller-side Pipeline Groovy, plugin listeners, SCM indexing, logging, job history rendering and garbage collection can consume controller CPU independently of agent work. Gather JVM thread dumps, GC metrics and high-cost operation evidence. Move computation to agents, reduce expensive plugin/job behavior or adjust resources only after root-cause analysis. + +## 41. What causes disk-full incidents on a Jenkins controller? + +*Topic: Troubleshooting* + +**Model answer:** Build records, archived artifacts, logs, Pipeline state, plugin data and workspaces on the controller can grow. Measure disk and inode use and file ownership, then adjust build retention and external artifact storage. Don't delete random job or Pipeline files to reclaim space because doing so may destroy durable execution state and recovery data. + +## 42. What distinguishes a queued job from a stuck Pipeline stage? + +*Topic: Troubleshooting* + +**Model answer:** A queued task has not acquired eligible execution resources; the queue item and Jenkins nodes explain why. A running but stuck stage may be blocked on an external command, hung tests, lock or unresponsive agent, with an executor possibly occupied. Diagnose schedule latency, step logs and runtime process separately. + +## 43. When should organizations run multiple Jenkins controllers instead of one? + +*Topic: Governance* + +**Model answer:** Consider separate controllers for trust isolation, business criticality, plugin compatibility, workload scale or recovery domains. They increase maintenance and developer discovery burdens, so standardize templates, metrics, credential policy and upgrade automation. Multiple controllers are distinct installations, not active-active replicas of a single Jenkins state. + +## 44. Why separate CI artifact production from runtime deployment? + +*Topic: Governance* + +**Model answer:** CI routinely executes developer-supplied code, making its controller and agent identities high-risk places for broad production permissions. Producing a verified immutable digest first, then promoting through a distinct CD authorization process narrows blast radius and improves auditability. Direct Jenkins deployment can still be reasonable with appropriately scoped roles and controls. + +## 45. What is a Jenkins platform SLO that isn't simply controller uptime? + +*Topic: Governance* + +**Model answer:** Measure real developer journeys, such as p90 time from commit to actionable result, eligible build completion rate, executor provisioning latency and artifact traceability. Jenkins can be reachable but unable to launch agents or retrieve dependencies. Define denominator/exclusions and distinguish infrastructure-induced failures from legitimate failing code. + +## 46. How do you govern a breaking change to a shared Jenkins library? + +*Topic: Governance* + +**Model answer:** Treat it as a fleet-level release: semantic contract, tests of known consuming Jenkinsfiles, canary cohorts, version pinning and rollback plan. Notify owners and record plugin version compatibility. A global unversioned change can fail hundreds of unrelated services, so avoid silent immediate promotion. + +## 47. What is the first design question when migrating Jenkins agents to Kubernetes? + +*Topic: Governance* + +**Model answer:** Which failure and trust assumptions change? Map current agent toolchains, special hardware, cache behavior, workspaces, credentials, average and burst capacity, build isolation and cold-start SLOs. Ephemeral Pods help cleanup but add Kubernetes API, network and image-pull dependencies. A migration that only swaps VMs for Pods misses platform semantics. + +[Return to Jenkins module](./) · [Hands-on labs](./labs) From 5b2a11f1cb03ba23af3992e9ebccc02b23cb1ada Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:23:22 +0530 Subject: [PATCH 12/15] docs(jenkins): add eight practical and safe production-operation labs --- docs/guides/jenkins/labs.md | 122 ++++++++++++++++++++++++++++++++++++ 1 file changed, 122 insertions(+) create mode 100644 docs/guides/jenkins/labs.md diff --git a/docs/guides/jenkins/labs.md b/docs/guides/jenkins/labs.md new file mode 100644 index 0000000..be3502e --- /dev/null +++ b/docs/guides/jenkins/labs.md @@ -0,0 +1,122 @@ +# Jenkins Hands-on Labs and Failure Drills + +The exercises build operational confidence, not just Pipeline syntax familiarity. Use a **disposable Jenkins controller and isolated agents**. Do not connect to production SCM, registries, cloud accounts or customer databases. Pin tested plugin versions and keep a snapshot of the sandbox controller before experimenting. + +## Lab 1: Controller, queue and executor lifecycle + +**Goal:** Trace a job from scheduling through executor allocation to workspace and build record. + +Set built-in-node executors to zero. Create one isolated agent with a `linux-small` label. Submit two jobs requesting that label when only one executor is available. + +```groovy +pipeline { + agent { label 'linux-small' } + options { timestamps(); timeout(time: 5, unit: 'MINUTES') } + stages { + stage('Hold') { steps { sh 'sleep 15' } } + } +} +``` + +**Expected:** one task runs and the other waits with a clear queue reason, even if other label pools are idle. Observe agent registration and build record timestamps. **Safety:** keep hold time small; do not intentionally saturate shared production agents. + +## Lab 2: Declarative stages, errors and bounded retries + +Create a test repository with `scripts/test.sh`. Build a Declarative Pipeline using stage-level agents, `post`, `timeout`, and parallel independent tests. Inject a deterministic failure in one branch; compare result with and without explicit fail-fast policy. + +```groovy +pipeline { + agent none + stages { + stage('Check') { + agent { label 'linux-small' } + steps { checkout scm; sh './scripts/test.sh' } + } + } + post { always { echo 'Record the result' } } +} +``` + +**Expected:** a required failed check causes an unsuccessful pipeline; the finalization step still runs. Record which agent/workspace each stage used. **Do not** paper over genuine failures with `catchError`. + +## Lab 3: Multibranch and untrusted pull request threat model + +Create a local/training repository and Multibranch Pipeline. Compare indexing, webhook delivery, PR build behavior and SCM credential access with a read-only identity. + +Exercise a change request that modifies a test script, and answer: What would that script be able to read if the agent used a release role? Which protections come from trusted Jenkinsfile retrieval, and which require isolated compute/credentials? + +**Expected:** no production credentials or host privileges exist in the test agent. Document which source refs are actually checked out and the branch trust configuration. **Safety:** do not add real secrets to prove exfiltration. + +## Lab 4: Ephemeral Kubernetes agent troubleshooting + +In a sandbox Kubernetes cluster with the Jenkins Kubernetes plugin installed, define a small agent Pod. Observe each state transition and time to executor. + +```bash +kubectl get pods -n ci -o wide -w +kubectl get events -n ci --sort-by=.lastTimestamp +kubectl get resourcequotas -n ci +kubectl logs -n ci AGENT_POD -c jnlp --tail=100 +``` + +Inject a *sandbox-only* label mismatch or overly restrictive quota. Compare queue errors from plugin provisioning with scheduler `FailedScheduling` events and inbound remoting connection errors. + +**Expected:** clearly distinguish Pod creation failure, Pending scheduling, failed image pull and healthy Pod without agent connection. **Safety:** never use production CI namespaces. + +## Lab 5: Shared Library version pinning and test boundaries + +Create a training library with `vars/runChecks.groovy`, then two tagged versions that print different permitted build messages. Pin a consumer Jenkinsfile to version one while advancing the default library branch. + +```groovy +@Library('training-ci@v1') _ +pipeline { + agent { label 'linux-small' } + stages { + stage('Test') { + steps { runChecks() } + } + } +} +``` + +**Expected:** the pinned consumer retains version-one behavior. Explain why trusted global libraries demand administrator-level SCM review, and why a library step accepting arbitrary shell commands is a risky public API. Test only untrusted sandbox library permissions. + +## Lab 6: Controller backup and isolated restore + +Create a non-production Jenkins controller with a demo Pipeline and non-sensitive test credential. Record core/plugin/Java versions, then take a consistent home snapshot and store key material separately according to the security design. Restore into a **network-isolated** second controller. + +**Expected:** Jenkins starts, jobs and build history are present, and test credentials are readable only after authorized key restoration. Record measured RTO and the latest restorable point. If key material is absent, explain why the encrypted credential state may not be recoverable. **Safety:** never copy real production `master.key` into an unsecured sandbox. + +## Lab 7: Controller 502 and queue incident tabletop + +Use a hypothetical situation: users report 502s, agent CPU is idle, the queue grows and a plugin update was deployed an hour ago. + +Construct an evidence-first timeline: reverse-proxy upstream errors, Jenkins uptime, JVM heap/GC, disk/inodes, job queue reasons, cloud agent errors, and recent plugin changes. + +```bash +df -h +df -i +free -h +vmstat 1 5 +# JVM inspection only with explicit authorized process PID: +jcmd PID GC.heap_info +jcmd PID Thread.print +``` + +**Expected:** list at least three distinct root causes and how to disprove each. Define safe mitigation, not a reflexive restart or random heap increase. + +## Lab 8: Build once, promote immutable artifact + +Implement a sandbox test/build job that records a Git commit SHA, test results and a local container image identity. Sketch a second controlled promotion stage that consumes an *approved digest*, not a mutable tag. + +```bash +git rev-parse HEAD +docker image inspect IMAGE_ID --format '{{json .RepoDigests}}' +``` + +**Expected:** explain source-to-digest traceability and why publishing secrets cannot be exposed to PR agents. For multi-architecture output, distinguish image index from target-specific digest. **Safety:** local registry or mock upload only; use no real cloud deploy credentials. + +## Lab completion rubric + +For every exercise record **hypothesis, input contract, expected result, actual evidence, failure mode, containment, recovery and access boundaries**. When a live environment is unavailable, do the exercise as a written tabletop and explicitly label unverified behavior. A green Jenkins UI or test job is not proof that an application deployment succeeded. + +[Module overview](./) · [47 advanced technical FAQs](./follow-ups) From 72f389bc1358dea1ab8c79b50634bcf3546f00fe Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:23:57 +0530 Subject: [PATCH 13/15] docs(jenkins): publish Module 11 index and learning path --- docs/guides/jenkins/index.md | 67 ++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 docs/guides/jenkins/index.md diff --git a/docs/guides/jenkins/index.md b/docs/guides/jenkins/index.md new file mode 100644 index 0000000..692c505 --- /dev/null +++ b/docs/guides/jenkins/index.md @@ -0,0 +1,67 @@ +# Jenkins Architecture, Pipelines and Production Operations + +**Module 11.** A practical senior-level engineering guide to understanding how Jenkins works and how to operate it safely at scale. The material covers the controller/agent execution model, Declarative and Scripted Pipelines, SCM branch discovery, ephemeral build infrastructure, BuildKit and artifacts, Shared Libraries, credential isolation, disaster recovery, JVM troubleshooting and CI platform governance. + +::: info Scope and audience +All incidents, organizations, repositories and infrastructure examples are **hypothetical**. Jenkins behavior depends on installed plugin versions, core/LTS line, hosting model and security configuration. Code examples are teaching fixtures, not deployment-ready configurations. Apply changes only in authorized test environments. +::: + +## Prerequisites + +- Basic Linux processes, storage, networking, Java process diagnostics and Git +- Containers, Kubernetes Pods and RBAC fundamentals +- Familiarity with a CI job running source checkout, tests and a build +- Ability to read simple Groovy and shell; no previous plugin development is required + +## Learning outcomes + +You should be able to: + +- Trace a queued Jenkins job to a specific eligible agent and executor. +- Explain controller Pipeline CPS coordination, durability limits and where Groovy work executes. +- Write bounded, testable Jenkinsfiles and choose between Declarative and Scripted syntax. +- Secure Multibranch/Organization Folder pipelines against untrusted changes. +- Provision and debug Kubernetes agents, workspace reuse and toolchain drift. +- Build traceable container artifacts with safe caches, digests and isolated publishing. +- Govern Shared Libraries by version, API design and trust level. +- Isolate credentials, trusted steps and production authorization. +- Recover a stateful Jenkins controller using backups, protected keys and tested upgrades. +- Diagnose UI 502s, queue bottlenecks, CPS failures, disk pressure and agent disconnections. +- Design Jenkins as one component of a governed internal developer platform. + +## Ten chapters + +1. [Controller, agents, executors and complete build request flow](./q1) +2. [Declarative Pipeline, Groovy CPS, parallelism and reliable stages](./q2) +3. [Multibranch, SCM events, webhooks and untrusted change requests](./q3) +4. [Kubernetes agents, provisioning, queue latency and isolation](./q4) +5. [Container image builds, BuildKit caching and artifact provenance](./q5) +6. [Shared Libraries, trusted code, versioning and governance](./q6) +7. [Jenkins security: credentials, authentication and agent trust](./q7) +8. [Controller lifecycle: JCasC, backups, upgrades and restore](./q8) +9. [Production troubleshooting: queue, JVM, storage and 502 errors](./q9) +10. [CI platform architecture, GitOps handoff, SLOs and trade-offs](./q10) + +## Practice and advanced questions + +- [Eight hands-on exercises and game-day scenarios](./labs) +- [47 advanced technical FAQs with complete model answers](./follow-ups) + +## Recommended learning sequence + +Start with Chapters 1–3 to understand job discovery and orchestration. Then use Chapters 4–6 to connect execution, artifacts and reusable library design. Continue through Chapters 7–9 to learn security, recovery and incident response, and finish with Chapter 10's platform architecture. Run the labs only in a disposable Jenkins instance and rehearse the FAQ explanations aloud. + +## Authoritative references + +- [Jenkins user handbook](https://www.jenkins.io/doc/book/) +- [Using agents](https://www.jenkins.io/doc/book/using/using-agents/) +- [Pipeline syntax](https://www.jenkins.io/doc/book/pipeline/syntax/) +- [Jenkins Pipeline scaling](https://www.jenkins.io/doc/book/pipeline/scaling-pipeline/) +- [Shared Libraries](https://www.jenkins.io/doc/book/pipeline/shared-libraries/) +- [Jenkins Kubernetes plugin](https://plugins.jenkins.io/kubernetes/) +- [Jenkins security](https://www.jenkins.io/doc/book/security/) +- [Backing up and restoring Jenkins](https://www.jenkins.io/doc/book/system-administration/backing-up/) +- [Configuration as Code](https://plugins.jenkins.io/configuration-as-code/) +- [Jenkins upgrade guide](https://www.jenkins.io/doc/upgrade-guide/) + +[← Knowledge Base](/guides/) · [Start with chapter 1](./q1) From ffecc59c179cb7938d5649d2ff94ecc149ff7cc9 Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:24:19 +0530 Subject: [PATCH 14/15] docs(jenkins): add Module 11 sidebar and full chapter links --- docs/.vitepress/config.mts | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index e8a9b0f..1ac14d9 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -182,6 +182,23 @@ export default withMermaid(defineConfig({ { text: 'Eight hands-on labs', link: '/guides/ai-assisted-engineering/labs' }, { text: '40 advanced FAQs', link: '/guides/ai-assisted-engineering/follow-ups' }, ] }, + { text: 'Module 11 · Jenkins Operations', link: '/guides/jenkins/', collapsed: true, items: [ + { text: 'Overview', link: '/guides/jenkins/' }, + { text: 'Chapters', collapsed: false, items: [ + { text: '1. Controller, agents and queue', link: '/guides/jenkins/q1' }, + { text: '2. Pipeline and CPS execution', link: '/guides/jenkins/q2' }, + { text: '3. Multibranch and SCM security', link: '/guides/jenkins/q3' }, + { text: '4. Kubernetes ephemeral agents', link: '/guides/jenkins/q4' }, + { text: '5. Docker builds and caching', link: '/guides/jenkins/q5' }, + { text: '6. Shared Libraries', link: '/guides/jenkins/q6' }, + { text: '7. Credentials and trust boundaries', link: '/guides/jenkins/q7' }, + { text: '8. Backups, JCasC and upgrades', link: '/guides/jenkins/q8' }, + { text: '9. Production troubleshooting', link: '/guides/jenkins/q9' }, + { text: '10. Jenkins platform governance', link: '/guides/jenkins/q10' }, + ] }, + { text: 'Eight hands-on labs', link: '/guides/jenkins/labs' }, + { text: '47 advanced FAQs', link: '/guides/jenkins/follow-ups' }, + ] }, ] }, search: { provider: 'local' }, From 3c97e2b3c616f8c490bfa261d6102bd0083a0adb Mon Sep 17 00:00:00 2001 From: Edwin Philip <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:24:34 +0530 Subject: [PATCH 15/15] docs(jenkins): integrate Module 11 into engineering knowledge-base index --- docs/guides/index.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/guides/index.md b/docs/guides/index.md index b6aa5f6..83e3197 100644 --- a/docs/guides/index.md +++ b/docs/guides/index.md @@ -46,8 +46,12 @@ Six consolidated chapters covering Python collections, JSON/log/CSV parsing, UTC Eight in-depth chapters on interpreting coding requirements, planning, bounded agent changes, independent verification, multi-file engineering and evidence-based handoffs. Includes [eight labs](/guides/ai-assisted-engineering/labs) and [40 advanced FAQs](/guides/ai-assisted-engineering/follow-ups). +### [Jenkins Architecture, Pipelines & Production Operations · Module 11](/guides/jenkins/) + +Ten in-depth chapters on Jenkins controller/agent architecture, Declarative and Scripted Pipelines, Multibranch SCM integration, dynamic Kubernetes agents, Docker builds and caching, Shared Libraries, credentials and supply-chain security, stateful controller recovery, production troubleshooting and CI platform governance. Includes [eight hands-on labs](/guides/jenkins/labs) and [47 advanced FAQs](/guides/jenkins/follow-ups). + ## Learning path -Modules 1–10 cover networking, Linux, Kubernetes, infrastructure as code, cloud platforms, reliability, platform engineering, automation and responsible AI-assisted development. +Modules 1–11 cover networking, Linux, Kubernetes, infrastructure as code, cloud platforms, reliability, platform engineering, automation, responsible AI-assisted development and reliable Jenkins CI infrastructure. [About the author](/about)