Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/.vitepress/config.mts
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,23 @@ export default withMermaid(defineConfig({
{ text: 'Eight hands-on labs', link: '/guides/ai-assisted-engineering/labs' },
{ text: '40 advanced FAQs', link: '/guides/ai-assisted-engineering/follow-ups' },
] },
{ text: 'Module 11 · Jenkins Operations', link: '/guides/jenkins/', collapsed: true, items: [
{ text: 'Overview', link: '/guides/jenkins/' },
{ text: 'Chapters', collapsed: false, items: [
{ text: '1. Controller, agents and queue', link: '/guides/jenkins/q1' },
{ text: '2. Pipeline and CPS execution', link: '/guides/jenkins/q2' },
{ text: '3. Multibranch and SCM security', link: '/guides/jenkins/q3' },
{ text: '4. Kubernetes ephemeral agents', link: '/guides/jenkins/q4' },
{ text: '5. Docker builds and caching', link: '/guides/jenkins/q5' },
{ text: '6. Shared Libraries', link: '/guides/jenkins/q6' },
{ text: '7. Credentials and trust boundaries', link: '/guides/jenkins/q7' },
{ text: '8. Backups, JCasC and upgrades', link: '/guides/jenkins/q8' },
{ text: '9. Production troubleshooting', link: '/guides/jenkins/q9' },
{ text: '10. Jenkins platform governance', link: '/guides/jenkins/q10' },
] },
{ text: 'Eight hands-on labs', link: '/guides/jenkins/labs' },
{ text: '47 advanced FAQs', link: '/guides/jenkins/follow-ups' },
] },
]
},
search: { provider: 'local' },
Expand Down
6 changes: 5 additions & 1 deletion docs/guides/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,12 @@ Six consolidated chapters covering Python collections, JSON/log/CSV parsing, UTC

Eight in-depth chapters on interpreting coding requirements, planning, bounded agent changes, independent verification, multi-file engineering and evidence-based handoffs. Includes [eight labs](/guides/ai-assisted-engineering/labs) and [40 advanced FAQs](/guides/ai-assisted-engineering/follow-ups).

### [Jenkins Architecture, Pipelines & Production Operations · Module 11](/guides/jenkins/)

Ten in-depth chapters on Jenkins controller/agent architecture, Declarative and Scripted Pipelines, Multibranch SCM integration, dynamic Kubernetes agents, Docker builds and caching, Shared Libraries, credentials and supply-chain security, stateful controller recovery, production troubleshooting and CI platform governance. Includes [eight hands-on labs](/guides/jenkins/labs) and [47 advanced FAQs](/guides/jenkins/follow-ups).

## Learning path

Modules 1–10 cover networking, Linux, Kubernetes, infrastructure as code, cloud platforms, reliability, platform engineering, automation and responsible AI-assisted development.
Modules 1–11 cover networking, Linux, Kubernetes, infrastructure as code, cloud platforms, reliability, platform engineering, automation, responsible AI-assisted development and reliable Jenkins CI infrastructure.

[About the author](/about)
287 changes: 287 additions & 0 deletions docs/guides/jenkins/follow-ups.md

Large diffs are not rendered by default.

67 changes: 67 additions & 0 deletions docs/guides/jenkins/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Jenkins Architecture, Pipelines and Production Operations

**Module 11.** A practical senior-level engineering guide to understanding how Jenkins works and how to operate it safely at scale. The material covers the controller/agent execution model, Declarative and Scripted Pipelines, SCM branch discovery, ephemeral build infrastructure, BuildKit and artifacts, Shared Libraries, credential isolation, disaster recovery, JVM troubleshooting and CI platform governance.

::: info Scope and audience
All incidents, organizations, repositories and infrastructure examples are **hypothetical**. Jenkins behavior depends on installed plugin versions, core/LTS line, hosting model and security configuration. Code examples are teaching fixtures, not deployment-ready configurations. Apply changes only in authorized test environments.
:::

## Prerequisites

- Basic Linux processes, storage, networking, Java process diagnostics and Git
- Containers, Kubernetes Pods and RBAC fundamentals
- Familiarity with a CI job running source checkout, tests and a build
- Ability to read simple Groovy and shell; no previous plugin development is required

## Learning outcomes

You should be able to:

- Trace a queued Jenkins job to a specific eligible agent and executor.
- Explain controller Pipeline CPS coordination, durability limits and where Groovy work executes.
- Write bounded, testable Jenkinsfiles and choose between Declarative and Scripted syntax.
- Secure Multibranch/Organization Folder pipelines against untrusted changes.
- Provision and debug Kubernetes agents, workspace reuse and toolchain drift.
- Build traceable container artifacts with safe caches, digests and isolated publishing.
- Govern Shared Libraries by version, API design and trust level.
- Isolate credentials, trusted steps and production authorization.
- Recover a stateful Jenkins controller using backups, protected keys and tested upgrades.
- Diagnose UI 502s, queue bottlenecks, CPS failures, disk pressure and agent disconnections.
- Design Jenkins as one component of a governed internal developer platform.

## Ten chapters

1. [Controller, agents, executors and complete build request flow](./q1)
2. [Declarative Pipeline, Groovy CPS, parallelism and reliable stages](./q2)
3. [Multibranch, SCM events, webhooks and untrusted change requests](./q3)
4. [Kubernetes agents, provisioning, queue latency and isolation](./q4)
5. [Container image builds, BuildKit caching and artifact provenance](./q5)
6. [Shared Libraries, trusted code, versioning and governance](./q6)
7. [Jenkins security: credentials, authentication and agent trust](./q7)
8. [Controller lifecycle: JCasC, backups, upgrades and restore](./q8)
9. [Production troubleshooting: queue, JVM, storage and 502 errors](./q9)
10. [CI platform architecture, GitOps handoff, SLOs and trade-offs](./q10)

## Practice and advanced questions

- [Eight hands-on exercises and game-day scenarios](./labs)
- [47 advanced technical FAQs with complete model answers](./follow-ups)

## Recommended learning sequence

Start with Chapters 1–3 to understand job discovery and orchestration. Then use Chapters 4–6 to connect execution, artifacts and reusable library design. Continue through Chapters 7–9 to learn security, recovery and incident response, and finish with Chapter 10's platform architecture. Run the labs only in a disposable Jenkins instance and rehearse the FAQ explanations aloud.

## Authoritative references

- [Jenkins user handbook](https://www.jenkins.io/doc/book/)
- [Using agents](https://www.jenkins.io/doc/book/using/using-agents/)
- [Pipeline syntax](https://www.jenkins.io/doc/book/pipeline/syntax/)
- [Jenkins Pipeline scaling](https://www.jenkins.io/doc/book/pipeline/scaling-pipeline/)
- [Shared Libraries](https://www.jenkins.io/doc/book/pipeline/shared-libraries/)
- [Jenkins Kubernetes plugin](https://plugins.jenkins.io/kubernetes/)
- [Jenkins security](https://www.jenkins.io/doc/book/security/)
- [Backing up and restoring Jenkins](https://www.jenkins.io/doc/book/system-administration/backing-up/)
- [Configuration as Code](https://plugins.jenkins.io/configuration-as-code/)
- [Jenkins upgrade guide](https://www.jenkins.io/doc/upgrade-guide/)

[← Knowledge Base](/guides/) · [Start with chapter 1](./q1)
122 changes: 122 additions & 0 deletions docs/guides/jenkins/labs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# Jenkins Hands-on Labs and Failure Drills

The exercises build operational confidence, not just Pipeline syntax familiarity. Use a **disposable Jenkins controller and isolated agents**. Do not connect to production SCM, registries, cloud accounts or customer databases. Pin tested plugin versions and keep a snapshot of the sandbox controller before experimenting.

## Lab 1: Controller, queue and executor lifecycle

**Goal:** Trace a job from scheduling through executor allocation to workspace and build record.

Set built-in-node executors to zero. Create one isolated agent with a `linux-small` label. Submit two jobs requesting that label when only one executor is available.

```groovy
pipeline {
agent { label 'linux-small' }
options { timestamps(); timeout(time: 5, unit: 'MINUTES') }
stages {
stage('Hold') { steps { sh 'sleep 15' } }
}
}
```

**Expected:** one task runs and the other waits with a clear queue reason, even if other label pools are idle. Observe agent registration and build record timestamps. **Safety:** keep hold time small; do not intentionally saturate shared production agents.

## Lab 2: Declarative stages, errors and bounded retries

Create a test repository with `scripts/test.sh`. Build a Declarative Pipeline using stage-level agents, `post`, `timeout`, and parallel independent tests. Inject a deterministic failure in one branch; compare result with and without explicit fail-fast policy.

```groovy
pipeline {
agent none
stages {
stage('Check') {
agent { label 'linux-small' }
steps { checkout scm; sh './scripts/test.sh' }
}
}
post { always { echo 'Record the result' } }
}
```

**Expected:** a required failed check causes an unsuccessful pipeline; the finalization step still runs. Record which agent/workspace each stage used. **Do not** paper over genuine failures with `catchError`.

## Lab 3: Multibranch and untrusted pull request threat model

Create a local/training repository and Multibranch Pipeline. Compare indexing, webhook delivery, PR build behavior and SCM credential access with a read-only identity.

Exercise a change request that modifies a test script, and answer: What would that script be able to read if the agent used a release role? Which protections come from trusted Jenkinsfile retrieval, and which require isolated compute/credentials?

**Expected:** no production credentials or host privileges exist in the test agent. Document which source refs are actually checked out and the branch trust configuration. **Safety:** do not add real secrets to prove exfiltration.

## Lab 4: Ephemeral Kubernetes agent troubleshooting

In a sandbox Kubernetes cluster with the Jenkins Kubernetes plugin installed, define a small agent Pod. Observe each state transition and time to executor.

```bash
kubectl get pods -n ci -o wide -w
kubectl get events -n ci --sort-by=.lastTimestamp
kubectl get resourcequotas -n ci
kubectl logs -n ci AGENT_POD -c jnlp --tail=100
```

Inject a *sandbox-only* label mismatch or overly restrictive quota. Compare queue errors from plugin provisioning with scheduler `FailedScheduling` events and inbound remoting connection errors.

**Expected:** clearly distinguish Pod creation failure, Pending scheduling, failed image pull and healthy Pod without agent connection. **Safety:** never use production CI namespaces.

## Lab 5: Shared Library version pinning and test boundaries

Create a training library with `vars/runChecks.groovy`, then two tagged versions that print different permitted build messages. Pin a consumer Jenkinsfile to version one while advancing the default library branch.

```groovy
@Library('training-ci@v1') _
pipeline {
agent { label 'linux-small' }
stages {
stage('Test') {
steps { runChecks() }
}
}
}
```

**Expected:** the pinned consumer retains version-one behavior. Explain why trusted global libraries demand administrator-level SCM review, and why a library step accepting arbitrary shell commands is a risky public API. Test only untrusted sandbox library permissions.

## Lab 6: Controller backup and isolated restore

Create a non-production Jenkins controller with a demo Pipeline and non-sensitive test credential. Record core/plugin/Java versions, then take a consistent home snapshot and store key material separately according to the security design. Restore into a **network-isolated** second controller.

**Expected:** Jenkins starts, jobs and build history are present, and test credentials are readable only after authorized key restoration. Record measured RTO and the latest restorable point. If key material is absent, explain why the encrypted credential state may not be recoverable. **Safety:** never copy real production `master.key` into an unsecured sandbox.

## Lab 7: Controller 502 and queue incident tabletop

Use a hypothetical situation: users report 502s, agent CPU is idle, the queue grows and a plugin update was deployed an hour ago.

Construct an evidence-first timeline: reverse-proxy upstream errors, Jenkins uptime, JVM heap/GC, disk/inodes, job queue reasons, cloud agent errors, and recent plugin changes.

```bash
df -h
df -i
free -h
vmstat 1 5
# JVM inspection only with explicit authorized process PID:
jcmd PID GC.heap_info
jcmd PID Thread.print
```

**Expected:** list at least three distinct root causes and how to disprove each. Define safe mitigation, not a reflexive restart or random heap increase.

## Lab 8: Build once, promote immutable artifact

Implement a sandbox test/build job that records a Git commit SHA, test results and a local container image identity. Sketch a second controlled promotion stage that consumes an *approved digest*, not a mutable tag.

```bash
git rev-parse HEAD
docker image inspect IMAGE_ID --format '{{json .RepoDigests}}'
```

**Expected:** explain source-to-digest traceability and why publishing secrets cannot be exposed to PR agents. For multi-architecture output, distinguish image index from target-specific digest. **Safety:** local registry or mock upload only; use no real cloud deploy credentials.

## Lab completion rubric

For every exercise record **hypothesis, input contract, expected result, actual evidence, failure mode, containment, recovery and access boundaries**. When a live environment is unavailable, do the exercise as a written tabletop and explicitly label unverified behavior. A green Jenkins UI or test job is not proof that an application deployment succeeded.

[Module overview](./) · [47 advanced technical FAQs](./follow-ups)
109 changes: 109 additions & 0 deletions docs/guides/jenkins/q1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Jenkins Architecture: Controller, Agents, Queue and Execution Flow

> **Engineering scenario:** A company operates hundreds of builds across different languages and architectures. Explain what the Jenkins controller owns, what agents execute, and what happens from a webhook to an archived artifact. Where can the entire system fail?

## Architectural model

Jenkins is an extensible automation server. Its controller serves the UI and API, handles authentication/authorization, owns job definitions and build records, coordinates Pipeline execution, tracks agent connections, schedules work and integrates plugins. **Executors run work on agents**. Configure zero executors on the built-in controller node in a production design so untrusted build code does not execute alongside its management plane.

```mermaid
flowchart TD
SCM["SCM webhook / periodic indexing"] --> WEB["Jenkins controller UI/API"]
WEB --> QUEUE["Queue: runnable tasks and resource constraints"]
QUEUE --> PICK{"Eligible online executor?"}
PICK -->|No| WAIT["Queue / dynamic provisioning"]
WAIT --> PICK
PICK -->|Yes| AGENT["Agent JVM and workspace"]
AGENT --> TOOLS["Compiler, tests, container build"]
TOOLS --> STORE["External artifact and test storage"]
AGENT --> LOG["Build log and result events"]
LOG --> HOME[("Controller JENKINS_HOME")]
WEB <--> HOME
```

This is a logical flow; some plugin actions, SCM probes, Groovy orchestration and metadata access execute on the controller. The `agent` directive and `node` step acquire an executor and normally a workspace. A Pipeline does not automatically reserve a single executor for its entire lifetime if it is designed with `agent none` and stage-level agents.

## Terms that must not be conflated

| Term | Meaning | Failure signature |
| --- | --- | --- |
| Controller | Coordinates jobs, plugins, users, history and Pipeline graph | UI/API slow, queue/flows stop progressing |
| Node | Registered machine or cloud-provisioned compute | Offline/mismatched labels |
| Agent | Remoting process providing execution on a node | Lost connection or launch failure |
| Executor | Slot for one concurrently allocated task | All busy even while CPU is idle |
| Workspace | Filesystem area for checkout/build on a node | Disk full, cross-job contamination, stale files |
| Queue | Jobs waiting for executor/resources/provisioning | Waiting for next available executor |
| Build record | Console output, artifacts/metadata and status | Storage pressure, slow job listing |

A label is a scheduling constraint, not a security boundary. Multiple executors on one agent can share filesystem and operating-system identity, so running unrelated untrusted tenants together remains risky even if the jobs have different workspaces.

### End-to-end sequence

```mermaid
sequenceDiagram
participant SCM as Source hosting
participant J as Jenkins controller
participant Q as Queue
participant K as Cloud/agent provisioner
participant A as Agent
participant R as Registry
SCM->>J: Webhook (repository/ref update)
J->>J: Validate event / evaluate branch policy
J->>Q: Schedule Pipeline run
Q->>K: Need node with eligible label
K->>A: Provision node or Pod, launch agent
A-->>J: Agent registers and supplies executors
J->>A: Checkout selected revision
A->>A: Build, test and package
A->>R: Publish immutable artifact
A-->>J: Step logs, status and metadata
J->>J: Persist result and notify
```

## Durability and CPS nuance

Pipeline's Groovy execution model supports persisted program state across some controller restarts. It is **not equivalent** to durable arbitrary agent processes: the step, agent availability, workspace and external commands determine what resumes safely. Pipeline CPS continuation state and build records create controller I/O and CPU demand; doing large data transformations in Groovy on the controller is a common scaling mistake. Offload heavy parsing or dependency operations to agents via `sh` or language-specific executables.

```groovy
pipeline {
agent none
options { timestamps(); timeout(time: 30, unit: 'MINUTES') }
stages {
stage('Verify') {
agent { label 'linux-small' }
steps {
checkout scm
sh 'make test'
}
}
}
}
```

This requires a configured eligible agent and a repository with a real test target.

## Production incident: agent nodes are online, builds remain queued

Check **why** an item is queued before adding capacity: labels, node mode, offline cause, executor busy state, throttling/resource locks, or cloud provisioning errors. Increasing controller CPU does not resolve a missing label. Likewise, adding agent Pods is ineffective if the Kubernetes API denies creation or the agent cannot connect to the controller.

```bash
# Read-only Linux diagnostics on an authorized controller host:
systemctl status jenkins
journalctl -u jenkins --since '15 minutes ago' --no-pager
df -h
df -i
# In the UI: Build Queue -> item -> 'Why is it not running?'
# In Manage Jenkins -> Nodes: labels, online status, executor utilization
```

The service commands apply to a systemd deployment, not every container or Helm installation.

## Trade-offs and platform leadership

Jenkins is flexible but carries stateful controller, plugin and upgrade responsibilities. Separate trusted release agents from untrusted PR agents; use ephemeral agents where appropriate; define retention and disaster-recovery objectives for the controller. Treat agent capacity and controller throughput as different resource budgets.

### Spoken explanation

> “Jenkins has a controller responsible for scheduling, pipeline coordination, auth, plugins and records; the agents execute actual workloads through executors and workspaces. A webhook queues a versioned job, Jenkins matches it to an eligible agent, runs build/test/publish stages, and persists logs and results. I would keep executors off the controller, isolate trust levels across agents, and diagnose queue constraints separately from controller performance. The controller is stateful, so backup, I/O and restart recovery remain essential.”

**References:** [Using Jenkins agents](https://www.jenkins.io/doc/book/using/using-agents/) · [Managing nodes](https://www.jenkins.io/doc/book/managing/nodes/) · [Scaling Pipelines](https://www.jenkins.io/doc/book/pipeline/scaling-pipeline/)
Loading
Loading