Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
241 changes: 128 additions & 113 deletions docs/.vitepress/config.mts

Large diffs are not rendered by default.

10 changes: 9 additions & 1 deletion docs/guides/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,16 @@ Eight comprehensive chapters covering GCP hierarchy, identity and federation, Sh

Eight in-depth chapters on GKE control plane, Standard/Autopilot, node scheduling, VPC-native networking/eBPF, Ingress/Gateway/NEGs, workload identity, CSI, autoscaling, upgrades and incident recovery. Includes [eight labs](/guides/gke/labs) and [40 difficult follow-ups](/guides/gke/follow-ups).

### [Cloud Architecture, Reliability & System Design · Module 7](/guides/reliability/)

Eight in-depth chapters covering SLIs/SLOs, high availability, failure domains, backpressure, durable transaction correctness, multi-region disaster recovery, capacity, data-layer resilience and incident leadership. Includes [40 difficult follow-ups](/guides/reliability/follow-ups) and [seven hands-on labs](/guides/reliability/labs).

### [CI/CD, GitOps & Platform Engineering · Module 8](/guides/platform-engineering/)

Eight chapters on trusted builds, GitOps reconciliation, progressive delivery, supply-chain security, federated identity, internal developer platforms, observability and multi-cluster governance. Includes [40 difficult follow-ups](/guides/platform-engineering/follow-ups) and [eight hands-on labs](/guides/platform-engineering/labs).

## Planned

Reliability, CI/CD and platform engineering, and Python data processing.
Module 9: Python coding and data processing.

[About the author](/about)
245 changes: 245 additions & 0 deletions docs/guides/platform-engineering/follow-ups.md

Large diffs are not rendered by default.

49 changes: 49 additions & 0 deletions docs/guides/platform-engineering/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# CI/CD, GitOps & Platform Engineering

**Module 8.** Eight comprehensive senior-level chapters on secure delivery architecture, pull-based deployment reconciliation, progressive rollout, artifact supply-chain security, federated identity, developer platforms, observability and multi-cluster governance.

::: info Public learning material
All company, repository, marketplace and cloud examples are hypothetical. Manifest fragments require appropriate controllers, versions, policies and environment-specific values. This module does not disclose or describe a particular employer's systems.
:::

## Prerequisites

Git, CI basics, containers, Kubernetes workload lifecycle, networking, IAM fundamentals, Terraform and a basic understanding of service-level objectives.

## Learning objectives

- Trace source review through reproducible or verifiable build, immutable digest and controlled promotion.
- Explain Argo CD reconciliation and handle drift, pruning and emergency intervention.
- Select canary, blue-green, rolling or feature-flag rollout strategies and business-aware gates.
- Threat-model build runners, dependencies, tokens, SBOMs, signatures and provenance.
- Isolate human, CI, CD and runtime identities using short-lived credentials and least privilege.
- Design usable self-service contracts, golden paths and safe cleanup.
- Define platform user-journey SLOs, DORA-style delivery trends and telemetry cost limits.
- Govern multi-cluster change ownership, upgrades, disaster recovery and incident response.

## Eight chapters

1. [CI/CD architecture, trust boundaries and immutable artifacts](./q1)
2. [GitOps, Argo CD, drift and ownership](./q2)
3. [Progressive delivery and controlled rollout analysis](./q3)
4. [Software supply-chain security, SBOM and provenance](./q4)
5. [Identity, secrets and environment isolation](./q5)
6. [Internal developer platforms and self-service contracts](./q6)
7. [Platform observability and engineering effectiveness](./q7)
8. [Multi-cluster governance, recovery and technical leadership](./q8)

## Practice and sequence

Begin with Q1-Q4 to establish the trusted delivery chain; continue with Q5-Q6 for security and developer experience; finish with Q7-Q8 for operating a platform as a product. Work through [eight labs](./labs) and [40 difficult follow-up questions with complete answers](./follow-ups).

## Authoritative references

- [GitHub Actions security](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions)
- [Argo CD user guide](https://argo-cd.readthedocs.io/en/stable/user-guide/)
- [Argo Rollouts analysis](https://argoproj.github.io/argo-rollouts/features/analysis/)
- [SLSA software supply chain framework](https://slsa.dev/)
- [Kubernetes Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)
- [CNCF platform engineering resources](https://tag-app-delivery.cncf.io/whitepapers/platforms/)
- [DORA](https://dora.dev/)

[← All engineering guides](/guides/) · [Start Q1 →](./q1)
100 changes: 100 additions & 0 deletions docs/guides/platform-engineering/labs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Module 8 · CI/CD, GitOps and Platform Engineering Labs

These exercises should be run in a **disposable repository and Kubernetes sandbox**, not with production credentials. For risky scenarios, use a tabletop simulation instead of a live failure.

## Lab 1: Design a trust-separated CI and CD pipeline

Create a repository with a pull-request workflow running lint/unit tests, a protected-main workflow building an immutable artifact, and a separate promotion PR updating a manifest digest.

```bash
git rev-parse HEAD
npm ci
npm test
docker build -t catalog-test:local .
docker image inspect catalog-test:local --format '{{.Id}}'
```

**Expected:** the test job cannot deploy, source revision and artifact digest are traceable, and the production update requires an approved change. **Safety:** no real cloud secrets in a PR job.

## Lab 2: Show GitOps drift and reconcile behavior

Install Argo CD only in a sandbox. Apply a sample Application with automated sync and explicit `prune: false`. Make an authorized manual change to a managed field, then compare behavior with self-heal disabled and enabled.

```bash
argocd app get catalog-preview
argocd app diff catalog-preview
argocd app history catalog-preview
kubectl get application catalog-preview -n argocd -o yaml
```

**Expected:** Synced and Healthy are different concepts; drift correction follows sync policy and owner behavior. Record how to suspend the right ApplicationSet owner during an emergency. **Safety:** do not experiment with automatic prune on shared namespaces.

## Lab 3: Progressive canary with rollback criteria

In a sandbox with Argo Rollouts installed, create an AnalysisTemplate that checks an instrumented error or success counter and pauses at 10% traffic. Inject failures only into the canary version. Verify that the controller stops or aborts according to the template.

```bash
kubectl argo rollouts get rollout catalog -n marketplace --watch
kubectl argo rollouts history catalog -n marketplace
```

**Expected:** candidate error signal produces an observable gate result. Verify actual traffic split and sample size, not just a configured `setWeight`. **Safety:** use test data and test routing only.

## Lab 4: Test OIDC trust boundaries on paper and in a sandbox

Draft two federated trust policies: one incorrectly accepts all repository branches; the other limits deployment to one repository and protected production environment. Evaluate whether a fork PR, arbitrary branch and approved main environment can assume the role.

**Expected:** untrusted runs are denied. Cloud role permissions remain least privilege even after the identity is authenticated. Do not issue or persist real long-lived production keys to prove the concept.

## Lab 5: Track SBOM, provenance and digest identity

Build a toy container using pinned dependencies. Generate an SBOM with an approved local tool, capture the image digest and create a mock provenance statement linking builder/source/digest. Change one build input and compare identity.

```bash
git rev-parse HEAD
docker image inspect catalog-test:local --format '{{json .RepoDigests}}'
```

**Expected:** a tag label alone cannot prove immutable content; signature and provenance verification require trusted signer/builder policy. A green scanner does not guarantee safe source. **Safety:** no secret environment variables in provenance/logs.

## Lab 6: Model an internal platform API with honest status

Define a fictional `ServiceEnvironment` contract with owner, TTL, resource class and exposure. In a design exercise, enumerate reconcile outcomes: Pending, Ready, Blocked, Deleting and Failed. Describe finalizer handling when DNS creation succeeds but volume provisioning fails.

```yaml
apiVersion: platform.example.com/v1alpha1
kind: ServiceEnvironment
metadata: {name: test-preview}
spec:
owner: catalog-team
class: ephemeral
ttlHours: 8
exposure: internal
```

**Expected:** stable identity, idempotent reconcile, useful status conditions and safe cleanup. This API is hypothetical and won't work without a CRD/controller.

## Lab 7: Build a platform user-journey SLI

Create synthetic event records for environment-request start, validation, reconciliation, readiness and user-access verification. Compute the fraction of eligible environments usable within ten minutes. Compare that with controller uptime.

**Expected:** the user-journey SLI can be poor when controller health is green; identify step latency with traces. Avoid high-cardinality request IDs as always-on Prometheus labels.

## Lab 8: Multi-cluster recovery tabletop

Simulate a bad organization-wide policy that breaks new Pod admission while existing workloads run. Identify which controller owns the change, separate production from nonproduction, preserve audit logs and choose a narrowly scoped recovery.

```bash
kubectl get validatingwebhookconfigurations
kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp
kubectl auth can-i create deployments -n marketplace
```

**Expected:** a documented independent break-glass path, controlled rollback or forward-fix, recovery of developer workflows and reconciliation into Git/IaC. Never disable all security admission in production as a lab.

## Lab completion criteria

For every lab write down the trust boundary, desired behavior, observables, failure mode, stop condition and recovery procedure. Distinguish **pipeline success**, **GitOps reconciliation** and **correct customer/developer outcome**.

[Module 8 index](./) · [40 challenging follow-ups](./follow-ups)
75 changes: 75 additions & 0 deletions docs/guides/platform-engineering/q1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Q1 · CI/CD Architecture, Reproducible Builds and Delivery Boundaries

> **Interviewer:** Design a secure and fast pipeline for 60 independently deployed microservices. What must CI prove, what does CD own, and how do you ensure the production artifact is the exact code that passed tests?

## First principles and system architecture

Continuous integration verifies a specific source revision and produces evidence that it can be built and tested. Continuous delivery promotes an approved **immutable artifact** into a controlled environment; continuous deployment automates that promotion subject to policy. CI success is not equivalent to business success in production. Draw trust boundaries around untrusted contribution, privileged release and cloud deployment.

```mermaid
flowchart TD
PR["Reviewed source revision"] --> VALIDATE["Lint, unit, contract, integration tests"]
VALIDATE --> BUILD["Isolated deterministic builder"]
BUILD --> SBOM["SBOM, scans, provenance"]
SBOM --> SIGN["Immutable digest and attestation"]
SIGN --> REG["Artifact registry"]
REG --> CHANGE["Promotion Git PR / release record"]
CHANGE --> APPROVE["Environment approval and policy"]
APPROVE --> CD["Deployment controller"]
CD --> CHECK["Smoke, SLO and business outcome verification"]
```

### Control boundaries

- **Source integrity:** protected branch, code review, CODEOWNERS, required checks, dependency update policy. Forked PR code must not receive deployment credentials.
- **Build isolation:** pin runtime and lockfile, use ephemeral runners where possible, restrict writeable shared caches, record build input and artifact identity.
- **Release authority:** only a narrowly authorized job or pull-based reconciler may deploy a reviewed digest to production. Do not use floating image tags as a trusted release identity.
- **Verification:** compare deployed digest to CI record; capture release version in traces; roll back according to pre-agreed health and business criteria.

### GitHub Actions CI example

```yaml
name: Verify
on:
pull_request:
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: {node-version: "20", cache: npm}
- run: npm ci
- run: npm test
- run: npm run build
```

This is an instructional CI job, not an attested production release pipeline. For real deployment, pin third-party actions to reviewed commit SHAs, isolate privileges and add organization-appropriate provenance, approvals and digest publication.

## Build once, promote by digest

```bash
docker build --pull -t registry.example.com/catalog:build-1234 .
docker push registry.example.com/catalog:build-1234
# Resolve immutable sha256 digest from registry/build output.
kubectl get deploy catalog-api -n marketplace \
-o jsonpath='{.spec.template.spec.containers[*].image}'
```

Tags can be mutated; a digest binds the deployment to bytes. Dependency lockfiles alone do not guarantee bit-for-bit reproducibility because builders, timestamps, operating-system packages and network downloads may vary. Separate **reproducibility** (same outputs) from **provenance** (trusted evidence of how outputs were produced).

## Test strategy and feedback loop

Use fast unit/static checks for every commit, interface/contract tests for compatibility, ephemeral integration environments for critical dependencies and a small number of valuable end-to-end user journeys. Measure queue wait, test flake rate, time to first useful failure and test failure classification. Optimizations must not let an untrusted cache contaminate privileged builds.

### Hypothetical incident

The `stable` Docker tag is repointed after CI tests commit A. Production later pulls commit B, which never passed release tests. Compare deployed RepoDigests with the build attestation, halt promotions and restore the known tested digest. Fix the promotion pipeline to record and enforce immutable identities.

### Senior trade-offs and spoken answer

> “I would isolate three identities: the reviewed source revision, the immutable artifact digest, and the approved environment deployment record. CI verifies the source and builds once on trusted infrastructure; CD promotes that exact digest with least privilege and observed SLO gates. I would optimize feedback speed without sacrificing trust boundaries or the ability to audit exactly what code is running.”

**References:** [GitHub Actions hardening](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions) · [SLSA](https://slsa.dev/)
92 changes: 92 additions & 0 deletions docs/guides/platform-engineering/q2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Q2 · GitOps Reconciliation, Argo CD, Drift and Ownership

> **Interviewer:** Explain what Argo CD actually reconciles. How does GitOps differ from simply running kubectl in CI? What happens when an SRE fixes a production incident manually but GitOps keeps reverting the change?

## Desired state and data plane

GitOps is a control-loop architecture, not a synonym for Git-hosted YAML. A reviewed configuration source defines desired state; a controller observes live state, computes the difference and converges where allowed. CI should produce and attest the artifact; the delivery repository references an immutable image digest; Argo CD applies declarative desired state using controlled cluster credentials.

```mermaid
sequenceDiagram
participant D as Developer
participant CI as CI build
participant R as Registry
participant G as Config Git
participant A as Argo CD
participant K as Kubernetes API
D->>CI: Merge reviewed commit
CI->>R: Push immutable digest
CI->>G: Propose promotion PR
G-->>A: Approved change observed
A->>K: Diff rendered desired and live
A->>K: Sync permitted resources
K-->>A: Watch state and health
```

## Meaning of status

**Synced** means the configured comparison rules find desired and observed resources aligned. It does not prove the application can complete business transactions. **Healthy** depends on resource-specific assessments and can miss downstream correctness. **Prune** enables deletion when a formerly managed resource is removed from Git; **selfHeal** attempts to correct drift initiated outside Git. Without an explicit sync trigger and self-heal, a manual patch can remain.

Argo CD operates with reconciliation intervals, webhooks and caches, so changes are not universally instantaneous. ApplicationSet controllers can recreate or override Application specifications. The emergency suspension mechanism must target the actual owner, not only a generated child object. <https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/>

## Application example

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: catalog-prod
namespace: argocd
spec:
project: catalog
source:
repoURL: https://github.com/example-org/deployment-config.git
targetRevision: main
path: apps/catalog/prod
destination:
server: https://kubernetes.default.svc
namespace: marketplace
syncPolicy:
automated:
prune: false
selfHeal: true
syncOptions:
- CreateNamespace=false
```

This needs the Argo CD CRDs/controller, a legitimate `catalog` AppProject, repo access and destination authorization. For a large team, scope sources, clusters, namespaces and resources with AppProjects and Kubernetes RBAC. Do not operate every team from a single unrestricted application credential.

## Who owns replicas, fields and secrets?

If HPA controls `spec.replicas` but GitOps insists on a fixed value, two controllers can repeatedly fight. Avoid unmanaged overlapping writers; adopt a defined ownership policy or targeted diff ignore when warranted. Operators also mutate annotations, generated ConfigMaps and CRD status. Ignore only known legitimate fields, not the entire spec. Do not commit plaintext production secrets; use an approved encryption or external secrets mechanism, and authorize the controller narrowly.

```mermaid
flowchart TD
SRC["Approved config Git"] --> ARGO["Argo reconciler"]
ARGO --> API["Kubernetes API"]
HPA["Autoscaler"] --> API
OP["Operator / admission controller"] --> API
API --> LIVE["Live object"]
LIVE -.->|"Observed changes"| ARGO
OWN["Field ownership contract"] -.-> ARGO
OWN -.-> HPA
OWN -.-> OP
```

## Hypothetical incident: emergency fix reverts

A production operator increases replicas or reverts an image manually while `selfHeal` remains active. The GitOps controller restores the broken Git version. The solution is a documented break-glass process: suspend the correct automated owner when authorized, approve the emergency revision in Git, or isolate the release path for rollback. Record the temporary override and reconcile it after recovery. Audit log and Git diff must explain what actually happened.

```bash
argocd app get catalog-prod
argocd app diff catalog-prod
argocd app history catalog-prod
kubectl get application catalog-prod -n argocd -o yaml
kubectl get applicationsets -n argocd
```

## Spoken answer

> “GitOps is declarative reconciliation from a reviewed desired-state source, separate from the software build. The controller observes Git and live resources and applies allowed changes, with policies for self-heal and pruning. I'd distinguish Synced from real application health and define field ownership alongside HPA and operators. During an incident, the emergency process must change the source of truth or safely suspend the correct reconciler so it cannot immediately undo the fix.”

**References:** [Argo CD sync policy](https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/) · [Argo CD projects](https://argo-cd.readthedocs.io/en/stable/user-guide/projects/)
Loading
Loading