Secure Kubernetes Platform on GKE

A private GKE cluster, a couple of workloads, and an AI agent that handles the security findings nobody wants to read.

Terraform builds it. GitHub Actions ships to it, keylessly. Each phase has a worklog, and decisions.md covers the why.

Closed on 2026-09-22. plan.md has the status table and what stayed optional. The site below still runs.

Live

sindrg.com

Architecture

flowchart TB
    User(["User"])
    Developer(["Developer"])
    DNS["Cloudflare DNS<br/>sindrg.com"]

    subgraph Delivery["Infrastructure and delivery"]
        Terraform["Terraform"]
        GitHub["GitHub Actions"]
    end

    subgraph GCP["Google Cloud"]
        Ingress["Global external<br/>Application Load Balancer<br/>reserved address"]
        Certs["Certificate Manager<br/>managed TLS"]
        Registry["Artifact Registry"]
        Identity["Workload<br/>Identity<br/>Federation"]
        Uptime["Uptime check<br/>three prober regions"]
        Observability["Cloud Logging<br/>Cloud Monitoring<br/>dashboard and alert policies"]
        Notify["Email notification<br/>channel"]
        SCC["Security Command Center"]
        PubSub["Pub/Sub findings subscription<br/>and dead letter topic"]
        Vertex["Vertex AI<br/>gemini-2.5-flash"]
        Ledger["Verdict ledger<br/>create-only bucket"]

        subgraph VPC["Custom VPC"]
            ControlPlane["GKE control plane<br/>DNS-only endpoint"]

            subgraph Cluster["GKE node pool, floor of two nodes"]
                Routes["Gateway and<br/>HTTPRoutes"]
                Nginx["nginx<br/>two replicas<br/>serves /"]
                Sky["sky<br/>two replicas<br/>serves /sky, /api, /static"]
                Guardrails["Pod Security, NetworkPolicy,<br/>quotas, disruption budgets"]
                Worker["triage worker<br/>agents namespace"]
            end

            NAT["Cloud NAT"]
        end
    end

    User --> DNS
    DNS --> Ingress
    Certs -. terminates TLS .-> Ingress
    Certs -. validated by a DNS record .-> DNS
    Ingress -- "Pod IPs via NEG" --> Nginx
    Ingress -- "Pod IPs via NEG" --> Sky
    Routes -. configures .-> Ingress
    Guardrails -. protects .-> Nginx
    Guardrails -. protects .-> Sky
    Developer --> Terraform
    Developer --> GitHub
    Terraform --> ControlPlane
    GitHub -. federates .-> Identity
    GitHub -. builds and pushes .-> Registry
    GitHub -. deploys .-> ControlPlane
    Registry -. images pulled by nodes .-> Cluster
    ControlPlane --> Cluster
    Cluster -- "node egress" --> NAT
    Cluster -. telemetry .-> Observability
    Uptime -- "probes /healthz every 60s" --> Ingress
    Uptime -. the result is the metric .-> Observability
    Observability -. opens an incident .-> Notify
    SCC -- "finding changes" --> PubSub
    Worker -- "pulls" --> PubSub
    Worker -- "only what the rules leave unmatched" --> Vertex
    Worker -- "records before notifying" --> Ledger
    Worker -. "verdict log entry" .-> Observability

    classDef external fill:#4B201D,stroke:#F28B82,color:#F8FAFC,stroke-width:2px
    classDef delivery fill:#493510,stroke:#FDD663,color:#F8FAFC,stroke-width:2px
    classDef workload fill:#123C2D,stroke:#81C995,color:#F8FAFC,stroke-width:2px
    classDef managed fill:#402060,stroke:#C58AF9,color:#F8FAFC,stroke-width:2px

    class User,Developer external
    class Terraform,GitHub delivery
    class ControlPlane,NAT,Routes,Nginx,Sky,Guardrails,Worker workload
    class Ingress,Registry,Identity,Uptime,Observability,Notify,SCC,PubSub,Vertex,Ledger managed

    style Delivery fill:#211A0D,stroke:#FDD663,color:#F8FAFC,stroke-width:2px
    style GCP fill:#101828,stroke:#8AB4F8,color:#F8FAFC,stroke-width:2px
    style VPC fill:#102A23,stroke:#81C995,color:#F8FAFC,stroke-width:2px
    style Cluster fill:#183B31,stroke:#A8DAB5,color:#F8FAFC,stroke-width:2px

Loading

Status

Complete, and closed on 2026-09-22. Milestones 1 to 3 are done, and Milestone 4 closed at Phase 15 and Phase 15b. Phases 16 to 19 are optional extensions and were not built; the plan says what would justify each, and lists the follow-ups still dated.

Area	State
Foundation	Private GKE on modular Terraform, custom VPC, Cloud NAT, DNS-only control plane
Workloads	Two behind one Gateway: nginx at two replicas, sky autoscaled from two to eight
Guardrails	Pod Security restricted, namespace budget, default-deny NetworkPolicies
Delivery	Keyless federation scoped to main, gated rollout, required checks, upstream CI checked before a pin moves
Ingress	Public Gateway on a custom domain, managed TLS, HTTP to HTTPS redirect
Resilience	Node floor of two, a disruption budget per workload, nightly maintenance window
Observability	Uptime check, dashboard as code, and three alerts: availability, a finding that needs a decision, and a finding that was dead-lettered
Proven	Both failure drills run and recorded
Hardened	One network, vulnerability scanning on, logs queryable, TLS 1.2 floor, rate limit, response security headers
Under load	Rollouts drop no requests, sky autoscales to 125 rps with no failures, nodes scale across three zones
Modelled	Eight trust boundaries with a threat model, measured rather than assumed, and scanned daily from outside
Streaming	Security Command Center findings reach a subscription in about two seconds, and park in a dead letter topic when nothing acknowledges them
Triaged	An agent in its own namespace settles findings by reviewed mapping, then by model, against a corpus compiled into its image. It holds four scoped grants and no cluster credential
Drilled	The worker is stopped at each of the three crash boundaries and recovers at each, notifying again rather than silently skipping. Every model failure path was made to happen
Evaluated	Rules alone against rules plus the model, on 25 reviewed findings with a sealed holdout, scored for citations that support the verdict, and rerun by CI rule when the prompt, cases or mapping change
Patched	The images this repository builds, measured by a scanner, with Dependabot moving each base image

Milestone 3 closed with eleven of the twelve findings in the threat model measured and closed across Phase 13 and Phase 14, the twelfth carrying a recorded acceptance.

Phase 15 is closed. The worker in ai-k8s settles what the reviewed mapping pairs, and asks Vertex AI about the rest through a one-permission role, within a token budget and a daily spend ceiling. Every failure path it adds was made to happen.

Whether the model is worth it is measured in slices 13 and 14, on a reviewed set of 25 findings asked five times each. At first the model tied the rules alone, trading false contradictions for real ones. With a contradiction required to land on a control that applies to the finding's resource, it gets 16 of 18 dev cases and 7 of 7 sealed holdout cases right on every run, against 14 and 5 for the rules, with no false contradiction in 125 runs.

Phase 15b patched the images whose vulnerabilities triage counts: frontend from 17 CRITICAL and HIGH to 0, and sky from 22 to 6, none of which has a fixed package yet.

Not built: cluster reads through an audited gateway, a first responder on alerts, findings as Kubernetes objects, and remediation by pull request. Their designs and decisions are in the plan. None has a requirement behind it yet, and the triage worker needs none of them.

Known limitations

Limitation	Why it stands
Terraform state is local, on one workstation	One operator and no automated apply. The decision gate for remote state is collaboration or automated apply, and neither arrived
The bootstrap order has not been rehearsed on an empty project	It is reconstructed from the worklogs in the operations reference
Zonal control plane	Cost. The node pool spans three zones; the control plane does not
Egress from agents admits any host on 443	NetworkPolicy cannot match hostnames. Recorded on boundary 3
Six HIGH vulnerabilities in sky	No fixed package exists. Accepted until Debian ships one
The evaluation is 25 cases, and the holdout shares its author	A check against tuning, not an independent sample. Slice 13 lists the rest
Event Threat Detection depends on a Security Command Center Premium trial	The tier is re-read when the trial ends. Standard drops the threat class of findings

Measured

Merge to Ready workload	59s
Deploy duration, median of twelve runs	70.5s
Alert detection floor	about 3 minutes
Running cost	kr461.81 a week, covered by credits
sky saturation, two replicas	40 requests a second, p95 230ms
Requests failed during a rollout at 20 rps	1.14%, error window up to 20.4s
Connection failures in a rollout, after preStop	0 across three rollouts, from 72
Closed-connection 503s in a ramp, after keep-alive	0 of 7,150, from 6
sky saturation, autoscaled to eight replicas	125 requests a second, p95 394ms, no failures
HPA decision to a Pod running on a new node	97s
Security Command Center finding change to a message on the subscription	about 2 seconds
Unacknowledged message to the dead letter topic	5 delivery attempts
Outstanding message returned after the worker is killed	about 25 seconds, on stream close rather than on the 120s deadline
Findings right on every one of five runs, rules plus the model	16 of 18 dev, 7 of 7 holdout. Rules alone: 14 and 5
False contradictions raised by the model	0 of 125 runs, from 15 before the check in code
Model call latency, p95	about 2 seconds
Model cost per call	about 0.003 USD, estimated from token counts
CRITICAL and HIGH vulnerabilities in the images built here	frontend 0, from 17. sky 6 with no fix, from 22
Crash to a recorded verdict on redelivery	four ledger states in 285ms

Method and evidence: Phase 8, Phase 10, Phase 12a, Phase 12b, Phase 12c, Phase 12d, Phase 15 and Phase 15b.

Platform capabilities

Domain	What exists	Decisions	Evidence
Networking	Custom VPC, private nodes, Cloud NAT, DNS-only control plane, Dataplane V2	Networking	Phase 1
Cluster	Zonal GKE Standard, autoscaling node pool, Shielded Nodes, Regular release channel	Cluster	Phase 1
Identity	Workload Identity Federation, dedicated node service account	Identity and access	Phase 1
Workload	demo namespace, two Deployments (nginx and sky), health probes, resource limits, ClusterIP Services	Infrastructure and configuration	Phase 2
Delivery	Credential-free pull request validation, required checks on main	Delivery	Phase 3
Policy	Pod Security restricted enforced, dedicated ServiceAccount, namespace budget, default-deny NetworkPolicies	Workload security	Phase 4, Phase 5
Images	Private Artifact Registry repository, immutable tags, retention policy, node read access	Images and supply chain	Phase 5
Deployment	Keyless GitHub Actions delivery for both workloads, federation scoped to main, namespaced pipeline RBAC, gated rollout	Delivery	Phase 6
Upstream tracking	Scheduled workflow that proposes the sky commit bump as a pull request, with the merge as the review	Upstream pin automation	Pin automation
Ingress	GKE Gateway on a reserved global address, container-native load balancing, Certificate Manager TLS, HTTP to HTTPS redirect, path routing to both workloads	Ingress and TLS	Phase 7
Observability	Cluster telemetry, uptime check on /healthz, one alert policy, dashboard as code, deployment and cost numbers	Observability	Phase 8
Resilience	Node floor of two, disruption budgets on both workloads, spread that survives a rollout, nightly maintenance window	Cluster	Phase 9
Failure drills	Deliberate outage with a measured three minute detection floor, and a failed rollout contained by maxUnavailable: 0	Observability	Phase 10
Hardening	Only gke-vpc remains, workload vulnerability scanning on, Log Analytics and one log-based metric	Workload security	Phase 11
Load and autoscaling	k6 harness on a throwaway load generator, preStop and keep-alive for clean rollouts, HPA on sky with requests and quota sized from measured load, nodes across three zones	Load and scaling	Phase 12a, 12b, 12c, 12d
Security baseline	Twelve ranked threat model findings, eleven closed with evidence, federation scoped to a ref, CSP on both paths, CAA on a signed zone, a measured rate limit	Workload security	Phase 13, Phase 14
Finding triage	Security Command Center findings over Pub/Sub with a dead letter policy, a worker that settles them by reviewed mapping and then by model, an append-only verdict ledger, and a scored evaluation of both	Agents	Phase 15

Documentation

• Implementation plan
• Architecture decisions
• Phase 1 infrastructure worklog
• Phase 2 workload worklog
• Phase 3 CI worklog
• Phase 4 guardrails worklog
• Phase 5 custom image worklog
• Phase 6 keyless delivery worklog
• Phase 7 gateway and TLS worklog
• Phase 8 observability worklog
• Phase 9 surviving a node worklog
• Phase 10 failure drills worklog
• Phase 11 hardening worklog
• Phase 12a load baseline worklog
• Phase 12b rollout baseline worklog
• Phase 12c rollouts and connections worklog
• Phase 12d autoscaling worklog
• Phase 13 security baseline worklog
• Phase 14 close the baseline worklog
• Phase 15 Security Command Center triage worklog
• Upstream pin automation worklog
• Manifest linting worklog
• Repository review worklog
• Availability drill postmortem
• Load test harness
• Troubleshooting log
• Operations reference
• Networking reference
• Kubernetes concepts reference
• kubectl command reference
• IAM and Workload Identity Federation reference
• Triage worker reference