diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index 75a9392..0e61403 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -22,7 +22,10 @@ export default withMermaid(defineConfig({ { text: 'Networking', link: '/guides/networking/' }, { text: 'Linux Internals', link: '/guides/linux-internals/' }, { text: 'Kubernetes', link: '/guides/kubernetes/' }, - { text: 'Terraform', link: '/guides/terraform/' } + { text: 'Terraform', link: '/guides/terraform/' }, + { text: 'GCP Fundamentals', link: '/guides/gcp/' }, + { text: 'GKE Architecture', link: '/guides/gke/' }, + { text: 'GKE Operations', link: '/guides/gke/' } ], sidebar: { '/guides/networking/': [ @@ -90,13 +93,44 @@ export default withMermaid(defineConfig({ { text: '45 difficult follow-ups', link: '/guides/terraform/follow-ups' } ] } ], + '/guides/gcp/': [ + { text: 'GCP Fundamentals and Production Engineering', link: '/guides/gcp/' }, + { text: 'Complete Module 5', items: [ + { text: '1. Resource hierarchy', link: '/guides/gcp/q1' }, + { text: '2. IAM and workload identity', link: '/guides/gcp/q2' }, + { text: '3. VPC, Shared VPC and firewall', link: '/guides/gcp/q3' }, + { text: '4. NAT, DNS and load balancing', link: '/guides/gcp/q4' }, + { text: '5. Compute Engine and MIGs', link: '/guides/gcp/q5' }, + { text: '6. Storage, artifacts and data', link: '/guides/gcp/q6' }, + { text: '7. Observability and SLOs', link: '/guides/gcp/q7' }, + { text: '8. Architecture and recovery', link: '/guides/gcp/q8' }, + { text: 'Hands-on labs', link: '/guides/gcp/labs' }, + { text: '40 challenging follow-ups', link: '/guides/gcp/follow-ups' } + ] } + ], + '/guides/gke/': [ + { text: 'GKE Architecture & Production Operations', link: '/guides/gke/' }, + { text: 'Complete Module 6', items: [ + { text: '1. Architecture and control plane', link: '/guides/gke/q1' }, + { text: '2. Node pools and scheduling', link: '/guides/gke/q2' }, + { text: '3. VPC-native networking', link: '/guides/gke/q3' }, + { text: '4. Ingress, Gateway and NEGs', link: '/guides/gke/q4' }, + { text: '5. Security and workload identity', link: '/guides/gke/q5' }, + { text: '6. CSI storage and StatefulSets', link: '/guides/gke/q6' }, + { text: '7. Autoscaling and QoS', link: '/guides/gke/q7' }, + { text: '8. Upgrades and incidents', link: '/guides/gke/q8' }, + { text: 'Hands-on labs', link: '/guides/gke/labs' }, + { text: '40 challenging follow-ups', link: '/guides/gke/follow-ups' } + ] } + ], '/guides/': [ { text: 'Learning Guides', items: [ { text: 'All guides', link: '/guides/' }, { text: 'Networking Fundamentals', link: '/guides/networking/' }, { text: 'Linux Internals', link: '/guides/linux-internals/' }, { text: 'Kubernetes Architecture', link: '/guides/kubernetes/' }, - { text: 'Terraform Architecture', link: '/guides/terraform/' } + { text: 'Terraform Architecture', link: '/guides/terraform/' }, + { text: 'GCP Fundamentals', link: '/guides/gcp/' } ] } ] }, diff --git a/docs/guides/gcp/follow-ups.md b/docs/guides/gcp/follow-ups.md new file mode 100644 index 0000000..5d8e1c0 --- /dev/null +++ b/docs/guides/gcp/follow-ups.md @@ -0,0 +1,254 @@ +# Module 5: 40 Challenging GCP Interview Follow-ups + +These questions extend the eight main chapters. Answers are designed for a senior platform/cloud engineer and explain mechanisms, debugging evidence and trade-offs, not just definitions. + +::: info Context +Examples are hypothetical. Verify exact APIs, features and CLI commands against the deployed Google Cloud configuration and current official documentation. +::: + +## 1. How does IAM inheritance work across organization, folder and project? + +**Topic:** Resource hierarchy + +**Model answer:** An allow binding on an ancestor can grant effective permissions on eligible descendants. A narrower project policy does not automatically remove an ancestor allow. Deny policies, IAM Conditions and Organization Policy constraints can further limit access. I'd inspect the complete resource ancestry and effective permissions, not only the project IAM policy. This is why folder-level Editor and Owner grants are particularly dangerous and should be avoided. Use groups, scoped predefined/custom roles and least privilege. + +## 2. Is a Google Cloud project equivalent to an AWS account? + +**Topic:** Resource hierarchy + +**Model answer:** Not exactly. Both support administrative boundaries, IAM, quota and billing attribution, but GCP projects live in a hierarchy under an organization/folder, and can share global VPC networks through Shared VPC host and service project attachments. AWS account and VPC models differ significantly. I wouldn't apply an AWS Organizations landing zone layout to GCP without considering policy inheritance, host-network ownership and service-specific resource scopes. + +## 3. A project has billing enabled and IAM permissions, but VM creation fails. What else could block it? + +**Topic:** Resource hierarchy + +**Model answer:** Compute Engine API enablement, regional or machine-family quota, zonal capacity exhaustion, organization policy denying an external IP or machine configuration, request validation, or a disabled/restricted service could block creation. I would inspect the exact Compute Engine operation error first. Quota and physical capacity are different: an approved quota increase does not guarantee VMs are available in a specific zone. + +## 4. Does a global VPC make all infrastructure globally available? + +**Topic:** Resource hierarchy + +**Model answer:** No. The VPC network resource is global, but its subnets are regional and VM instances zonal. A regional database or a node pool can still fail with its location. Interregional data movement still incurs latency and possible charges. A globally scoped network is an administrative/routing design, not automatic multi-region replication, data durability or application failover. + +## 5. What happens if the organization deletes a project by mistake? + +**Topic:** Resource hierarchy + +**Model answer:** Project deletion can disable resources and initiate a recovery-limited deletion lifecycle. The precise restoration window and resource effects are subject to current Google policies. I would immediately stop other automation, contact authorized project administrators, check project recovery procedures and retain audit evidence. A project restoration may not be sufficient to recover every application dependency or independently removed dataset; backups and recovery drills remain essential. + +## 6. What is the difference between Google IAM roles and a service account? + +**Topic:** IAM and identity + +**Model answer:** A service account is a principal identity, while a role is a set of permissions. IAM policy bindings associate roles with principals at a resource or ancestor scope. Creating a service account does not grant it storage access or Kubernetes RBAC. I'd create a dedicated principal for each relevant workload trust boundary and assign only the permissions needed on the narrowest resource. + +## 7. Explain ADC versus gcloud auth login. + +**Topic:** IAM and identity + +**Model answer:** gcloud auth login configures credentials for gcloud CLI operations; Application Default Credentials are discovered by Google client libraries through their documented lookup mechanism, including local ADC files or metadata credentials. The identities can differ on a developer laptop. In production, prefer metadata-based short-lived credentials or federation, rather than downloading long-lived JSON keys. A 403 investigation must identify the actual runtime principal, not assume the active gcloud account is used by the application. + +## 8. Why can a GKE Pod access Cloud Storage while it lacks Kubernetes API access? + +**Topic:** IAM and identity + +**Model answer:** Workload Identity Federation or service-account impersonation can give its Google principal bucket-level IAM rights, while Kubernetes RBAC separately controls verbs against the Kubernetes API server. Neither grants the other automatically. The Pod may call storage.objects.get yet receive Forbidden when listing Pods. I would review Google IAM principal bindings and KSA Kubernetes RBAC independently, including who can create Pods with the KSA. + +## 9. A valid OIDC assertion from CI fails to impersonate a Google service account. Where do you look? + +**Topic:** IAM and identity + +**Model answer:** Check the configured workload identity pool/provider, issuer, audience, token subject/attributes and attribute conditions. Verify the external principal matches the trust binding on the intended service account and has the appropriate workload identity user permission. Check the project number in qualified principal identifiers. Test the resulting principal without exposing token contents. A valid signed OIDC token alone doesn't imply authorization to impersonate arbitrary Google service accounts. + +## 10. A workload receives 403 even after Storage Object Viewer was assigned. Why? + +**Topic:** IAM and identity + +**Model answer:** The request might originate from a different principal, target a different bucket/project, require an additional permission such as list instead of get, hit an IAM condition or deny, or be blocked by VPC Service Controls or other storage policies. Also allow for IAM propagation. I'd inspect a single failed method/resource, actual token principal, bucket IAM and audit logs/Policy Troubleshooter. Granting Owner to silence the error hides the cause and creates a security risk. + +## 11. What is the scope of a VPC versus a subnet in Google Cloud? + +**Topic:** VPC and Shared VPC + +**Model answer:** A VPC network and its associated global routing/firewall configuration are global resources. Subnets are regional and supply IP ranges to eligible instances in zones of that region. Secondary alias-IP ranges can serve GKE Pods. This differs from AWS regional VPC plus zonal subnets. Location scope affects CIDR planning, resource ownership, troubleshooting and failure domains. + +## 12. Does a service project own the Shared VPC subnet used by its VMs? + +**Topic:** VPC and Shared VPC + +**Model answer:** No. The Shared VPC host project owns the network and subnets. An attached service project hosts eligible consumer resources, subject to subnet-use grants and other IAM permissions. This centralizes network governance but creates cross-project dependencies. I'd diagnose a creation failure by checking host/service attachment, host subnet permissions, service project IAM, org constraints, subnet region and remaining address capacity. + +## 13. How can two VMs on the same VPC fail to communicate? + +**Topic:** VPC and Shared VPC + +**Model answer:** The destination may not be listening, ingress/egress firewall or hierarchical policy may deny, the source/destination may have unexpected addresses, a route or policy path may be incompatible, or guest OS firewall/security may block. Even same-VPC location doesn't override policy. I'd inspect actual NIC/IP, routes, both directions of policy, guest listener and Connectivity Tests; sampled Flow Logs alone are not conclusive. + +## 14. Is Private Google Access equivalent to public Cloud NAT? + +**Topic:** VPC and Shared VPC + +**Model answer:** No. Private Google Access enables eligible resources without external IPv4 addresses to reach supported Google APIs/services through configured access paths. Public NAT translates eligible outbound flows to external destinations. Neither automatically exposes an inbound public service. A VM reading Cloud Storage may work with Private Google Access while HTTPS to a third-party API fails without an appropriate egress path. + +## 15. A VPN tunnel is up but applications cannot reach on-premises. What do you test? + +**Topic:** VPC and Shared VPC + +**Model answer:** Tunnel state is only one layer. Check Cloud Router BGP session, advertised/imported prefixes, dynamic route priorities, VPC/guest firewall, overlapping CIDRs, return routes, MTU/fragmentation and DNS. Cloud Router exchanges routing information but isn't in the packet path as a traditional transit VM. I would use packet/source-destination evidence and Connectivity Tests, not treat a green tunnel as end-to-end success. + +## 16. Why can Cloud NAT run out of connections even with idle CPUs? + +**Topic:** NAT DNS load balancing + +**Model answer:** Cloud NAT depends on available translated IP/port mappings and allocation policy, not just compute resources. Many short-lived flows, concentrated remote IP:port destinations, insufficient NAT addresses or too little allocated port capacity can cause new connections to fail. Measure NAT drop/port metrics, number of source VMs, SNAT behavior and connection churn. Connection reuse and bounded concurrency can reduce pressure; increasing replicas may worsen it. + +## 17. Does Cloud NAT accept unsolicited inbound requests from the public internet? + +**Topic:** NAT DNS load balancing + +**Model answer:** No. Public NAT is primarily outbound source translation for eligible instances and their return traffic. It doesn't by itself create a public listener on a VM without external IP. To expose a service, configure the appropriate external load balancer, proxy or other supported ingress endpoint, plus DNS, certificates, backend health and security controls. + +## 18. A public DNS answer works, but a VM resolves an internal address. Is this necessarily a bug? + +**Topic:** NAT DNS load balancing + +**Model answer:** Not necessarily. Cloud DNS private zones, response policies or forwarding can implement split-horizon resolution for associated VPCs, while public authoritative zones answer external clients. Validate which resolver the VM uses, zone visibility, DNS policy and query name. A private NXDOMAIN doesn't guarantee fallback to public records, and negative caching can preserve old failures until TTL expiry. + +## 19. Why can an HTTPS load balancer return 502 while the Pod reports Ready? + +**Topic:** NAT DNS load balancing + +**Model answer:** GKE Pod readiness and the Google Cloud backend service's health can be different states. Consider missing NEG membership, wrong backend port/protocol, failed external health checks, backend TLS mismatch, firewall denial, application crash or timeout. Inspect the exact LB type, backend health and logs, NEG membership and app traces. Pod readiness alone is insufficient evidence that the full client path works. + +## 20. When should you use an Application LB versus a Network LB? + +**Topic:** NAT DNS load balancing + +**Model answer:** Use Application Load Balancers for HTTP(S) semantics such as hostname/path routing and managed L7 proxying. Network Load Balancers serve transport-level needs including TCP/UDP variants, passthrough or proxy behavior depending on product. Check external/internal reachability, global versus regional support, client-IP preservation, TLS termination, backend type and connection characteristics. The word 'load balancer' does not define one universal packet path. + +## 21. A managed instance group is configured for nine replicas but only six are created. Why? + +**Topic:** Compute Engine + +**Model answer:** Investigate MIG operation errors and autoscaler recommendation versus actual target count, regional/zonal quota, machine availability, instance template correctness, subnetwork IP exhaustion, permission failures and distribution policies. A regional MIG spreads replicas but isn't a guarantee of machine availability in each zone. I'd examine the exact Compute Engine operation, not just utilization on existing instances. + +## 22. What is the difference between MIG autoscaling and autohealing? + +**Topic:** Compute Engine + +**Model answer:** Autoscaling changes desired instance count based on demand signals. Autohealing replaces instances judged unhealthy by a configured health check. Bad health check design can cause restart storms when a shared dependency fails. Both need appropriate initialization/warmup and frontend health integration; a newly created VM isn't automatically ready to serve real customer traffic. + +## 23. Why use instance templates rather than manually modifying running VMs? + +**Topic:** Compute Engine + +**Model answer:** Templates capture versioned, repeatable machine configuration. A MIG can roll a new template across replicas with controlled disruption and health checks. Manually editing each VM creates configuration drift and uncertain recovery. The template must still be secured: startup metadata should not contain long-lived secrets, images should be trusted, and changes should be canaried with rollback planning. + +## 24. Should every production VM run as Spot? + +**Topic:** Compute Engine + +**Model answer:** No. Spot VMs provide lower-cost interruptible capacity and can be reclaimed. They're suitable for fault-tolerant batch or stateless capacity backed by resilient baseline infrastructure. A lone transaction leader or only copy of durable state is inappropriate. Plan for preemption, graceful checkpointing, possible inability to reacquire capacity and workload-specific cost/availability trade-offs. + +## 25. A MIG rollout completes but API requests still fail. What did the rollout actually prove? + +**Topic:** Compute Engine + +**Model answer:** It mainly proves that the managed group performed its configured instance lifecycle actions and health assessments. It may not prove application business correctness, load-balancer backend health, dependency compatibility or TLS routing. Verify application-level synthetic transactions, p99/errors, backend health, database schema compatibility and connection draining across the rollout. + +## 26. Is Cloud Storage equivalent to a mounted POSIX filesystem? + +**Topic:** Storage and data + +**Model answer:** No. It is object storage with bucket/object semantics and well-defined strong-consistency operations, not ordinary arbitrary in-place POSIX file modification. Object naming and generation matter, and multi-object atomic transactions are not automatically available. For databases use an appropriate transactional data store or block device; for images, exports and immutable blobs, Cloud Storage is often suitable. + +## 27. How are Cloud Storage versioning, soft delete and retention different? + +**Topic:** Storage and data + +**Model answer:** Object Versioning preserves noncurrent versions under its configured behavior, soft delete allows recovery of recently deleted/overwritten objects for a retention interval, and retention policies enforce minimum preservation periods subject to lock/configuration. They solve different recovery/compliance needs and incur different costs. None replaces tested application-consistent backups or controls against credential/KMS loss. Always inspect actual bucket policy. + +## 28. Why can moving every object to Archive cost more than using Standard? + +**Topic:** Storage and data + +**Model answer:** Archive has minimum duration and retrieval/operation economics. Frequently read or rapidly deleted objects can incur early deletion and retrieval fees that outweigh lower storage GB price. Evaluate access rate, retention period, restore urgency and lifecycle transitions. Autoclass can help in suitable buckets, but pricing must be modeled with real workload behavior. + +## 29. Does Cloud SQL HA mean you can restore a deleted table to yesterday automatically? + +**Topic:** Storage and data + +**Model answer:** No. HA aims to restore service availability after instance/zone failure. It does not undo logical deletes or bad migrations; those changes can propagate. Backup and point-in-time recovery are separate capabilities requiring configuration and testing. Determine RPO/RTO, storage scope, backup retention and application reconciliation before asserting recoverability. + +## 30. Does Pub/Sub exactly-once delivery ensure an external payment runs exactly once? + +**Topic:** Storage and data + +**Model answer:** No. Delivery guarantees do not make a database commit and external side effect atomic. A worker can commit the payment record then crash before ack and receive the message again. Use idempotency keys/unique constraints and an outbox or equivalent reliable event pattern. Acknowledge after durable processing, implement bounded retry/dead-letter behavior and measure redelivery. + +## 31. A VM shows low CPU but p99 latency is high. What telemetry helps? + +**Topic:** Observability + +**Model answer:** Start from a customer-facing request SLI and traces to locate waiting time. Inspect application queues, DB connection pools, network/NAT errors and dependent service latency alongside CPU, memory and I/O pressure. A CPU graph says how much CPU was consumed, not where blocked requests wait. Check per-revision and per-region distributions and avoid averaging p99s. + +## 32. Why don't log absence and zero Cloud NAT log entries prove no traffic happened? + +**Topic:** Observability + +**Model answer:** Logs may be disabled, excluded, sampled, aggregated, delayed, retained in another project or unavailable for the specific event. VPC Flow Logs and some service logs do not guarantee every packet is individually visible. Inspect log settings and sink routing, and corroborate with API metrics, Connectivity Tests, packet captures where allowed and service responses. + +## 33. Why are high-cardinality metric labels dangerous? + +**Topic:** Observability + +**Model answer:** Labels like request ID, customer ID or unique URL can create huge time-series cardinality, memory/query load and observability billing. Keep dimensions such as service, region, status class and revision bounded. Use trace IDs and request IDs in logs/traces rather than metrics labels. Monitor telemetry costs and cardinality growth as part of platform operations. + +## 34. What is an error budget for a 99.9% SLO? + +**Topic:** Observability + +**Model answer:** For an event-based SLI, an objective of 99.9% allows 0.1% of eligible events to be noncompliant during the window. Error budget is not automatically downtime minutes unless the SLI specifically measures time-based availability. Burn rate compares observed noncompliance fraction to the allowed fraction; a 1.44% error fraction represents 14.4x burn relative to a 0.1% allowance. + +## 35. What is the difference between Cloud Logging and Cloud Audit Logs? + +**Topic:** Observability + +**Model answer:** Cloud Logging is the platform for logs, including application/VM logs and routed sinks. Cloud Audit Logs provide service-generated records of API and administrative activity under category-specific coverage and policy. Data Access logs are not universally enabled/visible in the same way as Admin Activity. Forensics need correct logging policy, retention, secure sinks and access to the right project. + +## 36. Does a second application region guarantee regional disaster recovery? + +**Topic:** Architecture and cost + +**Model answer:** No. The application may have a single-region transactional database, regional secrets, a dependent service, unrecoverable data or insufficient warm capacity. Define RPO/RTO, replication/restore design, service identity, failover trigger and split-brain controls. A global HTTPS frontend is not a substitute for application/data-plane recovery tests. + +## 37. Why is changing to a smaller VM type not automatically a cost optimization? + +**Topic:** Architecture and cost + +**Model answer:** Savings can be offset by throughput loss, added replicas, higher p99, retry traffic, cross-zone transfers and increased incident risk. Evaluate cost per successful business transaction, peak headroom, committed-use terms and SLO impact. Workload-based right-sizing with controlled benchmarks is preferable to lowering compute spend in isolation. + +## 38. Should a company hide AWS and GCP behind one universal platform module? + +**Topic:** Architecture and cost + +**Model answer:** Only for stable outcomes. Provider resource hierarchies, subnet scopes, IAM and workload federation, NAT and load-balancer behavior differ. A universal lowest-common-denominator module can conceal important reliability and security trade-offs. Provide consistent developer-facing capabilities where useful, but preserve provider-specific implementations and operational documentation. + +## 39. An incident affects 30% of users. When is failover to another region safe? + +**Topic:** Architecture and cost + +**Model answer:** First identify whether the secondary region is healthy, authorized, sufficiently provisioned and has compatible application code, dependencies and current data. Assess transaction writes and split-brain risk. Perform controlled routing changes only after verifying RPO/RTO and safety conditions; then monitor user-facing errors, latency and data correctness. Blindly shifting requests to a stale read-only replica can turn an availability incident into a data integrity incident. + +## 40. How would you evaluate a GCP landing zone six months after rollout? + +**Topic:** Architecture and cost + +**Model answer:** Measure time to provision approved workloads, IAM/policy violations, incidents caused by shared networking, quota/capacity failures, trace/SLO coverage, cost attribution and developer toil. Review how many resources remain outside IaC, whether backup/region recovery drills succeed and whether teams can self-serve safely. A high number of projects or policy checks isn't success by itself. + + +## Further practice + +For each question, explain a counterexample, identify the first command or evidence source you would check, and distinguish a root cause from a short-term mitigation. Reproduce only safe exercises in a disposable project. + +[← Module 5 index](./) · [Hands-on labs](./labs) diff --git a/docs/guides/gcp/index.md b/docs/guides/gcp/index.md new file mode 100644 index 0000000..5e1e1ae --- /dev/null +++ b/docs/guides/gcp/index.md @@ -0,0 +1,74 @@ +# Google Cloud Fundamentals and Production Engineering + +**Module 5 · Intermediate to senior platform/cloud engineering guide · Eight complete chapters** + +Google Cloud has familiar concepts for engineers coming from other providers, but its resource hierarchy, global VPC networking, project policies, IAM federation and load-balancing architecture require their own mental model. This field guide develops those fundamentals from first principles, then connects them to operational debugging and reliable system design. + +::: info Public education and accuracy +Examples use fictional marketplace infrastructure and placeholder resource names. They are teaching designs, not claims about a real production environment. Google Cloud services change rapidly: verify version, region, account policy, release stage, CLI syntax and pricing before deployment. Code snippets are educational and sometimes require prerequisites discussed in their chapters. +::: + +## Prerequisites + +Familiarity with Linux and TCP/IP, service deployment and a basic cloud model. Kubernetes and Terraform knowledge help with the examples but are not required to follow all conceptual material. For labs use a controlled Google Cloud sandbox with an approved budget, IAM rights and a cleanup plan. Read-only commands are sufficient for most investigations. + +## What you will learn + +- Explain organization, folders, projects, billing, API enablement, quotas and policy inheritance. +- Use IAM roles, service accounts, ADC and federation without long-lived keys. +- Design VPC/subnet, Shared VPC, firewall and hybrid routing based on actual resource scopes. +- Trace HTTPS ingress, backend health, DNS and Cloud NAT egress end to end. +- Operate Compute Engine templates, managed instance groups and autoscaling. +- Choose object storage, artifacts, relational transactions, messaging, caches and analytics deliberately. +- Correlate Cloud Monitoring, Logging, Trace and Audit Logs to a user-facing SLO. +- Evaluate multi-zone/multi-region reliability, disaster recovery, cost and security trade-offs. + +## Eight main chapters + +1. [Resource hierarchy, projects, governance and quotas](./q1): Administrative hierarchy, org policy, billing, API operations, failure cases. +2. [IAM, service accounts, ADC and workload identity](./q2): Authentication/authorization, token flow, GKE/CI federation, 403 diagnosis. +3. [VPC, Shared VPC, routes and firewall](./q3): Global VPC, regional subnets, firewall, hybrid networking and packet tracing. +4. [Cloud NAT, Cloud DNS and load balancing](./q4): HTTPS request sequence, NAT outbound mapping, proxy types, DNS and troubleshooting. +5. [Compute Engine and managed instance groups](./q5): Templates, VM boot, regional capacity, rolling updates, Spot and autohealing. +6. [Cloud Storage, Artifact Registry and data services](./q6): Objects, images, transactions, Pub/Sub idempotency and product selection. +7. [Observability, SLOs and incident diagnosis](./q7): Metrics, logs, traces, audit evidence, percentile and burn-rate reasoning. +8. [Architecture, security, DR and costs](./q8): Hypothetical marketplace design, failure domains, regional incident, trade-offs. + +## Interview and hands-on practice + +- [40 difficult interviewer follow-ups with complete model answers](./follow-ups) +- [Eight hands-on and tabletop labs, including expected observations and safety controls](./labs) + +## Study order + +Start with hierarchy and identity (Q1–Q2), which explains where permissions apply. Then work through network design and packet flow (Q3–Q4). Study compute and data guarantees (Q5–Q6), and finish with service-level observability and architecture decisions (Q7–Q8). Work through labs before answering the entire 40-question bank without notes. + +## AWS-to-GCP quick map + +| AWS concept | GCP concept | Caution | +| --- | --- | --- | +| Organizations / OUs / accounts | Organization / folders / projects | Not one-to-one isolation or inheritance | +| Regional VPC + zonal subnets | Global VPC + regional subnets | Different network scope | +| IAM roles + STS | IAM principals/roles + federation/impersonation | Different trust model | +| ALB / NLB | Application / Network Load Balancers | Product classes and datapaths differ | +| NAT Gateway | Public NAT | Port and translation models differ | +| EC2 + ASG | Compute Engine + MIG | Regional distribution semantics | +| S3 / ECR | Cloud Storage / Artifact Registry | Location, class and IAM details differ | +| RDS / Aurora | Cloud SQL and other DB products | HA/replication options differ | +| CloudWatch / X-Ray | Monitoring / Logging / Cloud Trace | Query/metric models vary | + +## Authoritative references + +- [Google Cloud Resource Manager hierarchy](https://cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy) +- [Cloud IAM overview](https://cloud.google.com/iam/docs/overview) +- [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) +- [Google VPC and Shared VPC](https://cloud.google.com/vpc/docs/vpc) +- [Public NAT](https://cloud.google.com/nat/docs/public-nat) +- [Cloud Load Balancing](https://cloud.google.com/load-balancing/docs/load-balancing-overview) +- [Compute Engine managed instance groups](https://cloud.google.com/compute/docs/instance-groups) +- [Cloud Storage](https://cloud.google.com/storage/docs/introduction) +- [Cloud SQL](https://cloud.google.com/sql/docs) +- [Cloud Monitoring](https://cloud.google.com/monitoring/docs/monitoring-overview) +- [Google Cloud architecture framework](https://cloud.google.com/architecture/framework) + +[← All modules](/guides/) · [Start with Q1 →](./q1) diff --git a/docs/guides/gcp/labs.md b/docs/guides/gcp/labs.md new file mode 100644 index 0000000..254cdbd --- /dev/null +++ b/docs/guides/gcp/labs.md @@ -0,0 +1,198 @@ +# Google Cloud Fundamentals: Eight Practical Labs + +The exercises reinforce the eight chapters. Each lab has a result to explain, not simply a command to run. + +::: warning Use only an approved sandbox +Google Cloud services can create billable resources. Before creating anything, choose an authorized project, set a budget alert, understand cleanup, avoid real credentials or PII in logs and never open test services publicly without permission. Read-only inspection labs are suitable when creation privileges are unavailable. +::: + +## Lab 1. Resource hierarchy, API and quotas + +**Goal:** Explain the boundaries between IAM, enabled service APIs, billing, quota and zone capacity. + +```bash +gcloud auth list +gcloud config get-value project +gcloud projects describe PROJECT_ID +gcloud services list --enabled --project=PROJECT_ID +gcloud billing projects describe PROJECT_ID +gcloud compute regions describe europe-west1 --project=PROJECT_ID +``` + +**Expected:** A project can exist without some APIs enabled. CPU quota is scoped by the relevant service/region, while resource availability can also be constrained by the zone. IAM roles on folders/organization can influence project access. + +**Challenge:** Describe how `API_DISABLED`, permission denied, quota exceeded and out-of-capacity would differ. Read operation error details instead of guessing. + +**Safety:** Do not change org policies, billing or quotas to complete the lab. + +## Lab 2. IAM and effective workload identity + +**Goal:** Show that authentication identity and resource authorization differ. + +On an authorized sandbox bucket, inspect: + +```bash +gcloud auth list +gcloud auth application-default print-access-token +gcloud storage buckets get-iam-policy gs://OWNED_BUCKET +gcloud projects get-iam-policy PROJECT_ID --format=json +``` + +**Expected:** The CLI and ADC identities can differ. A service account with no bucket access authenticates but may fail authorization. An applicable ancestor binding or deny can affect the result. + +**Challenge:** Draw an external CI OIDC assertion exchange through workload identity pool/provider, STS and optional service account impersonation. Describe audience, issuer and attribute restrictions. + +**Safety:** Never paste actual tokens into public tickets or CI logs. If printing a token for inspection, keep the terminal secure and avoid recording output. + +## Lab 3. VPC, Shared VPC and firewall path + +**Goal:** Trace a connection from one VM/subnet to another. + +Read-only inventory: + +```bash +gcloud compute networks list --project=PROJECT_ID +gcloud compute networks subnets list --project=PROJECT_ID +gcloud compute routes list --project=PROJECT_ID +gcloud compute firewall-rules list --project=PROJECT_ID +gcloud compute networks subnets list-usable --project=PROJECT_ID +``` + +**Expected:** The VPC is global and its subnetworks regional. A route existing does not guarantee firewall permission or application reachability. A Shared VPC consumer can use host-project subnets only with proper host/service attachment and IAM. + +**Challenge:** Draw both ingress and egress decisions, including guest firewall and service listener. Explain why Cloud Router isn't a per-packet forwarding VM. + +**Safety:** Do not broaden production firewall rules for debugging. Use Connectivity Tests or a permitted temporary test VM with narrow rules. + +## Lab 4. Cloud DNS, load balancer and Public NAT + +**Goal:** Separate the inbound HTTPS path from outbound internet translation. + +```bash +gcloud dns managed-zones list --project=PROJECT_ID +gcloud compute backend-services list --project=PROJECT_ID +gcloud compute url-maps list --project=PROJECT_ID +gcloud compute health-checks list --project=PROJECT_ID +gcloud compute routers list --project=PROJECT_ID +``` + +On a permitted host with relevant DNS configuration: + +```bash +dig example.org A +curl -I --connect-timeout 5 https://example.org/ +``` + +**Expected:** Public and private DNS zones can return different answers. A Cloud Application Load Balancer handles an inbound frontend/target proxy/backend chain. Public NAT handles eligible outbound source translation and does not make the VM directly reachable from the public Internet. + +**Challenge:** Differentiate a TLS/SNI failure, LB 502 and outbound Cloud NAT port exhaustion. Describe what NAT metrics and LB backend health would prove. + +**Safety:** No production NAT address allocation, DNS zone or LB changes are necessary; cloud provider cost and reachability requirements make creation a separate controlled exercise. + +## Lab 5. Compute Engine and managed instance group design + +**Goal:** Understand VM versus template versus MIG lifecycle without deploying an expensive regional fleet. + +```bash +gcloud compute instances list --project=PROJECT_ID +gcloud compute instance-templates list --project=PROJECT_ID +gcloud compute instance-groups managed list --project=PROJECT_ID +gcloud compute machine-types list --zones=europe-west1-b --limit=10 +``` + +**Expected:** An instance template specifies repeatable configuration, while a MIG controls replica lifecycle. Autoscaling controls count; autohealing replaces unhealthy instances. A regional MIG improves zone distribution, but doesn't eliminate quota and zonal capacity risk. + +**Challenge:** Design a rollout with a canary, application warmup, LB health check, surge capacity, draining and rollback. Explain why a Spot-only single-replica financial service is unsafe. + +**Safety:** Use design/paper exercise unless an authorized budgeted disposable MIG is available. + +## Lab 6. Cloud Storage, Artifact Registry and event idempotency + +**Goal:** Distinguish object storage, image artifacts, transactional data and queue delivery semantics. + +Read-only: + +```bash +gcloud storage buckets list --project=PROJECT_ID +gcloud artifacts repositories list --location=europe-west1 --project=PROJECT_ID +gcloud sql instances list --project=PROJECT_ID +gcloud pubsub subscriptions list --project=PROJECT_ID +``` + +If an owned test bucket is available: + +```bash +printf 'sample\n' > sample.txt +gcloud storage cp sample.txt gs://OWNED_BUCKET/lab/sample.txt +gcloud storage objects describe gs://OWNED_BUCKET/lab/sample.txt +``` + +**Expected:** Cloud Storage is object storage with generation/retention behavior, not a POSIX database. Artifact Registry contains deployable image/package artifacts. Pub/Sub at-least-once patterns mean consumer business effects must be idempotent. + +**Challenge:** Trace a worker committing a DB side effect then crashing before ack. Explain outbox, idempotency key and dead-letter routing. + +**Safety:** Only write into an owned sandbox bucket; cleanup may not immediately remove charges if soft delete/retention is enabled. + +## Lab 7. Logging, tracing and SLO calculation + +**Goal:** Translate a user-facing symptom to traces and resource evidence. + +```bash +gcloud logging read 'severity>=ERROR' \ + --project=PROJECT_ID --limit=20 --format=json +gcloud logging sinks list --project=PROJECT_ID +``` + +Choose a hypothetical SLO: 99.9% of valid checkout operations complete successfully within a specified bound over 30 days. + +Given 100,000 eligible events and 220 noncompliant events: +- Allowed noncompliant events under 99.9% = 100. +- Actual noncompliant fraction = 0.22%. +- Budget consumption = 220%, exceeding budget by 120%. +- Burn rate relative to steady-state allowed error fraction = 2.2x over that evaluated window. + +**Expected:** Explain difference between an event-based SLO and downtime minutes. A trace reveals nested latency; metrics show distributions; audit logs reveal administrative operations; logs can be sampled or excluded. + +**Challenge:** Explain why averaging regional p99 values is mathematically wrong. Use aggregate histograms/distributions instead. + +## Lab 8. Marketplace architecture and failure-recovery tabletop + +**Goal:** Defend a design and a response to failure. + +Assume: global external HTTPS API, regional Compute/GKE backends, transactional Cloud SQL, Cloud Storage media, Pub/Sub background work and Cloud NAT outbound partner calls. + +```mermaid +flowchart LR + C["Client"] --> LB["HTTPS Load Balancer"] + LB --> APP["Regional application replicas"] + APP --> DB["Transactional database"] + APP --> PUB["Pub/Sub"] + PUB --> WORK["Idempotent workers"] + APP --> NAT["Cloud NAT"] + NAT --> PART["External partner"] +``` + +**Failure injection (paper exercise):** One region's application replicas become unhealthy, while the secondary region remains healthy but the transactional primary database is still regional. + +Answer in order: +1. What user actions are failing and what is the data correctness risk? +2. What do backend health, traces, SQL status and errors say? +3. Is it safe to redirect read requests? What about writes? +4. What is the measured RPO/RTO versus required objective? +5. How do you prevent split-brain or duplicate partner side effects? +6. What are the exact operator approvals and rollback/failback conditions? +7. Which IaC, SLO and recovery changes prevent recurrence? + +**Expected:** A multi-region frontend does not guarantee a writable multi-region backend. The correct answer includes data state, authorization, capacity, failover sequencing and post-recovery business checks. + +## Completion checklist + +- Explain a failed GCP API request through credential, IAM, API enablement, org policy and quota. +- Trace packet forwarding and firewall decisions across Shared VPC projects. +- Distinguish NAT, Private Google Access, DNS and HTTPS LB. +- Design MIG rollout without equating VM startup to application readiness. +- Select object, transactional, analytical and queue services based on guarantees. +- Define a measurable event-based SLO and error budget. +- Demonstrate a safe regional recovery decision, including data correctness. + +[← Module 5](./) · [40 follow-up questions](./follow-ups) diff --git a/docs/guides/gcp/q1.md b/docs/guides/gcp/q1.md new file mode 100644 index 0000000..7f10281 --- /dev/null +++ b/docs/guides/gcp/q1.md @@ -0,0 +1,163 @@ +# Q1. Google Cloud Architecture, Resource Hierarchy and Governance + +> **Interviewer:** You have used other clouds, but now need to design a Google Cloud landing zone for multiple product teams. Explain the resource hierarchy, project boundaries, billing, IAM inheritance, APIs, quotas, regions, organizational policies and what happens when a project is deleted. + +## 1. Start with the control model + +Google Cloud is not one big collection of VMs. Every resource belongs to an administrative hierarchy, identity and policy model, and a service API. A project is the principal operational container for many Google Cloud resources, but it is **not** necessarily the boundary of the network. A Shared VPC can be hosted in a different project, while organization and folder policies apply from higher ancestors. + +```mermaid +flowchart TD + O["Organization"] --> SEC["Folder: security"] + O --> PROD["Folder: production"] + O --> NON["Folder: nonproduction"] + PROD --> PA["Project: marketplace-app"] + PROD --> PD["Project: marketplace-data"] + PROD --> PN["Project: shared-network"] + NON --> PS["Project: marketplace-stage"] + PA --> API["Compute / GKE / application APIs"] + PD --> DB["Managed databases / storage"] + PN --> NET["Shared VPC / firewall / routes"] + B["Billing account"] -.-> PA + B -.-> PD + B -.-> PN + B -.-> PS +``` + +Organization, folders and projects form a parent/child hierarchy; services' resources normally live under projects. IAM allow-policy inheritance flows down the hierarchy. An allow binding on a production folder may confer access in all descendant projects, so access granted high in the tree has wide blast radius. Organization Policy constraints can restrict what descendants may configure; their inheritance/evaluation is not identical to IAM allow policies. Deny policies are an additional IAM enforcement mechanism. and . + +**AWS mapping:** GCP organization/folder/project roughly resembles AWS Organizations/OUs/accounts for governance, but **a GCP project isn't an AWS account clone**. AWS accounts are generally stronger autonomous boundaries; Google Cloud IAM, VPC sharing and project/service policy models must be evaluated on their own terms. Cloud Billing accounts are not IAM organizations or network namespaces. + +## 2. Resource location and scope + +| Resource | Typical scope | Important consequence | +| --- | --- | --- | +| Organization / folder / project | Administrative hierarchy | Controls inherited permissions and policies | +| VPC network | Global | One VPC can contain subnets in multiple regions | +| Subnet | Regional | VMs in eligible zones in that region use its address range | +| VM instance | Zonal | Zone failure affects the VM | +| Regional managed instance group | Regional | Can place replicas across zones | +| Cloud Storage bucket | Selected location type | Data placement and redundancy follow bucket location | +| Global external Application Load Balancer | Global frontend; backend scopes vary | Multi-region backends possible depending on type | +| GKE Standard cluster | Zonal or regional by cluster configuration | Control-plane and node availability differ | + +Do not describe a global VPC as globally available application infrastructure: workloads and backends have their own zonal/regional failure domains. + +## 3. API enablement and identity + +Google services expose service APIs such as `compute.googleapis.com`, `container.googleapis.com` and `iam.googleapis.com`. A project may exist and have billing enabled while a required API is disabled. API enablement is not itself an IAM grant. + +```bash +gcloud auth login +gcloud auth list +gcloud config configurations list +gcloud config set project marketplace-sandbox +gcloud projects describe marketplace-sandbox +gcloud services list --enabled --project=marketplace-sandbox +gcloud services enable compute.googleapis.com --project=marketplace-sandbox +gcloud billing projects describe marketplace-sandbox +``` + +These commands are instructional. Never run `services enable` or billing changes in an unapproved project. For automation, prefer service account impersonation, metadata-based credentials or federation over personal user credentials. + +```mermaid +sequenceDiagram + participant D as Developer/CI + participant IAM as Auth/IAM + participant API as Google service API + participant P as Project + D->>IAM: Obtain appropriate credentials + D->>API: Invoke method on resource in project + API->>API: Authenticate principal + API->>IAM: Evaluate allow + deny + applicable constraints + IAM-->>API: Authorization decision + API->>P: Execute if API enabled, quota/location valid + P-->>API: Resource or error + API-->>D: Response + audit event +``` + +The diagram is conceptual; authorization, org policy, quota and validation may be enforced at different service stages. **A 403 can mean missing permission, API disabled, service policy, or billing/service restriction.** Read structured error details rather than guess. + +## 4. Roles and hierarchy + +- **Principal:** User, group, service account or eligible federated identity. +- **Role:** Collection of IAM permissions, for example `compute.instances.get`. +- **Policy binding:** Associates role with members on an applicable resource. +- **Predefined/custom/basic roles:** Prefer narrowly scoped predefined/custom roles to broad basic Owner/Editor. +- **IAM Conditions:** Additional conditional constraints on a binding when supported. +- **IAM Deny:** Can explicitly forbid permissions; inherited allow doesn't override applicable deny. +- **Organization Policy:** Governs permitted configurations, such as location, public exposure or allowed external identities. + +```bash +gcloud projects get-iam-policy marketplace-sandbox --format=json +gcloud organizations list +gcloud resource-manager folders list --organization=ORGANIZATION_ID +gcloud projects get-ancestors marketplace-sandbox +``` + +Permissions to enumerate an organization and its folders depend on caller access. A project user might not see its organizational ancestors in detail. + +## 5. Quotas, service limits and billing + +Quotas are frequently **project-, region- or resource-specific**. A team may have sufficient regional CPU quota but hit external IP, forwarding-rule, API rate, GPU or managed-service quotas. A quota increase is not a guarantee of zonal physical capacity. + +For production readiness record: +1. Service usage and enabled APIs. +2. Project-specific regional quota ceilings. +3. Requested maximum capacity and buffer. +4. Billing budgets/alerts and labels. +5. Organization-policy restrictions. +6. Disaster-recovery project/region strategy. +7. Ownership and escalation contacts. + +```bash +gcloud compute regions describe europe-west1 --project=marketplace-sandbox --format='table(quotas.metric,quotas.usage,quotas.limit)' +gcloud services list --enabled --project=marketplace-sandbox +gcloud asset search-all-resources --scope=projects/PROJECT_NUMBER +``` + +Quota command output varies by service/version; use the provider's Quotas & System Limits console/API for authoritative per-service values. Cloud Asset Inventory requires additional access. + +## 6. Projects, lifecycle and production safety + +Project deletion is a high-impact operation. Deleting a project affects resources, service access and related billing, with a recovery window and behavior subject to current policy. Don't model it as equivalent to deleting a folder in a source repository. Use deletion controls, separated administration, least privilege, asset inventory and documented recovery. + +An organization may separate: +- **Shared-network host project:** centrally managed networking. +- **Application service projects:** independently managed workloads. +- **Security/logging projects:** central security tools and sinks. +- **Data projects:** restricted data services and encryption controls. + +But more projects introduce IAM, service API and quota management overhead. Choose boundaries aligned with owners and trust domains. + +## 7. Production incident: no new VMs after a release + +**Hypothetical:** An autoscaler cannot add more Compute Engine workers. Cloud Monitoring shows low CPU usage on the existing fleet. + +**Evidence-driven investigation** +1. Examine instance/instance-group operations and the exact failure reason. +2. Verify project, zone, requested machine family and quota. +3. Distinguish `QUOTA_EXCEEDED` from `ZONE_RESOURCE_POOL_EXHAUSTED`, IAM authorization and disabled API. +4. Check org policy denying the requested external address, region or machine. +5. Assess fallback zones, different supported machine types and required workload topology. +6. Add appropriate quota/capacity headroom; do not simply raise a Kubernetes replica target. + +```bash +gcloud compute operations list --project=marketplace-sandbox --limit=20 +gcloud compute instance-groups managed list --project=marketplace-sandbox +gcloud logging read 'severity>=ERROR' --project=marketplace-sandbox --limit=20 +``` + +The error source is the operation and service API response, not necessarily Cloud Logging. + +## 8. Architecture trade-offs + +A single project for every environment is easy to start but broadens privileges, quota contention and billing attribution. A project per ephemeral deployment can be operationally costly. A lead engineer should define an intentionally limited project taxonomy, automation for provisioning and cleanup, organization policies, budgets and explicit service-project/host-project responsibilities. + +## Spoken answer + +> “I start with the resource hierarchy: organization, folders, projects and the service resources within them. Projects provide important isolation for billing, service APIs, IAM and quotas, but network boundaries can be shared. IAM access can be inherited from folders and organization; deny and organization policy can impose restrictions. I'd design separate production, nonproduction, network and security ownership where justified, use narrow roles and short-lived identities, and model regional failure domains and quotas early. When provisioning fails, I'd check the API's exact error, identity, organization policy, quota and physical capacity separately.” + +**Further reading:** [Resource hierarchy](https://cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy) · [IAM overview](https://cloud.google.com/iam/docs/overview) · [Organization Policy](https://cloud.google.com/resource-manager/docs/organization-policy/overview) · [Quotas](https://cloud.google.com/docs/quotas/overview) + +[Module index](./) · [Next: IAM and identities](./q2) diff --git a/docs/guides/gcp/q2.md b/docs/guides/gcp/q2.md new file mode 100644 index 0000000..f9a5078 --- /dev/null +++ b/docs/guides/gcp/q2.md @@ -0,0 +1,175 @@ +# Q2. IAM, Service Accounts, OAuth, ADC and Workload Identity Federation + +> **Interviewer:** A backend runs on GKE and needs to read from Cloud Storage. Explain how you would authenticate it without a JSON key. What is the difference between authentication, authorization, OAuth scopes, IAM role bindings, service account impersonation and workload identity? Show your diagnostic method for 403 errors. + +## 1. Two different questions + +**Authentication:** Which principal is the caller? OAuth access tokens, identity tokens and other supported credentials establish identity and audience. + +**Authorization:** May that principal perform `storage.objects.get` on the specific resource? IAM allow/deny, conditions, organizational constraints and any product-specific controls apply. + +```mermaid +flowchart TD + W["Workload / engineer / CI runner"] --> C["Credential acquisition"] + C --> A["Authenticated Google principal"] + A --> IAM["IAM policies on project or resource"] + IAM --> DEC{"Authorized? "} + DEC -->|Yes| API["Service API method"] + DEC -->|No| E["403 / policy denial"] + API --> OBJ["Specific resource operation"] +``` + +Roles contain permissions; principals are bound to roles at a scope. Permissions can inherit from higher resources, but resource-level data access still depends on the exact service's controls. + +## 2. Service accounts + +A **service account** is a Google identity for machine-to-machine access. Its email resembles `catalog-reader@marketplace-sandbox.iam.gserviceaccount.com`. Merely attaching a service account doesn't grant it permissions; assign the smallest roles on the intended resource. + +```bash +gcloud iam service-accounts create catalog-reader --project=marketplace-sandbox --display-name="Catalog read-only workload" +gcloud iam service-accounts describe catalog-reader@marketplace-sandbox.iam.gserviceaccount.com +``` + +For a sandbox bucket, grant bucket-scoped access instead of project-wide Storage Admin: + +```bash +gcloud storage buckets add-iam-policy-binding gs://EXAMPLE_CATALOG_BUCKET --member="serviceAccount:catalog-reader@marketplace-sandbox.iam.gserviceaccount.com" --role="roles/storage.objectViewer" +``` + +Use an actual owned bucket; this is not a public sample bucket. Avoid creating/downloadable JSON private keys except exceptional legacy integrations, with compensating controls and rotation. + +## 3. ADC and credential sources + +Application Default Credentials (ADC) are the standard discovery mechanism for Google client libraries. ADC is **not** itself a Google IAM role. In an environment with access to workload metadata, clients can obtain short-lived credentials without a key file. Local `gcloud auth login` and `gcloud auth application-default login` configure different credential stores. + +```bash +gcloud auth list +gcloud auth application-default print-access-token +gcloud config get-value project +gcloud auth print-identity-token --audiences=https://example.invalid +``` + +The identity-token command is illustrative and requires compatible credentials and an appropriate audience. Avoid printing real tokens into shared terminal histories or CI logs. + +**Access token versus ID token:** An access token authorizes Google API calls within granted permissions; an ID token primarily identifies the workload to an audience-verifying service, such as a protected HTTP endpoint. A valid token for one audience is not automatically valid for another service. + +## 4. Workload Identity Federation from external CI + +For a GitHub Actions runner, federation can exchange a trusted OIDC assertion for credentials through Security Token Service, then use direct resource permissions or service account impersonation: + +```mermaid +sequenceDiagram + participant CI as External CI workload + participant IDP as OIDC issuer + participant STS as Google STS + participant IAM as IAM / SA impersonation + participant API as Google Cloud API + CI->>IDP: Request signed OIDC assertion + IDP-->>CI: Assertion with claims + CI->>STS: Exchange assertion under WIF pool/provider + STS->>STS: Validate issuer, audience, attribute conditions + STS-->>CI: Federated short-lived credential + opt Impersonation model + CI->>IAM: Request target service account access token + IAM-->>CI: Short-lived SA token (if permitted) + end + CI->>API: Authenticated API request + API-->>CI: Authorized response or 403 +``` + +Restrict the provider to expected issuer, repository/ref/environment identity and audience, and grant only the required principal identities. Workload identity pools use **project numbers** in their fully qualified principal identifiers. For impersonation, grant the appropriate `roles/iam.workloadIdentityUser` role to the allowed principal on the target service account. Direct federated principal resource access is an alternative when the API supports it. See [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation). + +## 5. Workload Identity Federation for GKE + +A Kubernetes ServiceAccount (KSA) identity and Google IAM principal/Google Service Account (GSA) are different objects. + +```mermaid +flowchart LR + P["GKE Pod"] --> KSA["Kubernetes ServiceAccount"] + KSA --> MD["GKE identity metadata server"] + MD --> WIF["GKE WIF / token service"] + WIF --> IAM["Google IAM permission check"] + IAM --> GCS["Cloud Storage object"] +``` + +Depending on supported cluster settings, grant direct access to the GKE workload principal or configure service account impersonation. The sample KSA: + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: catalog-api + namespace: marketplace +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: catalog-api + namespace: marketplace +spec: + replicas: 2 + selector: + matchLabels: { app: catalog-api } + template: + metadata: + labels: { app: catalog-api } + spec: + serviceAccountName: catalog-api + containers: + - name: app + image: europe-west1-docker.pkg.dev/PROJECT_ID/apps/catalog-api:VERSION +``` + +The YAML does **not** by itself grant Cloud Storage access. Grant IAM permissions to the actual workload principal or impersonated GSA. Refer to the cluster workload pool, project number, KSA namespace/name and node metadata support. + +Kubernetes RBAC governs the Kubernetes API; IAM governs Google Cloud APIs. Neither automatically grants the other. + +## 6. Google IAM scope and ownership + +```bash +gcloud projects get-iam-policy marketplace-sandbox --format=json +gcloud storage buckets get-iam-policy gs://EXAMPLE_CATALOG_BUCKET +gcloud iam service-accounts get-iam-policy catalog-reader@marketplace-sandbox.iam.gserviceaccount.com +``` + +Be aware of IAM role binding drift: Terraform's `google_project_iam_policy` is authoritative over a whole policy; `google_project_iam_binding` authoritative for role membership; `google_project_iam_member` additive for a member/role binding. Two independent controllers writing authoritative policies can revoke each other. + +**OAuth scopes** are an additional legacy/VM access-token boundary in some workloads, not a substitute for IAM. On GKE, client scopes and node configuration can affect access paths, but workload identity should be the main Pod-level identity design. + +## 7. Troubleshoot a Cloud Storage 403 + +1. Identify exactly **which principal** sent the request; don't assume the pod's named KSA is mapped. +2. Confirm token type, expiry and intended audience. +3. Verify permission and role at the bucket/object scope, including resource ancestry, conditions and applicable denies. +4. Inspect project number, KSA namespace/name, workload pool and WIF provider attributes. +5. Check storage-specific restrictions (VPC Service Controls, bucket policies, object retention, requester pays if relevant). +6. Use Policy Troubleshooter and audit logs, while accounting for audit-log coverage/configuration. + +```bash +gcloud storage ls gs://EXAMPLE_CATALOG_BUCKET +gcloud policy-intelligence troubleshoot-policy iam //storage.googleapis.com/projects/_/buckets/EXAMPLE_CATALOG_BUCKET --principal-email=catalog-reader@marketplace-sandbox.iam.gserviceaccount.com --permission=storage.objects.list +gcloud logging read 'protoPayload.status.code=7' --project=marketplace-sandbox --limit=20 +``` + +The Policy Troubleshooter command and resource format depend on current `gcloud` components and permissions; check `gcloud policy-intelligence troubleshoot-policy iam --help` before using. Logs may be absent when applicable Data Access audit logs aren't enabled. + +## 8. Common failures and trade-offs + +| Symptom | Likely branch | +| --- | --- | +| 401 | Missing, invalid or expired authentication | +| 403 | Identity authentic but lacks permission or blocked by policy | +| CI worked locally but fails in pipeline | Different ADC principal / audience / impersonation rights | +| GKE Pod works on one node pool only | Metadata server/WIF node configuration or networking | +| Token good but request denied | Role scope, condition, VPC SC, service-specific controls | +| Broad Owner role fixes issue | Unsafe masking of missing least-privilege permission | + +**AWS comparison:** GCP Workload Identity Federation relates conceptually to AWS STS web-identity role assumption, but IAM role trust policies and Google IAM principal bindings have distinct formats and evaluation rules. Treat them separately. + +## Spoken answer + +> “I'd first identify the calling principal, then trace how its short-lived credential was issued and which permission is needed for the exact Cloud Storage resource. On GKE I'd use Workload Identity Federation rather than mounting JSON keys, with a dedicated Kubernetes ServiceAccount and narrow bucket IAM. For external CI I'd restrict OIDC federation to expected repository or environment claims. A 403 should be diagnosed through identity, role scope, conditions, deny and product-specific policy; adding Project Owner is never a satisfactory root-cause fix.” + +**Further reading:** [Service accounts](https://cloud.google.com/iam/docs/service-account-overview) · [ADC](https://cloud.google.com/docs/authentication/application-default-credentials) · [WIF for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) · [IAM Conditions](https://cloud.google.com/iam/docs/conditions-overview) + +[Previous](./q1) · [Next: VPC and Shared VPC](./q3) diff --git a/docs/guides/gcp/q3.md b/docs/guides/gcp/q3.md new file mode 100644 index 0000000..40e84c7 --- /dev/null +++ b/docs/guides/gcp/q3.md @@ -0,0 +1,186 @@ +# Q3. VPC Networking, Shared VPC, Routes, Firewalls and Hybrid Connectivity + +> **Interviewer:** Design networks for a multi-project service. Explain global VPC scope, regional subnets, routes, hierarchical firewall policy, Shared VPC, Private Google Access, hybrid connectivity and how to trace a VM-to-VM packet. Why does the AWS mental model often mislead engineers on GCP? + +## 1. Resource scope is fundamental + +A GCP VPC network is global; its subnets are regional. A subnet's primary IPv4 range serves eligible VMs and other primary-address consumers; it may have secondary ranges for alias IPs, such as VPC-native GKE Pods and Services. + +```mermaid +flowchart TB + V["Global VPC: marketplace-network"] --> EU["Regional subnet: europe-west1"] + V --> US["Regional subnet: us-central1"] + EU --> VM1["VM in zone b"] + EU --> VM2["VM in zone c"] + EU --> POD["Secondary Pod IP CIDR"] + US --> VM3["VM in another region"] +``` + +GCP automatically provides subnet routes and supports additional static/dynamic routes. Intra-VPC network reachability remains subject to firewall rules and applicable service security. Do not describe VPC global scope as promising low-latency or failure-free interregional traffic. See [VPC specifications](https://cloud.google.com/vpc/docs/vpc). + +**AWS contrast:** AWS VPC is regional with zonal subnet placement. GCP networks can hold subnets across regions in one VPC; resources are not all global. + +## 2. Create custom-mode network and subnet + +```bash +gcloud compute networks create marketplace-vpc \ + --subnet-mode=custom --project=marketplace-sandbox +gcloud compute networks subnets create app-ew1 \ + --network=marketplace-vpc \ + --region=europe-west1 \ + --range=10.40.0.0/20 \ + --enable-private-ip-google-access \ + --enable-flow-logs \ + --project=marketplace-sandbox +gcloud compute networks describe marketplace-vpc --project=marketplace-sandbox +gcloud compute networks subnets list --project=marketplace-sandbox +``` + +All examples assume a disposable sandbox; creating resources may incur cost. Auto-mode networks predefine subnets; custom mode offers explicit CIDR and region planning and is generally easier to govern for mature platforms. + +Plan CIDRs across VPCs, VPN/Interconnect networks, peered networks and secondary ranges. Overlap can break routing/peering and complicate migrations. Address-family support and IPv6 details vary by configuration. + +## 3. Packet route from one VM to another + +```mermaid +sequenceDiagram + participant A as VM A in europe-west1 + participant F as VPC firewall evaluation + participant R as GCP virtual network routes + participant B as VM B in us-central1 + A->>F: Packet source/destination, protocol, port + F->>F: Evaluate applicable egress controls + F->>R: Route lookup / virtual forwarding + R->>F: Destination path + F->>F: Apply destination ingress controls + F->>B: Deliver if route and security permit + B-->>A: Return flow subject to stateful policies +``` + +This is a logical policy/routing model, **not** a claim that packets traverse centralized firewall or router VMs. Google Cloud virtual networking implements distributed forwarding and firewall evaluation. VPC firewall rules are stateful for permitted connections, with details depending on the rule and protocol. + +### Route decision + +Google Cloud determines the applicable routes using route categories, specificity and documented precedence. Never assume the simple “longest prefix always wins over every class of route” model when policy-based routes and system-generated routes are involved. + +```bash +gcloud compute routes list \ + --filter='network:marketplace-vpc' \ + --project=marketplace-sandbox +gcloud compute firewall-rules list \ + --filter='network:marketplace-vpc' \ + --project=marketplace-sandbox +``` + +For BGP-learned dynamic routes, inspect Cloud Router status and advertised/learned routes. + +## 4. Firewall model + +GCP supports VPC firewall rules and hierarchical/network firewall policies in supported environments. Evaluation includes rule priorities, match conditions, implied policy and higher-level enforcement. Rules can target instances by service account or network tags, with limitations according to firewall type. + +```bash +gcloud compute firewall-rules create allow-internal-api \ + --network=marketplace-vpc \ + --direction=INGRESS \ + --priority=1000 \ + --action=ALLOW \ + --rules=tcp:8080 \ + --source-ranges=10.40.0.0/20 \ + --target-tags=internal-api \ + --project=marketplace-sandbox +``` + +This uses a broad internal source range solely to illustrate command syntax, not a complete production least-privilege policy. Network tags aren't a cryptographic identity; constrain who may set them. Where supported, service-account-targeted rules improve association with workload identities. + +**Implicit rule intuition:** In a VPC, implied ingress denial and implied egress allowance exist, unless higher-priority explicit applicable policy changes behavior. Check the current firewall policy hierarchy instead of assuming an empty VPC is fully isolated. + +### Rule evaluation and GKE + +Node firewall rules, GKE network policies and cloud LB health-check ranges operate at different layers. A VM can be reachable from peers but fail health checks if firewall policy blocks the health-check source ranges relevant to its LB type. + +## 5. Shared VPC + +Shared VPC centralizes network administration while letting workloads belong to different **service projects**. The host project owns shared VPC and subnets; an attached service project may run VM/GKE workloads using granted host subnets. + +```mermaid +flowchart TD + ORG["Organization"] --> HOST["Host project: network-foundation"] + ORG --> APP["Service project: catalog"] + ORG --> DATA["Service project: analytics"] + HOST --> VPC["Shared VPC + subnetworks"] + APP --> A["Catalog VMs / GKE"] + DATA --> D["Analytics VMs"] + A -.->|"Uses permitted host subnets"| VPC + D -.->|"Uses permitted host subnets"| VPC +``` + +Service-project teams require suitable roles such as subnetwork use permissions on host resources, plus ordinary permissions to create workloads in their service project. Network admins retain control of subnet routing and firewall policy. Host/service attachment is a project-level relationship; current Shared VPC policy imposes organization constraints. See [Shared VPC](https://cloud.google.com/vpc/docs/shared-vpc). + +### Why adopt Shared VPC? + +- Central networking and address allocation. +- One policy/firewall governance model. +- Independent service-team project/billing/resource IAM. +- Cleaner interservice private routing. + +**Trade-offs:** Shared failure domains, network dependency on host project, more complex IAM and potential cross-team change coordination. It is not appropriate if teams require fully independent trust/routing domains. + +## 6. Private Google Access is not Public NAT + +Private Google Access allows eligible VMs without external IPs to reach supported Google APIs/services through internal routing/address mechanisms. It doesn't provide general outbound internet access to arbitrary third-party services. + +Public NAT provides eligible outbound internet access to external destinations using managed source translation. Private Service Connect provides private connectivity to supported services through endpoints/service attachments and has different consumer-producer semantics. VPC Service Controls are a data exfiltration security perimeter, **not** a packet routing feature. + +```text +VM without external IP -> Google API: + eligible PGA/private endpoint path may suffice +VM without external IP -> external partner API: + requires appropriate egress path, commonly Public NAT +VM -> private producer service: + PSC/private network path if supported +``` + +## 7. Hybrid networking + +Cloud VPN creates encrypted IPsec tunnels for supported routing topologies. Cloud Interconnect provides dedicated or partner connectivity. Cloud Router exchanges BGP routes for supported dynamic routing modes, but isn't an ordinary appliance that forwards each data packet. + +```mermaid +flowchart LR + O["On-premises network"] <-->|"VPN or Interconnect"| H["Hybrid edge"] + H <--> CR["Cloud Router BGP control plane"] + H --> V["GCP VPC data path"] + V --> S["Regional subnet workloads"] +``` + +For a hybrid incident, distinguish tunnel health, BGP session status, imported/advertised routes, CIDR overlap, firewall rules, MTU and return routes. A green VPN tunnel by itself does not prove application reachability. + +## 8. Production scenario: one project cannot reach a shared backend + +**Hypothetical:** Catalog runs in service project A and data services in service project B. Both should use Shared VPC, but catalog times out. + +Runbook: +1. Confirm both service projects attached to correct host. +2. Check source/destination subnet and actual VM interfaces. +3. Check route, local/global firewall policy and service identity targeting. +4. Capture VPC Flow Logs from relevant subnets; remember sampling and aggregation mean missing log entries aren't proof that packets never existed. +5. Run Network Intelligence Center Connectivity Tests to evaluate modeled paths. +6. Check application listener, process, TLS and downstream security. + +```bash +gcloud compute networks subnets list-usable \ + --project=marketplace-sandbox +gcloud compute routes list --project=marketplace-sandbox +gcloud compute firewall-rules list --project=marketplace-sandbox +gcloud logging read 'resource.type="gce_subnetwork"' \ + --project=marketplace-sandbox --limit=30 +``` + +The log filter is illustrative; Flow Logs may appear in specific logging resource types and projects depending on configuration. Use observed log metadata and Connectivity Tests instead of assuming a universal resource-type filter. + +## Spoken answer + +> “I'd start with the global VPC and regional subnet model, then separate network routing from firewall policy and workload identity. Shared VPC lets a central host project govern network resources while service projects own workloads, provided the required cross-project IAM is in place. For a failed connection I'd trace source IP and subnet, route choice, ingress/egress firewall, service listener and return traffic. I'd distinguish Private Google Access from NAT and remember Cloud Router is a routing control plane, not an in-path NAT appliance. Hybrid links add BGP, MTU and return-route failure modes.” + +**Further reading:** [VPC](https://cloud.google.com/vpc/docs/vpc) · [Shared VPC](https://cloud.google.com/vpc/docs/shared-vpc) · [Routes](https://cloud.google.com/vpc/docs/routes) · [Firewall](https://cloud.google.com/firewall/docs/firewalls) · [Hybrid Connectivity](https://cloud.google.com/network-connectivity/docs) + +[Previous](./q2) · [Next: NAT, DNS and load balancing](./q4) diff --git a/docs/guides/gcp/q4.md b/docs/guides/gcp/q4.md new file mode 100644 index 0000000..be113ea --- /dev/null +++ b/docs/guides/gcp/q4.md @@ -0,0 +1,177 @@ +# Q4. Cloud NAT, Cloud DNS, Load Balancing and End-to-End Request Flow + +> **Interviewer:** A user reaches a public HTTPS marketplace API in Google Cloud, and the backend calls an external payment provider. Trace both directions through DNS, TLS, load balancing, VPC connectivity, firewall rules and NAT. Explain how to diagnose timeouts, 502s, NAT port exhaustion and DNS failures. + +## 1. Don't collapse products into “one load balancer” + +Google Cloud load balancing offers Application (HTTP/S, Layer 7) and Network (Layer 4, proxy and passthrough) products, each with external/internal, global/regional and sometimes cross-region deployments. Feature support and backend location depend on the exact class. External global Application Load Balancers can support multi-region backends; internal regional L7 LBs have different reachability rules. See [Cloud Load Balancing overview](https://cloud.google.com/load-balancing/docs/load-balancing-overview). + +### Layer 7 conceptual resources + +```mermaid +flowchart TD + DNS["Public DNS name"] --> VIP["External frontend IP + forwarding rule"] + VIP --> TP["HTTPS target proxy / TLS certificate"] + TP --> URL["URL map / routing rules"] + URL --> BS["Backend service"] + BS --> HC["Health check"] + BS --> B1["Instance group / NEG backend A"] + BS --> B2["Instance group / NEG backend B"] + WAF["Cloud Armor policy, if configured"] -.-> BS +``` + +The forwarding rule points at the frontend target; the target proxy handles TLS; URL map selects backend service; backend service defines health, capacity/session features and backends. Some load balancer types expose a different resource chain. **Not every product's packet flow looks like this.** + +## 2. HTTPS request end to end + +```mermaid +sequenceDiagram + participant U as Browser + participant D as Recursive/public DNS + participant G as GFE or Envoy frontend + participant B as Backend VM/NEG + participant S as Downstream service + U->>D: Query api.example.invalid + D-->>U: A/AAAA frontend address + U->>G: TCP/TLS handshake, SNI and ALPN + G->>G: Evaluate routing/security policy + G->>B: New backend request/connection + B->>S: Authorized dependency call + S-->>B: Response + B-->>G: HTTP response + G-->>U: HTTP response +``` + +This is a logical proxy-based L7 path. Client TLS may terminate at the frontend, while backend traffic can use HTTP or separately configured HTTPS. Backend health and real request success are distinct. For passthrough L4 products, the packet destination/source handling differs; don't assume TLS terminates in the LB. + +### Production failure isolation + +- DNS lookup failure: resolver/delegation/zone/policy path. +- TCP connect failure: frontend listener, reachability or policy. +- TLS failure: certificate/SNI/ALPN or protocol mismatch. +- 502/503: backend service routing/health, wrong named port, backend TLS, deployment health, rate limits or application response errors. +- High TTFB: proxy queueing, backend saturation, dependency latency. +- Inconsistent regions: routing, health, service capacity and policy. + +```bash +dig +short api.example.invalid A +dig +trace api.example.invalid +curl -Iv --connect-timeout 5 https://api.example.invalid/ +openssl s_client -connect api.example.invalid:443 -servername api.example.invalid +gcloud compute backend-services list --project=marketplace-sandbox +gcloud compute url-maps list --project=marketplace-sandbox +gcloud compute health-checks list --project=marketplace-sandbox +``` + +The domain is a documentation-only domain and intentionally doesn't resolve to a real service. For backend health, select the correct `--global` or `--region` API and inspect the specific backend service. + +## 3. Cloud DNS + +Cloud DNS supports public zones and private zones, with inbound/outbound forwarding and server policies for supported hybrid environments. Public authoritative DNS answers external recursive resolvers; a private zone is visible to associated authorized VPC networks. Overlapping private zones and forwarding policy can change which answers VMs receive. + +```mermaid +flowchart TD + Q["VM stub resolver"] --> M["Google metadata DNS resolution"] + M --> Z{"Applicable private zone / policy?"} + Z -->|Yes| P["Private Cloud DNS zone / forwarding"] + Z -->|No| R["Public recursive resolution"] + P --> A["Private answer or forwarding result"] + R --> B["Public DNS result"] +``` + +Private DNS lookup precedence and forwarding details are implementation-specific; see documentation instead of assuming public fallback after every private NXDOMAIN. + +```bash +gcloud dns managed-zones list --project=marketplace-sandbox +gcloud dns record-sets list --zone=ZONE_NAME --project=marketplace-sandbox +gcloud dns policies list --project=marketplace-sandbox +dig @169.254.169.254 internal.example. A +``` + +The last test assumes Linux VM metadata resolver behavior; actual query results require a configured zone. Diagnose negative caching and TTL changes at recursive resolvers too. Split-horizon can cause Cloud VMs and public clients to resolve different addresses. + +## 4. Public NAT and outbound connections + +Public NAT translates supported outbound connections from eligible private instances using external IPv4 addresses and allocated source ports. It **doesn't create a listening inbound public endpoint** on a private VM. Cloud NAT is distributed and managed; Cloud Router carries its configuration and routing-related control data but isn't the packet forwarding appliance. + +```mermaid +sequenceDiagram + participant A as Backend VM / GKE node + participant N as Public NAT + participant P as Partner API + A->>N: TCP SYN from private address + ephemeral port + N->>N: Allocate NAT external IP/source port mapping + N->>P: TCP SYN with translated source + P-->>N: SYN/ACK to NAT external mapping + N-->>A: Reverse-translate reply + A->>N: Application request + N->>P: Translated packets + P-->>A: Replies via NAT mapping +``` + +This is conceptual; there is no mandatory separate NAT middlebox VM. For GKE, determine whether Pod sources are SNATed to node IPs first; NAT allocation and logs can vary with source mode. + +### Why can NAT ports run out? + +Port allocations are finite. Connection churn, many source VM instances, concentrated connections to the same remote IP/port, static/minimum port allocations and NAT address supply can create port allocation failures even with spare CPU/RAM. Watch NAT packet-drop and port-allocation metrics. Unlike a proxy, NAT doesn't automatically queue/retry a failed application request. + +```bash +gcloud compute routers nats list --router=marketplace-router --region=europe-west1 --project=marketplace-sandbox +gcloud compute routers nats describe marketplace-nat --router=marketplace-router --region=europe-west1 --project=marketplace-sandbox +gcloud compute routers get-nat-mapping-info marketplace-router --region=europe-west1 --project=marketplace-sandbox +``` + +These operations require existing router/NAT and appropriate IAM permissions. Mapping inspection may require selecting a VM in some CLI/API versions; check help and actual schema. + +## 5. Configure Public NAT (example) + +```bash +gcloud compute routers create marketplace-router --network=marketplace-vpc --region=europe-west1 --project=marketplace-sandbox +gcloud compute routers nats create marketplace-nat --router=marketplace-router --region=europe-west1 --auto-allocate-nat-external-ips --nat-all-subnet-ip-ranges --enable-logging --project=marketplace-sandbox +``` + +Review logging cost, min/max port allocation mode, supported subnet coverage and firewall/route behavior before production. Automatic NAT IP allocation doesn't eliminate port planning. + +## 6. Private Google Access, DNS and Private Service Connect + +- **Private Google Access:** Supported Google API access for eligible VMs without external IP. +- **Public NAT:** Outbound internet connectivity using shared external address/port translation. +- **Private Service Connect:** Private endpoint/service attachment architecture for supported Google or producer services. +- **Cloud DNS:** Name resolution; it does not itself grant network reachability. +- **Cloud Armor:** Security policy attached to supported load-balancing backends; not a generic VM firewall. + +These components have different failure modes; using one does not imply the others are configured. + +## 7. Hypothetical incident: checkout cannot call partner API + +**Symptoms:** Internal health checks pass; newly initiated outbound partner calls time out during a high-volume auction close. + +**Investigation** +1. Confirm DNS resolves the expected partner destination from affected nodes. +2. Separate connection timeout from TLS timeout and application 429/5xx. +3. Check source subnet, private/default route, NAT region and eligible source range. +4. Inspect allocated NAT ports, packet drops, translation IPs and destination concentration. +5. Examine ephemeral port churn, connection pooling and HTTP keepalive. +6. Verify partner allowlisted outbound IPs and firewall egress rules. +7. Mitigate via appropriate port/address capacity, connection reuse and bounded concurrency; don't blindly increase Pod replica count because more sources/connections can worsen exhaustion. + +```bash +getent ahosts partner.example.invalid +curl -sv --connect-timeout 3 https://partner.example.invalid/health +ss -s +ss -tan state time-wait | head +``` + +The partner domain and endpoint are hypothetical. Use approved diagnostic endpoints; retries of payments must preserve idempotency. + +### Another incident: 502 while Pods show Ready + +Inspect backend health with the relevant LB API, check NEG membership and health-check settings, backend firewall ranges, application port and recent rollout. A Kubernetes readiness probe does not guarantee Google load-balancer health. + +## Spoken answer + +> “For external HTTPS I'd trace public DNS, the frontend address, TLS termination, URL map and backend service before reaching a VM or NEG. For the backend's outbound partner call I'd separately trace DNS, routes, egress firewall and Cloud NAT source-port translation. Google L7 and L4 load balancers have different termination models, so I'd name the exact product. For 502s I'd inspect backend health and application errors; for outbound connection timeouts I'd check NAT coverage, allocated ports, connection churn and partner allowlisting. I'd validate the actual source address rather than assume AWS NAT behavior maps directly to GCP.” + +**Further reading:** [Load balancing](https://cloud.google.com/load-balancing/docs/load-balancing-overview) · [Cloud DNS](https://cloud.google.com/dns/docs/overview) · [Public NAT](https://cloud.google.com/nat/docs/public-nat) · [Connectivity Tests](https://cloud.google.com/network-intelligence-center/docs/connectivity-tests/concepts/overview) + +[Previous](./q3) · [Next: Compute Engine](./q5) diff --git a/docs/guides/gcp/q5.md b/docs/guides/gcp/q5.md new file mode 100644 index 0000000..a65e5c7 --- /dev/null +++ b/docs/guides/gcp/q5.md @@ -0,0 +1,167 @@ +# Q5. Compute Engine, Instance Templates, Managed Instance Groups and Autoscaling + +> **Interviewer:** Design a fault-tolerant catalog API on Compute Engine. Explain VM creation, machine families, zones, boot images, service accounts, disks, instance templates, managed instance groups, health checks, autoscaling and rolling replacement. How would you run it during a zonal outage or a Spot VM reclamation? + +## 1. Compute Engine resource layers + +A VM is a zonal compute instance whose definition includes machine type, image/boot disk, network interfaces, metadata, attached service account and disk/storage settings. Creating a VM invokes the Compute Engine API to place it on available infrastructure in the chosen zone; requesting sufficient CPU quota is not proof that the zone has spare capacity for every machine family. + +```mermaid +flowchart TD + IMG["Image / OS family"] --> IT["Instance template"] + CFG["Machine type / disks / network / identity"] --> IT + IT --> MIG["Regional managed instance group"] + MIG --> AZ1["Zone A: VM replica"] + MIG --> AZ2["Zone B: VM replica"] + MIG --> AZ3["Zone C: VM replica"] + HC["Health check / autohealing policy"] --> MIG + AS["Autoscaler"] --> MIG + MIG --> BS["Backend service / traffic"] +``` + +A VM is not a container. Operating system, patching, process supervision, OS Login and runtime bootstrap remain explicit responsibilities. + +## 2. Machine family and disk decisions + +Choose from general-purpose, compute-optimized, memory-optimized, accelerator-optimized and specialized instances using a documented benchmark. Compare vCPU/memory, network bandwidth, disk IOPS/throughput, guest OS support, committed-use discounts and availability by region/zone. + +Storage options differ: +- Persistent Disk / Hyperdisk: managed persistent block options, subject to attachment, performance and redundancy semantics. +- Local SSD: high-performance local data, not a durable source of truth across loss of the VM. +- Cloud Storage: object API, not equivalent to a local POSIX filesystem. +- Regional disks: replication/availability properties differ from ordinary zonal disks and must be evaluated with supported machine/service configuration. + +### Instance disk inventory + +```bash +gcloud compute machine-types list --zones=europe-west1-b --limit=15 +gcloud compute disks list --project=marketplace-sandbox +gcloud compute disks describe DATA_DISK --zone=europe-west1-b --project=marketplace-sandbox +gcloud compute images list --filter='family:debian' --limit=10 +``` + +## 3. Create a disposable private VM + +```bash +gcloud compute instances create catalog-dev --project=marketplace-sandbox --zone=europe-west1-b --machine-type=e2-standard-2 --subnet=app-ew1 --no-address --image-family=debian-12 --image-project=debian-cloud --service-account=VM_RUNTIME_SA_EMAIL --scopes=https://www.googleapis.com/auth/cloud-platform --tags=internal-api +``` + +This presumes networking, IAM, machine availability and OS image support. It creates billable resources. Prefer appropriate OS Login and IAP TCP tunneling instead of assigning a public IP or opening SSH to the world. + +```bash +gcloud compute instances describe catalog-dev --zone=europe-west1-b --project=marketplace-sandbox +gcloud compute ssh catalog-dev --tunnel-through-iap --zone=europe-west1-b --project=marketplace-sandbox +``` + +IAP access requires IAM and network policy, including allowed IAP proxy source ranges where appropriate. The VM service account is for workload APIs and should have narrow roles; `cloud-platform` OAuth scope does not grant permissions beyond IAM. + +## 4. Instance templates versus mutable VMs + +Instance templates describe the configuration from which MIG instances are created. Treat them as effectively immutable versions. A change to template configuration normally means creating a new template and rolling the group forward, not hand-editing each instance. + +Example: + +```bash +gcloud compute instance-templates create catalog-template-v2 --project=marketplace-sandbox --machine-type=e2-standard-2 --subnet=app-ew1 --no-address --image-family=debian-12 --image-project=debian-cloud --service-account=VM_RUNTIME_SA_EMAIL --scopes=https://www.googleapis.com/auth/cloud-platform --metadata-from-file=startup-script=./startup.sh +``` + +Review flags against the actual gcloud version and whether the template includes a full subnet URL in a Shared VPC. Avoid placing long-lived secrets in metadata or startup scripts. Use Secret Manager with authorized access. + +## 5. Managed instance groups + +**Zonal MIG:** replicas in one zone. **Regional MIG:** places managed VMs across multiple zones in a region and supports broader failure-domain resilience. + +A MIG can manage homogeneous instances from templates, health-based repair, rolling updates and autoscaling. An unmanaged instance group provides grouping for externally managed VMs but not the same automatic lifecycle features. Google Cloud documentation describes both. . + +```mermaid +sequenceDiagram + participant OP as Engineer/CI + participant MIG as Managed instance group + participant CP as Compute Engine control plane + participant VM as Replacement VM + participant LB as Load balancer + OP->>MIG: Update instance template + MIG->>CP: Replace instances in configured waves + CP->>VM: Boot + metadata + startup + VM-->>MIG: Healthy under checks + MIG->>LB: Backend membership/health converges + MIG->>CP: Continue update of old replicas +``` + +The exact update controller, health-check policies and LB membership propagation are asynchronous. Template rollout progress isn't proof an application business request succeeds. + +## 6. Autoscaling and autohealing are different + +**Autoscaling:** adjusts replica count based on supported signals, such as CPU, LB serving capacity, Cloud Monitoring metrics or schedules. **Autohealing:** recreates VMs that fail the configured health check. + +An application can be alive but deadlocked, so health checks should reflect real progress without amplifying shared dependency outages. A database outage that causes every node's autohealing probe to fail can cause a fleet-wide restart storm. + +```bash +gcloud compute instance-groups managed list --project=marketplace-sandbox +gcloud compute instance-groups managed describe catalog-mig --region=europe-west1 --project=marketplace-sandbox +gcloud compute instance-groups managed list-instances catalog-mig --region=europe-west1 --project=marketplace-sandbox +gcloud compute instance-groups managed list-errors catalog-mig --region=europe-west1 --project=marketplace-sandbox +``` + +Regional versus zonal flags must match the real MIG. Quotas, firewall rules, instance-template errors and lack of zone capacity can prevent autoscaler fulfillment. + +## 7. Rolling updates and surge capacity + +A deployment should account for: +1. New VM boot and application warmup. +2. Health check passing. +3. LB backend registration. +4. Draining old connections. +5. Database compatibility between versions. +6. Maximum unavailable and surge capacity. +7. Rollback when a template has a defect. + +```mermaid +flowchart TD + N["New template revision"] --> CAN["Deploy limited canary"] + CAN --> H{"Health + synthetic transactions OK?"} + H -->|No| RB["Stop rollout / restore prior template"] + H -->|Yes| WAVES["Advance rolling waves"] + WAVES --> VERIFY["Verify SLO and capacity after replacement"] +``` + +A safe rollout depends on the actual MIG update policy, health behavior and LB draining, not simply choosing “rolling.” + +## 8. Spot VM capacity + +Spot VMs are discounted, interruptible capacity. They can be preempted, subject to current Compute Engine behavior, with stop/delete termination action and limited advance notice. Use for fault-tolerant stateless serving with baseline capacity, batch processing, or checkpointed jobs. They are not suitable for a single unrecoverable leader or sole persistent writer. + +Compare with AWS Spot instances conceptually, but do not assume identical pricing, interruption signals or capacity pools. + +## 9. Troubleshooting case: checkout VMs never become healthy + +Hypothetical evidence: +- Regional MIG desired=9, healthy=3. +- LB backend health fails for new version. +- VM systemd logs show app binding only to 127.0.0.1:8080; backend health reaches VM NIC address. + +Evidence path: +1. Confirm the instance template and startup script actually deployed. +2. Inspect guest application listener and service logs. +3. Check health-check protocol/port/path and firewall source. +4. Inspect LB backend health separately from MIG health. +5. Correct application bind address or health config. +6. Canary new image; verify real checkout requests, then proceed. + +```bash +gcloud compute instances get-serial-port-output VM_NAME --zone=europe-west1-b --project=marketplace-sandbox +gcloud compute operations list --project=marketplace-sandbox --limit=20 +sudo systemctl status catalog-api +ss -lntp +curl -v http://127.0.0.1:8080/health +``` + +The final three commands run inside the VM. Avoid using `curl localhost` alone to prove remote network reachability. + +## Spoken answer + +> “I'd choose a regional managed instance group for a stateless API that must survive a zone failure, with versioned instance templates, health-based autohealing and autoscaling. I separate machine/OS provisioning from application readiness and load-balancer health. For rollouts, I'd stage replacement waves and validate end-to-end traffic and dependency compatibility. If VMs don't scale, I'd distinguish quota from physical zonal capacity and check the template and health checks. Spot can lower costs for interruptible workloads, but baseline capacity and checkpointing must be deliberate.” + +**Further reading:** [Compute Engine](https://cloud.google.com/compute/docs) · [MIGs](https://cloud.google.com/compute/docs/instance-groups) · [Autoscaling](https://cloud.google.com/compute/docs/autoscaler) · [Spot VMs](https://cloud.google.com/compute/docs/instances/spot) + +[Previous](./q4) · [Next: Storage, data and artifacts](./q6) diff --git a/docs/guides/gcp/q6.md b/docs/guides/gcp/q6.md new file mode 100644 index 0000000..08a84e9 --- /dev/null +++ b/docs/guides/gcp/q6.md @@ -0,0 +1,169 @@ +# Q6. Cloud Storage, Artifact Registry, Databases and Asynchronous Services + +> **Interviewer:** Design how a marketplace stores catalog images, application artifacts, transactional orders, event processing and analytical data in Google Cloud. Compare Cloud Storage, Persistent Disk, Artifact Registry, Cloud SQL, Spanner, Firestore, Bigtable, BigQuery, Memorystore and Pub/Sub. What guarantees do they provide and what do they not? + +## 1. Select by data model, not product popularity + +| Requirement | Typical Google Cloud service | Main question | +| --- | --- | --- | +| Images, exports, immutable blobs | Cloud Storage | Object durability, location, lifecycle, access | +| Container images and packages | Artifact Registry | Build provenance, IAM, scanning, regional dependency | +| PostgreSQL/MySQL transactional app | Cloud SQL | HA, backups, connection limits, PITR | +| Relational transactions at significant global scale | Spanner | Data model, consistency, cost, topology | +| Document/key-value app data | Firestore | Query shape, indexes, transactional boundaries | +| High-throughput wide-column/key access | Bigtable | Key design, access latency, column-family model | +| Analytical warehouse | BigQuery | Columnar analytics, ingestion and query cost | +| Cache / low-latency ephemeral data | Memorystore | Cache invalidation and authoritative storage | +| Event transport / decoupled workers | Pub/Sub | At-least-once delivery, ordering options, idempotency | + +The choices overlap in some use cases. Model transaction invariants, scale, RPO/RTO, query patterns and operational overhead first. + +## 2. Cloud Storage fundamentals + +Cloud Storage is an object storage system: **bucket + object name + generation** identify relevant object versions. It is not automatically a POSIX filesystem. Object operations offer strong consistency guarantees under documented semantics, but don't assume atomic multi-object transactions or traditional rename operations. + +```mermaid +flowchart TD + C["Client or workload"] --> IAM["IAM / signed URL / credential check"] + IAM --> B["Bucket: region / dual-region / multi-region"] + B --> O1["Object: catalog/photo1.jpg"] + B --> O2["Object: exports/2026/report.csv"] + B --> P["Retention, lifecycle, versioning, soft delete"] +``` + +### Location and storage classes + +A bucket's location influences data placement, redundancy, latency and egress. Storage classes include Standard, Nearline, Coldline and Archive, each with different access cost and minimum storage durations. Additional newer classes such as Rapid can have specialized scope and availability; inspect the current pricing and product documentation before applying the older four-class model universally. + +Nearline/Coldline/Archive have retrieval and minimum-duration economics. A bulk reclassification may incur operation/retrieval/early-deletion fees. Cloud Storage Autoclass can optimize class transitions for supported buckets. + +### Secure bucket example + +```bash +gcloud storage buckets create gs://EXAMPLE_CATALOG_BUCKET \ + --location=europe-west1 \ + --uniform-bucket-level-access \ + --public-access-prevention \ + --project=marketplace-sandbox +gcloud storage buckets describe gs://EXAMPLE_CATALOG_BUCKET +gcloud storage cp ./example.jpg gs://EXAMPLE_CATALOG_BUCKET/demo/example.jpg +gcloud storage ls gs://EXAMPLE_CATALOG_BUCKET/demo/ +``` + +The bucket name must be globally unique and owned by the sandbox project. Cloud Storage public access prevention and uniform bucket-level access are distinct controls. IAM role `roles/storage.objectViewer` can be scoped to the bucket for application reads. + +### Soft delete, retention and object versioning + +Cloud Storage soft delete can retain deleted/overwritten objects for recovery; eligible buckets are generally created with default soft-delete retention, but actual policy and duration are configurable. Object Versioning and retention locks serve different objectives. A backup plan should specify whether accidental overwrite, malicious delete, cryptographic-key loss or regional disaster is in scope. + +```bash +gcloud storage objects describe gs://EXAMPLE_CATALOG_BUCKET/demo/example.jpg +gcloud storage buckets describe gs://EXAMPLE_CATALOG_BUCKET \ + --format=json +``` + +Don't apply lifecycle delete rules to legal/financial data without retention approval. + +## 3. Artifact Registry + +Artifact Registry stores container images and other supported build artifacts in named repositories. A GKE workload pulls `europe-west1-docker.pkg.dev/PROJECT_ID/apps/catalog-api:VERSION`. Image reference location, IAM and network reachability all affect pull success. + +```mermaid +sequenceDiagram + participant CI as Build pipeline + participant AR as Artifact Registry + participant G as GKE node/runtime + CI->>AR: Push signed/scanned image + AR-->>CI: Immutable digest + G->>AR: Pull image by digest + AR-->>G: OCI layers + G->>G: Start container +``` + +For reproducibility, deploy by digest rather than mutable tags when feasible. Plan artifact retention/cleanup and billable regional storage/egress. Supply-chain policy may include vulnerability scanning, signing/attestation and controlled registries. GKE nodes require pull permissions on the correct repository or project. + +```bash +gcloud artifacts repositories list --location=europe-west1 --project=marketplace-sandbox +gcloud artifacts docker images list europe-west1-docker.pkg.dev/PROJECT_ID/apps --include-tags +``` + +## 4. Cloud SQL and transaction integrity + +Cloud SQL provides managed MySQL, PostgreSQL and SQL Server variants with service-specific HA/backup capabilities. Managed doesn't mean connection pooling, schema compatibility, index tuning and replica lag disappear. + +```mermaid +flowchart LR + API["Checkout API"] --> POOL["Bounded connection pool"] + POOL --> SQL["Cloud SQL primary"] + SQL --> HA["High-availability standby"] + SQL --> BR["Backups / PITR logs"] + SQL --> RR["Optional read replica"] +``` + +**Important:** HA failover, read replicas and backups solve different problems. A replica can be stale or replicate logical corruption. PITR is for restoring data to an earlier point, while HA is about restoring availability. Design RPO/RTO and recovery tests explicitly. + +```bash +gcloud sql instances list --project=marketplace-sandbox +gcloud sql instances describe orders-db --project=marketplace-sandbox +gcloud sql backups list --instance=orders-db --project=marketplace-sandbox +gcloud sql operations list --instance=orders-db --project=marketplace-sandbox +``` + +Networking options include supported private IP paths, Private Service Connect integrations and authorized Cloud SQL connection mechanisms. Don't assume VPC-native private reachability without configuring relevant connectivity and IAM. + +## 5. Pub/Sub delivery is not a database transaction + +Pub/Sub decouples publishers from subscribers with topics and subscriptions. Delivery guarantees and ordering vary by subscription type and supported feature configuration. Design consumers to handle redelivery, out-of-order events where applicable and poison messages. + +```mermaid +sequenceDiagram + participant A as Order API + participant DB as Transactional DB + participant O as Outbox publisher + participant P as Pub/Sub topic + participant W as Worker + A->>DB: Commit order + outbox event + O->>DB: Read undelivered event + O->>P: Publish message + P-->>O: Publish acknowledgment + P->>W: Deliver message + W->>W: Validate ID / idempotency + W-->>P: Ack after durable processing +``` + +This depicts an outbox pattern, not a built-in Cloud SQL-to-Pub/Sub atomic commit. Publish and business transaction can't be assumed atomic merely because both calls succeeded. Use database outbox or another proven integration when avoiding lost events matters. + +A subscriber may acknowledge only after durable side effect. If it crashes after committing business work but before ack, it may process the event again. Use a deduplication key or transactional uniqueness constraint. + +## 6. Spanner, Bigtable, Firestore and BigQuery + +- **Spanner:** relational SQL with transactional semantics and distributed architecture; carefully evaluate interregion latency, schema and pricing. +- **Bigtable:** wide-column low-latency large-scale key access. Row-key distribution is critical; poor sequential hot key design can create hotspotting. +- **Firestore:** document data with query/index design considerations and transactional constraints; assess index/write costs. +- **BigQuery:** analytical warehouse, not a substitute for low-latency OLTP order processing. Control scan/query cost, partitioning and access. +- **Memorystore:** an in-memory data service; don't treat a cache as source of truth unless the application explicitly tolerates that failure model. + +AWS conceptual analogues include RDS/Aurora, DynamoDB, S3, ECR, Redshift and SQS/SNS, but consistency, indexing, networking and identity semantics vary. + +## 7. Production incident: duplicate charges after queue redelivery + +**Hypothetical:** A worker processes a payment-completed message, commits the ledger operation, crashes before ack, then receives the message again. A retry blindly repeats the side effect. + +**Fix:** Store an idempotency key or unique transaction ID in the transactional system; verify durable processing before acknowledgement. Set bounded retries, dead-letter routing, alerting and replay policy. Exactly-once *delivery* options for some Pub/Sub subscription modes do not automatically make an external payment side effect exactly once. + +**Diagnosis:** +```bash +gcloud pubsub topics list --project=marketplace-sandbox +gcloud pubsub subscriptions list --project=marketplace-sandbox +gcloud logging read 'severity>=ERROR' --project=marketplace-sandbox --limit=50 +``` + +Inspect message IDs, business event IDs, ack deadlines, redelivery rate, worker retry count and database uniqueness constraints. Avoid printing payment payload PII into logs. + +## Spoken answer + +> “I'd separate durable objects, transactional business state, low-latency caches, events and analytics. Cloud Storage suits images and exports, Artifact Registry stores deployable packages, Cloud SQL or another transactional database holds orders, Pub/Sub decouples workers and BigQuery supports analysis. The key design question is each service's consistency and failure guarantee. HA, backups and replicas aren't interchangeable, and message redelivery must not duplicate payments. I'd enforce bucket IAM and retention, immutable image deployment, bounded DB connections, idempotent consumers and tested restores.” + +**Further reading:** [Cloud Storage](https://cloud.google.com/storage/docs/introduction) · [Storage classes](https://cloud.google.com/storage/docs/storage-classes) · [Soft delete](https://cloud.google.com/storage/docs/soft-delete) · [Artifact Registry](https://cloud.google.com/artifact-registry/docs) · [Cloud SQL](https://cloud.google.com/sql/docs) · [Pub/Sub](https://cloud.google.com/pubsub/docs) + +[Previous](./q5) · [Next: Observability](./q7) diff --git a/docs/guides/gcp/q7.md b/docs/guides/gcp/q7.md new file mode 100644 index 0000000..cb167f6 --- /dev/null +++ b/docs/guides/gcp/q7.md @@ -0,0 +1,185 @@ +# Q7. Cloud Monitoring, Logging, Tracing, Audit and Operational Reliability + +> **Interviewer:** A marketplace API's p99 latency rises while Compute Engine CPU remains low. Explain Google Cloud observability, resource metrics, logs, trace propagation, SLOs and audit logs. How would you isolate latency without assuming the infrastructure is healthy merely because VMs are Running? + +## 1. Observability begins with user impact + +A reliable system needs a clear signal of **what the user experiences** and enough correlated telemetry to locate where progress stops. CPU percent, VM Running state and GKE Pod readiness aren't substitutes for a successful checkout or catalog API response. + +Use the RED method at the service level: +- **Rate:** Requests or work units per second. +- **Errors:** Failed or invalid operations, including correctness failures not necessarily reported as 5xx. +- **Duration:** p50/p95/p99 latency and latency distributions. + +Use USE on resources: +- **Utilization:** How busy the resource is. +- **Saturation:** Queues, resource pressure and waiting. +- **Errors:** Drops, throttle and service API errors. + +```mermaid +flowchart TD + U["User request / business SLI"] --> S["Service metrics and traces"] + S --> M["Cloud Monitoring: metrics / dashboards / alerts"] + S --> L["Cloud Logging: structured logs"] + S --> T["Cloud Trace: distributed spans"] + M --> I["Incident investigation"] + L --> I + T --> I + A["Cloud Audit Logs"] --> I + I --> FIX["Root cause + safe mitigation + regression check"] +``` + +## 2. Metrics + +Cloud Monitoring stores time-series data, including platform service metrics and application/custom metrics. Labels help segment by service, region, status and revision. High-cardinality labels such as raw request IDs or customer IDs can create extreme cost and query overhead; keep unique values in traces/logs instead. + +```bash +gcloud monitoring metrics list --project=marketplace-sandbox --filter='metric.type=starts_with("compute.googleapis.com/instance/")' +gcloud monitoring time-series list --project=marketplace-sandbox --filter='metric.type="compute.googleapis.com/instance/cpu/utilization"' --interval-start-time="2026-10-25T00:00:00Z" --interval-end-time="2026-10-25T01:00:00Z" +``` + +The availability/flags of these commands can vary by Cloud SDK release; `gcloud monitoring --help` or Monitoring API/console is authoritative. The interval is intentionally an example, not current incident timing. + +### Histograms and percentiles + +A service with p50 30 ms and p99 2 s has a tail problem. Average latency can hide an overloaded minority of instances or slow downstream calls. + +For reliable percentiles use distributions/histograms (with suitable buckets and aggregation) rather than taking the arithmetic average of per-instance p99s. Percentiles cannot generally be combined by simple averaging. + +## 3. Structured logs and Logs Router + +A useful JSON log includes time, severity, service, revision, trace ID, span ID, request outcome, sanitized error category and latency. Avoid secrets, access tokens, raw payment details and customer PII. + +```json +{ + "severity": "ERROR", + "service": "checkout-api", + "revision": "r42", + "trace_id": "sample-trace-id", + "operation": "submit_order", + "error_type": "dependency_timeout", + "latency_ms": 1250 +} +``` + +Google Cloud Logging supports logs from infrastructure and apps. Logs Router determines how incoming entries reach configured sinks and storage destinations. Sinks can send selected entries to log buckets or supported destinations such as BigQuery, Cloud Storage and Pub/Sub. The destination writer requires the right IAM permission. + +```mermaid +flowchart LR + A["VM / GKE / Cloud service logs"] --> R["Logs Router"] + R --> B["Log bucket + retention"] + R --> C["Filtered BigQuery sink"] + R --> D["Long-term Cloud Storage sink"] + B --> E["Logs Explorer / query"] +``` + +### Logs are not guaranteed complete by merely enabling a service + +Check agent/logging configuration, sampling, exclusion filters, sink permissions, retention and quotas. VPC Flow Logs may be sampled/aggregated. Cloud Audit **Data Access** log availability depends on service, policy and configuration. The absence of a log entry alone is not proof an event never happened. + +```bash +gcloud logging read 'resource.type="gce_instance" AND severity>=ERROR' --project=marketplace-sandbox --limit=30 --format=json +gcloud logging sinks list --project=marketplace-sandbox +gcloud logging buckets list --location=global --project=marketplace-sandbox +``` + +Log bucket locations can differ; inspect relevant region-specific locations too. + +## 4. Distributed tracing + +A request can spend time in: +1. External LB/proxy. +2. Application accept queue and worker scheduling. +3. User authentication/authorization. +4. Database connection acquisition. +5. Query execution. +6. External partner HTTP call. +7. Response serialization and egress. + +```mermaid +sequenceDiagram + participant U as Client + participant API as API handler + participant DB as Database + participant P as Partner + U->>API: Request with trace context + API->>DB: Query (child span) + DB-->>API: 120 ms + API->>P: Downstream HTTPS (child span) + P-->>API: 1400 ms + API-->>U: Response + Note over API: Parent duration includes downstream work +``` + +Propagate trace context using supported OpenTelemetry/W3C headers. Spans should reflect causal relationships, not just unrelated logs with timestamps. Sampling reduces costs, but it can miss rare incidents; align sampling strategy with traffic and high-value error paths. + +## 5. Cloud Audit Logs + +Audit logs help answer **who did what to which resource, when and via which API**. Core categories include Admin Activity, Data Access and System Event, with service-specific coverage. A human console change, Terraform apply and service account API action can produce different audit evidence. + +```bash +gcloud logging read 'logName:"cloudaudit.googleapis.com" AND protoPayload.methodName:"setIamPolicy"' --project=marketplace-sandbox --limit=20 +``` + +Inspect `principalEmail`, service name, method, resource, status and caller context when available. Audit visibility/retention is not identical for every project or service. Protect audit sinks from alteration by workload administrators. + +## 6. SLIs, SLOs and error budgets + +**SLI:** Quantitative measure, e.g., fraction of valid checkout requests completed under 500 ms. + +**SLO:** Objective for that metric over a window, e.g., 99.9% compliant events over rolling 30 days. + +**Error budget:** Allowed fraction of noncompliant events: `1 - objective` under that SLO definition. For an event-count SLI of 99.9%, budget = 0.1% of eligible events. **Do not** mechanically convert that into “minutes of downtime” when the SLI measures request latency or correctness rather than time-window availability. + +**Burn rate:** How quickly the error budget is being spent relative to the steady-state allowance. + +```text +SLO target = 99.9% +Allowed noncompliant fraction = 0.001 +Actual error fraction over short window = 0.0144 +Burn rate = 0.0144 / 0.001 = 14.4x +``` + +For alerting, pair a short window with a longer confirmation window to catch sustained customer-impacting failures while limiting noise. Alert on symptoms/SLO burn, then use resource alerts as diagnostic context. + +## 7. Alerting and incident response + +A strong alert has: +- Service or product impact. +- Threshold and duration, not merely a transient data point. +- Links to runbooks/dashboards. +- Responsible owner and severity. +- No sensitive payloads. +- Sufficient sampling/metric freshness context. + +An overly aggressive page on every high CPU instance creates noise. A checkout success-rate burn alert is closer to customer impact. + +## 8. Production incident: slow checkout with healthy CPU + +**Hypothetical:** +- 99th percentile jumped from 150 ms to 3 seconds. +- API CPU only 30%. +- New connections to Cloud SQL increased after a deployment. +- DB connection pool reaches limit, requests wait before executing queries. + +**Investigation** +1. Confirm RED signals and the affected region/revision. +2. Inspect trace spans to locate request waiting before SQL operations. +3. Compare Cloud SQL connections, lock waits and query latency. +4. Inspect code/config change to pool size and replica counts. +5. Reduce unbounded request concurrency or rollback, preserving transaction idempotency. +6. Verify p99/error budget recovery and correlate with DB sessions. + +Other possible causes include NAT port pressure, LB unhealthy endpoints, IAM token refresh, DNS timeouts and quota throttling. Don't select one without evidence. + +## 9. Observability cost and data governance + +Metrics label cardinality, verbose logging, log retention, cross-region export, tracing sampling and BigQuery scan size affect cost. Centralize restricted audit evidence while allowing teams their own service dashboards. A lead engineer balances diagnostic value with access/privacy and budget rather than blindly collecting every packet/payload. + +## Spoken answer + +> “I begin with a user-facing SLI and RED metrics, then locate the time spent using correlated traces, structured logs and resource metrics. In Google Cloud I'd examine Cloud Monitoring, Cloud Logging, Cloud Trace and Cloud Audit Logs, while remembering that some logs and traces are sampled or filtered. I'd define SLOs that measure business-relevant availability or latency and alert on burn rate, not just VM CPU. During a p99 incident I'd use trace spans to distinguish queueing, database contention and remote network waits, then verify recovery in the same user-facing signals.” + +**Further reading:** [Cloud Monitoring](https://cloud.google.com/monitoring/docs/monitoring-overview) · [Cloud Logging](https://cloud.google.com/logging/docs/overview) · [Cloud Trace](https://cloud.google.com/trace/docs/overview) · [Audit Logs](https://cloud.google.com/logging/docs/audit) · [SLOs](https://cloud.google.com/stackdriver/docs/solutions/slo-monitoring) + +[Previous](./q6) · [Next: architecture and cost](./q8) diff --git a/docs/guides/gcp/q8.md b/docs/guides/gcp/q8.md new file mode 100644 index 0000000..ccb4e93 --- /dev/null +++ b/docs/guides/gcp/q8.md @@ -0,0 +1,195 @@ +# Q8. Senior GCP Architecture, Security, Costs and Production Failure Modes + +> **Interviewer:** Design a secure, highly available marketplace platform on Google Cloud. Defend project boundaries, networking, identity, regional resilience, database selection, scale-out, backups, cost trade-offs and operational controls. Then explain how you'd lead a regional incident without making unsafe recovery assumptions. + +## 1. Begin with constraints + +Architecture should follow concrete requirements: +- Maximum acceptable downtime, RTO and RPO. +- Expected baseline/peak requests and concurrent connections. +- User regions and latency objective. +- Data residency and security classification. +- Read/write transaction consistency. +- Migration and operational team capacity. +- Cost ceiling and recovery funding. +- Expected failure domains: VM, zone, region, dependency, identity and operator error. + +An attractive diagram is not a substitute for a verifiable SLO and failure model. + +## 2. A plausible marketplace foundation + +```mermaid +flowchart TD + U["Global users"] --> DNS["Cloud DNS public zone"] + DNS --> LB["Global external Application Load Balancer"] + WAF["Cloud Armor policy"] -.-> LB + LB --> B1["Regional application backend: primary"] + LB --> B2["Regional application backend: secondary"] + B1 --> DB["Transactional managed database"] + B1 --> PUB["Pub/Sub / durable event processing"] + PUB --> WK["Worker fleet"] + B1 --> GCS["Cloud Storage catalog objects"] + B1 --> SEC["Workload identity / Secret Manager"] + B1 --> OBS["Monitoring / Logging / Trace"] + IAM["Organization / folder IAM & policies"] -.-> B1 + NET["Shared VPC / firewall / Cloud NAT"] -.-> B1 +``` + +**Hypothetical reference model, not verified production infrastructure.** The secondary backend doesn't imply transactional cross-region failover is solved. A DB's actual replication, promotability and application consistency must be designed separately. + +### Project boundaries + +- Network host project: global VPC, regional subnet ranges, Shared VPC, firewall governance and hybrid connectivity. +- Application service projects: deployment/compute, runtime IAM, quotas. +- Data project: database/storage and narrow network access. +- Observability/security projects: protected audit logs, sinks and governance. +- Nonproduction projects: quota, cost and IAM separated from production. + +Avoid indiscriminate folder-wide `roles/editor` access. Organization Policy, IAM deny, resource-level bindings and private networking each solve different problems. + +## 3. Routing and traffic security + +Request path: + +```mermaid +sequenceDiagram + participant U as Browser + participant F as External HTTPS LB + participant APP as Regional backend + participant DB as Transactional DB + participant NAT as Public NAT + participant API as External partner + U->>F: DNS then TLS/HTTPS + F->>APP: HTTP(S) backend routing + APP->>DB: Transactional read/write + DB-->>APP: Committed/failed result + APP->>NAT: Outbound partner call + NAT->>API: Translated TCP connection + API-->>NAT: Response + NAT-->>APP: Reverse translation + APP-->>F: Application response + F-->>U: HTTPS response +``` + +Cloud NAT only applies to eligible egress flows; internal DB traffic does **not** traverse public NAT. LB backends may be MIGs or NEGs and the precise proxy/data path depends on LB class. Use HTTPS or mTLS between components when required by the threat model; VPC placement alone isn't end-to-end authorization. + +## 4. Availability and failure domains + +| Layer | Failure | Design lever | Residual risk | +| --- | --- | --- | --- | +| Single VM | OS hang/disk failure | Autohealing + multiple replicas | Faulty health check restarts fleet | +| Zone | Zone unavailable | Regional MIG or regional GKE/node spread | Stateful zonal disk attachment | +| Region | Widespread service outage | Secondary region + planned failover | Data lag, DNS/LB decision, provider dependency | +| Database | Primary outage/corruption | HA + backups/PITR + tested restore | Shared bad writes/replicated corruption | +| Identity | IAM change/metadata outage | Least privilege, audit, recovery access | Shared org policy change | +| External provider | Partner timeout | Bounded retries, circuit breaker, idempotency | Uncertain transaction result | +| Deployment | Broken release | Canary + rollback + schema compatibility | Already committed incompatible data | + +Avoid claiming multi-zone deployment equals multi-region disaster recovery. Likewise a multi-region HTTPS frontend doesn't automatically provide a writable secondary transactional database. + +## 5. Scaling as a latency pipeline + +From load increase to ready capacity: + +```text +Metric production + -> Autoscaler decision + -> Cloud quota/capacity selection + -> VM/GKE node creation + -> OS/container bootstrap + -> Application warmup + -> Health check success + -> LB registration + -> Requests served +``` + +Predictable promotional or auction peaks often justify scheduled pre-scaling and baseline headroom. Reactive autoscaling cannot retroactively remove startup and quota delays. + +### Data and queue scaling + +Do not increase application replicas without considering database connection limits, Pub/Sub processing capacity and external API quotas. The service may become faster at accepting work but slower or less correct at finishing it. Bound concurrency, use backpressure and preserve business idempotency. + +## 6. Regional disaster recovery + +Write down: +- Recovery point objective (RPO): acceptable durable data loss. +- Recovery time objective (RTO): acceptable time to restore service. +- Data source of truth and replication direction. +- Cross-region restore locations, keys, permissions and dependencies. +- Failover trigger/approval and split-brain prevention. +- Planned failback and consistency verification. +- Disaster recovery drills with measured recovery times. + +A secondary region isn't ready merely because Terraform resources exist there. It must have compatible application release, secrets, IAM, network access, data copy and tested traffic switching. + +## 7. Cost architecture + +The meaningful total cost includes: +- Compute hours, machine families and Spot/committed-use strategy. +- Storage GB, operations, retrieval and soft-delete retention. +- Inter-region, interzone and internet egress. +- Cloud NAT gateway/data processing, public IPs and LB resources. +- Cloud SQL/Spanner capacity and backups. +- Logging/Monitoring labels, metrics retention and traced events. +- Unused reserved capacity, duplicate staging environments and snapshots. + +Rightsizing shouldn't eliminate resilience headroom. A smaller regional footprint may save money until one zonal event causes hours of downtime. + +### Guardrails + +Use budgets and alerts, resource labels/tags, quotas, scoped project ownership, infrastructure plan cost checks, scheduled temporary environment cleanup and monthly service-level chargeback. Cloud billing labels aren't guaranteed to capture every cost dimension; reconcile with billing export. + +```bash +gcloud billing accounts list +gcloud billing projects describe marketplace-sandbox +gcloud compute instances list --project=marketplace-sandbox +gcloud compute disks list --project=marketplace-sandbox +gcloud storage buckets list --project=marketplace-sandbox +gcloud services list --enabled --project=marketplace-sandbox +``` + +These are inventory commands; identity permissions determine visibility. + +## 8. Incident: public API fails only in one region + +**Hypothetical:** A deployment succeeds but 30% of global users experience 503s. Secondary regional backends are healthy; primary region experiences unhealthy endpoints after a node pool change. + +**Senior incident sequence** +1. Classify business impact: checkout/bidding errors, data correctness, affected regions. +2. Freeze risky deployments and identify the rollout revision. +3. Inspect Cloud LB backend health by region, forwarding rule, route and policy. +4. Determine whether failure is at application, health check, NEGs/MIGs, IAM, network or underlying capacity. +5. Verify DB write path and whether secondary region can truly serve writes safely. +6. Shift traffic only when data consistency, capacity and service dependency readiness are validated. A blind failover can make an outage a data incident. +7. Validate error rate, p99 and transaction outcomes. +8. Record timeline, cause, contributing conditions and corrective controls. + +```bash +gcloud compute backend-services list --project=marketplace-sandbox +gcloud compute url-maps list --project=marketplace-sandbox +gcloud compute health-checks list --project=marketplace-sandbox +gcloud logging read 'severity>=ERROR' --project=marketplace-sandbox --limit=50 +gcloud compute operations list --project=marketplace-sandbox --limit=25 +``` + +Use `--global` or `--region` on appropriate backend subcommands; different LB classes have different APIs. + +## 9. A lead-level answer must include rejected alternatives + +Consider and explain: +- **GKE versus MIGs:** Container orchestration/self-service versus simpler VM lifecycle, team capabilities and workload shape. +- **Cloud SQL versus Spanner:** Ordinary relational application needs versus distributed scale/availability and cost. +- **Regional only versus multi-region:** Budget and simplicity versus RTO/RPO requirements. +- **Shared VPC versus independent VPCs:** Central governance versus fault/isolation boundaries. +- **Spot versus standard capacity:** Cost versus reclamation and startup risk. +- **Centralized policy versus team autonomy:** Consistency/security versus workflow friction. + +Choose based on required outcomes and measured constraints, not cloud-fashion preferences. + +## Spoken answer + +> “I'd begin with SLO, RTO/RPO, transaction semantics, peak load and data residency. I'd use a governed resource hierarchy with separate network, service and data ownership, narrow IAM and workload identity, and an ingress/load-balancing design matched to the application's traffic. Regional compute distribution protects against zones; a region-level recovery requires a separate plan for data, secrets, dependencies and cutover. I'd favor managed services when they reduce operational risk without violating requirements, use quota and cost headroom, and exercise failure recovery. During an incident I'd correlate load-balancer health, service metrics and application transactions before making a potentially unsafe region switch.” + +**Further reading:** [Google Cloud architecture framework](https://cloud.google.com/architecture/framework) · [Well-Architected reliability](https://cloud.google.com/architecture/framework/reliability) · [Cost optimization](https://cloud.google.com/architecture/framework/cost-optimization) · [Security foundations](https://cloud.google.com/architecture/security-foundations) + +[Previous](./q7) · [Module index](./) · [Challenge questions](./follow-ups) diff --git a/docs/guides/gke/follow-ups.md b/docs/guides/gke/follow-ups.md new file mode 100644 index 0000000..da850d3 --- /dev/null +++ b/docs/guides/gke/follow-ups.md @@ -0,0 +1,254 @@ +# GKE Architecture and Operations: 40 Difficult Follow-ups + +This interview bank accompanies all eight chapters of Module 6. Each answer emphasizes the actual Kubernetes/GKE guarantee, where the guarantee ends, how the mechanism works and what evidence would establish a production root cause. + +::: info Version scope +GKE APIs, release channels, ComputeClass support, load-balancer behavior and storage/networking features change over time. Answers reflect general platform principles and documented product behavior. Verify your deployed GKE version and cluster settings before operational use. +::: + +## 1. If the GKE API server is unavailable, do already running Pods immediately stop serving? + +**Topic:** Architecture + +**Model answer:** Not automatically. Processes and previously programmed data-plane routing can continue if nodes and dependencies remain healthy. New scheduling, controller reconciliation and status updates can fail while the API is unavailable; API-dependent applications may also be affected. Prolonged control-plane unavailability weakens recovery as nodes or Pods fail. I'd test real user requests and monitor node/control-plane connectivity rather than conflate API health with immediate application health. + +## 2. Why does successful kubectl apply not imply successful application delivery? + +**Topic:** Architecture + +**Model answer:** Apply success establishes that the API accepted the desired object, after appropriate authentication, authorization and admission. Deployment/ReplicaSet controllers must still create Pods, scheduler must bind them, kubelet must pull images and start containers, and readiness plus cloud backend health must converge. I'd check status/conditions at each stage and then execute a synthetic request through the real load balancer. + +## 3. Can you run Autopilot-managed workloads inside a Standard GKE cluster? + +**Topic:** Architecture + +**Model answer:** In supported GKE versions, yes: ComputeClasses can select Autopilot mode workloads in Standard clusters, with GKE-managed node pools for those workloads alongside Standard-managed capacity. The exact supported version and workload constraints matter. I would not describe Standard versus Autopilot as a rigid workload-level binary and would check the feature matrix before assuming privileged/node-specific apps are supported. + +## 4. Does a regional GKE cluster automatically put each Deployment replica in a different zone? + +**Topic:** Architecture + +**Model answer:** No. A regional control-plane configuration and node placement do not themselves guarantee a particular Deployment's zone spread. Scheduler behavior depends on eligible nodes, topology spread constraints, affinity, taints and available capacity. For critical replicas I'd inspect actual Pod zone labels, configure an appropriate spread policy and verify behavior when one zone is unavailable, while acknowledging a hard spread can block recovery. + +## 5. How do GKE DNS-based control-plane endpoints differ from IP-based endpoints? + +**Topic:** Architecture + +**Model answer:** The DNS-based endpoint is a GKE API access mode integrated with Google Cloud IAM and supported service perimeter controls, while IP-based endpoints expose separate private/public reachability and authorized-network settings. A private-node cluster may still have a public API endpoint depending on configuration. I'd inspect cluster network isolation settings and current client authentication before claiming the API can only be accessed from within a VPC. + +## 6. Pods remain Pending on nodes with 30% CPU utilization. How can that be correct? + +**Topic:** Node pools + +**Model answer:** The scheduler primarily uses requested resources against node allocatable capacity, not current CPU utilization. Existing requests, DaemonSet overhead and reservations may consume schedulable CPU despite idle cores. Hard taints, node affinity, topology and volume zones can also block placement. I'd inspect FailedScheduling events, node Allocated resources and the workload's requests before changing the HPA. + +## 7. Does a toleration direct a workload toward a tainted node pool? + +**Topic:** Node pools + +**Model answer:** No. A toleration allows a Pod to be scheduled despite a matching taint; it does not make that node preferable. A positive selector or node affinity directs placement. Without one, the Pod could land on any other feasible eligible node. For specialized pools I'd combine taints/tolerations with appropriate affinity and document autoscaler implications. + +## 8. Can hard topology spread reduce availability during a zone outage? + +**Topic:** Node pools + +**Model answer:** Yes. With DoNotSchedule and strict maxSkew, Pods may remain Pending if eligible topology domains lack capacity or a zone is unavailable. That can preserve distribution at the cost of scaling/rollout availability. I'd evaluate ScheduleAnyway, capacity headroom and failure-domain design based on SLO; a schedule constraint should not be blindly tightened without a zone-loss test. + +## 9. Why does GKE node autoscaling fail even when HPA adds replicas? + +**Topic:** Node pools + +**Model answer:** HPA controls Pod desired count, not physical nodes. GKE must find a provisionable node shape meeting CPU, memory, taints, affinity, volumes and topology within node pool bounds, quotas, node IP/Pod IP space and actual zonal capacity. I would inspect autoscaler events and Compute Engine operations to identify the first unsatisfied constraint, not simply raise max node count. + +## 10. Will a PodDisruptionBudget prevent a Spot VM reclamation? + +**Topic:** Node pools + +**Model answer:** No. A PDB governs relevant voluntary evictions via the Eviction API. Spot capacity loss and sudden node failure are involuntary disruptions that can still remove Pods. For Spot workloads I'd use replication, backpressure, retry/idempotency, safe checkpoints and sufficient non-Spot baseline capacity as appropriate, while testing interruption recovery. + +## 11. Why can a /16 Pod range be exhausted long before 65,536 Pods are running? + +**Topic:** Networking + +**Model answer:** GKE VPC-native clusters allocate blocks of Pod IP addresses to nodes according to configured max Pods per node. For example, a common /24 allocation per node turns a /16 into at most 256 such node allocations, not a free pool assigned one IP at a time. Primary subnet capacity and GKE limits can reduce capacity further. I'd calculate using deployed max-pods settings and inspect subnet allocations. + +## 12. In GKE Dataplane V2, why might you not find kube-proxy iptables rules? + +**Topic:** Networking + +**Model answer:** Dataplane V2 uses a Cilium-based eBPF network data path, with anetd programming service routing and NetworkPolicy. kube-proxy is not used for Service routing there. Troubleshooting should inspect the enabled dataplane, Kubernetes Service/EndpointSlice state, relevant anetd health/logging and supported network-policy telemetry rather than conclude missing kube-proxy rules are the fault. + +## 13. Does a node query EndpointSlices on every packet sent to a ClusterIP? + +**Topic:** Networking + +**Model answer:** No. Controllers/agents observe API objects and program a local service endpoint mapping or kernel data-plane state ahead of packet arrival. A connection to ClusterIP is translated/forwarded to a selected Pod backend according to the node dataplane. The API server is not synchronously consulted for every SYN. If routing fails, inspect endpoint selection and dataplane convergence. + +## 14. Why can a Pod-to-Pod connection fail despite an allowed VPC firewall rule? + +**Topic:** Networking + +**Model answer:** Kubernetes NetworkPolicy can deny the traffic, the destination app might not listen, DNS can resolve an unexpected target, or CNI/eBPF routing can be broken. Conversely an allowed NetworkPolicy doesn't override a blocking VPC firewall. I'd check source/destination Pod IP, routing, both policy layers and app listener, correlating network-policy logs and packet evidence where permitted. + +## 15. A Pod can call Cloud Storage but not a third-party HTTPS API. What does that suggest? + +**Topic:** Networking + +**Model answer:** Supported Google API access can work through Private Google Access or another private endpoint path without providing general public internet egress. The third-party call may require eligible Cloud NAT, default route and egress firewall permission. I'd check actual Pod or node SNAT source, NAT subnet coverage/port metrics, destination DNS and TCP/TLS stage before blaming the Google IAM credential. + +## 16. Why do GKE-managed L7 NEG backends differ from a traditional NodePort load-balancing path? + +**Topic:** Ingress and LB + +**Model answer:** GCE_VM_IP_PORT NEGs can register Pod IP:port endpoints directly in Cloud backend services, so the Google L7 load balancer can target Pods without necessarily sending each request through node NodePort and kube-proxy. The GKE controller configures LB resources from Kubernetes objects. A Service type LoadBalancer uses a different Network LB model and should not be assumed to share that same data path. + +## 17. What does cloud.google.com/load-balancer-neg-ready represent? + +**Topic:** Ingress and LB + +**Model answer:** For supported GKE Ingress with container-native load balancing, the controller adds a readiness gate so Pod readiness can consider backend load-balancer endpoint health, not just the kubelet's application probe. A locally healthy Pod can remain not Ready if the LB health check fails. This can intentionally stall a rollout and avoid routing new traffic to unhealthy endpoints. Support differs by controller/configuration. + +## 18. A Pod returns 200 on localhost /ready but the global HTTPS LB reports 502. What next? + +**Topic:** Ingress and LB + +**Model answer:** I'd inspect the URL map/backend service chosen by host and path, NEG membership and endpoint IP:port, health-check path/port/protocol, firewall for health-check sources and backend TLS behavior. A localhost probe does not prove a remote listener or correct LB routing. I would compare Google backend health with Pod conditions and examine actual request logs before changing probes. + +## 19. How do Gateway and HTTPRoute differ from Ingress? + +**Topic:** Ingress and LB + +**Model answer:** Ingress is an older HTTP(S) host/path abstraction with controller-specific annotations. Gateway API separates an infrastructure listener/Gateway from application routes and route-attachment permissions, with GatewayClass selecting an implementation such as a particular GKE LB family. Cross-namespace references may require ReferenceGrant. I'd choose based on controller support, security ownership, LB capabilities and migration constraints rather than YAML length. + +## 20. Why doesn't terminationGracePeriodSeconds alone guarantee zero dropped connections? + +**Topic:** Ingress and LB + +**Model answer:** Pod termination hooks, readiness changes, EndpointSlice/NEG removal and LB backend draining propagate asynchronously. The app must stop accepting work and finish in-flight requests, and cloud LB connection-draining/timeout settings must match protocol needs. preStop consumes the grace period and SIGKILL can still end unfinished work. I'd test rolling replacements with live connections and measure actual error rates. + +## 21. Is a Kubernetes ServiceAccount a Google IAM service account? + +**Topic:** Security + +**Model answer:** No. A KSA is a Kubernetes workload identity for Kubernetes API/auth and federation integration. A Google IAM service account is a principal authorized for Google Cloud services. GKE Workload Identity Federation can grant access directly to the KSA principal or allow it to impersonate a Google service account. They remain distinct identities with different RBAC/IAM control planes. + +## 22. When is GKE Workload Identity Federation direct principal access preferable to GSA impersonation? + +**Topic:** Security + +**Model answer:** Direct resource IAM grants to the workload principal avoid an additional Google service account identity and can be simpler when the target service supports them. GSA impersonation may be needed for APIs or integrations that have limitations with direct federated principals. Both require narrow IAM on the intended resource; impersonation requires the correct workload identity user binding and KSA annotation. + +## 23. A GKE Pod gets 403 from Cloud Storage after migration to another node pool. What is your first investigation? + +**Topic:** Security + +**Model answer:** Identify the actual Pod KSA and runtime principal, confirm the new node/pool supports the configured GKE metadata server and workload identity path, inspect bucket IAM on the exact object permission and check IAM conditions/VPC Service Controls. For impersonation verify both IAM binding and KSA annotation. Do not grant Project Owner merely because the Pod moved. + +## 24. Why is namespace-admin a potentially sensitive permission even with narrow IAM on Kubernetes API resources? + +**Topic:** Security + +**Model answer:** A namespace administrator may create Pods, mount Secrets, exec into workloads or select a privileged KSA in the same namespace, yielding effective access beyond read-only object permissions. Admission constraints, ServiceAccount ownership, Pod security, secrets access and cloud identity bindings must be analyzed together. Namespace isolation is not automatically a strong multi-tenant trust boundary. + +## 25. How can a validating admission webhook become a production outage? + +**Topic:** Security + +**Model answer:** If an admission webhook is unavailable and its failurePolicy is Fail, matching API operations can be rejected or timed out, preventing new Pods or deployments even while existing workloads continue to run. I'd restore webhook backend health first, inspect matching rules/timeouts and use narrowly scoped emergency policy changes only with security approval. Testing fail-open/fail-closed behavior is part of platform reliability. + +## 26. Why does ReadWriteOnce not guarantee single-Pod access? + +**Topic:** Storage + +**Model answer:** RWO is generally a one-node read-write access-mode constraint; under compatible driver behavior, more than one Pod on that node may use the same volume. ReadWriteOncePod provides stronger single-Pod use where CSI support exists, but database-level fencing and transaction correctness remain separate. Choosing an access mode isn't equivalent to distributed consensus. + +## 27. Why is WaitForFirstConsumer useful for zonal Persistent Disks? + +**Topic:** Storage + +**Model answer:** Immediate binding can create a disk in a zone before the Pod's scheduler has considered all node affinity, spread and resource constraints. WaitForFirstConsumer allows volume provisioning to account for a consuming Pod's placement. It reduces stranded volumes and unschedulable Pods but still depends on available nodes, quota and CSI operations. + +## 28. Does a regional Persistent Disk allow simultaneous writers in two zones? + +**Topic:** Storage + +**Model answer:** Not generally. Regional PD replicates disk data across selected zones for availability, but attachment and writer safety must follow supported semantics. It doesn't implement active-active database transactions or automatically elect a safe writer after failure. Use fencing and database recovery, and treat regional replication as distinct from independent backups. + +## 29. A PVC says Bound yet a Pod is stuck ContainerCreating with Multi-Attach. What does Bound tell you? + +**Topic:** Storage + +**Model answer:** Only that the PVC is bound to a PV/backing volume, not that the volume has successfully attached and mounted to the selected node. Inspect VolumeAttachment, Pod events, disk zone, CSI controller/node logs and old node attachment. Do not force-detach until you know the old writer cannot resume. Otherwise you risk filesystem/database corruption. + +## 30. Are CSI snapshots sufficient for an application-consistent database restore? + +**Topic:** Storage + +**Model answer:** Not automatically. A snapshot may be crash-consistent and omit unflushed application buffers or transactions. A robust database backup includes coordination with WAL/checkpoints or database-aware tooling, encryption/retention, access to keys and tested restoration. A snapshot that exists but cannot be restored within RTO is not adequate disaster recovery. + +## 31. Ten replicas at 80% CPU utilization with HPA target 50% imply what basic count? + +**Topic:** Autoscaling + +**Model answer:** The simplified HPA formula is ceil(current replicas times current metric divided by target metric), so ceil(10×80/50)=16. Actual HPA behavior can adjust for missing metrics, not-yet-ready Pods, tolerance and stabilization policies. I'd verify HPA conditions and reported current/desired values rather than insist every cycle must produce exactly 16. + +## 32. Why can doubling CPU requests make HPA scale down even when traffic is unchanged? + +**Topic:** Autoscaling + +**Model answer:** CPU utilization is usage divided by requests. A Pod using 400m has 80% utilization when requesting 500m but 40% when requesting 1000m. HPA may recommend fewer replicas despite unchanged demand. That is why tuning requests and HPA utilization signals must be coordinated; for some services traffic/concurrency metrics are more appropriate. + +## 33. How can CPU throttling create high p99 with low node CPU utilization? + +**Topic:** Autoscaling + +**Model answer:** A CPU limit typically maps to a cgroup quota. Multi-threaded work can consume its quota early during a scheduling period and then be throttled even while the host has spare cores. Inspect cgroup cpu.stat nr_throttled/throttled_usec and service latency by revision, then benchmark changes to quotas/requests. Adding replicas may help or may amplify dependent service pressure. + +## 34. Can VPA and HPA fight each other? + +**Topic:** Autoscaling + +**Model answer:** Yes. VPA can alter CPU requests and HPA CPU utilization is relative to those requests, so each controller changes the other's feedback signal. GitOps also can fight HPA if it continually restores a fixed replicas field. Define ownership and use VPA recommendation-only or complementary scaling signals where appropriate, then load-test stability. + +## 35. Why can application scale-out worsen Cloud SQL latency? + +**Topic:** Autoscaling + +**Model answer:** More replicas may create more DB connections, concurrent queries and lock contention. The bottleneck can move from application CPU to shared database capacity, worsening p99 despite more Pods. I'd cap per-Pod connections and concurrency, use backpressure, monitor successful transactions and tune scaling against end-to-end dependency capacity. + +## 36. What does enrolling in a GKE release channel change? + +**Topic:** Upgrades and recovery + +**Model answer:** A channel controls the version rollout availability/maturity track and maintenance-related options, but it doesn't provide indefinite version pinning. GKE automatically upgrades clusters under lifecycle/security policy, and available channels/targets vary over time. I'd select based on compatibility and support, configure appropriate maintenance windows/exclusions and test deprecations ahead of production. + +## 37. Can PDB prevent a Deployment rollout from violating its minimum available count? + +**Topic:** Upgrades and recovery + +**Model answer:** A PDB primarily controls certain voluntary Eviction API operations, such as node drains. Deployment rolling updates follow their own strategy and can make Pods unavailable independently of PDB enforcement. PDB also can't prevent sudden node failures. Safe availability needs rollout settings, readiness, capacity, topology and app draining in addition to PDBs. + +## 38. Existing Pods serve after a GKE upgrade, but new Pods fail admission. What is the likely boundary? + +**Topic:** Upgrades and recovery + +**Model answer:** The API and runtime can be healthy while a webhook, policy, CRD schema or removed Kubernetes API version rejects new objects. Inspect API/admission errors, webhook configurations/backends, CRDs and release deprecations before restarting healthy Pods. If Pods were admitted but fail sandbox/container startup, move to kubelet, runtime, image, CNI and CSI. + +## 39. Does a regional GKE cluster guarantee regional disaster recovery? + +**Topic:** Upgrades and recovery + +**Model answer:** No. A regional GKE cluster improves resilience across zones in one region. Multi-region recovery requires separate deployment/cluster capacity, image/secret/identity availability, network routing, data replication/restore and transaction-writer fencing. A global LB can't make a single-region database writable in another region. Test actual RTO/RPO under failure. + +## 40. How would you triage a service outage immediately after node-pool upgrade? + +**Topic:** Upgrades and recovery + +**Model answer:** Start with business SLI, freeze further disruptive changes and compare old/new nodes. Identify first broken stage: admission, scheduling, runtime, image pull, CNI, CSI, probes, NEG/backend health or app dependency. Inspect Pod events, node conditions, GKE operations, workload logs and LB health. Mitigate only the failing boundary, validate real transactions, and document regression checks before resuming rollout. + + +## Practicing effectively + +For each question, try a 60-to-90-second answer, then identify the exact Kubernetes condition, Cloud API object or metric you'd inspect first. Distinguish diagnosis from mitigation and predict the cost or failure mode of your proposed fix. + +[← Module 6](./) · [Hands-on labs](./labs) diff --git a/docs/guides/gke/index.md b/docs/guides/gke/index.md new file mode 100644 index 0000000..d710bbd --- /dev/null +++ b/docs/guides/gke/index.md @@ -0,0 +1,76 @@ +# GKE Architecture, Networking and Production Operations + +**Module 6 · Intermediate to Senior/Lead · Eight complete chapters** + +This guide explores how managed Kubernetes behaves on Google Cloud: managed control plane, Standard and Autopilot operations, node pools, VPC-native Pod routing, eBPF, Gateway/Ingress with NEGs, workload identity, CSI storage, autoscaling, upgrades and recovery. + +It builds on [Kubernetes architecture](../kubernetes/) and [GCP Fundamentals](../gcp/), but emphasizes the Google-specific operational integration rather than repeating generic Kubernetes concepts. + +::: info Examples and version scope +All cloud projects, marketplace workloads, account identifiers and incidents are hypothetical for public engineering education. Snippets are educational and may require existing clusters, namespaces and cloud policies. Exact behavior varies with GKE version, cluster mode, release channel and enabled features. Confirm the target environment and official Google Cloud documentation before applying changes. +::: + +## Prerequisites + +- Kubernetes Deployments, Services, scheduler, etcd/control plane and Pod lifecycle basics. +- Subnetting, CIDR, TCP, DNS and L4/L7 load-balancing knowledge. +- Google Cloud projects, IAM, VPC, Cloud NAT and Compute Engine fundamentals. +- Read-only cluster and GCP inspection access, or a budgeted disposable sandbox. + +## Learning objectives + +After completing the guide, you should be able to: + +- Trace the full managed control-plane-to-Pod lifecycle and identify failure boundaries. +- Choose Standard/Autopilot modes based on actual node, security and workload needs. +- Diagnose scheduling, node auto-provisioning and quota/IP capacity bottlenecks. +- Explain Pod alias IPs, ClusterIP service routing, Dataplane V2 and network policies. +- Trace Google Cloud L7 frontend to NEG Pod endpoint with LB health and readiness gates. +- Configure least-privilege GKE workload identity and separate Google IAM from Kubernetes RBAC. +- Explain CSI provisioning, zonal/regional storage and safe failure recovery. +- Design HPA/VPA/KEDA/node scaling without overloading databases or remote APIs. +- Plan GKE upgrades, observe SLOs and handle cluster or regional recovery safely. + +## Chapters + +1. [GKE architecture, control plane, Standard versus Autopilot](./q1) + End-to-end Kubernetes API/controller/scheduler/kubelet request flow; regional and zonal modes. +2. [Node pools, scheduling and capacity](./q2) + Resource requests, node taints/affinity, topology, ComputeClasses, quotas and autoscaler failure. +3. [VPC-native networking, Services, DNS and eBPF](./q3) + Pod secondary IP sizing, ClusterIP, Dataplane V2, NetworkPolicy, Pod SNAT and Cloud NAT. +4. [Ingress, Gateway API, NEGs and load-balancer health](./q4) + HTTPS request path, Pod-direct routing, backend health, NEG readiness gates and draining. +5. [IAM, workload identity, RBAC and Pod security](./q5) + Direct GKE federation, GSA impersonation, admission, isolation and 403 troubleshooting. +6. [CSI storage, StatefulSets and recoverability](./q6) + PVC/PV/StorageClass, WaitForFirstConsumer, zonal/regional disks, attachment and snapshots. +7. [Autoscaling, resource management and latency](./q7) + HPA math, VPA feedback, throttling, memory pressure and downstream capacity. +8. [GKE upgrades, production incidents and disaster recovery](./q8) + Release channels, node rolling upgrades, PDBs, observability, SLOs and recovery runbooks. + +## Practice + +- [40 challenging technical follow-ups with complete model answers](./follow-ups) +- [Eight labs and tabletop exercises with expected observations and safety notes](./labs) + +## Recommended study sequence + +Study Q1–Q2 for architecture, scheduling and node operations. Q3–Q4 build packet-by-packet intuition from internal Pod routing to external client load balancing. Q5–Q6 cover identities, security and storage, then Q7–Q8 connect autoscaling and upgrades to measurable reliability. Finish by explaining each lab incident aloud before opening the model answers. + +## Verified reference documentation + +- [GKE modes of operation](https://cloud.google.com/kubernetes-engine/docs/concepts/choose-cluster-mode) +- [GKE control plane network isolation](https://cloud.google.com/kubernetes-engine/docs/concepts/network-isolation) +- [GKE cluster autoscaler](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler) +- [VPC-native clusters / Pod CIDRs](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips) +- [GKE Dataplane V2](https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2) +- [GKE load balancing and NEGs](https://cloud.google.com/kubernetes-engine/docs/concepts/about-load-balancing) +- [GKE Gateway API](https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-api) +- [Ingress backend health](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-routing-security) +- [Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) +- [PersistentVolume and CSI](https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes) +- [GKE release channels](https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels) + +[← All guides](/guides/) · [Begin Module 6 Q1 →](./q1) diff --git a/docs/guides/gke/labs.md b/docs/guides/gke/labs.md new file mode 100644 index 0000000..e634d60 --- /dev/null +++ b/docs/guides/gke/labs.md @@ -0,0 +1,207 @@ +# Module 6: Eight GKE Labs and Production Tabletop Exercises + +These exercises accompany the eight chapters. A successful lab means you can explain the mechanism and expected failure evidence, not just execute a command. + +::: warning Safety and cost +Use an approved disposable GCP project/cluster with a budget and teardown plan. GKE cluster nodes, load balancers, regional disks, logging and NAT create charges. The commands below use placeholders and are primarily read-only unless marked. Never deliberately break production ingress, force-detach a disk, disable admission security, expose the API endpoint or drain critical nodes for practice. Autopilot/Standard feature support differs. +::: + +## Lab 1. GKE API, controllers and lifecycle + +**Goal:** Trace an accepted Deployment through ReplicaSet, scheduler, kubelet and readiness. + +```bash +gcloud container clusters list --project=PROJECT_ID +gcloud container clusters get-credentials CLUSTER \ + --location=europe-west1 --project=PROJECT_ID +kubectl config current-context +kubectl get --raw=/readyz +kubectl get deployment,replicaset,pods -A +kubectl -n marketplace get events --sort-by=.lastTimestamp +``` + +In a sandbox namespace, deploy a simple approved HTTP image with `replicas: 2`, requests and readiness probes. Inspect: + +```bash +kubectl -n marketplace describe deployment demo +kubectl -n marketplace get rs,pods -o wide +kubectl -n marketplace describe pod POD +``` + +**Expected:** API acceptance precedes Pod readiness. Deployment owns ReplicaSet, ReplicaSet owns Pod; the scheduler binds Pod to node; kubelet starts it. A successful apply is not proof of a healthy user request. Track `metadata.ownerReferences` and readiness conditions. + +**Challenge:** Explain whether already-running Pods keep serving when Kubernetes API becomes unavailable, and what new scheduling/reconciliation stops. + +## Lab 2. Standard pools, scheduling and failure domains + +**Goal:** Distinguish requests from utilization and hard from soft placement. + +```bash +kubectl get nodes -o wide +kubectl get nodes -L cloud.google.com/gke-nodepool,topology.kubernetes.io/zone +kubectl describe node NODE +kubectl -n marketplace describe pod POD +gcloud container node-pools list --cluster=CLUSTER \ + --location=europe-west1 --project=PROJECT_ID +``` + +In a paper exercise, a Pod requests 3 CPU, a candidate node has 4 vCPU capacity but only 2.5 CPU allocatable free after reservations and existing requests. **Expected:** Scheduler rejects the Pod even if measured host CPU is 15%. + +Try a dry-run manifest with `nodeSelector` that matches no node (in a disposable namespace, only if authorized), then inspect FailedScheduling without creating expensive new capacity: + +```bash +kubectl -n sandbox apply -f scheduling-demo.yaml +kubectl -n sandbox describe pod demo-pod +``` + +The YAML must be constructed for your sandbox; don't force an unschedulable Pod into production. + +**Challenge:** Contrast nodeSelector, taints, topology spread and PriorityClass; explain why node autoscaler might fail to satisfy a hard constraint. + +## Lab 3. Service dataplane, Pod CIDRs and DNS + +**Goal:** Inspect VPC-native subnet ranges and understand virtual ClusterIP routing. + +```bash +gcloud container clusters describe CLUSTER \ + --location=europe-west1 --project=PROJECT_ID +gcloud compute networks subnets describe SUBNET \ + --region=europe-west1 --project=PROJECT_ID +kubectl get nodes -o wide +kubectl -n marketplace get pods -o wide +kubectl -n marketplace get svc,endpointslices,networkpolicy +kubectl -n kube-system get daemonsets +``` + +**Expected:** Nodes use primary subnet addresses, Pods use secondary alias-IP ranges. A Service ClusterIP is a virtual VIP backed by EndpointSlices. In Dataplane V2, `anetd`/eBPF handles Service routing rather than kube-proxy iptables rules. + +**Calculation challenge:** Pod range /17 with node allocation /24 offers at most 128 per-node blocks from that range. Explain why 128 nodes can be the limiting factor long before all addresses are in live Pods. Confirm actual max-Pod settings before applying the formula elsewhere. + +**Network policy challenge:** In a sandbox namespace only, create a default-deny policy, demonstrate DNS/connectivity failures, then add a narrowly scoped DNS egress rule and app peer ingress. Clean up policies afterwards. Don't apply default deny to an existing production namespace. + +## Lab 4. GKE Ingress, Gateway and NEG backend health + +**Goal:** Compare Kubernetes readiness with Google LB backend readiness. + +```bash +kubectl -n marketplace get ingress,gateway,httproute,svc,endpointslices +kubectl -n marketplace describe pod POD +kubectl -n marketplace get pod POD -o jsonpath='{.status.conditions}' +gcloud compute backend-services list --project=PROJECT_ID +gcloud compute network-endpoint-groups list --project=PROJECT_ID +gcloud compute health-checks list --project=PROJECT_ID +``` + +Some CRDs are unavailable if Gateway isn't installed/enabled; a missing resource type isn't proof the cluster is broken. + +```mermaid +flowchart LR + C["Client HTTPS"] --> LB["Google Application LB"] + LB --> BACK["Backend service"] + BACK --> NEG["NEG: Pod IP:port"] + NEG --> POD["Ready app container"] + K["Kubernetes / NEG controller"] -.-> NEG + HC["Cloud health check"] -.-> BACK +``` + +**Expected:** A Pod might pass local readiness but fail an external backend health check due to path, port, TLS or firewall mismatch. On supported Ingress NEGs the readiness gate may remain false and a rollout can stall. + +**Challenge:** Trace a 502 and show why blindly replacing Gateway with NodePort changes the data path and security/operational semantics. Creating new public LBs for a lab costs money and should be avoided unless expressly budgeted. + +## Lab 5. GKE Workload Identity and RBAC + +**Goal:** Prove Google Cloud IAM and Kubernetes RBAC are separate. + +```bash +kubectl -n marketplace get serviceaccount +kubectl -n marketplace get role,rolebinding +kubectl auth can-i get pods -n marketplace +kubectl auth can-i get secrets -n marketplace +gcloud storage buckets get-iam-policy gs://OWNED_BUCKET +``` + +For a **sandbox only**, create a dedicated KSA and grant the exact bucket viewer role to its GKE WIF principal according to the cluster's workload pool. Run an application using a compatible Google Cloud client and observe successful read access to that bucket while unauthorized Kubernetes API operations remain denied. + +**Expected:** Identity path is KSA -> metadata server/federation -> Google IAM authorization. If using GSA impersonation, both the IAM workload identity user binding and KSA annotation matter. Granting bucket access does not imply Kubernetes cluster-admin. + +**Safety:** Don't print tokens, grant Project Owner or give all namespace Pods a shared privileged identity. A `kubectl auth can-i --as` test requires authorized impersonation permissions. + +## Lab 6. CSI, zonal storage and failure recovery + +**Goal:** Inspect the complete PVC -> PV -> disk -> attach/mount lifecycle. + +```bash +kubectl -n marketplace get pvc +kubectl get storageclass +kubectl get pv,volumeattachments +kubectl -n marketplace get events --sort-by=.lastTimestamp +kubectl -n marketplace describe pod STATEFUL_POD +gcloud compute disks list --project=PROJECT_ID +``` + +**Expected:** PVC Bound does not prove disk attachment to the scheduled node. `WaitForFirstConsumer` allows topology-aware provisioning. A zonal PD binds a workload to eligible zone capacity; a regional PD adds two-zone replication but does not grant concurrent safe DB writers. + +**Tabletop fault:** Old database node is unreachable, new Pod logs Multi-Attach. Explain why forcibly detaching may create dual-writer corruption, how to establish fencing, and how to distinguish disk replication from backup. + +**Safety:** Do not create/delete PVCs that contain application data or trigger real multi-attach incidents on shared clusters. + +## Lab 7. Autoscaling, CPU quota and database pressure + +**Goal:** Compute HPA recommendations and reason about node and dependency limits. + +```bash +kubectl -n marketplace get hpa +kubectl -n marketplace describe hpa bidding-api +kubectl -n marketplace top pods +kubectl -n marketplace get events --sort-by=.lastTimestamp +``` + +**Calculation:** 10 current Pods, 80% average CPU utilization relative to request, 50% target -> `ceil(10*80/50)=16` suggested Pods before tolerance and other HPA behavior. + +**Observation:** A Pod using 400m with 500m CPU request is at 80% utilization; change request to 1000m without changing usage and HPA reports 40%. Explain why VPA/HPA CPU feedback can conflict. + +**Tabletop incident:** HPA doubles replicas, Cloud SQL connection limit is reached, p99 worsens. Propose bounded per-Pod pools, backpressure, pre-scaling and request/queue signals while preserving idempotency. + +**Safety:** Prefer dry-run calculations and existing metrics to triggering uncontrolled scale tests. + +## Lab 8. Upgrade readiness and full incident tabletop + +**Goal:** Plan GKE upgrades without treating control-plane success as product success. + +```bash +gcloud container clusters describe CLUSTER --location=europe-west1 --project=PROJECT_ID +gcloud container node-pools list --cluster=CLUSTER \ + --location=europe-west1 --project=PROJECT_ID +gcloud container operations list --location=europe-west1 --project=PROJECT_ID +kubectl get nodes -o wide +kubectl get pdb -A +kubectl get validatingwebhookconfigurations +kubectl get crd +``` + +**Review:** Release channel, version skew, maintenance windows/exclusions, deprecated APIs, add-on/CSI compatibility, PDBs, zone headroom and Pod IP space for surge. + +**Tabletop failure:** +1. Control plane upgrades successfully. +2. Old Pods continue serving. +3. New Pods are rejected by an outdated admission webhook. +4. A second symptom appears: a CSI node plugin fails to mount a zonal disk on new nodes. + +Describe what to check first in each failure path, how to prevent uncontrolled rollouts, and why a blanket control-plane downgrade is not a safe recovery plan. + +**Regional DR extension:** Secondary region cluster exists but database writer remains in primary region. Define RPO/RTO, data replication/restore, credentials, frontend routing, writer fencing and synthetic transaction verification before traffic cutover. + +## Completion review + +- Trace Deployment API acceptance through controllers, scheduler, kubelet and LB eligibility. +- Explain Standard, Autopilot and supported Autopilot workloads on Standard. +- Diagnose FailedScheduling from node allocatable, taints, topology and IP ranges. +- Distinguish kube-proxy routing from Dataplane V2 eBPF. +- Explain NEG direct-to-Pod routing and health/readiness gates. +- Design direct GKE workload identity or permitted GSA impersonation. +- Recover safely from CSI attachment failures and verify backups. +- Compute HPA decisions and explain database-dependent scale limits. +- Upgrade with version/channel policy, headroom and app compatibility. +- Defend regional DR with data correctness and controlled failover. + +[← Module 6 index](./) · [40 challenging follow-ups](./follow-ups) diff --git a/docs/guides/gke/q1.md b/docs/guides/gke/q1.md new file mode 100644 index 0000000..a683163 --- /dev/null +++ b/docs/guides/gke/q1.md @@ -0,0 +1,141 @@ +# Q1. GKE Architecture, Control Plane, Standard and Autopilot + +> **Interviewer:** Starting from a Deployment YAML, trace the request through GKE's managed control plane to the running Pod. Compare Standard and Autopilot, regional versus zonal clusters, API endpoints, ownership and failure modes. + +## First principles: Kubernetes with a cloud operations contract + +GKE runs Kubernetes integrated with Google Compute Engine, IAM, networking, storage, observability and load balancing. The managed control plane runs the API server, etcd-backed state, scheduler and Kubernetes controllers. In Standard clusters, teams have control of node pools and node configuration. In Autopilot, Google manages more of the node resources and applies stronger workload restrictions and defaults. + +**Managed doesn't mean automatically available applications.** Image bugs, invalid readiness, incompatible schema changes, unsuitable quotas, IAM, Pod resource specifications and transaction consistency are still platform/application responsibilities. + +```mermaid +flowchart TD + ENG["Engineer / CI / GitOps"] --> API["Managed Kubernetes API server"] + API --> AUTH["Authenticate, authorize, admit"] + AUTH --> STATE["Control plane desired state"] + STATE --> DC["Deployment controller"] + DC --> RS["ReplicaSet controller"] + RS --> POD["Pod objects"] + POD --> SCHED["Scheduler: node binding"] + SCHED --> KUBE["Node kubelet"] + KUBE --> CRI["Runtime: images, containers"] + KUBE --> NET["CNI / storage setup"] + CRI --> READY["Probes and readiness"] + NET --> READY + READY --> TRAFFIC["Endpoint and LB backend eligibility"] +``` + +Controllers do not run as one synchronous request chain. They observe accepted API objects and reconcile asynchronously. A successful `kubectl apply` is only one stage. + +## End-to-end Deployment lifecycle + +1. The client establishes TLS to its configured GKE Kubernetes API endpoint and authenticates with supported Google identity/cluster credentials. +2. Kubernetes RBAC authorizes the operation. Admission checks applicable policy and mutations. +3. The API server stores the Deployment. Deployment controller manages ReplicaSet; ReplicaSet controller creates Pods. +4. The scheduler evaluates hard resource/topology/taint constraints, scores feasible nodes and binds Pods. +5. Kubelet observes bound Pod specs, prepares sandbox/network/volume resources, pulls images through the runtime and launches containers. +6. Readiness probes and applicable readiness gates determine Pod status. EndpointSlices and cloud controllers asynchronously update routing/backends. +7. Application requests can flow only after the relevant network, load balancer and health-control states converge. + +```mermaid +sequenceDiagram + participant C as Engineer + participant API as GKE API server + participant D as Deployment/ReplicaSet controllers + participant S as Scheduler + participant K as kubelet + participant G as GKE cloud integrations + C->>API: Apply Deployment + API-->>C: Accepted + API-->>D: Desired state observed + D->>API: Create ReplicaSet and Pods + API-->>S: Unschedulable Pod ready for binding + S->>API: Bind to selected node + API-->>K: Node-assigned Pod + K->>G: Prepare network, volumes and image + G-->>K: Prerequisites ready + K->>K: Start containers + run probes + K->>API: Update Pod conditions + Note over API,G: LB backend health reconciles separately +``` + +## Standard, Autopilot and ComputeClasses + +| Topic | Standard | Autopilot | +| --- | --- | --- | +| Node management | User-configured node pools | Google-managed node provisioning | +| Machine/OS tuning | Greater flexibility | Constrained to supported platform model | +| Privileged/host-level workloads | More configurable | More restrictive | +| Workload resource governance | Configurable | Managed defaults and enforcement | +| Capacity planning | Explicit pool autoscaling, optional node auto-provisioning | Workload-led automated capacity | +| Best fit | Specialized runtime/network/agent requirements | Conventional production workloads | + +**Current nuance:** Supported Standard clusters can also host Autopilot-managed workloads using ComputeClasses and provider-managed nodes. Don't describe Standard and Autopilot as mutually exclusive at every workload level. See [GKE modes](https://cloud.google.com/kubernetes-engine/docs/concepts/choose-cluster-mode) and [Autopilot workloads on Standard](https://cloud.google.com/kubernetes-engine/docs/concepts/about-autopilot-mode-standard-clusters). + +Choose Standard for specialized node control, host-level agents, device support, workload constraints or operational settings not available in Autopilot. Choose Autopilot to reduce node lifecycle toil when workloads fit its supported model. Neither automatically fixes bad application resource requests or gives free capacity. + +## Regional and zonal failure domains + +```mermaid +flowchart TD + R["Regional GKE cluster"] --> CP["Managed regional control plane"] + R --> ZA["Zone A worker capacity"] + R --> ZB["Zone B worker capacity"] + R --> ZC["Zone C worker capacity"] + ZA --> A["Pod A"] + ZB --> B["Pod B"] + ZC --> C["Pod C"] + DB["Shared regional database"] -.-> A + DB -.-> B + DB -.-> C +``` + +A regional cluster improves regional control-plane/node placement resilience subject to cluster and node-pool configuration. A multi-zone control plane doesn't guarantee multi-zone application replicas. The scheduler also needs appropriate topology spread, capacity and network/IP availability. Even three replicas across zones can share one failed database. + +## API endpoint/network isolation + +GKE control-plane access can use DNS-based and IP-based endpoints, with supported private/public endpoint settings and authorized networks. A cluster's private nodes do not automatically prove that its control plane accepts only private IP clients. DNS-based endpoints integrate Google IAM and can use VPC Service Controls policy. IP-based endpoints have separate network reachability/allowlisting characteristics. + +```bash +gcloud container clusters list --project=PROJECT_ID +gcloud container clusters describe CLUSTER --location=europe-west1 --project=PROJECT_ID +gcloud container clusters get-credentials CLUSTER --location=europe-west1 --project=PROJECT_ID +kubectl config current-context +kubectl cluster-info +kubectl get --raw=/readyz +kubectl auth can-i create deployments -n marketplace +kubectl get nodes -o wide +``` + +Verify current context before operations. Authentication, Google IAM permission to obtain cluster credentials, Kubernetes RBAC and network reachability are distinct checks. + +## Failure investigation: object accepted but Pods unavailable + +Hypothetical: Deployment `desired=4, available=0`. Classify by lifecycle boundary: + +- No ReplicaSet: deployment controller or malformed/blocked object. +- ReplicaSet but no Pods: ReplicaSet controller, namespace ResourceQuota or admission. +- Pending Pods: scheduling constraints, allocatable, taints, node pools or autoscaler. +- ContainerCreating: CNI/CSI/runtime/image prep. +- CrashLoopBackOff: application/process/probe or config. +- Running, not Ready: readiness probe, startup and NEG readiness gates. +- Ready but 502: Google LB backend health, networking and application failure. + +```bash +kubectl -n marketplace describe deployment catalog-api +kubectl -n marketplace get rs,pods -o wide +kubectl -n marketplace describe pod POD +kubectl -n marketplace get events --sort-by=.lastTimestamp +``` + +## Architecture trade-offs + +Tenants need more than namespaces: RBAC, Pod Security/admission, NetworkPolicy, quotas, dedicated workload identities, and sometimes separate projects/clusters. GKE manages infrastructure components but leaves compatibility, workload security and SLOs to engineering teams. + +## Spoken answer + +> “GKE follows the standard Kubernetes reconciliation model with managed Google Cloud integrations. A client authenticates to the API, RBAC and admission accept the Deployment, controllers create Pod objects, the scheduler selects a node and kubelet starts them through runtime, networking and storage integrations. Readiness and cloud backend health are asynchronous after that. Standard provides deeper node control, while Autopilot manages more node operations and enforces different workload constraints. I'd choose the mode and regional topology from actual application requirements, then troubleshoot from the first broken lifecycle stage rather than restarting the cluster.” + +**References:** [GKE operation modes](https://cloud.google.com/kubernetes-engine/docs/concepts/choose-cluster-mode) · [GKE network isolation](https://cloud.google.com/kubernetes-engine/docs/concepts/network-isolation) · [Kubernetes architecture](https://kubernetes.io/docs/concepts/architecture/) + +[Module index](./) · [Next: node pools](./q2) diff --git a/docs/guides/gke/q2.md b/docs/guides/gke/q2.md new file mode 100644 index 0000000..8586e73 --- /dev/null +++ b/docs/guides/gke/q2.md @@ -0,0 +1,184 @@ +# Q2. GKE Node Pools, Scheduling, Runtime and Capacity + +> **Interviewer:** HPA increases an application from 10 to 60 replicas, yet Pods remain Pending while nodes show low CPU usage. Explain node pools, kubelet/runtime responsibilities, resource requests, taints, affinity, zones, ComputeClasses, autoscaler and quota/capacity limits. + +## 1. Workers, node pools and the scheduler + +In GKE Standard, worker nodes are Compute Engine VMs running kubelet, container runtime and GKE system components. A node pool groups VMs with common configuration such as machine type, OS image, node service account, labels, taints and scaling/upgrade policies. + +Scheduler feasibility is based on **allocatable resources and requests**, not current real-time utilization. A node can have idle CPU but no schedulable requested CPU headroom. + +```mermaid +flowchart TD + DEP["Deployment HPA: desired 60"] --> P["Create Pod objects"] + P --> S["Scheduler filtering"] + S -->|"Eligible capacity"| NODE["Node binding"] + S -->|"Insufficient or incompatible"| WAIT["Pending / FailedScheduling"] + WAIT --> CA["Cluster autoscaler"] + CA --> CHECK{"Node shape, quotas, Pod IPs, zone available?"} + CHECK -->|Yes| NEW["Create suitable Compute Engine capacity"] + CHECK -->|No| BLOCK["Scale-up blocked"] + NEW --> S + NODE --> K["kubelet starts and readies Pod"] +``` + +### Capacity versus usage + +A 4-vCPU node does not offer four full CPUs to application requests if kube/system reservations and DaemonSet allocations consume part of allocatable. `kubectl top nodes` reports utilization, whereas `kubectl describe node` reports allocatable and requested resource commitments. + +```bash +kubectl get nodes -o wide +kubectl describe node NODE +kubectl top nodes +kubectl -n marketplace describe pod POD +kubectl -n marketplace get events --sort-by=.lastTimestamp +``` + +## 2. Hard and soft placement constraints + +| Construct | Scheduling meaning | Trap | +| --- | --- | --- | +| nodeSelector | Required node labels | No matching nodes | +| required nodeAffinity | Hard node constraint | No provisionable eligible shape | +| preferred nodeAffinity | Preference | Not a hard guarantee | +| taint + toleration | Taint repels; toleration permits | Toleration does not attract | +| podAntiAffinity | Separate selected Pods | May block under zonal shortage | +| topologySpreadConstraints | Bound distribution skew | DoNotSchedule can block recovery | +| PriorityClass | Preemption priority | Can't create missing GPU or IPs | + +A high-priority Pod can preempt lower-priority Pods when doing so satisfies hard constraints, but it cannot schedule onto the wrong zone or acquire absent hardware. + +## 3. Multi-zone workload example + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: catalog-api + namespace: marketplace +spec: + replicas: 6 + selector: + matchLabels: { app: catalog-api } + template: + metadata: + labels: { app: catalog-api } + spec: + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: { app: catalog-api } + containers: + - name: api + image: europe-west1-docker.pkg.dev/PROJECT_ID/apps/catalog-api:VERSION + ports: + - containerPort: 8080 + resources: + requests: { cpu: 500m, memory: 512Mi } + limits: { memory: 1Gi } +``` + +This is a **hypothetical manifest**, not an available image. Hard spread can reduce availability if one zone has no capacity. `ScheduleAnyway` is a softer alternative when scale availability is more important than strict skew. + +## 4. Standard node pools and version-dependent options + +```bash +gcloud container node-pools list \ + --cluster=CLUSTER --location=europe-west1 --project=PROJECT_ID + +gcloud container node-pools describe api-pool \ + --cluster=CLUSTER --location=europe-west1 --project=PROJECT_ID + +kubectl get nodes -L cloud.google.com/gke-nodepool,topology.kubernetes.io/zone +``` + +A conceptual sandbox creation command: + +```bash +gcloud container node-pools create api-pool \ + --cluster=CLUSTER --location=europe-west1 --project=PROJECT_ID \ + --machine-type=e2-standard-4 \ + --service-account=NODE_SA_EMAIL \ + --num-nodes=1 \ + --enable-autoscaling --min-nodes=1 --max-nodes=5 +``` + +**Verify min/max scope** for regional pools in your deployed CLI: per-zone node count and total-pool bounds may use different flags/semantics. The node service account must have required IAM permissions. The command incurs cost and should only run in approved sandboxes. + +Some nodes use Container-Optimized OS, others supported Ubuntu images. Host package availability and runtime debugging differ. Autopilot restricts direct node access. Node upgrades may change kernel, cgroups, runtime, logging and CSI behavior even with no application manifest change. + +## 5. HPA versus node scaling + +HPA changes a workload's `scale` desired replicas. GKE cluster autoscaling adds/removes nodes when Pod requests and scheduling indicate need. **HPA does not allocate Compute Engine capacity.** + +```mermaid +sequenceDiagram + participant MET as Metrics pipeline + participant H as HPA + participant API as API server + participant S as Scheduler + participant CA as GKE autoscaler + participant CE as Compute Engine + participant K as kubelet + MET-->>H: Demand above target + H->>API: Increase replica count + API-->>S: Pending Pods + S-->>CA: Unschedulable workload + CA->>CE: Provision matching node resources + CE-->>K: VM boot + K->>API: Node Ready + S->>API: Pod bound + K->>K: Image pull, app startup, probes +``` + +Total scale-out delay = metric collection + HPA reconciliation + scheduling + node acquisition if needed + boot/image download + warmup + endpoint/LB readiness. For predictable campaigns or close-time spikes, pre-warm enough capacity instead of relying exclusively on reactive scaling. + +## 6. Why node autoscaling may not happen + +- Node pool already at max. +- Compute Engine CPU/GPU quota exhausted. +- Requested VM shape unavailable in that zone. +- Cluster primary or Pod secondary range lacks assignable IPs. +- Required node affinity/taint incompatible with all provisionable pools. +- Pod requests larger than any available shape. +- Volume affinity restricts scheduling to unavailable zone. +- Too many constrained node pools; conflicting ComputeClass selection. +- API/organization policy rejects new instances. + +**Autopilot** automates node provisioning but can't bypass resource and quota constraints. In current Standard, node auto-provisioning and ComputeClasses can create/manage pools for appropriate workloads. Feature support and node pool counts vary with GKE version. See [node auto-provisioning](https://cloud.google.com/kubernetes-engine/docs/concepts/node-auto-provisioning). + +## 7. Spot and failure tolerance + +Spot worker nodes can reduce cost for interruptible capacity. Design PodDisruptionBudgets for voluntary evictions, graceful termination, workload replication, retry safety and rebalancing. PDBs do **not** prevent an involuntary Spot reclaim or node failure. Stateful databases require durable data and fencing independently of node lifetime. + +## 8. Incident: 35% CPU, 80 Pending Pods + +Hypothetical scheduler output: + +```text +0/9 nodes are available: +3 Insufficient cpu, +3 node(s) had untolerated taint, +3 node(s) didn't match Pod node affinity +``` + +Determine which constraints are blockers and whether changing a Pod's request, affinity or class is safe. Check GKE autoscaler events, node-pool bounds and Compute Engine operations. Do not blindly raise `maxReplicas`; that creates more pending work and may worsen quota/database pressure. + +```bash +kubectl -n marketplace describe pod POD +kubectl describe node NODE +kubectl -n marketplace get events --sort-by=.lastTimestamp +gcloud container operations list --location=europe-west1 --project=PROJECT_ID +``` + +Then verify new nodes joined, Pods reached readiness, LB endpoints are healthy, and user-facing p99/error rate returned to normal. + +## Spoken interview answer + +> “I treat HPA scale-out and node provisioning as separate controllers. Scheduler uses Pod requests and hard constraints against node allocatable, not low average CPU. If Pods remain Pending, I'd inspect FailedScheduling, taints, affinity, topology, volume zones and pool limits, then see whether GKE can provision a matching node given IPs, quotas and capacity. Standard offers direct node-pool control; Autopilot manages more of that but doesn't bypass scheduling. Ultimately I'd measure time to traffic-ready capacity, not just the number of new VMs.” + +**References:** [GKE cluster autoscaling](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler) · [Node pool auto-creation](https://cloud.google.com/kubernetes-engine/docs/concepts/node-auto-provisioning) · [Kubernetes scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/) + +[Previous](./q1) · [Next: GKE networking](./q3) diff --git a/docs/guides/gke/q3.md b/docs/guides/gke/q3.md new file mode 100644 index 0000000..5c0d533 --- /dev/null +++ b/docs/guides/gke/q3.md @@ -0,0 +1,193 @@ +# Q3. GKE Networking: VPC-native Pods, Services, DNS and Dataplane V2 + +> **Interviewer:** Trace a packet from one Pod to another in GKE, then from a Pod to a ClusterIP Service, and finally from a private Pod to an external API. Explain alias IP allocations, Service programming, NetworkPolicy, Dataplane V2 eBPF, kube-dns/Cloud DNS, IP masquerading and Cloud NAT. + +## 1. VPC-native addressing from first principles + +Google Cloud VPCs are global, with regional subnets. GKE VPC-native clusters can allocate: +- **Node addresses:** From the subnet primary IP range. +- **Pod addresses:** From a secondary IP range as alias-IP blocks allocated to nodes. +- **Service ClusterIP:** From a Service address range; implementation varies with GKE networking mode and newer managed-Service-range features. + +```mermaid +flowchart TD + V["GCP VPC global network"] --> S["Regional subnet: primary node IP range"] + S --> N1["Node A: primary 10.40.x.x"] + S --> N2["Node B: primary 10.40.y.y"] + V --> P["Pod secondary IPv4 range"] + P --> P1["Alias CIDR allocated to Node A"] + P --> P2["Alias CIDR allocated to Node B"] + P1 --> A["Pod A"] + P2 --> B["Pod B"] + V --> C["Service IP allocation / virtual VIPs"] +``` + +**Critical capacity distinction:** GKE reserves a block of Pod addresses per node according to max Pods per node. The common 110 Pods/node setting results in a /24 per node; not only the actual live Pod addresses count against the secondary range. A /16 range that allocates /24 blocks supports at most 256 node blocks in that range before other topology constraints. Default/settings differ between Standard and Autopilot; calculate actual limits instead of assuming 256 universally. Read [VPC-native clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips). + +## 2. Example IP calculation + +Suppose: +- Pod secondary range: 10.48.0.0/17, 32,768 addresses. +- Configured maximum Pods per node: 110. +- GKE allocated Pod block per node: /24 = 256 addresses. + +The theoretical pool has `2^(24 - 17) = 128` /24 blocks, so at most **128 nodes** from that particular Pod address range. Actual node limit can be lower due to primary subnet capacity, resource quotas, cluster limits, other users of the range and GKE-specific accounting. + +For the GKE Autopilot max-32-Pods/node model, block sizing may be different. Use the documented formula and deployed version details. + +## 3. Pod-to-Pod datapath + +```mermaid +sequenceDiagram + participant P1 as Pod A 10.48.x.x + participant V1 as Node A dataplane + participant G as GCP VPC fabric + participant V2 as Node B dataplane + participant P2 as Pod B 10.48.y.y + P1->>V1: Packet with source Pod IP, destination Pod IP + V1->>V1: CNI / egress policy and route + V1->>G: Forward toward destination node alias IP range + G->>V2: Deliver to node owning target Pod CIDR + V2->>V2: Ingress policy / Pod interface lookup + V2->>P2: Deliver packet +``` + +This is logical and omits implementation-specific encapsulation/routing optimizations. In a VPC-native cluster, Pod alias addresses are routable by GCP VPC under supported configurations. VPC firewall policy, NetworkPolicy, node dataplane and guest processing still affect reachability. + +## 4. Pod-to-Service flow + +A Kubernetes ClusterIP is usually a **virtual service address**, not a permanent VM owning that IP. Kubernetes EndpointSlices describe selected backend Pods, and the cluster dataplane chooses a backend using programmed routing/NAT. + +```mermaid +sequenceDiagram + participant C as Client Pod + participant DNS as Cluster DNS + participant DP as Node Service dataplane + participant E as EndpointSlice + participant B as Backend Pod + C->>DNS: Query catalog.marketplace.svc.cluster.local + DNS-->>C: Service ClusterIP + C->>DP: TCP SYN to ClusterIP:8080 + DP->>E: Backend set already programmed from API state + DP->>DP: Choose endpoint and translate/forward + DP->>B: TCP SYN to actual Pod IP:8080 + B-->>C: Return traffic via conntrack/dataplane +``` + +The dataplane doesn't query the EndpointSlice API synchronously for each SYN; controller/agent observes API changes and programs backend maps/rules **ahead of packet arrival**. The arrow is conceptual backend selection, not a per-packet API call. + +### Legacy versus Dataplane V2 + +- Legacy GKE dataplane commonly uses kube-proxy/iptables programming for Services and Calico for NetworkPolicy. +- GKE Dataplane V2 uses Cilium-derived eBPF programs and **does not use kube-proxy for Service routing**. +- `anetd` DaemonSet in `kube-system` programs the eBPF dataplane in Dataplane V2. +- Kubernetes NetworkPolicy is built into Dataplane V2; additional policy logging is available. +- GKE does not support adding arbitrary custom eBPF programs to nodes using Dataplane V2 because they can interfere with system programs. + +Check the cluster's actual dataplane mode before relying on `iptables-save` or assuming kube-proxy exists. See [Dataplane V2](https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2). + +## 5. NetworkPolicy is not a VPC firewall + +Kubernetes NetworkPolicy selects Pods/namespaces and controls supported ingress/egress traffic. VPC firewall policies operate on VPC/node/cloud resource identity and routes. Both may apply to a communication path, but solving one denial doesn't automatically solve the other. + +Example default deny for Pods in one namespace: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-deny + namespace: marketplace +spec: + podSelector: {} + policyTypes: [Ingress, Egress] +``` + +**Important:** Default-deny egress can block DNS, cloud metadata, auth and application dependencies. Add narrowly scoped DNS and workload requirements only after mapping egress. Some metadata connectivity has implementation-specific NetworkPolicy treatment, so validate the actual dataplane. + +Example permit catalog API callers by namespace label: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-frontends + namespace: marketplace +spec: + podSelector: + matchLabels: { app: catalog-api } + policyTypes: [Ingress] + ingress: + - from: + - namespaceSelector: + matchLabels: { tenant-role: frontend } + ports: + - protocol: TCP + port: 8080 +``` + +Namespace labels are access-control inputs. Limit who can set them. + +## 6. DNS resolution + +An application usually queries a configured cluster DNS service through `/etc/resolv.conf`. Search domains like `marketplace.svc.cluster.local` and `svc.cluster.local` allow short names; CoreDNS/kube-dns architecture and Cloud DNS integration depend on the GKE DNS provider/configuration. Cloud DNS VPC scope may change cross-cluster behavior compared with cluster-scoped DNS. + +```bash +kubectl -n marketplace exec deploy/catalog-api -- cat /etc/resolv.conf +kubectl -n marketplace get svc,endpointslices +kubectl -n kube-system get deploy,svc | grep -E 'dns|coredns' +kubectl -n marketplace run dns-test --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default.svc.cluster.local +``` + +If the application image is distroless, use an authorized ephemeral debug container. DNS `NXDOMAIN` is not a TCP timeout. Distinguish stub resolver, cluster DNS backend, upstream recursive/private zone, network policy egress and negative TTL caching. + +## 7. Pod egress and IP masquerading + +```mermaid +sequenceDiagram + participant P as GKE Pod + participant N as GKE node + participant NAT as Cloud NAT + participant X as Partner API + P->>N: Connect to public destination + N->>N: Decide Pod source preservation vs SNAT to node IP + N->>NAT: Route eligible private egress + NAT->>NAT: Translate to public NAT address/port + NAT->>X: Outbound connection + X-->>P: Return over established translation path +``` + +This path depends on GKE IP-masquerade configuration, cluster mode, NAT subnet coverage, default routes and firewalls. Cloud NAT port allocation source identities differ depending on whether Pods are SNATed first. An additional Cloud NAT address is not always the right fix when application churn and connection pooling are the real problem. + +## 8. Common production failure patterns + +| Symptom | First hypothesis | +| --- | --- | +| Pending new nodes after scale-out | Primary/Pod secondary IP range or quota | +| ClusterIP times out but Pod IP works | Service rules/eBPF, EndpointSlice selection, policy | +| DNS lookup fails from one namespace | Default-deny policy, kube-dns/service reachability | +| External API times out at peak | Egress firewall, Cloud NAT port capacity, connection churn | +| Pod-to-Pod works within node but not across nodes | Node dataplane, VPC routing/firewall, alias range | +| `iptables` rules absent | Dataplane V2 may use eBPF, not kube-proxy | + +### Investigation commands + +```bash +gcloud container clusters describe CLUSTER --location=europe-west1 --project=PROJECT_ID +gcloud compute networks subnets describe SUBNET --region=europe-west1 --project=PROJECT_ID +kubectl get nodes -o wide +kubectl -n marketplace get pods -o wide +kubectl -n marketplace get svc,endpointslices,networkpolicy +kubectl -n kube-system get daemonsets +kubectl -n marketplace describe pod POD +``` + +Use tcpdump and host-level utilities only with appropriate privileges and on modes that permit them. A packet capture can distinguish SYN loss, RST, TLS failure and application timeouts; avoid using OCR-like logs as a substitute for source/destination evidence. + +## Spoken interview answer + +> “GKE VPC-native networking uses primary subnet addresses for nodes and secondary alias IP ranges for Pods, so Pod IP allocation is a node-block capacity problem. A ClusterIP is a virtual Service; the node dataplane forwards to endpoints already programmed from EndpointSlices. Dataplane V2 uses eBPF with anetd rather than kube-proxy. I'd diagnose networking by separating DNS, Pod-to-Pod, Service routing, NetworkPolicy, VPC firewall and egress masquerading/Cloud NAT. I would validate exact packet source addresses and CIDR capacity before simply increasing node pools.” + +**References:** [VPC-native GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips) · [Dataplane V2](https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2) · [Cloud NAT](https://cloud.google.com/nat/docs/public-nat) + +[Previous](./q2) · [Next: Ingress, Gateway and NEGs](./q4) diff --git a/docs/guides/gke/q4.md b/docs/guides/gke/q4.md new file mode 100644 index 0000000..2eb8057 --- /dev/null +++ b/docs/guides/gke/q4.md @@ -0,0 +1,231 @@ +# Q4. GKE Ingress, Gateway API, NEGs and Load-Balancer Lifecycle + +> **Interviewer:** A browser reaches a Kubernetes backend through a Google Cloud HTTPS load balancer. Trace DNS, TLS, forwarding rules, routing, backend services, NEGs, health checks, EndpointSlices, readiness gates and graceful termination. Explain Ingress versus Gateway API versus Service type LoadBalancer. + +## 1. Kubernetes API object is not the cloud datapath + +GKE has controllers that watch Kubernetes objects and provision/configure Google Cloud load-balancing resources. Those controllers are **control-plane reconcilers**, not necessarily in-path proxies that receive every user request. + +For supported GKE L7 Ingress/Gateway configurations, cloud backend services can use Pod-direct `GCE_VM_IP_PORT` Network Endpoint Groups (NEGs). Cloud load balancer sends requests to the Pod endpoint IP:port, rather than necessarily to a NodePort then through a ClusterIP Service. + +```mermaid +flowchart TD + DNS["Public DNS"] --> IP["External forwarding rule / VIP"] + IP --> TLS["Target HTTPS proxy: TLS cert, ALPN"] + TLS --> MAP["URL map or Gateway routing"] + MAP --> B["Google Cloud backend service"] + B --> HC["Health check configuration"] + B --> N["GCE_VM_IP_PORT NEG endpoints"] + N --> P1["Pod A IP:8080"] + N --> P2["Pod B IP:8080"] + IN["GKE Ingress/Gateway controller"] -.-> TLS + IN -.-> MAP + IN -.-> B + IN -.-> N + SV["Kubernetes Service / Pod selectors"] -.-> N +``` + +The diagram is a **proxy L7** path, not a generic GKE Service LoadBalancer path. GKE documentation notes all GKE-managed L7 load balancers, including Ingress and Gateway, use container-native backend routing; `type: LoadBalancer` Services have different network load-balancer behavior. See [GKE load balancing](https://cloud.google.com/kubernetes-engine/docs/concepts/about-load-balancing). + +## 2. Complete user request flow + +```mermaid +sequenceDiagram + participant B as Browser + participant D as Public recursive DNS + participant F as Google L7 frontend + participant H as Backend service/health + participant P as GKE Pod endpoint + participant DB as Database + B->>D: Query api.example.invalid + D-->>B: Global/regionally scoped frontend address + B->>F: TCP/TLS handshake with SNI + ALPN + F->>F: Certificate, URL/host routing, security policies + F->>H: Select healthy backend + H->>P: Backend HTTP or HTTPS request to Pod IP:port + P->>DB: Application query + DB-->>P: Result + P-->>H: HTTP response + H-->>F: Response + F-->>B: HTTPS response +``` + +The backend's own TLS is independent of frontend termination and must be configured deliberately. A successful browser TLS handshake does not prove the Pod or database works. + +## 3. Ingress, Gateway API and LoadBalancer Service + +| Mechanism | Main abstraction | Cloud integration | +| --- | --- | --- | +| `Ingress` | HTTP(S) host/path routing | GKE Ingress controller provisions supported Application LBs | +| Gateway API `Gateway` + `HTTPRoute` | Listener ownership + route attachment/traffic policy | GKE Gateway controller; GatewayClass selects LB family | +| `Service type=LoadBalancer` | Expose a Service at L4 | GKE network load-balancer product, not automatically L7 NEG | +| Standalone NEG | Direct Pod endpoints used by independently managed LB | Explicit controller/ownership coordination | + +Gateway API separates infrastructure-facing Gateway/listener ownership from application HTTPRoute routing and permissions. A `ReferenceGrant` may be required for supported cross-namespace object references. GatewayClass must match the desired cloud LB family (for example `gke-l7-global-external-managed` for a supported global external Application LB). + +## 4. A minimal HTTP backend and Gateway API example + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: catalog-api + namespace: marketplace +spec: + replicas: 3 + selector: + matchLabels: { app: catalog-api } + template: + metadata: + labels: { app: catalog-api } + spec: + containers: + - name: api + image: europe-west1-docker.pkg.dev/PROJECT_ID/apps/catalog-api:VERSION + ports: + - containerPort: 8080 + readinessProbe: + httpGet: { path: /ready, port: 8080 } + periodSeconds: 5 +--- +apiVersion: v1 +kind: Service +metadata: + name: catalog-api + namespace: marketplace +spec: + selector: { app: catalog-api } + ports: + - port: 80 + targetPort: 8080 +``` + +An illustrative **HTTP-only** Gateway for a sandbox (not a secure public HTTPS production endpoint): + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: marketplace-gateway + namespace: marketplace +spec: + gatewayClassName: gke-l7-global-external-managed + listeners: + - name: http + protocol: HTTP + port: 80 + allowedRoutes: + namespaces: + from: Same +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: catalog-route + namespace: marketplace +spec: + parentRefs: + - name: marketplace-gateway + rules: + - matches: + - path: { type: PathPrefix, value: /catalog } + backendRefs: + - name: catalog-api + port: 80 +``` + +**Caveats:** Actual GatewayClass/controller feature support, static IP, HTTPS certificates, BackendPolicy health checks and allowedRoutes policies need environment-specific configuration. A real public production Gateway should provide HTTPS, correct DNS, certificate lifecycle and security controls. Do not deploy this demo publicly unchanged. + +## 5. Ingress and NEG readiness gates + +In some supported GKE Ingress/NEG deployments, the controller injects a Pod readiness gate: + +```text +cloud.google.com/load-balancer-neg-ready +``` + +It represents the load balancer controller's assessment of NEG endpoint/backend health in addition to kubelet container probes. A Pod can pass `/ready` inside Kubernetes yet remain not Ready until external NEG readiness is satisfied. Misconfigured backend health checks can **stall rollouts** rather than immediately send broken new traffic. See [GKE Ingress readiness gates](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-routing-security). + +```mermaid +sequenceDiagram + participant DEP as Deployment controller + participant API as API server + participant NEG as NEG controller + participant LB as Cloud LB health checker + participant POD as New Pod + DEP->>API: Create new Pod + API-->>NEG: Endpoint eligible for NEG + NEG->>LB: Register new endpoint IP:port + LB->>POD: Probe health check path and port + POD-->>LB: Healthy + LB-->>NEG: Backend health status + NEG->>API: Set neg-ready readiness gate + API-->>DEP: Pod becomes Ready/Available + DEP->>API: Remove old replica within rollout strategy +``` + +This sequence simplifies GKE controller behavior and timing. **Do not assume all Gateway configurations inject exactly the same readiness gate behavior as Ingress**; inspect Pod conditions, backend policies and supported controller semantics. + +## 6. Health checks, graceful draining and rolling updates + +A safe path requires: +1. Readiness reflects ability to accept traffic. +2. Cloud backend health checks use the correct port and endpoint. +3. Pod termination stops admitting new work. +4. Endpoint/NEG withdrawal propagates. +5. In-flight requests drain within the allowed grace period. +6. Application/database schema supports overlap of versions. +7. Remaining replicas have enough capacity. + +```yaml +spec: + terminationGracePeriodSeconds: 90 + containers: + - name: api + image: europe-west1-docker.pkg.dev/PROJECT_ID/apps/catalog-api:VERSION + lifecycle: + preStop: + exec: + command: ["/bin/sh", "-c", "wget -qO- http://127.0.0.1:8080/drain && sleep 10"] +``` + +This is an **illustrative Pod spec fragment** and requires app endpoint/tools. A 10-second sleep is not a universal propagation guarantee. preStop consumes termination grace, and endpoint updates occur asynchronously. For long-lived connections, investigate actual LB timeout/connection drain features, application protocol and graceful shutdown. + +## 7. Troubleshooting health divergence + +```bash +kubectl -n marketplace get ingress,gateway,httproute,svc,endpointslices +kubectl -n marketplace describe gateway marketplace-gateway +kubectl -n marketplace describe httproute catalog-route +kubectl -n marketplace describe pod POD +kubectl -n marketplace get pod POD -o jsonpath='{.status.conditions}' +gcloud compute backend-services list --project=PROJECT_ID +gcloud compute network-endpoint-groups list --project=PROJECT_ID +gcloud compute health-checks list --project=PROJECT_ID +``` + +Use the right `--global` or `--region` backend service API for the LB class. Some commands/CRDs may be absent if Gateway API isn't enabled/supported in that cluster. + +### Incident A: Pod Ready but external 502 + +Identify the exact URL map/backend service and whether the requested host/path reaches the intended backend. Verify endpoints in the NEG, health check response, firewall health check access, backend HTTP vs HTTPS, actual Pod listener, app error and Cloud Armor policy. Internal Pod readiness is only one piece of evidence. + +### Incident B: rollout stuck when new Pods healthy locally + +Inspect Pod condition `cloud.google.com/load-balancer-neg-ready`. Cloud LB health check may target wrong path, wrong port, blocked firewall or insufficient warmup. Fix cloud health/backends rather than lowering readiness requirements or blindly increasing rollout surge. + +### Incident C: two controllers edit the same load balancer + +Terraform provisions an external backend while GKE Gateway/Ingress controller also reconciles it. Changes oscillate or controllers recreate resources. Establish one authoritative owner or use a supported standalone NEG integration with documented division of responsibilities. Never assume controller-created cloud resources are safe to manage manually. + +## 8. Senior design trade-offs + +Choose Ingress for a simpler existing HTTP exposure contract when supported. Gateway API provides richer ownership/route attachment abstractions and product-specific policy. `LoadBalancer` Service suits appropriate L4 exposure. Don't choose based on manifest brevity alone; account for LB location, client IP semantics, security policy, certificate renewal, rate controls, request timeout, backend endpoint mode, cost and migration strategy. + +## Spoken interview answer + +> “I'd first name the load balancer class, because GKE Application and Network LBs differ substantially. In a GKE-managed L7 path, DNS resolves the frontend, the target proxy handles HTTPS, route selection chooses a backend service, and a NEG can expose Pod IP:port endpoints directly. The Ingress or Gateway controller manages cloud resources but doesn't have to proxy user packets itself. I'd examine both Kubernetes probes and cloud backend health, including NEG readiness gates where supported. During deployments, endpoint convergence and draining are asynchronous, so I would verify a real request path rather than equating Pod Running with customer availability.” + +**References:** [GKE load balancing](https://cloud.google.com/kubernetes-engine/docs/concepts/about-load-balancing) · [Gateway API](https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-api) · [Ingress routing and security](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-routing-security) + +[Previous](./q3) · [Next: GKE security](./q5) diff --git a/docs/guides/gke/q5.md b/docs/guides/gke/q5.md new file mode 100644 index 0000000..3013ca6 --- /dev/null +++ b/docs/guides/gke/q5.md @@ -0,0 +1,217 @@ +# Q5. GKE Security: IAM, RBAC, Workload Identity, Admission and Isolation + +> **Interviewer:** A GKE catalog Pod must read one Cloud Storage bucket but must not list Kubernetes Secrets or access other teams' services. Explain Google's IAM model versus Kubernetes RBAC, Workload Identity Federation, network policy, Pod Security, admission, image provenance and a production 403 incident. + +## 1. Four distinct boundaries + +1. **Google Cloud IAM:** Authorizes Google Cloud API operations on projects, service accounts, buckets and other services. +2. **Kubernetes RBAC:** Authorizes Kubernetes API operations such as get Pods or create Deployments. +3. **Network isolation:** NetworkPolicy controls supported Pod traffic; GCP VPC firewall policies control applicable network flows. +4. **Admission / runtime controls:** Pod security, container privileges, allowed images, host access, quotas and custom policies. + +A Kubernetes ServiceAccount is not the same object as a Google IAM service account. + +```mermaid +flowchart TD + POD["Application Pod"] --> KSA["Kubernetes ServiceAccount"] + KSA --> RBAC["Kubernetes RBAC, when calling K8s API"] + KSA --> META["GKE metadata server / federation"] + META --> IAM["Google Cloud IAM principal"] + IAM --> GCS["Cloud Storage API"] + POD --> NP["Kubernetes NetworkPolicy"] + NP --> FW["VPC firewall and route path"] + ADM["API server admission / Pod Security"] -.-> POD +``` + +A Pod allowed to read Cloud Storage can still be Forbidden from the Kubernetes API. Conversely a namespace administrator doesn't automatically gain Google Project Owner, but permission to create workloads using a privileged KSA/GSA can create an **effective privilege escalation** path. Review who can use each ServiceAccount. + +## 2. GKE Workload Identity Federation: direct access + +GKE Workload Identity Federation maps Kubernetes workload identities to IAM principals in an identity pool. With direct resource authorization, assign a role to the KSA principal identifier rather than attach a JSON key file. + +```text +principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/marketplace/sa/catalog-api +``` + +Use the actual numerical project number and project ID. Restrict to one KSA, one namespace and the intended resource. + +Example (requires existing bucket, cluster WIF and permissions): + +```bash +kubectl -n marketplace create serviceaccount catalog-api + +gcloud storage buckets add-iam-policy-binding \ + gs://OWNED_CATALOG_BUCKET \ + --member="principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/marketplace/sa/catalog-api" \ + --role="roles/storage.objectViewer" +``` + +Don't grant `roles/storage.admin` project-wide if the workload only reads one bucket. For APIs where direct federation has limitations, use supported impersonation. + +## 3. Alternative: impersonate a Google IAM service account + +The KSA is annotated with a GSA email, and the identity relationship has an IAM binding allowing impersonation. + +```bash +gcloud iam service-accounts add-iam-policy-binding \ + catalog-reader@PROJECT_ID.iam.gserviceaccount.com \ + --role="roles/iam.workloadIdentityUser" \ + --member="serviceAccount:PROJECT_ID.svc.id.goog[marketplace/catalog-api]" + +kubectl -n marketplace annotate serviceaccount catalog-api \ + iam.gke.io/gcp-service-account=catalog-reader@PROJECT_ID.iam.gserviceaccount.com +``` + +Both IAM binding and KSA annotation are required for this GSA impersonation approach. The GSA itself also needs permission to read the target bucket. Don't conflate this with direct KSA-principal permissions. + +```mermaid +sequenceDiagram + participant P as Pod Google client library + participant M as GKE metadata server + participant F as Google federation / IAM + participant S as Google SA credentials API + participant G as Cloud Storage + P->>M: Request application default token + M->>F: Verify KSA workload identity + alt Direct federated principal + F-->>P: Authorized federated access path + else GSA impersonation configured + F->>S: Exchange for permitted GSA token + S-->>P: GSA access token + end + P->>G: storage.objects.get + G-->>P: Object or 403 +``` + +The diagram is conceptual; metadata server returns credentials via the client path and API token formats/flows vary by integration. See [Authenticate GKE workloads](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity). + +## 4. Kubernetes RBAC and impersonation + +A namespace-specific read-only Role: + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: audit-reader + namespace: marketplace +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: configmap-reader + namespace: marketplace +rules: + - apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "list", "watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: audit-reader-config + namespace: marketplace +subjects: + - kind: ServiceAccount + name: audit-reader + namespace: marketplace +roleRef: + kind: Role + name: configmap-reader + apiGroup: rbac.authorization.k8s.io +``` + +Test: + +```bash +kubectl auth can-i get configmaps -n marketplace +kubectl auth can-i list secrets -n marketplace +kubectl auth can-i list configmaps -n marketplace \ + --as=system:serviceaccount:marketplace:audit-reader +``` + +Impersonation tests require permission to impersonate that principal. A RoleBinding can refer to a ClusterRole while granting its **namespaced resource rules** in one namespace. RBAC evaluation also includes cluster-scoped resources, impersonation, `bind`, `escalate` and operational paths through `pods/exec`. + +## 5. Pod Security and admission + +Pod Security Admission can enforce baseline or restricted policies by namespace labels: + +```bash +kubectl label namespace marketplace \ + pod-security.kubernetes.io/enforce=restricted \ + pod-security.kubernetes.io/audit=restricted \ + pod-security.kubernetes.io/warn=restricted +``` + +Verify application compatibility before enforcement. Some infrastructure DaemonSets need privileged access and should use separately governed namespaces with deliberate policies. + +Workload hardening: + +```yaml +securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault +containers: + - name: app + image: europe-west1-docker.pkg.dev/PROJECT_ID/apps/app:VERSION + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + resources: + requests: + cpu: 250m + memory: 256Mi + limits: + memory: 512Mi +``` + +Fragment only; attach container securityContext inside a full Pod spec. Different node OS and application permissions may require compatible UID, filesystem and capabilities. + +Admission webhooks and ValidatingAdmissionPolicy can enforce approved images, labels, privileged controls, quotas and standards. A fail-closed unreachable webhook can block new Pod creation cluster-wide even when existing Pods are running. Scope webhooks and test failure policies/recovery. + +## 6. NetworkPolicy and tenancy + +A namespace label and Kubernetes NetworkPolicy can constrain peer traffic, but the CNI must enforce it, and not every metadata/network path has identical policy semantics. With GKE Dataplane V2, NetworkPolicy enforcement is built into the eBPF path, with supported network-policy logging. Keep application authorization even if network traffic is limited: a reachable Pod may still be an unauthorized caller. + +## 7. Image and secrets supply chain + +Artifact Registry IAM controls who can push or pull container images. Deploying by OCI digest improves immutability. Binary Authorization and attestations can enforce supply-chain provenance for supported GKE deployment workflows. Admission/image scanning is not a guarantee against malicious runtime behavior. + +Kubernetes Secrets are base64 representations, not encryption by themselves. Protect: +- Kubernetes API access and etcd/control-plane encryption features. +- Secret Manager/CSI integrations, runtime permissions and rotation. +- Workload identity and node service account grants. +- Logging and debugging paths so secrets aren't accidentally printed. + +A node service account with overly broad permissions can become a fallback risk if workload identity isn't configured correctly. + +## 8. Production 403 investigation + +A hypothetical catalog Pod logs `storage.objects.get: permission denied` after a node-pool change. + +1. Check actual workload KSA from Pod spec, not Deployment defaults. +2. Confirm GKE WIF and metadata server availability on that node/pool. +3. Verify cluster workload pool, project number, namespace/name, IAM binding and GSA annotation if impersonating. +4. Check token audience/expiry and client library ADC behavior. +5. Check exact bucket/object permissions, IAM Conditions/Deny, VPC Service Controls and Cloud Storage restrictions. +6. Correlate Cloud Audit Logs and policy troubleshooting where supported. +7. Fix the correct role or identity, not by granting basic project Owner. + +```bash +kubectl -n marketplace get pod POD -o jsonpath='{.spec.serviceAccountName}' +kubectl -n marketplace describe serviceaccount catalog-api +kubectl -n marketplace describe pod POD +gcloud storage buckets get-iam-policy gs://OWNED_CATALOG_BUCKET +gcloud container node-pools describe POOL --cluster=CLUSTER \ + --location=europe-west1 --project=PROJECT_ID +``` + +## Spoken interview answer + +> “I separate Kubernetes API permissions from Google Cloud API permissions and network controls. On GKE I'd assign an individual KSA to the workload and use Workload Identity Federation for bucket access, either directly to the KSA principal or via a narrowly permitted Google service account. I'd restrict who can launch workloads under that identity. Admission and Pod Security limit privilege escalation; NetworkPolicy limits network paths, not application authorization. For a 403, I'd verify the actual runtime identity, token path, exact resource/permission, IAM conditions and service-specific policy before changing grants.” + +**References:** [WIF for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) · [Configure WIF](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) · [Kubernetes RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) · [Pod Security](https://kubernetes.io/docs/concepts/security/pod-security-standards/) + +[Previous](./q4) · [Next: storage and CSI](./q6) diff --git a/docs/guides/gke/q6.md b/docs/guides/gke/q6.md new file mode 100644 index 0000000..653d50d --- /dev/null +++ b/docs/guides/gke/q6.md @@ -0,0 +1,204 @@ +# Q6. GKE Storage, CSI, StatefulSets and Data Recovery + +> **Interviewer:** A StatefulSet Pod is stuck on volume attachment after node replacement. Explain PV/PVC/StorageClass, Compute Engine Persistent Disk CSI, immediate versus delayed binding, zonal versus regional volumes, snapshots, StatefulSet identity and recovery safety. Why doesn't a Bound PVC guarantee mountability? + +## 1. Storage is a distinct lifecycle from Pods + +A Pod may be destroyed/recreated while its data persists. Kubernetes models desired storage through: + +- **PersistentVolumeClaim (PVC):** A workload request for size, access mode and storage class. +- **PersistentVolume (PV):** A bound/presented volume with capacity and backing volume identity. +- **StorageClass:** Dynamic provisioning settings and topology/binding behavior. +- **CSI controller plugin:** Provisions, deletes, resizes and may attach/detach storage. +- **CSI node plugin:** Stages and publishes/mounts the volume on a node. +- **VolumeAttachment:** Tracks controller-level attachment when supported. + +```mermaid +flowchart TD + STS["StatefulSet ordinal"] --> POD["Pod ledger-0"] + STS --> PVC["PVC data-ledger-0"] + PVC --> PV["PersistentVolume"] + PV --> DISK["Compute Engine zonal or regional disk"] + CTRL["PD CSI controller"] -.-> DISK + NODE["PD CSI node plugin"] --> MOUNT["Mounted filesystem in Pod"] + DISK --> MOUNT +``` + +StatefulSets give stable ordinals, associated PVCs and rollout order. They don't implement database replication, quorum, fencing or transaction correctness. A three-replica StatefulSet is not automatically a reliable three-node database. + +## 2. Dynamic provisioning path + +```mermaid +sequenceDiagram + participant APP as StatefulSet/Pod + participant API as Kubernetes API + participant S as Scheduler + participant PROV as CSI external provisioner + participant CSI as PD CSI controller + participant G as Compute Engine disk API + participant K as kubelet / CSI node + APP->>API: Create Pod and PVC + API-->>S: Pod scheduling request + Note over S,PROV: WaitForFirstConsumer coordinates topology + S->>API: Choose eligible zone/node constraints + PROV->>CSI: Request volume creation + CSI->>G: Create disk in appropriate zone(s) + G-->>CSI: Disk ID + PROV->>API: PV/PVC binding + API-->>K: Bound Pod and volume + Note over CSI,K: Attach controller step if required + K->>K: Stage and mount filesystem + K->>APP: Start container with volume +``` + +Actual provisioning and binding order depends on StorageClass binding mode, scheduler integration and CSI driver behavior. Some volumes don't require attach operations. **PVC Bound means identity/capacity binding, not that a disk is attached/mounted successfully.** + +## 3. WaitForFirstConsumer and topology + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: pd-balanced-delayed +provisioner: pd.csi.storage.gke.io +parameters: + type: pd-balanced + csi.storage.k8s.io/fstype: ext4 +volumeBindingMode: WaitForFirstConsumer +allowVolumeExpansion: true +reclaimPolicy: Retain +``` + +This is an illustrative class using the supported Persistent Disk CSI driver. `WaitForFirstConsumer` lets scheduler constraints help choose a valid zone before provisioning, reducing disks created in zones where Pods cannot run. + +**Immediate binding trap:** A volume may be provisioned in zone A before a Pod with hard node affinity to zone B is scheduled. Both resources exist but cannot be brought together. Delayed binding is especially valuable for zonal disks. See [GKE Persistent Disk CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver). + +## 4. PVC and StatefulSet example + +```yaml +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: ledger + namespace: marketplace +spec: + serviceName: ledger-headless + replicas: 3 + selector: + matchLabels: { app: ledger } + template: + metadata: + labels: { app: ledger } + spec: + containers: + - name: ledger + image: europe-west1-docker.pkg.dev/PROJECT_ID/apps/ledger:VERSION + volumeMounts: + - name: data + mountPath: /var/lib/ledger + volumeClaimTemplates: + - metadata: + name: data + spec: + storageClassName: pd-balanced-delayed + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 20Gi +``` + +A real StatefulSet also needs an appropriate governing headless Service, app configuration, secrets, replication/quorum/fencing and backup policy. The image is fictional. + +## 5. Access-mode semantics + +| Mode | Meaning | Pitfall | +| --- | --- | --- | +| RWO | Read-write access from one node | Not necessarily one Pod; multiple Pods on same node may access | +| ROX | Read-only many nodes where driver supports | Not a database writer | +| RWX | Read-write many nodes where driver supports | PD CSI default block disks don't provide generic shared POSIX RWX | +| RWOP | Read-write access restricted to one Pod with compatible CSI | Still not a distributed transaction/fencing protocol | + +A CSI provider must actually support the requested mode. Don't simply change `ReadWriteOnce` to `ReadWriteMany` when a two-node mount fails. + +## 6. Zonal versus regional persistent disks + +**Zonal PD:** Disk resides in one zone. Pods attaching it must run on eligible nodes in that zone. If zone is unavailable, a Pod scheduled elsewhere usually can't attach the zonal disk. + +**Regional PD:** Backing disk replicates synchronously between two supported zones and is designed to tolerate some zonal failures. Attachment/writer orchestration remains important; replication doesn't automatically make an active-active multi-writer database. + +Illustrative regional StorageClass: + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: regional-pd +provisioner: pd.csi.storage.gke.io +parameters: + type: pd-balanced + replication-type: regional-pd +volumeBindingMode: WaitForFirstConsumer +allowedTopologies: + - matchLabelExpressions: + - key: topology.gke.io/zone + values: [europe-west1-b, europe-west1-c] +``` + +The `topology.gke.io/zone` key and regional PD support require compatible GKE/CSI and zones; validate current machine/disk constraints. Regional replication provides zone resilience, not backups or protection from deletion/corruption. See [Regional PD](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/regional-pd). + +## 7. Snapshots versus backups + +CSI snapshot objects (`VolumeSnapshotClass`, `VolumeSnapshot`, `VolumeSnapshotContent`) use external snapshot-controller integrations where supported. A storage snapshot may be **crash-consistent**, not guaranteed application-consistent. For databases, coordinate flush/checkpoints/WAL and restore validation. + +Recovery requirements: +- RPO/RTO. +- Snapshot/backup location and access permissions. +- Key material and independent restore rights. +- Cross-region and off-cluster recovery. +- Consistent database transactions. +- Explicit cleanup/retention. +- Drill at least one full restore. + +A cloud disk's retention policy and a StatefulSet's persistent claim lifecycle can preserve data after Pod or StatefulSet deletion, but they don't replace recovery backups. + +## 8. Incident: Multi-Attach error during node replacement + +Hypothetical: + +```text +PVC Bound, Pod ContainerCreating +FailedAttachVolume: Multi-Attach error +Disk attached to previous node in zone A +New Pod scheduled to node B +``` + +Investigate: + +```bash +kubectl -n marketplace describe pod ledger-0 +kubectl -n marketplace get pvc +kubectl get pv +kubectl get volumeattachments +kubectl -n marketplace get events --sort-by=.lastTimestamp +kubectl -n kube-system get pods | grep -Ei 'csi|pd' +gcloud compute disks list --project=PROJECT_ID +gcloud compute disks describe DISK --zone=europe-west1-b --project=PROJECT_ID +``` + +Different CSI drivers create resources in multiple namespaces. Inspect the relevant controller/node driver logs with an authorized cluster view. + +**Do not force-detach a disk while the old database writer may still be active.** Establish fencing or verify the node/instance is dead and cannot resume writing. Deleting PVC or PV to “fix” attach is potentially destructive. + +Potential causes: stale old attachment, CSI controller failure, zone affinity mismatch, unsupported simultaneous attachment, node failed mid-shutdown, volume operation quota/API error. + +## 9. Senior architecture decision + +For mission-critical transactional databases, prefer a managed service where requirements/latency allow, or use a properly engineered database operator with proven replication and backup. A StatefulSet plus three PVs isn't a substitute for consensus, failover logic, disaster recovery and data correctness. + +## Spoken interview answer + +> “I'd separate Pod lifecycle from storage lifecycle. A PVC asks for storage; a PV represents the bound volume; the CSI driver provisions, attaches and mounts through controller and node components. Delayed binding aligns zonal provisioning with scheduling. Bound doesn't prove attach and mount succeeded. For a Multi-Attach incident I'd inspect VolumeAttachments, Pod events, disk topology and the previous node, and fence an old writer before detaching storage. Regional PD improves zonal resilience but not database consistency or backup. I'd verify a restore rather than equate snapshots with recoverability.” + +**References:** [GKE PV overview](https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes) · [PD CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver) · [Regional PD](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/regional-pd) + +[Previous](./q5) · [Next: autoscaling and resource management](./q7) diff --git a/docs/guides/gke/q7.md b/docs/guides/gke/q7.md new file mode 100644 index 0000000..69d0b3a --- /dev/null +++ b/docs/guides/gke/q7.md @@ -0,0 +1,223 @@ +# Q7. GKE Autoscaling, Resource Management, QoS and Performance + +> **Interviewer:** A bidding API's p99 latency spikes during a scheduled event. HPA says 20 replicas, 10 Pods are Pending, nodes show spare CPU, and the database is near connection limits. Explain requests, limits, QoS, CPU throttling, memory OOM, HPA/VPA, node autoscaling, KEDA and reliable scale-out. + +## 1. Model three separate scaling questions + +**Horizontal scaling:** How many Pod replicas should exist? HPA or an event-driven scaler adjusts workload replica count. + +**Vertical sizing:** How much CPU/memory should each Pod request? Resource analysis, VPA recommendations and workload benchmarking inform sizing. + +**Infrastructure capacity:** Where do those Pods run? GKE Standard cluster autoscaling/node auto-provisioning or Autopilot-managed capacity responds to eligible scheduling demands. + +```mermaid +flowchart TD + DEMAND["User traffic / queue growth"] --> METRICS["Application and resource metrics"] + METRICS --> HPA["HPA / event scaler: Pod replicas"] + HPA --> SCHED["Scheduler"] + SCHED -->|Enough capacity| PODS["Running / Ready Pods"] + SCHED -->|Unschedulable| CA["GKE node autoscaler"] + CA --> NODES["Compute Engine nodes"] + NODES --> PODS + VPA["VPA sizing / resource recommendations"] -.-> SCHED + PODS --> DB["Shared database / APIs"] +``` + +The controllers operate asynchronously, with different feedback delays and ownership. + +## 2. CPU requests, limits and memory + +- **CPU request:** Scheduler placement entitlement; HPA CPU utilization denominator if resource utilization is configured. +- **CPU limit:** Linux cgroup CPU quota in typical implementations. A multi-threaded app can exhaust quota and be throttled while host has idle cores. +- **Memory request:** Scheduling/pressure considerations. +- **Memory limit:** Cgroup memory ceiling; crossing it can cause OOM kill after reclaim. +- **Ephemeral-storage request/limit:** Placement and local disk accounting where supported. + +```yaml +resources: + requests: + cpu: 500m + memory: 512Mi + limits: + memory: 1Gi +``` + +This intentionally omits CPU limit, allowing CPU bursting under available capacity and applicable policy. It is **not** always the right choice: some multi-tenant or cost-isolation environments require quotas/limits. Use actual load tests and SLO data. + +### QoS classes + +- **Guaranteed:** Applicable CPU and memory requests equal limits for all relevant containers. +- **Burstable:** Some requests/limits but not Guaranteed. +- **BestEffort:** No CPU/memory requests or limits. + +Guaranteed does **not** prevent OOM when the Pod exceeds its memory cgroup limit. Container OOMKilled and kubelet eviction under node pressure are distinct. PriorityClass/Pod QoS influence scheduling and eviction, but neither guarantees immunity to node failure. + +## 3. HPA math + +Simplified HPA calculation: + +```text +desiredReplicas = ceil(currentReplicas × currentMetric / targetMetric) + +Example: + 10 Pods + average CPU utilization 80% of requests + target CPU utilization 50% + +desired = ceil(10 × 80/50) = 16 Pods +``` + +The real controller accounts for missing metrics, not-yet-ready Pods, tolerance, scaling policies and stabilization windows. + +**CPU utilization is relative to requests**, not node capacity. A Pod using 400m with 500m CPU request reports 80% of request; with 1000m request and identical usage it reports 40%. Changing VPA CPU request recommendations can alter HPA decisions without changing workload demand. + +```yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: bidding-api + namespace: marketplace +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: bidding-api + minReplicas: 6 + maxReplicas: 60 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 60 + behavior: + scaleDown: + stabilizationWindowSeconds: 300 +``` + +This requires a functional metrics pipeline and CPU requests. Use observed traffic concurrency or request/queue metrics where CPU isn't a good leading indicator. + +## 4. HPA multiple metrics and missing data + +When multiple metrics are configured, HPA derives desired counts per metric and generally chooses the highest recommendation, subject to error handling and behavior constraints. + +If CPU recommends 8 and requests-per-second recommends 14, HPA may select 14. If one metric fails and another recommends downscaling, the controller may conservatively skip that downscale. Read HPA conditions rather than assuming a bug. + +```bash +kubectl -n marketplace get hpa +kubectl -n marketplace describe hpa bidding-api +kubectl -n marketplace top pods +kubectl -n marketplace describe deployment bidding-api +``` + +## 5. Vertical Pod Autoscaling + +VPA estimates suitable CPU/memory requests from workload usage and policy. Depending on supported mode/version, it can recommend or apply resource adjustments through admission or in-place/eviction mechanisms. Don't describe VPA as changing runtime CPU consumption itself or assume every provider/version supports all update modes. + +**Feedback-loop hazard:** HPA scaling on CPU utilization and VPA modifying CPU requests can alter each other's control signals. A common conservative strategy uses VPA for recommendations and HPA for replica scaling, then applies right-sizing through controlled rollout. Other combinations require intentional complementary metrics and testing. + +## 6. Event and queue scaling + +Queue depth can be more directly connected to worker demand than CPU utilization. + +Suppose 20,000 queued events and target 500 events/worker. A simple backlog-per-worker goal suggests ~40 replicas, but processing time, arrival rate, allowed deadlines and retry behavior also matter. + +KEDA is an optional third-party extension, not core GKE or always installed. With a compatible scaler it can expose queue/external metrics to HPA and support scale-to-zero under supported configurations. Workload startup and node startup delays can make reactive scale-from-zero unsuitable for low-latency user endpoints. + +```mermaid +sequenceDiagram + participant Q as Queue / traffic + participant K as KEDA / metrics adapter + participant H as HPA + participant A as API server + participant S as Scheduler + participant N as GKE autoscaler + Q-->>K: Queue depth grows + K-->>H: External metric + H->>A: Increase replicas + A-->>S: New Pods + S-->>N: Some Pods unschedulable + N->>S: New nodes available + S->>A: Bind remaining Pods +``` + +## 7. Scale-out is not instantaneous + +End-to-end added capacity: + +```text +Request/queue measurement ++ metric publication/scrape ++ HPA calculation ++ Pod object creation ++ scheduler wait ++ optional VM/node provisioning ++ image pull and container startup ++ readiness and cloud NEG/backend health += time before additional traffic is safely handled +``` + +For a predictable auction closing period, pre-scale ahead of known demand and maintain reserve capacity across zones. Reactive HPA is part of the system, not the entire capacity plan. + +## 8. Downstream amplification and feedback safety + +More Pods can create: +- More Cloud SQL connections and lock contention. +- More NAT source/connection churn. +- More queue consumers increasing API throttle/rate-limit errors. +- More egress spend. +- More startup load from image pulls and cache misses. + +Avoid an unbounded retry storm: use bounded concurrency, request deadlines, exponential backoff with jitter and business idempotency. Monitor **successful transactions per second**, not just requests accepted or Pod count. + +## 9. Troubleshoot CPU throttling and memory + +On a permitted Standard Linux worker/container context: + +```bash +cat /sys/fs/cgroup/cpu.max +cat /sys/fs/cgroup/cpu.stat +cat /sys/fs/cgroup/memory.current +cat /sys/fs/cgroup/memory.max +cat /sys/fs/cgroup/memory.events +cat /proc/pressure/cpu +``` + +Paths are cgroup-v2 oriented and may vary by node OS/runtime. Privileged node access is limited or unavailable in Autopilot. Use Cloud Monitoring and runtime metrics when direct cgroup reading isn't available. + +Kubernetes event inspection: + +```bash +kubectl -n marketplace get pods -o wide +kubectl -n marketplace describe pod POD +kubectl -n marketplace logs POD --previous +kubectl -n marketplace get events --sort-by=.lastTimestamp +``` + +**OOMKilled** is different from **Evicted**. A container may exceed its own memory limit even if the node has free memory. `kubectl top` averages may not capture a short transient peak. + +## 10. Hypothetical incident: Pods scale while checkout gets slower + +Observed: +- Requests increase 4x. +- HPA expands 8 to 25 ready replicas. +- Database connections rise from 250 to 1,200. +- Cloud SQL max connections approached; request queue time jumps. +- New Pods increase pressure on the same database. + +Response: +1. Confirm p99/error impact by endpoint and revision. +2. Trace waiting time to DB connection acquisition/transaction locks. +3. Cap connection pool per Pod and global concurrency. +4. Evaluate whether scaling should be limited by backend capacity rather than CPU alone. +5. Pre-scale safely, warm caches, preserve backpressure and test with modeled peak traffic. +6. Verify actual completed business operations and error-budget recovery. + +## Spoken interview answer + +> “GKE autoscaling involves three loops: workload replicas, per-Pod resource sizing and available nodes. HPA computes desired replicas from target metrics, but doesn't create node capacity. VPA changes sizing recommendations, and GKE autoscaler responds to scheduling needs within quota/IP/capacity limits. I'd measure throttling, memory/OOM and startup latency, and I'd pre-scale for predictable demand. Most importantly I'd check databases and dependent APIs because making more Pods can worsen the bottleneck. Successful scale-out means sustainable customer throughput and latency, not just a larger replica count.” + +**References:** [Kubernetes HPA](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) · [GKE cluster autoscaler](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler) · [Kubernetes resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) · [KEDA](https://keda.sh/docs/) + +[Previous](./q6) · [Next: GKE upgrades and production incidents](./q8) diff --git a/docs/guides/gke/q8.md b/docs/guides/gke/q8.md new file mode 100644 index 0000000..84dd459 --- /dev/null +++ b/docs/guides/gke/q8.md @@ -0,0 +1,236 @@ +# Q8. GKE Upgrades, Observability, Incident Response and Disaster Recovery + +> **Interviewer:** You operate production GKE clusters supporting a marketplace. Design safe Kubernetes/GKE upgrades and maintenance. Explain release channels, node upgrades, PDBs, admission/CSI/CNI compatibility, logging and monitoring. Then diagnose a cluster-wide incident and discuss regional disaster recovery. + +## 1. A GKE upgrade changes several systems + +Upgrading isn't only changing an API version. Production depends on: + +```mermaid +flowchart TD + CHANGE["GKE upgrade"] --> CP["Managed control plane and APIs"] + CHANGE --> N["Worker kubelet, OS, container runtime"] + N --> CNI["Network dataplane and policy"] + N --> CSI["CSI storage mounting"] + CP --> CRD["Admission webhooks, controllers and CRDs"] + CRD --> APP["Application workloads"] + CSI --> APP + CNI --> APP + APP --> SLO["User-facing success/latency/error budget"] +``` + +A cluster upgrade can break an obsolete Kubernetes API, a custom webhook schema, a CSI driver/node dependency or application scheduling topology. A control plane that reports healthy does not prove image pulls, new Pods or real user requests will succeed. + +## 2. Release channels and lifecycle + +GKE release channels include **Rapid, Regular, Stable and Extended**, each balancing earlier availability against rollout maturity and supported lifecycle. GKE automatically upgrades clusters over time under its documented policy; a Standard cluster not enrolled in a release channel is not a way to freeze upgrades forever, and “No channel” is being deprecated for current lifecycle schedules. + +- Choose channels based on organization's compatibility and maintenance policy. +- Use maintenance windows and appropriately scoped exclusions for required testing or planned freezes, without assuming indefinite postponement. +- Test early patches and deprecations in representative nonproduction. +- Review GKE release notes and versions supported in each channel. +- Keep versions in supported skew for control plane, kubelet and add-ons. +- Emergency security updates and end-of-support rules can override planned maintenance assumptions. + +The exact availability, exclusions, auto-upgrade target and supported versions change over time. See [GKE release channels](https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels) and [version lifecycle](https://cloud.google.com/kubernetes-engine/docs/concepts/versioning). + +## 3. Upgrade stages and canary capacity + +```mermaid +sequenceDiagram + participant O as Platform operator + participant G as GKE cluster manager + participant API as Kubernetes API + participant N as Node pools + participant LB as Backend health + O->>G: Review versions + maintenance policy + O->>API: Validate deprecated APIs and controllers + G->>API: Managed control-plane upgrade + G->>N: Start node pool surge/blue-green upgrade + N->>API: Cordon/drain eligible old nodes + API->>API: Evict within PDB and controller constraints + N->>N: Boot new nodes, kubelet/runtime + N->>LB: New Pods become healthy/backends ready + O->>O: Monitor SLO, events, cloud operations +``` + +This is a conceptual GKE maintenance sequence; managed auto-upgrade scheduling, surge/blue-green support and sequencing depend on cluster mode/pool settings. Control plane upgrades are distinct from node pool upgrades. Some node replacement is managed under GKE upgrade policy without an operator-issued drain command. + +### Surge and blue-green + +**Surge upgrades:** Add replacement node capacity while draining/removing old capacity according to configured `maxSurge` and `maxUnavailable` policy for supported pools. + +**Blue-green node upgrades:** Create a new pool-like replacement generation, transition workloads and retain old capacity during a configurable soak period under supported configurations. + +Both require quotas, secondary/primary IP space, capacity and working readiness/draining. A lack of surge capacity can delay an otherwise supported upgrade. + +## 4. PDB, eviction and graceful termination + +```yaml +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: bidding-api + namespace: marketplace +spec: + minAvailable: 4 + selector: + matchLabels: { app: bidding-api } +``` + +This controls permitted voluntary evictions through the Eviction API. It doesn't prevent node power failure, involuntary loss, all direct deletions or Deployment rolling-update unavailability, which uses Deployment strategy. + +A PDB requiring `minAvailable: 4` with only four healthy replicas can block planned drains; that's working as configured, not necessarily a Kubernetes bug. Increase safe replica capacity or plan maintenance. + +### Pod termination sequence + +```mermaid +sequenceDiagram + participant API as API server + participant EP as Endpoint controller / cloud reconciler + participant K as kubelet + participant APP as Application + API-->>EP: Pod termination observed + API-->>K: Pod deletion/grace deadline + K->>APP: preStop hook if configured + K->>APP: Send termination signal + APP->>APP: Stop accepting + drain sessions + EP->>EP: Remove/mark backend from traffic + K->>APP: SIGKILL if deadline expires +``` + +Endpoint withdrawal and hook execution are asynchronous. A hard-coded 5-second sleep is not proof that Cloud Load Balancer endpoint propagation completed. TerminationGracePeriod, app SIGTERM behavior, LB connection-draining settings and session lengths must be designed together. + +## 5. Upgrade preparation checklist + +1. Inventory GKE versions, release channel, control-plane endpoint and node pool OS/runtime. +2. Read target GKE/Kubernetes release notes and removed/deprecated APIs. +3. Audit CRDs, admission policies/webhooks, CNI, CSI, gateways/ingress and security agents. +4. Verify ServiceAccounts/IAM, node service account, Artifact Registry access and policy changes. +5. Check node, CPU/GPU and **Pod IP secondary range** headroom for surge replacements. +6. Review PDBs, readiness gates, replicas and zone spread. +7. Test application and cluster add-ons in a representative staging cluster/node pool. +8. Canary node pool or selected workload class, observe errors/p99/latency and backend health. +9. Document mitigation and compatible forward-fix strategy, not an assumed control-plane downgrade. +10. Confirm post-upgrade API and workload operations through real synthetic business requests. + +```bash +gcloud container clusters list --project=PROJECT_ID +gcloud container clusters describe CLUSTER --location=europe-west1 --project=PROJECT_ID +gcloud container node-pools list --cluster=CLUSTER --location=europe-west1 --project=PROJECT_ID +gcloud container operations list --location=europe-west1 --project=PROJECT_ID +kubectl get nodes -o wide +kubectl get pdb -A +kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations +kubectl get crd +``` + +The exact supported release channel maintenance commands/fields should be validated with the current Cloud SDK and provider documentation, not copied from an old playbook. + +## 6. Observability for GKE + +Start from service SLIs (request success, duration, correctness), not “all nodes Ready.” + +Correlate: +- Cloud Monitoring GKE node/pod/container and autoscaling metrics. +- Kubernetes events, resource conditions and container restart reasons. +- Cloud Logging: workload, kubelet/system, controller and audit entries where available. +- Network policy logs on Dataplane V2. +- GKE operation logs for cluster/node changes and failures. +- Google LB backend health/NEG and application traces. +- Database/service dependency latency and errors. + +```bash +kubectl get events -A --sort-by=.lastTimestamp | tail -50 +kubectl get pods -A -o wide +kubectl top nodes +kubectl -n marketplace logs deploy/catalog-api --previous +gcloud logging read 'severity>=ERROR' --project=PROJECT_ID --limit=50 +``` + +Logs may be filtered, sampled or absent depending on configured collection. `--previous` works when a previously terminated container instance's logs remain accessible; it is not a historical log archive. + +## 7. Production troubleshooting hierarchy + +```mermaid +flowchart TD + START["Service unavailable"] --> POD{"Pod created?"} + POD -->|No| CONTROL["Controllers, admission, quota"] + POD -->|Yes| BOUND{"Scheduled?"} + BOUND -->|No| SCHED["Requests, taints, topology, pools"] + BOUND -->|Yes| RUN{"Container running?"} + RUN -->|No| RUNTIME["Image, runtime, CNI, CSI"] + RUN -->|Yes| READY{"Pod ready?"} + READY -->|No| PROBE["Probes, app startup, NEG gate"] + READY -->|Yes| CLIENT{"Client request successful?"} + CLIENT -->|No| NETWORK["Service, eBPF, Gateway, LB, dependency"] + CLIENT -->|Yes| SLO["Measure p99, errors and correctness"] +``` + +### Diagnostic map + +| Symptom | Investigate | +| --- | --- | +| API kubectl timeout | Endpoint/DNS/authorized networks/API availability | +| Forbidden | Identity, IAM to cluster API and Kubernetes RBAC | +| Pending | Scheduler resources, taints, topology, PVC binding | +| ContainerCreating | CSI mounts, CNI sandbox, image setup | +| ImagePullBackOff | Artifact Registry IAM, image location/digest, network | +| CrashLoopBackOff | Process exit, probe, missing config, OOM | +| Ready but 502 | NEG/backend health, LB routing/health, app upstream | +| New nodes fail | Quota, zonal capacity, node SA, subnet/Pod CIDRs | +| Network only one namespace | NetworkPolicy, DNS, service mesh policy | +| Stateful Pod Multi-Attach | Old writer fencing, VolumeAttachment, zone | +| Rolling upgrade blocked | PDB, surge capacity, health gates, constraints | + +## 8. Hypothetical incident: upgrade healthy, new Pods broken + +Situation: +- GKE managed control plane upgraded. +- Existing Pods serve traffic. +- Newly scheduled Pods report `CreateContainerConfigError` or admission rejection. +- Helm/GitOps release fails, but no API-server outage exists. + +Investigation: +1. Compare failure stage: API admission versus kubelet container config. +2. Inspect Pod events, missing Secret/ConfigMap, removed API fields or webhook incompatibility. +3. Check CRD schemas/controller versions and deployment revisions. +4. Confirm node pool images, registry auth and kubelet versions if container creation begins but fails. +5. Mitigate by restoring compatible webhook/controller or config, rather than restarting all healthy application Pods. +6. Validate ordinary create/scale/rollout operations and end-to-end transaction success. + +## 9. Regional disaster recovery for GKE + +A regional GKE cluster is not a cross-region failover service. Disaster recovery may involve: +- Secondary cluster and its node/IP capacity. +- IAM/WIF principal design for both projects/clusters. +- Container registry availability. +- DNS/LB multi-cluster routing and backend health. +- Kubernetes state/application manifests and Secrets recovery. +- CSI and database data restoration/replication. +- Preventing simultaneous active writers or duplicate payment processing. + +```mermaid +flowchart LR + USERS["Global clients"] --> LB["Global HTTPS frontend"] + LB --> PRI["Primary region GKE"] + LB -.-> SEC["Secondary region GKE"] + PRI --> DB["Transactional primary database"] + DB -.-> REPL["DR copy / restore"] + SEC -.-> REPL + PLAN["Failover runbook with RPO/RTO and fencing"] -.-> LB +``` + +Dashed lines mean planned standby/replication paths, not guaranteed automatic database failover. Before shifting writes, verify current data state and leader/writer fencing. Practice restore/failover and measure actual RTO/RPO. + +## 10. Senior/lead trade-offs + +Choose release channel and maintenance policy based on support and compliance, not avoidance of every upgrade. Maintain compatibility tests and declarative manifests. Reserve enough node and IP capacity for maintenance. Protect app availability with correct readiness/draining and PDBs but understand what they do not guarantee. Make cluster debugging a lifecycle-boundary decision rather than a broad “restart things” response. + +## Spoken interview answer + +> “I'd approach GKE upgrades as a stack: managed API compatibility, node OS/runtime, CNI/CSI, admission, ingress and application contracts. I'd use a supported release channel, maintenance controls and a staging/canary upgrade to validate actual Pod startup, backend health and transactions. For worker changes I'd check PDBs, surge and Pod IP capacity, zone spread and drain time. If an incident occurs I'd trace the first failed transition from API admission through scheduler, kubelet, networking, readiness and client response. A regional cluster alone doesn't solve regional data recovery; that needs separate RPO/RTO, replicated data, identity, routing and fencing.” + +**References:** [GKE release channels](https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels) · [GKE upgrades](https://cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster) · [Kubernetes disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) · [GKE operations](https://cloud.google.com/kubernetes-engine/docs/how-to/logging) + +[Previous](./q7) · [Module index](./) · [Challenge questions](./follow-ups) diff --git a/docs/guides/index.md b/docs/guides/index.md index 88f4dc4..f903458 100644 --- a/docs/guides/index.md +++ b/docs/guides/index.md @@ -22,8 +22,16 @@ Twelve in-depth chapters covering Kubernetes control-plane internals, etcd, sche Twelve in-depth chapters covering Terraform execution, state, modules, AWS/GCP infrastructure, cloud identity, import/refactoring, drift recovery, security and CI/CD. Includes [45 senior follow-ups](/guides/terraform/follow-ups) and [eight labs](/guides/terraform/labs). +### [Google Cloud Fundamentals and Production Engineering · Module 5](/guides/gcp/) + +Eight comprehensive chapters covering GCP hierarchy, identity and federation, Shared VPC, NAT/DNS/load balancing, Compute Engine, storage/data, observability, and high-availability architecture. Includes [eight practical labs](/guides/gcp/labs) and [40 senior follow-ups](/guides/gcp/follow-ups). + +### [GKE Architecture, Networking and Production Operations · Module 6](/guides/gke/) + +Eight in-depth chapters on GKE control plane, Standard/Autopilot, node scheduling, VPC-native networking/eBPF, Ingress/Gateway/NEGs, workload identity, CSI, autoscaling, upgrades and incident recovery. Includes [eight labs](/guides/gke/labs) and [40 difficult follow-ups](/guides/gke/follow-ups). + ## Planned -GCP, GKE operations, reliability, CI/CD and platform engineering, and Python data processing. +Reliability, CI/CD and platform engineering, and Python data processing. [About the author](/about)