From 37d81342c6800dd69987aa551cdb5f8dec9aa583 Mon Sep 17 00:00:00 2001 From: EDWIN PHILIP <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 18:58:03 +0530 Subject: [PATCH 01/42] docs(python): restore consolidated index.md on updated main --- docs/guides/python-automation/index.md | 49 ++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 docs/guides/python-automation/index.md diff --git a/docs/guides/python-automation/index.md b/docs/guides/python-automation/index.md new file mode 100644 index 0000000..c1e428f --- /dev/null +++ b/docs/guides/python-automation/index.md @@ -0,0 +1,49 @@ +# Module 9. Python and Shell Automation for Platform Engineers + +Practical, intermediate-level coding skills for working with API responses, logs, CSV exports, timestamps and infrastructure inventories. The focus is straightforward, reliable **data processing and automation**, not framework development or competitive algorithm tricks. + +::: info Public educational guide +All services, endpoints, inputs and incidents are synthetic. Commands and snippets must be adapted and tested before real operational use. No exercise assumes access to private systems or a particular coding platform. +::: + +## Prerequisites + +Basic Python syntax and functions, shell execution, Linux files/pipes and simple HTTP concepts. You do not need advanced algorithms or application framework experience. + +## Learning objectives + +- Choose lists, dictionaries, sets and Counters appropriately, with basic complexity awareness. +- Validate nested JSON, parse text logs and aggregate CSV safely. +- Handle UTC-aware timestamp windows and deduplicate by meaningful record identity. +- Fetch paginated APIs with clear error, timeout and retry semantics. +- Build small CLI utilities with useful tests and nonzero failure exit codes. +- Use Bash, `jq`, `awk`, `grep`, `sort` and `uniq` effectively, and know when Python is a better fit. + +## Six consolidated chapters + +1. [Core Python: data structures, sorting and deduplication](./q1) +2. [Parsing JSON, logs and CSV](./q2) +3. [Timestamps, generators, batching and concurrency](./q3) +4. [HTTP APIs, cursors, error handling and retries](./q4) +5. [Complete CLI automation: validation, testing and results](./q5) +6. [Linux shell processing: Bash, jq, awk, sed and pipelines](./q6) + +## Practice + +- [Six practical data-processing labs](./labs) +- [36 challenging questions with full model answers](./follow-ups) + +## Study sequence + +Begin with Python structures and parsing, then handle time and remote APIs. Use the CLI chapter to combine those fundamentals into a maintainable tool. Compare the same processing tasks with shell in the final chapter, and complete the labs with small fixtures and independent expected outputs. Use [Module 10](/guides/ai-assisted-engineering/) for a separate, deeper discussion of AI-assisted problem solving and Plan–Build–Review. + +## Language and tool references + +- [Python standard library](https://docs.python.org/3/library/) +- [Python collections](https://docs.python.org/3/library/collections.html) +- [Python json](https://docs.python.org/3/library/json.html) +- [Python csv](https://docs.python.org/3/library/csv.html) +- [Bash manual](https://www.gnu.org/software/bash/manual/) +- [jq manual](https://jqlang.github.io/jq/manual/) + +[← All guides](/guides/) · [Start chapter 1](./q1) From 23f87215f946ef876c6e0eeef15a06abb85f5095 Mon Sep 17 00:00:00 2001 From: EDWIN PHILIP <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 18:58:07 +0530 Subject: [PATCH 02/42] docs(python): restore consolidated q1.md on updated main --- docs/guides/python-automation/q1.md | 193 ++++++++++++++++++++++++++++ 1 file changed, 193 insertions(+) create mode 100644 docs/guides/python-automation/q1.md diff --git a/docs/guides/python-automation/q1.md b/docs/guides/python-automation/q1.md new file mode 100644 index 0000000..e12149d --- /dev/null +++ b/docs/guides/python-automation/q1.md @@ -0,0 +1,193 @@ +# Q1. Core Python, Collections, Sorting and Deduplication + +This chapter covers the Python concepts used most often in operational scripts. Work through both examples, then practice explaining record identity, grouping, complexity, input validation and deterministic output. + +## Part A. Python Fundamentals for Infrastructure Automation + +> **Interviewer:** You receive a list of service-health events. Which Python data structures would you use to count unhealthy services, remove duplicate hosts and report the worst offenders? Explain your approach and its complexity. + +### Start with the data contract + +Platform scripts typically transform API responses, log records or configuration, rather than invent a new algorithm. Clarify the schema, input size, missing fields, duplicates, ordering, and intended output before writing code. + +Use Python's standard library wherever reasonable: + +| Structure | Good for | Typical operations | +| --- | --- | --- | +| `list` | Ordered records, repeated values | Append amortized O(1), scan O(n) | +| `dict` | Lookups and grouping by key | Expected lookup/update O(1) | +| `set` | Membership and deduplication | Expected membership/add O(1) | +| `tuple` | Small immutable records, multiple returns | Fixed-size structure | +| `Counter` | Counts per key | Expected O(n) over input | +| `defaultdict` | Groups or nested buckets | Expected O(n) over input | + +These are *average/expected* costs for hashing, not a guarantee for adversarial hashing or every operation. Sorting n records costs O(n log n). Memory complexity matters as much as time when input grows. + +```mermaid +flowchart LR + IN["List of health records"] --> VALID["Validate required fields"] + VALID --> COUNT["Counter: failures per service"] + VALID --> HOST["Set: distinct affected hosts"] + COUNT --> ORDER["Sort counts descending"] + HOST --> OUT["Concise report"] + ORDER --> OUT +``` + +### Working solution + +```python +from collections import Counter + +events = [ + {"service": "api", "host": "node-a", "status": "error"}, + {"service": "api", "host": "node-a", "status": "error"}, + {"service": "worker", "host": "node-b", "status": "ok"}, + {"service": "checkout", "host": "node-c", "status": "error"}, +] + +def summarize_failures(records: list[dict]) -> tuple[list[tuple[str, int]], int]: + failures = Counter() + hosts = set() + + for record in records: + if not isinstance(record, dict): + raise ValueError("each event must be an object") + service = record.get("service") + host = record.get("host") + status = record.get("status") + if not all(isinstance(v, str) and v for v in (service, host, status)): + raise ValueError("service, host and status must be non-empty strings") + if status == "error": + failures[service] += 1 + hosts.add(host) + + ranked = sorted(failures.items(), key=lambda item: (-item[1], item[0])) + return ranked, len(hosts) + +print(summarize_failures(events)) +## ([('api', 2), ('checkout', 1)], 2) +``` + +#### Reasoning and edge cases + +The script counts *events*, not unique failures. The duplicate record for node-a produces two failures while the set counts node-a once. If duplicates represent retry delivery and should not count twice, require an event ID and deduplicate before counting. A host count is across all failing services; if the question asks for per-service distinct hosts, use a dictionary of sets instead. + +Time is O(n + k log k), where n is the number of records and k the number of distinct failing services. Auxiliary space is O(k + h), for distinct services and hosts. + +#### Interview follow-ups + +- **Why not a list for membership?** A list scan is O(n), while a set normally gives expected O(1) membership checks. +- **Is `dict` insertion order guaranteed?** Modern Python preserves insertion order, but sorting is required for count ranking. +- **What if `status` has new values?** Define acceptable statuses; either reject unknown values or explicitly classify them, rather than silently treating all unknown states as healthy. +- **What if input is a generator?** The `for` loop works without materializing the entire iterable, although the return annotation could be broadened to `Iterable[dict]`. + +### Practical test cases + +```python +assert summarize_failures([]) == ([], 0) +assert summarize_failures([{"service":"s","host":"h","status":"ok"}]) == ([], 0) +assert summarize_failures([{"service":"s","host":"h","status":"error"}]) == ([("s", 1)], 1) +``` + +### Spoken answer + +> “I would first confirm whether failures are counted per event or per unique incident. A Counter handles frequency, and a set tracks distinct hosts without repeated linear scans. I'd validate the record shape so a malformed response doesn't produce misleading statistics. The scan is linear and only sorting the distinct services adds a logarithmic factor. I'd test empty input, duplicates and unknown statuses.” + +**Reference:** [Python collections](https://docs.python.org/3/library/collections.html) + + +--- + +## Part B. Filtering, Grouping, Sorting and Deduplication + +> **Interviewer:** A monitoring API sends duplicate incident updates. For each service, keep the latest incident status and return the services with the highest number of still-open incidents. How do you make the result deterministic? + +### Define record identity separately from arrival order + +If records have an `incident_id`, a later update should replace an earlier update for the **same incident**. A `service` name alone is not a unique incident ID. If timestamps tie, define a deterministic tie-breaking rule rather than letting API page order decide. + +This example accepts only valid incident updates. The latest event is selected by timestamp, with status as a tie-breaker for this exercise. Real event sources should instead provide a monotonic revision or event sequence number; two conflicting updates with the same timestamp may be genuinely ambiguous. + +```mermaid +flowchart TD + REC["Incident events"] --> VALID["Validate ID, service, state, timestamp"] + VALID --> LATEST["Dictionary: latest event per incident"] + LATEST --> OPEN["Filter incidents currently open"] + OPEN --> GROUP["Counter by service"] + GROUP --> RANK["Rank by count and service name"] +``` + +### Working solution + +```python +from collections import Counter +from datetime import datetime, timezone + +def summarize_open_incidents(records: list[dict]) -> list[tuple[str, int]]: + latest = {} + + for index, record in enumerate(records): + if not isinstance(record, dict): + raise ValueError(f"record {index} must be an object") + ident = record.get("incident_id") + service = record.get("service") + state = record.get("state") + stamp = record.get("updated_at") + if not isinstance(ident, str) or not ident: + raise ValueError("missing incident_id") + if not isinstance(service, str) or not service: + raise ValueError("missing service") + if state not in ("open", "resolved"): + raise ValueError("invalid incident state") + if not isinstance(stamp, str): + raise ValueError("invalid timestamp") + try: + time = datetime.fromisoformat(stamp.replace("Z", "+00:00")) + except ValueError as exc: + raise ValueError("invalid timestamp") from exc + if time.tzinfo is None or time.utcoffset() is None: + raise ValueError("timestamp must contain a timezone") + order = (time.astimezone(timezone.utc), state) + current = latest.get(ident) + if current is None or order > current[0]: + latest[ident] = (order, service, state) + + counts = Counter( + service for _, service, state in latest.values() if state == "open" + ) + return sorted(counts.items(), key=lambda item: (-item[1], item[0])) + +events = [ + {"incident_id":"i1","service":"api","state":"open", + "updated_at":"2026-01-01T10:00:00Z"}, + {"incident_id":"i2","service":"worker","state":"open", + "updated_at":"2026-01-01T10:02:00Z"}, + {"incident_id":"i1","service":"api","state":"resolved", + "updated_at":"2026-01-01T10:05:00Z"}, +] +print(summarize_open_incidents(events)) +## [('worker', 1)] +``` + +#### Why use a dictionary? + +The dictionary stores only one candidate state per incident, so duplicate updates don't count twice. After the pass, group the currently-open latest states. This avoids repeated O(n) scanning per incident. For n updates and i incidents, the scan is expected O(n), with O(i) memory; sorting s services is O(s log s). + +#### Validation and semantics + +If the API permits a service reassignment, which service owns the incident for historical reporting? For a current-state dashboard, the last event's service is suitable; for chargeback/audit reports the service identity may need to be immutable and validation should reject conflicting owners. + +#### Edge cases + +- Empty records returns an empty list. +- Duplicate identical records do not change counts. +- Out-of-order records still work because selection compares timestamps. +- An older `open` event cannot reopen a newer `resolved` incident. +- Conflicting equal-timestamp events are not reliably resolved by alphabetic state ordering. A real API should supply revision numbers. + +### Spoken answer + +> “I'd identify an incident by its stable ID, not its service. I would keep the latest update for each incident, filter those still open and count them per service. A dictionary makes deduplication efficient and sorting makes output stable. I'd ask what defines latest when timestamps tie and whether incident ownership can change, because those details determine correctness more than the Python syntax.” + +**Reference:** [Python sorting how-to](https://docs.python.org/3/howto/sorting.html) + From cd0feb5d986dd259bac14aa7bc65c3f2911b0d03 Mon Sep 17 00:00:00 2001 From: EDWIN PHILIP <38521046+EdwinPhilip@users.noreply.github.com> Date: Sun, 25 Oct 2026 18:58:10 +0530 Subject: [PATCH 03/42] docs(python): restore consolidated q2.md on updated main --- docs/guides/python-automation/q2.md | 338 ++++++++++++++++++++++++++++ 1 file changed, 338 insertions(+) create mode 100644 docs/guides/python-automation/q2.md diff --git a/docs/guides/python-automation/q2.md b/docs/guides/python-automation/q2.md new file mode 100644 index 0000000..a24e6db --- /dev/null +++ b/docs/guides/python-automation/q2.md @@ -0,0 +1,338 @@ +# Q2. Real-World JSON, Logs and CSV Parsing + +This chapter combines structured API validation, unstructured log parsing and streaming CSV aggregation. Compare failure policies and choose the simplest parser for each input contract. Retain complete exercise implementations instead of writing one universal parser. + +## Part A. Parsing JSON and Validating Nested API Responses + +> **Interviewer:** An API returns a JSON document containing Kubernetes-like workloads. Print every unhealthy workload by namespace. How will you handle missing fields, unexpected types, invalid JSON and partial API failures? + +### A parser has two responsibilities + +**Syntax parsing** converts bytes/text into Python objects. **Schema validation** checks that objects have the expected meaning. Successful `json.loads()` does not imply the response contains `items`, that `items` is a list, or that `readyReplicas` is a useful integer. + +The standard library returns Python `dict`, `list`, `str`, `int`, `float`, `bool` and `None`. JSON booleans become Python booleans. Python `bool` is a subclass of `int`, so use `type(value) is int` when rejecting boolean replica counts. + +```mermaid +sequenceDiagram + participant A as HTTP/API response + participant P as JSON decoder + participant V as Schema validation + participant T as Transformation + A->>P: Response body + alt Invalid JSON + P-->>A: Decode error with location + else Parsed + P->>V: Python containers + V->>T: Accepted records only + T-->>A: Deterministic report + end +``` + +### Reference solution + +```python +import json + +sample = """{ + "items": [ + {"metadata": {"name": "api", "namespace": "store"}, + "spec": {"replicas": 3}, "status": {"readyReplicas": 2}}, + {"metadata": {"name": "worker", "namespace": "jobs"}, + "spec": {"replicas": 1}, "status": {"readyReplicas": 1}} + ] +}""" + +def find_unready(payload: str) -> list[str]: + try: + document = json.loads(payload) + except json.JSONDecodeError as exc: + raise ValueError(f"invalid JSON at position {exc.pos}") from exc + + if not isinstance(document, dict) or not isinstance(document.get("items"), list): + raise ValueError("expected an object containing an items list") + + result = [] + for index, item in enumerate(document["items"]): + if not isinstance(item, dict): + raise ValueError(f"item {index} must be an object") + metadata = item.get("metadata") + spec = item.get("spec") + status = item.get("status", {}) + if not isinstance(metadata, dict) or not isinstance(spec, dict): + raise ValueError(f"item {index} lacks metadata/spec objects") + if not isinstance(status, dict): + raise ValueError(f"item {index} has invalid status") + + name = metadata.get("name") + namespace = metadata.get("namespace", "default") + desired = spec.get("replicas", 1) + ready = status.get("readyReplicas", 0) + + if not isinstance(name, str) or not name: + raise ValueError(f"item {index} lacks a workload name") + if not isinstance(namespace, str) or not namespace: + raise ValueError(f"item {index} has invalid namespace") + if type(desired) is not int or type(ready) is not int or min(desired, ready) < 0: + raise ValueError(f"item {index} has invalid replica counts") + + if ready < desired: + result.append(f"{namespace}/{name}: {ready}/{desired} ready") + + return sorted(result) + +print(find_unready(sample)) +## ['store/api: 2/3 ready'] +``` + +This is a **simplified workload report**, not a universal definition of Kubernetes deployment health. During rollouts, fields such as available replicas, observed generation, conditions and old ReplicaSets matter. A real client also has to account for authorization errors, pagination and resource-version consistency. + +### Strict versus lenient parsing + +Strict failure is appropriate for a small authoritative config or security-sensitive decision. For a noisy batch log with thousands of independent records, collecting validation errors and continuing may be preferable. State the behavior explicitly and return errors alongside successful records instead of quietly dropping malformed input. + +#### Common mistakes + +- `item["metadata"]["name"]` without validating object shape raises confusing exceptions. +- `str(item.get("name"))` turns missing data into the literal name `"None"`. +- `except Exception: pass` hides provider outages and malformed records. +- Treating `status.readyReplicas` missing as healthy instead of zero can mask unready workloads. + +### Tests and reasoning + +```python +assert find_unready('{"items": []}') == [] +assert find_unready(sample) == ["store/api: 2/3 ready"] +try: + find_unready('{"items": "not-a-list"}') +except ValueError: + pass +else: + raise AssertionError("expected validation failure") +``` + +Time is O(n + k log k) because the report sorts k unready workloads after inspecting n items. Space includes the decoded JSON tree; for very large JSON use paged APIs or newline-delimited JSON where available rather than forcing the whole payload into memory. + +### Spoken answer + +> “I would decode JSON, then validate its expected structure before reading nested keys. I wouldn't assume the API always returns the same shape or that a missing readiness field means healthy. I'd define whether malformed records stop the operation or are collected separately. Finally, I'd test valid, empty, missing, malformed and partially unhealthy data. For large responses I'd handle pagination and memory limits.” + +**Reference:** [Python json](https://docs.python.org/3/library/json.html) + + +--- + +## Part B. Log Parsing with Strings and Regular Expressions + +> **Interviewer:** Given a large application log, count 5xx responses per service. The file contains invalid lines, unusual paths and missing fields. How would you solve it without crashing or loading the entire file? + +### Prefer structured logs when available + +JSON Lines (one JSON object per line) is easier to validate than free-form text. When the input is legacy text, establish a **documented format** before using regular expressions. Do not write a giant regex that guesses every conceivable log layout. + +Sample log contract: + +```text +2026-10-25T10:00:00Z service=api status=200 path=/items +2026-10-25T10:00:01Z service=checkout status=503 path=/pay +2026-10-25T10:00:02Z service=checkout status=502 path=/pay +bad line +``` + +```mermaid +flowchart LR + F["Open UTF-8 log"] --> LINE["One line at a time"] + LINE --> MATCH{"Matches documented format?"} + MATCH -->|No| BAD["Count malformed lines"] + MATCH -->|Yes| VALID["Parse status and service"] + VALID --> COUNT["Aggregate 500–599"] + COUNT --> REPORT["Sorted report + invalid count"] + BAD --> REPORT +``` + +### Working solution with bounded memory + +```python +import re +from collections import Counter +from io import StringIO +from typing import TextIO + +PATTERN = re.compile( + r"^(?P