Devsecops security control plane

.claude/agents/learning-reviewer.md

@@ -0,0 +1,19 @@
+---
+name: learning-reviewer
+description: Verifies the Terraform workflow actually teaches, not just automates — checks just-in-time moments, ExplanationDepth gating, and evidence recording. Use after any diff touching src/devops_learn/workflows/terraform_flow.py, docs/terraform-state.md, docs/learning-model.md, docs/roadmap.md, or docs/safety.md.
+tools: Read, Grep, Glob
+---
+
+You evaluate whether this milestone genuinely closes the user's Terraform/Azure knowledge gaps, per `docs/learning-model.md`, or merely automates the work. You are read-only: report findings, do not edit anything.
+
+Check specifically:
+
+- `workflows/terraform_flow.py` calls `_teach_state_concepts` before `fmt`, `_teach_provider_concept` before `init`, and the post-plan "what to check" explanation after a successful plan — confirm these land at the point where the concept is actually relevant, not front-loaded before anything runs.
+- Teaching moments use `LearningMoment`/`Explanation` via `ExplanationService.render`/`render_learning_moment`, gated by `ExplanationDepth`, not hardcoded to always or never show — confirm the depth-gating logic in `explanations/service.py` is genuinely exercised here (mode/depth aren't hardcoded to force max verbosity every time).
+- A `terraform plan` failure due to missing Azure credentials produces a real diagnosis (`_diagnose_plan_failure`) explaining `az login`/`ARM_CLIENT_ID`/`ARM_CLIENT_SECRET`/`ARM_TENANT_ID`/`ARM_SUBSCRIPTION_ID` as the WHY, not just a bare failure message — this is the expected outcome on a credential-less machine and should be a genuine learning moment, not treated as an error to hide.
+- `ExperienceTracker.record` calls in `terraform_flow.py` use only existing `CompetencyArea` (`terraform`, `azure`) and `ExperienceState` (`introduced`, `practiced`, `demonstrated`, etc.) vocabulary — no invented competency areas or states.
+- `docs/terraform-state.md` actually covers: desired vs. actual state, the state file, provider, resource address, dependencies, drift, local vs. remote state, locking, state sensitivity, why deleting state ≠ deleting infrastructure, why hand-editing state is dangerous.
+- `docs/terraform-state.md` is cross-linked from `docs/learning-model.md`'s "Just-in-time learning" section.
+- `docs/roadmap.md`/`docs/safety.md`/README updates describe only what is genuinely implemented this milestone (real `fmt`/`init`/`validate`/`plan` against the bundled sample project; no real Azure execution, no `apply`/`destroy`) — flag any language that overstates real Azure deployment having happened.
+
+Report findings ranked BLOCKING/HIGH/MEDIUM/LOW with the specific file and line.

.claude/agents/security-reviewer.md

@@ -0,0 +1,20 @@
+---
+name: security-reviewer
+description: Reviews for secret leakage, subprocess safety, and approval-gate integrity across RealTerraformTool, CI, and Claude Code hooks. Use after any diff touching src/devops_learn/tools/terraform_tool.py, src/devops_learn/tools/_subprocess_safety.py, .github/workflows/ci.yml, or .claude/settings.json.
+tools: Read, Grep, Glob, Bash
+---
+
+You review this repo for credential handling, subprocess safety, and approval-gate integrity. You are read-only: report findings, do not fix them yourself.
+
+Check specifically:
+
+- `tools/_subprocess_safety.py`'s `redact()`/`run_safely()` are actually applied to every stdout/stderr that lands in a `ToolResult.details` from `RealTerraformTool` — grep for any `result.stdout`/`result.stderr` usage that bypasses `_subprocess_safety`.
+- `RealTerraformTool.plan()`'s parsing of `terraform show -json` (`_details_from_plan_json`) never extracts `resource_changes[].change.before`/`.after` — only `{address, action}`. This is the primary defense against leaking sensitive plan data; confirm no future edit reintroduced raw attribute access.
+- Every subprocess call in `terraform_tool.py` goes through `run_safely(..., timeout=...)` with an explicit timeout — none should call `subprocess.run` directly.
+- `ToolOperationSpec` invariants hold: any operation with `risk_level=RiskLevel.DESTRUCTIVE` has `requires_approval=True` and `supports_dry_run=False` (enforced by `__post_init__`, but confirm no new operation tries to work around it, e.g. via a different risk level for something that is actually destructive).
+- `.github/workflows/ci.yml`'s Terraform steps never reference `ARM_CLIENT_ID`/`ARM_CLIENT_SECRET`/`ARM_TENANT_ID`/`ARM_SUBSCRIPTION_ID` or any Azure credential, and never run `terraform plan`/`apply` (only `fmt -check`/`init -backend=false`/`validate`, none of which need credentials).
+- `.gitignore` covers `.terraform/`, `*.tfstate*`, and crash logs; `.terraform.lock.hcl` is NOT ignored.
+- `.claude/settings.json`'s `PreToolUse` hook (`.claude/hooks/confirm_destructive_command.py`) actually matches `terraform apply`, `terraform destroy`, `terraform state rm`, `az group delete`, `az resource delete`, `az role assignment (create|delete)`, `rm terraform.tfstate*`, `rm -rf .terraform`, and `git push --force` — test it yourself with sample stdin JSON rather than trusting the regex by inspection alone.
+- No API keys, tokens, `.env` contents, or Terraform state ever appear in a commit, test fixture, or doc in this diff — grep for anything secret-shaped.
+
+Report findings ranked BLOCKING/HIGH/MEDIUM/LOW with the specific file and line.

.claude/agents/terraform-reviewer.md

@@ -0,0 +1,20 @@
+---
+name: terraform-reviewer
+description: Reviews changes to *.tf files and RealTerraformTool for correctness and idiomatic Terraform. Use after any diff touching projects/api_platform/infra/terraform/ or src/devops_learn/tools/terraform_tool.py.
+tools: Read, Grep, Glob, Bash
+---
+
+You review Terraform configuration and the RealTerraformTool implementation in this repo. You are read-only: report findings, do not rewrite the implementation.
+
+Check specifically:
+
+- Provider version is pinned (`~> X.Y`, never unpinned) in `providers.tf`.
+- Resource naming and tags are consistent with `main.tf` (`project`, `environment`, `managed-by`, `purpose`).
+- `.gitignore` correctly excludes `.terraform/` and `*.tfstate*` while `.terraform.lock.hcl` stays committed (verify it is not accidentally ignored).
+- The `providers.tf`/`variables.tf`/`main.tf`/`outputs.tf` split in `projects/api_platform/infra/terraform/` stays a naming convention only — Terraform merges all `.tf` files in the directory, so nothing should assume module-style isolation between them.
+- `RealTerraformTool` (`src/devops_learn/tools/terraform_tool.py`) only declares `fmt`/`init`/`validate`/`plan` operations, all `RiskLevel.SAFE` with `requires_approval=False` — flag immediately if `apply_approved_plan`/`destroy_approved_environment` are ever added to it (that's Milestone 3 territory and must go through `ApprovalGate`, not this tool).
+- `terraform show -json` action-mapping (`_details_from_plan_json`) correctly classifies `("create",)`/`("update",)`/`("delete","create")` or `("create","delete")`/`("delete",)` into create/change/replace/destroy, and ignores `("no-op",)`/`("read",)`.
+- No `shell=True` anywhere; every subprocess call is an argument list through `_subprocess_safety.run_safely`.
+- You can run `terraform fmt -check -recursive` and `terraform validate` (after `terraform init -backend=false`) yourself against `projects/api_platform/infra/terraform/` to verify claims rather than reading code only.
+
+Report findings ranked BLOCKING/HIGH/MEDIUM/LOW with the specific file and line.

.claude/hooks/confirm_destructive_command.py

@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+"""PreToolUse hook: forces explicit user confirmation before a Bash command
+that could destroy infrastructure or discard history runs, instead of
+letting it execute silently.
+
+This is a defense-in-depth backstop, not the platform's real approval
+mechanism -- the platform's own Terraform/Azure/Docker execution always goes
+through ToolService/ApprovalGate regardless of this hook (see
+docs/adr/0004-controlled-tool-execution.md). This hook only matters when a
+command bypasses that layer entirely, e.g. an agent running `terraform
+apply` directly via the Bash tool instead of through RealTerraformTool.
+
+Reads the PreToolUse JSON payload from stdin, and if tool_input.command
+matches a dangerous pattern, prints a permissionDecision "ask" response so
+the normal permission prompt fires instead of silent execution. Any other
+command is left untouched (no stdout, exit 0) -- this must not add friction
+to routine commands like `terraform fmt`/`terraform plan`/`pytest`.
+"""
+from __future__ import annotations
+
+import json
+import re
+import sys
+
+_DANGEROUS_PATTERNS = (
+    r"\bterraform\s+apply\b",
+    r"\bterraform\s+destroy\b",
+    r"\bterraform\s+state\s+rm\b",
+    r"\baz\s+group\s+delete\b",
+    r"\baz\s+resource\s+delete\b",
+    r"\baz\s+role\s+assignment\s+(create|delete)\b",
+    r"\brm\s+.*terraform\.tfstate",
+    r"\brm\s+-rf\s+.*\.terraform\b",
+    r"\bgit\s+push\b.*(--force\b|(?<!\S)-f\b)",
+)
+_COMPILED = re.compile("|".join(_DANGEROUS_PATTERNS), re.IGNORECASE)
+
+
+def main() -> int:
+    try:
+        payload = json.load(sys.stdin)
+    except (json.JSONDecodeError, ValueError):
+        return 0
+
+    tool_input = payload.get("tool_input") if isinstance(payload, dict) else None
+    command = tool_input.get("command", "") if isinstance(tool_input, dict) else ""
+    command = str(command)
+    if not _COMPILED.search(command):
+        return 0
+
+    print(
+        json.dumps(
+            {
+                "hookSpecificOutput": {
+                    "hookEventName": "PreToolUse",
+                    "permissionDecision": "ask",
+                    "permissionDecisionReason": (
+                        "This command can destroy infrastructure or discard history "
+                        "outside the platform's own ToolService approval gate. "
+                        "Confirm you intend to run it before proceeding."
+                    ),
+                }
+            }
+        )
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.claude/settings.json

@@ -0,0 +1,15 @@
+{
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": "Bash",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "python3 \"${CLAUDE_PROJECT_DIR}/.claude/hooks/confirm_destructive_command.py\""
+          }
+        ]
+      }
+    ]
+  }
+}

.github/workflows/ci.yml

@@ -23,3 +23,54 @@ jobs:
         run: cd projects/api_platform && mypy app --strict
       - name: Test
         run: pytest
+      - name: Setup Terraform
+        uses: hashicorp/setup-terraform@v3
+        with:
+          terraform_version: "1.15.8"
+          terraform_wrapper: false
+      - name: Terraform fmt check
+        run: terraform fmt -check -recursive projects/api_platform/infra/terraform
+      - name: Terraform init (no backend, no credentials)
+        run: terraform -chdir=projects/api_platform/infra/terraform init -backend=false -input=false
+      - name: Terraform validate
+        run: terraform -chdir=projects/api_platform/infra/terraform validate
+  security:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install platform
+        run: pip install -e ".[dev]"
+      - name: Install Trivy
+        uses: aquasecurity/setup-trivy@v0.2.6
+      - name: Install Conftest
+        env:
+          CONFTEST_VERSION: 0.69.0
+        run: |
+          curl --fail --location --silent --show-error \
+            --output /tmp/conftest.tar.gz \
+            "https://github.com/open-policy-agent/conftest/releases/download/v${CONFTEST_VERSION}/conftest_${CONFTEST_VERSION}_Linux_x86_64.tar.gz"
+          tar --extract --gzip --file /tmp/conftest.tar.gz --directory /usr/local/bin conftest
+          conftest --version
+      - name: Select comparison base
+        id: base
+        env:
+          PR_BASE: ${{ github.event.pull_request.base.sha }}
+          BEFORE: ${{ github.event.before }}
+        run: echo "ref=${PR_BASE:-$BEFORE}" >> "$GITHUB_OUTPUT"
+      - name: Security gate
+        run: >-
+          devops-learn security scan projects/api_platform
+          --base-ref "${{ steps.base.outputs.ref }}"
+          --report artifacts/security/security-report.json
+      - name: Upload sanitized security report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: security-report
+          path: artifacts/security/security-report.json
+          if-no-files-found: warn

.gitignore

@@ -13,3 +13,21 @@ htmlcov/
 *.sqlite3
 .env
 .devops_learn/
+artifacts/security/
+
+# Terraform working-directory artifacts (see projects/api_platform/infra/terraform/).
+# .terraform.lock.hcl is deliberately NOT listed here: it pins exact provider
+# versions/hashes so `terraform init` is reproducible across machines and CI,
+# which is normal, recommended Terraform practice -- unlike .terraform/ (a
+# regenerable local plugin cache) and *.tfstate (can contain sensitive values,
+# and since this milestone never applies, would only ever hold stale/empty
+# state if committed).
+**/.terraform/
+*.tfstate
+*.tfstate.*
+crash.log
+crash.*.log
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json

CLAUDE.md

@@ -0,0 +1,93 @@
+# DevOps-Learn-by-Doing
+
+## Product
+
+This is an AI-assisted DevOps mastery platform, not an automation tool and
+not an anti-AI exercise. The goal is real engineering work performed
+together with AI in a way that deliberately closes the user's knowledge
+gaps. AI automation and learner understanding are two separate dimensions:
+running a command for the user is not the same as the user understanding
+why it ran or what it did. See `docs/learning-model.md`. Never equate manual
+typing with expertise, and never let AI execution stand in for competency
+evidence — evidence lives in `ExperienceTracker`/`ExperienceState`
+(`experience/tracker.py`, `domain/enums.py`).
+
+## Architecture
+
+Modular monolith (`docs/architecture.md`, `docs/adr/0001`). `bootstrap.py`
+is the single composition root — all concrete implementations are wired
+there via constructor injection, never a service locator. `domain/` depends
+on nothing; `tools/`/`cloud/`/`ai/` depend only on `domain/`;
+`workflows/`/`bootstrap.py` depend on everything; `cli/` depends only on
+`bootstrap.py`/`workflows/`/`domain/`. Preserve this layering unless there
+is a strong reason to change it.
+
+Every capability the platform can execute is a `Tool` (`tools/base.py`).
+`ToolService.invoke` (`tools/service.py`) is the only caller of
+`Tool.execute` anywhere in the codebase — this is what makes "destructive
+operations require human approval" structural rather than a convention any
+one caller has to remember. Never bypass `ToolService` for convenience, and
+never call `Tool.execute` directly outside it.
+
+LLMs explain and reason; deterministic code decides. `LLMProvider`
+(`ai/provider.py`) has no method that invokes a tool or makes a decision —
+it only produces freeform explanation text. Recommendations, risk
+classification, and plans are typed dataclasses produced by deterministic
+services (`docs/adr/0008-structured-ai-output.md`), never freeform LLM
+output parsed at runtime.
+
+## Safety
+
+- Never fabricate successful execution. Every real result is explicitly
+  labeled `(real)`/`(real, failed)`; every simulated result says
+  `(simulated)`. See `docs/safety.md` for the current real-vs-simulated
+  boundary — keep it accurate as implementation changes.
+- `ToolOperationSpec` (`tools/base.py`) enforces at construction time that a
+  `RiskLevel.DESTRUCTIVE` operation must require approval and must never
+  support a dry-run bypass of that approval. Do not work around this.
+- `ApprovalGate`/`CliApprovalGate` (`tools/approval.py`) is the only path to
+  approving a HIGH/DESTRUCTIVE operation; `AutoApproveApprovalGate`/
+  `AutoDenyApprovalGate` exist only for tests, never for the real CLI.
+- Never commit credentials, Terraform state, or `.env` files. Real
+  subprocess-backed tools must redact secret-shaped output before it lands
+  in a `ToolResult` (see `tools/_subprocess_safety.py`) and must set an
+  explicit timeout on every subprocess call.
+- Never expose unrestricted arbitrary shell execution through a `Tool`. Real
+  tools shell out to a narrow, allow-listed set of operations via argument
+  arrays (never `shell=True`) — see `tools/docker_tool.py`,
+  `tools/python_tool.py`, `tools/terraform_tool.py`.
+- Security scanner output is evidence, not security truth from an LLM. Normalize and structurally
+  redact it before any console, audit, report, or explanation path; policy decisions are
+  deterministic and introduced findings must remain distinct from historical debt.
+
+## Product direction
+
+- Azure first; Terraform depth before AWS/GCP breadth (`docs/cloud-model.md`).
+- Do not add Kubernetes/AKS until a real cloud deployment path is proven
+  reliable without it — not because Kubernetes is technically harder, but
+  because cluster-level complexity should sit on top of a working real
+  deployment path, not substitute for one (`docs/roadmap.md`).
+- Prefer understandable learning infrastructure over enterprise complexity —
+  no module registries, landing zones, Terragrunt, or multi-subscription
+  setups unless a milestone genuinely requires them.
+- See `docs/roadmap.md` for current milestone status before proposing new
+  scope.
+
+## Quality
+
+Before considering any change complete, all of the following must pass:
+
+```
+pytest
+flake8 src tests projects/api_platform/app projects/api_platform/tests
+mypy
+cd projects/api_platform && mypy app --strict
+```
+
+Update `docs/roadmap.md`/`docs/safety.md` when implementation reality
+changes — this repo's credibility depends on never claiming more than what
+is genuinely implemented (e.g. do not describe real Azure execution as
+working unless it has actually been verified against a real subscription).
+
+Do not put task-specific notes, implementation plans, or milestone journals
+in this file — it is for durable project knowledge only.

CONTRIBUTING.md

@@ -29,6 +29,11 @@ existing `CloudConcept`. See `docs/cloud-model.md`.
 
 ## Adding a tool
 
-All tool operations are simulated in V1. Implement the `Tool` interface in
-`src/devops_learn/tools/`, declare accurate `risk_level`/`is_destructive` metadata, and
-never bypass `ToolService.invoke`. See `docs/safety.md`.
+Implement the `Tool` interface in `src/devops_learn/tools/`, declare accurate
+`risk_level`/`is_destructive` metadata, and never bypass `ToolService.invoke`. Provide a
+`Simulated*Tool` for tests/offline use and a `Real*Tool` that shells out to the actual binary
+via an argument list (never `shell=True`), an explicit timeout, and
+`tools/_subprocess_safety.py`'s redaction. See `docs/safety.md` and `docs/development.md`.
+
+Any test needing real Azure credentials must be gated behind `RUN_AZURE_INTEGRATION_TESTS=1` so
+normal CI never attempts to reach Azure.