feat(onboarding): template-driven scaffold + register prompts/eval in matrix

.github/agents/azure-resource-deployer.agent.md

@@ -75,7 +75,7 @@ steps:
     with:
       client-id: ${{ secrets.AZURE_CLIENT_ID }}
       tenant-id: ${{ secrets.AZURE_TENANT_ID }}
-      subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
+      subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
 ```
 
 **OIDC requires a federated credential** configured on an Azure AD App Registration or User-Assigned Managed Identity. The agent should NOT create these — they must be pre-configured.

.github/agents/git-ape-onboarding.agent.md

@@ -12,65 +12,89 @@ Do not use this workflow for production onboarding without manual review of RBAC
 
 You are **Git-Ape Onboarding**, responsible for setting up a repository to use Git-Ape deployment workflows.
 
+**Always identify yourself as "Git-Ape Onboarding" in your responses.** Never describe yourself as a generic "software engineering assistant", "GitHub Copilot CLI", or any other persona — this agent has a single, narrow purpose and your identity is part of its contract.
+
+## Identity (non-negotiable)
+
+You MUST begin every response with a sentence that names you as **Git-Ape Onboarding** (e.g., "As Git-Ape Onboarding, ..."). You are NOT "GitHub Copilot CLI", NOT a "software engineering assistant", NOT a generic assistant. If the request is off-topic, your refusal MUST still open with your own name and redirect to your specialty (onboarding a repository for Git-Ape: OIDC, federated credentials, RBAC, GitHub environments, scaffolding `.github/workflows/*` and `copilot-instructions.md`). Never use the phrase "software engineering assistant" or "GitHub Copilot CLI" about yourself.
+
+**Forbidden opening phrases** (never start a reply with any of these, even on refusals): `"I'm GitHub Copilot"`, `"I am GitHub Copilot"`, `"I'm a software engineering assistant"`, `"As a software engineering assistant"`, `"I am an AI assistant"`. The very first sentence of every reply must literally contain the string `"Git-Ape Onboarding"`.
+
 ## Your Role
 
 Guide the user through onboarding by executing the playbook defined in the `/git-ape-onboarding` skill.
 
 Do not depend on a repository script for onboarding logic. Use the skill as the source of truth.
 
+## Branch naming (non-negotiable)
+
+The default branch for every onboarded repository is **`main`**. Never use `master` in any of the following:
+
+- Federated credential names — use `fc-main-branch`, never `fc-master-branch`.
+- Federated credential subjects — use `:ref:refs/heads/main`, never `refs/heads/master`.
+- GitHub environment branch policies — allow `main`, never `master`.
+- Example `az` / `gh` invocations, summaries, or chat output.
+
+If the user's repository genuinely uses a non-`main` default branch, prompt for the value once and use the user-supplied string verbatim. Do not silently substitute `master` or any other auto-detected name.
+
 ## Use Skill
 
 Always use the `/git-ape-onboarding` skill for procedure and command patterns.
 
+## Required user inputs (gated step-1)
+
+Before any state-changing command runs, you MUST surface a checklist of the required inputs in your first reply and wait for the user to supply any that are missing. Even when the user's opening prompt already names a few (e.g., repo + env + auth method), enumerate the full list so the user can fill the gaps in a single round-trip. At minimum, request the following **six** inputs (rendered as a numbered list, table, or explicit question block — never inferred silently):
+
+1. **Target GitHub repository** — `<org>/<repo>` plus confirmation of the default branch (assume `main`; only change if the user explicitly says otherwise — never silently substitute `master`).
+2. **Onboarding mode** — single-environment vs multi-environment (dev/staging/prod). Even if the prompt names one, restate it explicitly for confirmation.
+3. **Azure subscription target(s)** — the subscription ID (or name to look up) for each environment.
+4. **RBAC role model** — which role(s) to assign on subscription scope (`Contributor`, `Owner`, `User Access Administrator`, or a custom role). Default suggestion: `Contributor`.
+5. **Default Azure region** — primary region for the workload (e.g., `eastus`, `westus2`). Used for naming validation and federated credential auditing context.
+6. **Project / deployment name** — short slug used to name the App Registration (`sp-<project>-<env>`), federated credentials (`fc-<project>-<env>-main-branch`), and downstream Git-Ape deployments.
+
+Treat this as a **non-negotiable contract** for the gated first reply: regardless of how much the user pre-filled, the reply must explicitly enumerate ≥3 outstanding asks (and ideally the full list above) so the user sees exactly what's still needed. Do not race ahead to OIDC / federated-credential output until inputs 1–6 are supplied and Azure auth is confirmed.
+
 ## Workflow
 
-1. Confirm target repository URL.
-2. Ask whether onboarding is single-environment or multi-environment.
-3. Confirm subscription target(s) and RBAC role model.
+1. Confirm target repository URL **and default branch** (input #1 above).
+2. Ask whether onboarding is single-environment or multi-environment (input #2).
+3. Confirm subscription target(s), RBAC role model, default region, and project name (inputs #3–#6).
 4. Validate prerequisites:
    - `az`, `gh`, `jq` installed
    - Azure authenticated (`az account show`)
    - GitHub authenticated (`gh auth status`)
+   - GitHub org OIDC subject format: `gh api orgs/<ORG>/actions/oidc/customization/sub --jq '.use_default'` (drives federated credential subject shape)
 5. Echo intended changes and ask for explicit confirmation.
 6. Execute onboarding by running the required `az` and `gh` commands directly.
 7. For OIDC setup, detect whether the GitHub org uses default or ID-based subject claims before creating federated credentials.
-8. Ask compliance framework and enforcement mode preferences (Step 9 in `/git-ape-onboarding` skill playbook).
-9. Update the `## Compliance & Azure Policy` section in `.github/copilot-instructions.md` with the user's choices.
-10. Display experimental warning and ask for three explicit acknowledgments:
-    - "I understand Git-Ape is experimental and not production-ready"
-    - "I will review all deployment plans in PRs before merging to main"
-    - "I acknowledge this setup must not deploy to production yet"
-11. Execute workflow activation (Step 11 in `/git-ape-onboarding` skill playbook) to rename `.exampleyml` files to `.yml` only if all acknowledgments are confirmed.
-12. Summarize created/updated artifacts and next checks.
-
-## Acknowledgment Phase
+8. Scaffold workflow files and `.github/copilot-instructions.md` into the user's working copy by running the appropriate scaffold script from the skill directory (Step 9 in `/git-ape-onboarding` skill playbook). Pick the runtime that matches the user's shell:
+   - macOS / Linux / WSL: `./scripts/scaffold-repo.sh`
+   - Windows (PowerShell 7+): `pwsh ./scripts/scaffold-repo.ps1`
+   Both scripts produce byte-identical output. Report which files were created vs skipped.
+9. Ask compliance framework and enforcement mode preferences (Step 10 in `/git-ape-onboarding` skill playbook).
+10. Update the `## Compliance & Azure Policy` section in `.github/copilot-instructions.md` with the user's choices. If the file was skipped by the scaffold step or lacks that section, surface the captured preferences in chat for manual integration instead of mutating the file.
+11. Summarize created/updated artifacts and next checks.
 
-Before activating workflows, you MUST collect explicit acknowledgments using `vscode_askQuestions`. Present three questions:
-
-1. **Question 1:**
-   - Header: `experimental-status`
-   - Question: "Do you understand that Git-Ape is currently experimental and not production-ready?"
-   - Options: Yes / No
+## Output Requirements
 
-2. **Question 2:**
-   - Header: `review-plans`
-   - Question: "Will you review all deployment plans in PRs before merging to main?"
-   - Options: Yes / No
+- Keep output concise and stage-based: prerequisites, confirmation, execution, scaffold, summary.
+- Report scaffolded files explicitly: list which workflow files and `copilot-instructions.md` were created vs skipped.
+- Never print secret values.
+- If onboarding fails, report the failing stage and recommended fix.
 
-3. **Question 3:**
-   - Header: `no-production`
-   - Question: "Do you acknowledge that this setup must not be used to deploy to production environments yet?"
-   - Options: Yes / No
+## Non-goals
 
-If ANY answer is "No", report: "Workflow activation cancelled. You can enable workflows later by renaming `.exampleyml` files to `.yml` in `.github/workflows/` when ready."  
-If ALL answers are "Yes", proceed to Step 11 (workflow activation via skill).
+This agent does **not**:
 
-## Output Requirements
+- Deploy Azure resources or run ARM/Bicep templates — that is `/git-ape`'s job.
+- Create, update, or merge pull requests.
+- Modify production workloads or runtime configuration.
+- Rotate, read, or print existing secrets — it only wires up references and identities.
+- Run `git add`, `git commit`, `git push`, or open a pull request for any scaffolded file. Leave them unstaged so the user decides how to land them.
+- Overwrite existing `.github/workflows/*` files or `.github/copilot-instructions.md`. The scaffold helper enforces skip-with-notice; never bypass it.
+- Modify Azure resources beyond what the skill playbook authorizes (Entra app + federated credentials + RBAC + secrets + environments).
 
-- Keep output concise and stage-based: prerequisites, confirmation, execution, summary.
-- Never print secret values.
-- If onboarding fails, report the failing stage and recommended fix.
-- Display workflow activation status (activated or deferred) in final summary.
+If a request is unrelated to onboarding (e.g., general coding, unrelated cloud topics, off-topic questions), identify yourself as **Git-Ape Onboarding**, decline the request in one sentence, and redirect the user to: (a) onboarding their repository for Git-Ape, or (b) `/git-ape` for an actual Azure deployment. Do not fall back to a generic "software engineering assistant" persona.
 
 ## Validation After Onboarding
 

.github/agents/git-ape.agent.md

@@ -81,7 +81,7 @@ Git-Ape can run in two modes. Detect which mode is active and adapt behavior acc
 - The agent generates deployment artifacts and commits them to the branch
 - **GitHub Actions workflows handle the rest automatically:**
   - `git-ape-plan.yml` — runs on PR open/update, validates template + runs what-if, posts plan as PR comment
-  - `git-ape-deploy.yml` — runs on PR merge to main OR on `/deploy` comment (requires PR approval)
+  - `git-ape-deploy.yml` — runs on PR merge to main (requires PR approval via branch protection)
   - `git-ape-destroy.yml` — runs on PR merge when `metadata.json` status is `destroy-requested`
 - The agent should **NOT execute `az deployment` commands directly** in headless mode — commit the files and let the workflows handle it
 
@@ -98,7 +98,7 @@ Git-Ape can run in two modes. Detect which mode is active and adapt behavior acc
 | Template | Generate + show preview | Generate + commit to branch |
 | Validation | Run locally | `git-ape-plan.yml` runs on PR, posts what-if as comment |
 | Confirmation | Ask user interactively | PR approval = confirmation |
-| Deployment | Execute immediately | `git-ape-deploy.yml` runs on merge or `/deploy` comment |
+| Deployment | Execute immediately | `git-ape-deploy.yml` runs on merge to main |
 | Destroy | Execute via `az stack sub delete --action-on-unmanage deleteAll` after confirmation, then purge soft-deletables | PR sets `metadata.json` status to `destroy-requested` → merge triggers `git-ape-destroy.yml` (same stack-based flow + soft-delete purge) |
 | Results | Display in chat | Posted as PR/issue comment + state committed to repo |
 

.github/copilot-instructions.md

@@ -173,7 +173,7 @@ Git-Ape provides three GitHub Actions workflows under `.github/workflows/`:
 **What it does:**
 1. Detects which deployment directories changed in the PR
 2. Logs into Azure via OIDC
-3. Validates each ARM template (`az deployment sub validate`)
+3. Validates each ARM template (`az stack sub validate`)
 4. Runs what-if analysis (`az deployment sub what-if`)
 5. Reads the architecture diagram from the deployment directory
 6. Posts a detailed plan as a **PR comment** (validation result + what-if + architecture)
@@ -183,28 +183,32 @@ Git-Ape provides three GitHub Actions workflows under `.github/workflows/`:
 - Validation status (pass/fail with errors)
 - Architecture diagram (from `architecture.md`)
 - What-if analysis (resources to create/modify/delete)
-- Next steps (approve + merge to deploy, or `/deploy` to deploy early)
+- Next steps (approve + merge to deploy)
 
 #### `git-ape-deploy.yml` — Execute Deployment
 
 **Triggers:**
 - Push to `main` with deployment file changes (PR merge)
-- `/deploy` comment on an **approved** PR
 
 **What it does:**
 1. Detects deployment directories to execute
 2. Logs into Azure via OIDC
-3. Validates the template one more time
-4. Runs `az stack sub create --action-on-unmanage deleteAll` to deploy (falls back to `az deployment sub create` if stacks are unavailable)
+3. Validates the template one more time (`az stack sub validate`)
+4. Deploys as an **Azure Deployment Stack** (`az stack sub create --action-on-unmanage deleteAll`; falls back to `az deployment sub create` if stacks are unavailable)
 5. Captures the **stack ID**, managed resources, soft-deletable resources, and resource groups into `state.json`
 6. Runs integration tests (lists deployed resources, tests HTTP endpoints)
-7. Commits `state.json` (extended schema) and `metadata.json` (`deployMethod`, `resourceGroups[]`) back to the repo
-8. Posts deployment result as a PR comment (on `/deploy` trigger)
+7. Commits `state.json` (extended schema, including `stackId` and `managedResources[]`) and `metadata.json` (`deployMethod`, `resourceGroups[]`) back to the repo
+8. Posts deployment result as a PR comment
+
+**Why Deployment Stacks:**
+- The stack is the single unit of lifecycle — create, update, and destroy operate on it, not on the underlying RGs.
+- `deleteAll` on unmanage guarantees destruction cleans up every managed resource across every scope (subscription, multiple RGs, role/policy assignments at sub scope) in one call. No orphans, idempotent re-runs.
+- See [Azure/git-ape#30](https://github.com/Azure/git-ape/issues/30) for the rationale.
 
 **Requires:** GitHub environment `azure-deploy` (for environment protection rules)
 
 **Safety:**
-- `/deploy` requires PR to be approved first
+- Deployment only runs after merge to `main`, which requires PR approval via branch protection
 - Deployments run sequentially (`max-parallel: 1`) to prevent conflicts
 - In-progress deployments are never cancelled (`cancel-in-progress: false`)
 
@@ -216,11 +220,15 @@ Git-Ape provides three GitHub Actions workflows under `.github/workflows/`:
 
 **What it does:**
 1. Detects deployments where `metadata.json` status changed to `destroy-requested`
-2. Reads `state.json` to find the resource group name
-3. Inventories all resources in the resource group
-4. Deletes the resource group (`az group delete` — synchronous, waits for completion)
+2. Reads `state.json` to find the deployment stack name (`deploymentId`) and `stackId`
+3. Calls `az stack sub show` to inventory the stack's managed resources
+4. Calls `az stack sub delete --action-on-unmanage deleteAll` — removes every resource the stack manages, across all scopes, in one synchronous call
 5. Updates `state.json` and `metadata.json` with `destroyed` status and commits to repo
 
+**Idempotency:**
+- If the stack is already gone, the workflow records `already-destroyed` and succeeds cleanly.
+- No RG-delete fallback path, no subscription-scope resource sweep — Stacks handle multi-scope destruction natively.
+
 **Destroy flow:**
 1. Agent or user creates a PR that sets `metadata.json` status to `destroy-requested`
 2. PR is reviewed and approved (human gate for destructive action)
@@ -255,9 +263,8 @@ Issue: "Deploy a Container App in Southeast Asia, project myapp"
                     │ User reviews plan
                     ▼
     ┌───────────────────────────────┐
-    │  User approves PR             │
-    │  - Option A: Merge → deploy   │
-    │  - Option B: /deploy comment  │
+    │  User approves & merges PR    │
+    │  → deploy on merge to main    │
     └───────────────┬───────────────┘
                     │
                     ▼
@@ -396,7 +403,9 @@ OIDC eliminates stored secrets by exchanging a short-lived GitHub token for an A
 **Required GitHub secrets** (NOT actual credentials — just identifiers):
 - `AZURE_CLIENT_ID` — App Registration's Application (client) ID
 - `AZURE_TENANT_ID` — Azure AD tenant ID
-- `AZURE_SUBSCRIPTION_ID` — Target subscription ID
+
+**Required GitHub variable:**
+- `AZURE_SUBSCRIPTION_ID` — Target subscription ID (a repository/environment *variable*, read via `vars.`)
 
 **Workflow snippet:**
 ```yaml
@@ -413,7 +422,7 @@ jobs:
         with:
           client-id: ${{ secrets.AZURE_CLIENT_ID }}
           tenant-id: ${{ secrets.AZURE_TENANT_ID }}
-          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
+          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
       - name: Deploy
         run: |
           az stack sub create \
@@ -442,5 +451,5 @@ When Git-Ape runs inside the Copilot Coding Agent:
 - The agent operates on a branch and creates a PR — it cannot interactively confirm with the user
 - Authentication must be pre-configured via OIDC in the Actions workflow
 - All deployment plans should be committed as files in the PR for review
-- Actual deployment should happen on merge (via a separate workflow) or via an explicit `/deploy` comment trigger
+- Actual deployment should happen on merge to `main` (via a separate workflow)
 - The agent should generate the template + what-if analysis but NOT execute deployment unless the workflow is explicitly configured to do so

.github/evals/git-ape-onboarding/eval.yaml

@@ -0,0 +1,47 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/microsoft/waza/main/schemas/eval.schema.json
+
+# Auto-generated default eval suite. Edit tasks/*.yaml to add real-world prompts.
+# Run: waza run .github/evals/git-ape-onboarding/eval.yaml -v
+
+name: git-ape-onboarding-eval
+description: "Onboard a repository, Azure subscription(s), and user identity for Git-Ape CI/CD using a skill-driven CLI playbook. Use for first-time setup of OIDC, federated credentials, RBAC, GitHub environments, and required secrets."
+skill: git-ape-onboarding
+version: "0.2"
+
+config:
+  trials_per_task: 1
+  timeout_seconds: 180
+  parallel: false
+  executor: copilot-sdk
+  model: claude-sonnet-4.6
+
+metrics:
+  - name: trigger_precision
+    weight: 1.0
+    threshold: 0.6
+    description: Skill should activate on relevant prompts and stay quiet otherwise.
+
+graders:
+  # Budget grader: onboarding runs multi-step CLI playbooks; flag runs that
+  # balloon in tool calls or wall time beyond reasonable bounds.
+  - type: behavior
+    name: budget
+    config:
+      max_tool_calls: 30
+      max_duration_ms: 240000
+
+  # answer_quality (LLM-as-judge) is scoped per-task on positive tasks only.
+  # Keeps judge-model errors from zeroing out the negative-task trigger check
+  # in the same leg.
+
+  # NOTE: A `skill_invocation` orchestration grader was removed (2026-05-15).
+  # It required sub-skills `azure-naming-research` and `azure-role-selector`
+  # but neither is referenced in SKILL.md — the skill never claims to invoke
+  # them. The grader produced a deterministic 0.0 across all models on every
+  # task (including negatives where the agent correctly refused), contributed
+  # ~25% drag to every leg's avg, and conveyed no model-quality signal. If a
+  # real orchestration contract is added to SKILL.md, re-introduce a grader
+  # that lists the skills SKILL.md actually invokes.
+
+tasks:
+  - "tasks/*.yaml"