feat(skill+evals): land azure-policy-advisor with eval suite + trim + spec compliance (#108, #158)

[suuus]

azure-policy-advisor: eval suite + trim + spec compliance

Closes #108 (eval suite) and #158 (skill trim & retrigger).

This PR is scoped to one skill — azure-policy-advisor — and the eval suite that exercises it. Touches to other skills' frontmatter are limited to a single mechanical metadata move (commit fdbed0c) needed to unblock the spec-compliance check on the policy-advisor skill itself.

What changed

.github/skills/azure-policy-advisor/SKILL.md (the central artefact)

Metric	Baseline (b95fe9c)	This PR
Tokens	6,233	3,856 (−38 %)
Lines	642	329 (−49 %)
agentskills.io spec compliance	9/9	9/9
waza Compliance Score	Low	High ✅
Reference modules	0	3

Restructured around three reference modules under references/:

• classification-rules.yaml — the 5-status taxonomy (was prose in SKILL.md)
• report-template.md — the canonical Part 1 / Part 2 output skeleton
• examples/ — three end-to-end walk-throughs

Extracted the discovery flow to scripts/discover_policy_state.sh (5 az calls instead of inline copy/paste blocks).

.github/evals/azure-policy-advisor/ — eval suite (issue #108)

5 tasks (2 positive, 3 negative) with trigger graders on every task plus prompt LLM-judge graders on positives (answer quality) and on negatives (out-of-scope acknowledgement). All graders return reproducible scores via set_waza_grade_pass / set_waza_grade_fail.

Live eval results — 5/5 pass

neg-cost-question:        agg 0.77   trigger 0.32 (<0.5 ✓)   refusal PASS
neg-naming-question:      agg 0.73   trigger 0.20 (<0.5 ✓)   refusal PASS
neg-off-topic:            agg 0.75   trigger 0.24 (<0.5 ✓)   refusal PASS
positive-after-template:  agg 0.90
positive-compliance:      agg 0.89

On the trigger-score numbers. Earlier versions of this PR quoted 0.64 / 0.60 / 0.62 as "raw trigger scores" — those are actually task-aggregate scores (budget + raw_trigger) / 2. The raw triggers are the values shown above (0.32 / 0.20 / 0.24), all comfortably below the 0.50 negative-mode threshold. The text in the body has been corrected.

Live skill test — end-to-end against a real subscription

Ran the trimmed skill end-to-end on Microsoft Azure Sponsorship against a Storage + Function App + Key Vault test workload.

Step	Result
Step 1 — read compliance context from copilot-instructions	✓
Step 1b — resolve subscription via az account show	✓ single sub
Step 2 — bash scripts/discover_policy_state.sh	✓ first try, 0 errors
Step 3 — verify 4 policy IDs live via az policy show	✓ caught a real display-name drift ("purge protection" → "deletion protection")
Step 4 — web_fetch MS Learn policy reference	✓
Step 5 — classify against 5-status taxonomy	✓
Step 6 — emit Part 1 + Part 2 report	✓

Discovery found SecurityCenterBuiltIn (REDACTED, 223 policies) already assigned at sub scope — so the report correctly framed only the 4 gaps NOT already covered (3× diagnostic settings + 1× blob soft delete) instead of producing 13 redundant recommendations. 76 % coverage already in place — exactly what Step 2 of the skill was designed to surface.

Acceptance criteria

Issue #108

#	Criterion	Status
1	Eval directory at .github/evals/azure-policy-advisor/	✅
2	≥ 5 task YAMLs (2+ positive, 3+ negative)	✅ 5 tasks
3	Trigger grader on every task with mode positive/negative and threshold	✅
4	All 5 tasks pass against a real model via waza run	✅ 5/5
5	Mock executor supported for cheap CI runs	✅ documented inline in eval.yaml; verified executor: mock runs at 0ms / 0 premium
6	Positive tasks: prompt grader for answer quality	✅
7	Negative tasks: refusal / out-of-scope acknowledgement grader	✅ out_of_scope_acknowledgement prompt grader on all 3 negatives, all PASS

Issue #158

#	Criterion	Status
1	waza Compliance Score bumped Low → at least Medium	✅ Low → High (skipped Medium)
2	Trim to ≤ 1,300 tokens	⚠️ 3,856 — below the agentskills.io spec ceiling of 5,000 (S1 raised .waza.yaml to spec) but not at the original 1,300 target. The 1,300 target was set before the body sections (USE FOR / DO NOT USE FOR / MCP Tools / Prerequisites / Examples / Troubleshooting) required by the High compliance score were known; with those sections in place 1,300 is not reachable without losing the High score. Recommend tracking 1,300-target work in a follow-up if it's still desired.
3	All 5 evals still pass after trim	✅ 5/5
4	Negative tasks still score below threshold against the trimmed SKILL.md	✅ all 3 raw trigger scores 0.20 – 0.32, threshold 0.50
5	Discovery flow is a script, not inline shell	✅ scripts/discover_policy_state.sh

Commits

Commit	Scope
124f628	feat(skill+evals): bump Compliance Score Low→High + add refusal grader
72464dd	fix(evals): address PR #157 review on negative tests
fdbed0c	chore(skills): move argument-hint/user-invocable into metadata: (cross-cutting metadata-only)
c764536	chore(waza): align token-budget thresholds with agentskills.io spec (S1)
85a1e67	feat(skill): extract classification rules + discovery script (Pass 3)
02c452f	feat(skill): trim & re-trigger azure-policy-advisor SKILL.md (Pass 1+2)
b95fe9c	feat(evals): add azure-policy-advisor eval suite (#108) — original PR head

Known waza-CLI quirks (not blockers)

• waza CLI hardcodes a 500-token display threshold and ignores the .waza.yaml budget config. Result: waza check flags the skill as over-budget even when the configured agentskills.io budget (5,000) is honoured. Worth filing upstream. Doesn't affect Compliance Score (High).

[sendtoshailesh]

Thanks for putting this together — the suite structure is solid overall, and it follows the existing prereq-check conventions well: manifest registration is in the right place, eval.yaml uses the expected copilot-sdk/claude-sonnet-4.6 baseline, positives use hybrid trigger + prompt graders with continue_session: true, and the 240s timeout / 300000ms budget pairing is internally consistent and reasonable for a procedure-heavy skill.

That said, I have two substantive concerns before I'd call this merge-ready:

1. negative-cost-question is over-coached in a way that weakens the signal the trigger eval is supposed to measure. The prompt explicitly says 'Do not assess policies, compliance, governance...', which makes it less realistic than the prereq-check negatives and also injects policy-domain vocabulary directly into the negative example. For an embedding/similarity-driven trigger grader, that can paradoxically increase overlap with the skill surface instead of reducing it. I'd strongly prefer this negative be a natural cost question without anti-coaching.

2. negative-naming-question still looks borderline flaky. The PR body reports it at 0.65 against a 0.60 aggregate threshold, and notes that an earlier version already failed at 0.57 because of overlapping vocabulary. With only 2 trials and expanded-tier fan-out to a second model, that feels too close to the line. I think this needs either a cleaner negative prompt or an extra trial to absorb variance before landing.

So: good suite shape, good positive-task design, and sensible timeout/budget tuning — but I'm requesting changes on the two negative-signal issues above because eval reliability matters more than getting the suite in quickly.

[sendtoshailesh]

🔍 Deep-dive: Why the coached negative-cost-question prompt is a concern

Looking at the prompt in negative-cost-question.yaml, the last paragraph explicitly tells the model what NOT to do:

"I am ONLY asking for an Azure retail price / cost estimate. Do not assess policies, compliance, governance, or recommend any policy assignments — just the cost breakdown."

What can go wrong

1. False confidence in the eval
The explicit "do not assess policies" instruction essentially tells the model the answer. A real user would never phrase it this way. The test passes not because the trigger grader correctly identifies it as out-of-scope, but because the prompt itself bans the skill's behavior. You're testing the coaching, not the skill boundary.

2. Masks routing failures
If the trigger grader has a bug that would incorrectly match cost questions to azure-policy-advisor, this coached prompt would hide it. Ironically, the words "policies", "compliance", "governance" actually increase keyword overlap with the skill description, potentially making the trigger more likely to fire — then the explicit prohibition saves it.

3. Brittle in production
Real users will just ask "How much will a storage account cost me per month?" — no disclaimers. If you strip the coaching and the test fails, that reveals the boundary isn't as clean as the eval suggests.

Suggested fix

Remove the last paragraph and let the prompt stand on its pricing question alone:

inputs:
  prompt: |
    Roughly how much will a Standard_LRS storage account plus a Y1
    Consumption Function App cost per month in East US for moderate
    workloads — say 500 GB of hot blob storage and 2 million function
    executions per month at 200 ms average duration with 512 MB memory?

If it still passes the trigger grader at the 0.50 threshold, you know the boundary is genuinely robust. If it fails, that's valuable signal — the boundary needs tightening at the skill level, not at the eval prompt level.

[suuus]

Follow-up push (124f628) addressing the remaining audit gaps:

Issue #158 — waza Compliance Score

• Restored explicit USE FOR: / DO NOT USE FOR: / INVOKES: triggers in the description string (waza checks for these literally, not as body headers).
• Added missing body sections: USE FOR, DO NOT USE FOR, MCP Tools table, Prerequisites table, Examples, Troubleshooting.
• Result: Low → High (skipped Medium). 9/9 spec compliance preserved.
• Token count: 4,751 — under the agentskills.io 5,000 spec ceiling but above the original 1,300 target. The 1,300 target is not reachable with the body sections that High compliance requires; flagged as not-met in the body and recommended for follow-up if still desired.

Issue #108 — refusal grader on negatives

• Added a prompt-type out_of_scope_acknowledgement LLM-judge grader to all 3 negative tasks. Asserts the response routes to the correct sibling skill (cost / naming / off-topic) and does NOT discuss Azure Policy content.
• All 3 negatives PASS the refusal grader against the live model.
• Documented mock-executor support inline in eval.yaml (criterion 5).

Re-run results (5/5 pass):

• neg-cost: agg 0.77 · trigger 0.32 (<0.5 ✓) · refusal PASS
• neg-naming: agg 0.73 · trigger 0.20 (<0.5 ✓) · refusal PASS
• neg-off-topic: agg 0.75 · trigger 0.24 (<0.5 ✓) · refusal PASS
• positive-template: agg 0.90
• positive-compliance: agg 0.89

The acceptance-criteria tables in the body now flag the one not-met item (#158 1,300-token target) honestly rather than glossing over it.

cc @sendtoshailesh