epic: migrate all scanner rules to the evaluate() coverage contract #380

Activity

1. parthrohit22

   self-assigned this

   on Oct 4, 2026
2. parthrohit22

   added

   enhancementNew feature or requestNew feature or request

   coreCore team ownership not for studentsCore team ownership not for students

   priority: highImportant, should be fixed in the current sprintImportant, should be fixed in the current sprint

   roadmapPlanned feature track, not a current bugPlanned feature track, not a current bug

   on Oct 4, 2026
3. SHAURYAKSHARMA24 commented on Oct 9, 2026

   

   SHAURYAKSHARMA24

   on Oct 9, 2026

   Collaborator

   More actions

   @m-khan-97 @Vishnu2707, a question on scope and ownership for the piece that sits next to this epic.

   #381 and #382 get the evaluation layer right. When an inventory call fails (403, 429, expired credential, a pager that dies halfway), migrated rules now report ERROR/INVENTORY_UNAVAILABLE and show up in failed_rule_ids, where dev reports KV-006 as NOT_APPLICABLE. I checked this by driving the real AzureClient with the SDK clients patched to fail. Compliance already treats ERROR/UNKNOWN as not passing.

   What none of the open work covers yet is the scan result itself. In every one of those failure cases the scan still finishes as completed with score 100, on dev and on the PR heads, because score_findings() only counts findings and the scan has no notion of what it couldn't see. /api/score returns "status": "OK", and the dashboard shows the 100. Drift has the same problem from the other side: it diffs the last two scans with findings, across subscriptions, so a finding that wasn't observed shows up as REMOVED. That's the gap the #305 gate describes ("failed or incomplete evidence cannot produce PASS, a clean scan, or score 100"), and the evaluate() migration alone doesn't close it.

   What I'd like to propose, as a separate issue/PR on top of #381:

   • Add a coverage block to the scan result and keep three things apart: rule execution (rules loaded vs. rules with usable outcomes), inventory completeness per collector (only for collectors that can report failure, i.e. the list_* ones from feat(scanner): add the evaluate() contract foundation (#369) #381; NOT_MEASURABLE for the rest), and resource assessment (PASS/FAIL vs. resources actually observed, migrated rules only). No percentage unless its denominator is something we actually observed.
   • Add a score_status (COMPLETE / INCOMPLETE / NOT_MEASURABLE) next to the score in the scan result, /api/score and the dashboard. The score formula stays as it is, so risk and coverage remain separate numbers, but a score is never shown without saying whether it covers everything. Nothing gets a PASS it didn't earn.
   • Fix drift to compare within one subscription and report pairs whose rule is ERROR/UNKNOWN in the latest scan as "not observed" rather than removed.

   The tests would be deterministic and offline: empty subscription, compliant, violating, 403, 429, expired credential, failure mid-pagination, one collector failing while others succeed, a previously failing resource becoming unobservable, and a retry restoring coverage.

   On #326: lifecycle should read the same signal (evaluations plus failed_rule_ids) rather than a separate outcome list. I've left details on that PR. The coverage block wouldn't depend on #326 either way.

   I'm happy to own this if it fits how you want the gate handled. I'd start the design once you're both OK with the direction, and the implementation once #381 merges. If you'd rather it live under #305 or with someone already working in this area, that's fine too.