Skill Being Reviewed
Skill name: nist-csf-assessment
Skill path: skills/compliance/nist-csf-assessment/
False Positive Analysis
Benign-looking scenario that can be over-scored as mature supply-chain risk management:
assessment_scope: B2B SaaS platform
current_evidence:
GV.OC-05:
external_dependencies:
- primary_cloud_provider
- email_delivery_provider
- payment_processor
- customer_support_saas
GV.SC-04:
supplier_inventory:
- name: primary_cloud_provider
criticality: critical
- name: payment_processor
criticality: high
GV.SC-05:
supplier_contracts:
- security_addendum: present
- breach_notice_clause: present
- confidentiality_clause: present
GV.SC-07:
annual_vendor_review: completed
claimed_result:
GV.SC: "3 - Repeatable"Why this is a false positive:
The current skill asks whether suppliers are inventoried, prioritized, included in contracts, assessed before acquisition, monitored during the relationship, and handled after the conclusion of a partnership. Those prompts are directionally correct, but they can still let an assessor score supply-chain risk as mature when the only evidence is an inventory plus contract clauses.
That misses whether the organization has evidence for concentration risk, fourth-party dependency visibility, supplier exit/offboarding controls, alternate-provider feasibility, and operational resilience if a critical supplier fails or is terminated. A supplier can be known and contractually governed while still creating an untested single point of failure.
Coverage Gaps
Missed variant 1: critical supplier concentration and substitutability are not evaluated
critical_services:
authentication:
supplier: identity_saas_a
backup_supplier: none
tested_failover: false
payments:
supplier: payment_processor_a
backup_supplier: "contract signed, not integrated"
tested_failover: false
email_delivery:
supplier: email_saas_a
backup_supplier: email_saas_b
tested_failover: falseWhy it should be caught:
GV.OC-05 and GV.SC-04 identify dependencies and supplier criticality, but the skill does not require concentration-risk fields such as sole-source dependency, viable substitute, switching time, tested failover, contractual portability, or residual business impact. Without those fields, a high-quality supplier inventory can hide that recovery depends on a supplier that cannot be replaced in an outage or contract dispute.
Missed variant 2: fourth-party and subprocessor dependencies are not mapped to critical services
Primary supplier: customer_support_saas
Critical service: support ticket intake and customer PII handling
Known contract evidence:
- DPA present
- security addendum present
Missing fourth-party evidence:
- hosting cloud provider
- AI summarization provider
- observability/logging provider
- outsourced support subcontractor
- data residency for each subprocessor
- notice and approval path for subprocessor changes
Why it should be caught:
GV.SC-07 and GV.SC-09 require understanding and monitoring supplier risk over the relationship and product/service life cycle. For SaaS and managed services, material cyber risk often sits in fourth parties and subprocessors. The skill should require a dependency chain evidence table so supplier criticality is not evaluated only at the direct vendor layer.
Missed variant 3: supplier exit/offboarding is mentioned but not evidenced
Supplier termination checklist:
- contract ended
- final invoice paid
Missing technical evidence:
- SSO app disabled
- SCIM/API tokens revoked
- shared Slack/Teams channels removed
- vendor VPN account disabled
- webhook secrets rotated
- customer data export verified
- data deletion certificate received
- DNS/CNAME/vendor-hosted subdomain removed
- backups and support attachments covered by deletion/retention terms
Why it should be caught:
GV.SC-10 says C-SCRM plans should include provisions after a partnership or service agreement ends, but the current process only asks the question at a high level. It should require exit evidence for identity, network access, data return/deletion, DNS/vendor-hosted assets, secrets, integrations, and retained backup/support data. Otherwise a terminated supplier can retain access or data while the assessment marks GV.SC-10 as covered.
Missed variant 4: supplier incident participation is not tested
Contract says supplier will notify incidents within 72 hours.
Missing operational proof:
- named incident contact and escalation path
- evidence package expectations
- joint tabletop or notification drill
- supplier status page / API health dependency in incident runbooks
- recovery-time dependencies for customer-facing services
Why it should be caught:
GV.SC-08 requires relevant suppliers and third parties to be included in incident planning, response, and recovery. A contract clause is weaker evidence than an exercised escalation path or tabletop. The skill should separate contractual incident clauses from tested operational participation.
Edge Cases
- Cloud providers can be both critical suppliers and infrastructure platforms; the report should capture shared-responsibility evidence, support plan dependency, region/service concentration, and backup-region limits.
- A supplier may be low spend but high criticality, such as DNS, identity, code-signing, package registry, payment routing, or transactional email.
- Vendor marketplaces and app integrations can create suppliers that bypass procurement, so SSO/OAuth/SaaS discovery evidence matters.
- Open-source or package ecosystem dependencies may not have contracts, but they can still be critical third-party dependencies that need owner, substitute, and monitoring evidence.
- Supplier exit evidence often lives across procurement, legal, IAM, IT, DNS, and application teams; the skill should allow
not_evaluable_owner_unavailable rather than silently lowering or guessing the score.
Remediation Quality
Recommended additions:
- Add a
Supplier Concentration and Substitutability Matrix with supplier, service dependency, criticality, sole-source status, replacement option, switching time, tested failover, contractual portability, and residual impact.
- Add a
Fourth-Party / Subprocessor Chain table with direct supplier, fourth party, service/data handled, region, change-notice mechanism, evidence source, and monitoring owner.
- Add a
Supplier Exit Evidence checklist covering identity revocation, network access, API tokens, webhooks, DNS/vendor-hosted assets, data export, data deletion/retention, backups, and support artifacts.
- Add supplier incident drill fields: named contacts, escalation SLA, joint tabletop date, evidence package expected, recovery dependency, and not-evaluable reason.
- Add not-evaluable reason codes for missing supplier owner, missing fourth-party list, missing exit evidence, missing failover test, and missing supplier incident contact.
Comparison to Other Tools
| Tool / Source |
Catches this? |
Notes |
| NIST CSF 2.0 GV.SC |
Partial |
Defines C-SCRM outcomes, including monitoring across the life cycle and provisions after relationship end, but the local skill needs evidence fields to operationalize those outcomes. |
| NIST SP 1305 C-SCRM Quick Start Guide |
Partial |
Focuses on using the CSF GV.SC category to establish and operate C-SCRM capability; the skill should translate that into auditable review fields. |
| NIST SP 800-161 Rev. 1 |
Partial |
Provides deeper C-SCRM practices across the organization and system life cycle, but the current skill does not capture concentration, fourth-party, or exit evidence explicitly. |
| Generic vendor-risk questionnaires |
Partial |
Often ask for contracts, attestations, and questionnaires, but frequently miss technical offboarding and tested substitutability unless customized. |
Overall Assessment
Strengths:
- Strong CSF 2.0 structure and useful GOVERN coverage.
- Correctly includes GV.SC-01 through GV.SC-10 and identifies supplier inventory, contracts, due diligence, monitoring, incident planning, and relationship conclusion as assessment topics.
- Good guardrails against fabricated CSF IDs and CSF 1.1 / 2.0 terminology drift.
Needs improvement:
- Supplier criticality should include concentration risk and substitutability, not only inventory and priority.
- Fourth-party and subprocessor dependencies need explicit evidence mapping for SaaS, cloud, support, monitoring, AI, and managed-service suppliers.
- GV.SC-10 needs technical offboarding evidence, not just a high-level partnership-conclusion question.
- Supplier incident readiness should distinguish contract clauses from exercised escalation and recovery participation.
Priority recommendations:
- Add supplier concentration and tested substitutability fields to GV.OC-05 and GV.SC-04.
- Add fourth-party/subprocessor chain evidence to GV.SC-07 and GV.SC-09.
- Add supplier exit/offboarding evidence gates to GV.SC-10.
- Add supplier incident drill evidence to GV.SC-08.
- Add
not_evaluable_* reason codes for supplier evidence controlled by procurement, legal, IAM, or vendor owners.
Sources Checked
This review is distinct from #91/#93 because it focuses on supply-chain concentration, fourth-party, exit, and supplier incident evidence, not CSF Tier/profile scoring or evidence confidence generally. It is distinct from #712 because it focuses on C-SCRM operational dependency evidence, not privacy/civil-liberties impact.
Bounty Info
Skill Being Reviewed
Skill name:
nist-csf-assessmentSkill path:
skills/compliance/nist-csf-assessment/False Positive Analysis
Benign-looking scenario that can be over-scored as mature supply-chain risk management:
Why this is a false positive:
The current skill asks whether suppliers are inventoried, prioritized, included in contracts, assessed before acquisition, monitored during the relationship, and handled after the conclusion of a partnership. Those prompts are directionally correct, but they can still let an assessor score supply-chain risk as mature when the only evidence is an inventory plus contract clauses.
That misses whether the organization has evidence for concentration risk, fourth-party dependency visibility, supplier exit/offboarding controls, alternate-provider feasibility, and operational resilience if a critical supplier fails or is terminated. A supplier can be known and contractually governed while still creating an untested single point of failure.
Coverage Gaps
Missed variant 1: critical supplier concentration and substitutability are not evaluated
Why it should be caught:
GV.OC-05 and GV.SC-04 identify dependencies and supplier criticality, but the skill does not require concentration-risk fields such as sole-source dependency, viable substitute, switching time, tested failover, contractual portability, or residual business impact. Without those fields, a high-quality supplier inventory can hide that recovery depends on a supplier that cannot be replaced in an outage or contract dispute.
Missed variant 2: fourth-party and subprocessor dependencies are not mapped to critical services
Why it should be caught:
GV.SC-07 and GV.SC-09 require understanding and monitoring supplier risk over the relationship and product/service life cycle. For SaaS and managed services, material cyber risk often sits in fourth parties and subprocessors. The skill should require a dependency chain evidence table so supplier criticality is not evaluated only at the direct vendor layer.
Missed variant 3: supplier exit/offboarding is mentioned but not evidenced
Why it should be caught:
GV.SC-10 says C-SCRM plans should include provisions after a partnership or service agreement ends, but the current process only asks the question at a high level. It should require exit evidence for identity, network access, data return/deletion, DNS/vendor-hosted assets, secrets, integrations, and retained backup/support data. Otherwise a terminated supplier can retain access or data while the assessment marks GV.SC-10 as covered.
Missed variant 4: supplier incident participation is not tested
Why it should be caught:
GV.SC-08 requires relevant suppliers and third parties to be included in incident planning, response, and recovery. A contract clause is weaker evidence than an exercised escalation path or tabletop. The skill should separate contractual incident clauses from tested operational participation.
Edge Cases
not_evaluable_owner_unavailablerather than silently lowering or guessing the score.Remediation Quality
Recommended additions:
Supplier Concentration and Substitutability Matrixwith supplier, service dependency, criticality, sole-source status, replacement option, switching time, tested failover, contractual portability, and residual impact.Fourth-Party / Subprocessor Chaintable with direct supplier, fourth party, service/data handled, region, change-notice mechanism, evidence source, and monitoring owner.Supplier Exit Evidencechecklist covering identity revocation, network access, API tokens, webhooks, DNS/vendor-hosted assets, data export, data deletion/retention, backups, and support artifacts.Comparison to Other Tools
Overall Assessment
Strengths:
Needs improvement:
Priority recommendations:
not_evaluable_*reason codes for supplier evidence controlled by procurement, legal, IAM, or vendor owners.Sources Checked
This review is distinct from #91/#93 because it focuses on supply-chain concentration, fourth-party, exit, and supplier incident evidence, not CSF Tier/profile scoring or evidence confidence generally. It is distinct from #712 because it focuses on C-SCRM operational dependency evidence, not privacy/civil-liberties impact.
Bounty Info