From 8fd83bca5fb1429f41a7e6a9c0f763d40f13c93b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 30 Aug 2026 11:44:12 +0000 Subject: [PATCH 01/18] feat(threat-intel): add CISA KEV catalog ingestion adapter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a new threat-intel adapter alongside the existing STIX/MISP/TAXII/ OpenCTI family: POST /api/threat-intel/cisa-kev fetches the CISA Known Exploited Vulnerabilities catalog (a single well-known public JSON feed, https://www.cisa.gov/known-exploited-vulnerabilities-catalog) and upserts one `cve`-typed ThreatIndicator per entry, keyed by CVE ID, severity escalated to Critical when CISA has tied the CVE to a known ransomware campaign (High otherwise — mere inclusion in the catalog already signals CISA-confirmed active exploitation). - src/kev_import.rs: pure parser (`parse_kev_document`), mirroring the misp_import/opencti_import/stix_import module shape exactly. Accepts the real catalog shape (`{"vulnerabilities": [...]}`) or a bare array. KEV entries carry no IP/domain/URL/hash observable, so `dnsbl` stays empty (kept only for ThreatFeedImport parity). - src/lib.rs: `import_kev_feed` handler follows the phishing-database fetch-by-URL pattern (not the MISP/STIX/OpenCTI paste-a-document pattern) since KEV has one canonical, stable, publicly known URL that every consumer wants pulled automatically rather than hand-relayed. Reuses the existing `fetch_text_feed`/`apply_threat_feed_import` plumbing untouched. `validate_http_url` gained a third `allowed_hosts` parameter (was hardcoded to the phishing-database allowlist) so KEV gets its own `www.cisa.gov`-only default, matching the SSRF-safety posture phishing-database already has (opt-out via `allow_non_default_hosts`); all other call sites updated. - crates/waf-ids-core: new buyer-evidence-manifest endpoint entry. - docs/architecture.md, docs/security/compliance-mapping.md: documented alongside the other threat-intel adapters. Deliberately not adding a cargo-fuzz target: none of the five existing JSON-import adapters (misp_import, opencti_import, stix_import, suricata_eve, coraza_audit) have one today — CLAUDE.md's fuzzed-surface list (request scorer, state deserializer, admin-token parser, DNSBL zone export) doesn't cover this adapter family. Matched the existing convention instead: a `parse_never_panics_on_arbitrary_text` unit test, same as its siblings. Verified: cargo fmt --check, cargo test --locked --workspace (all pass except the pre-existing root-sandbox-only `load_surfaces_state_rewrite_failures` flake tracked by PR #93, unrelated to this change), cargo clippy --locked --workspace --all-targets -- -D warnings. --- crates/waf-ids-core/src/lib.rs | 8 + docs/architecture.md | 2 +- docs/security/compliance-mapping.md | 2 +- src/kev_import.rs | 223 ++++++++++++++++ src/lib.rs | 379 +++++++++++++++++++++++++++- 5 files changed, 606 insertions(+), 8 deletions(-) create mode 100644 src/kev_import.rs diff --git a/crates/waf-ids-core/src/lib.rs b/crates/waf-ids-core/src/lib.rs index e3788698..abd5dcc6 100644 --- a/crates/waf-ids-core/src/lib.rs +++ b/crates/waf-ids-core/src/lib.rs @@ -1292,6 +1292,14 @@ fn buyer_evidence_endpoints() -> Vec { "OpenCTI observable/indicator JSON ingest into threat indicators and DNSBL (admin-auth)", false, ), + buyer_evidence_endpoint( + "cisa_kev_ingest", + "POST", + "/api/threat-intel/cisa-kev", + "application/json", + "CISA Known Exploited Vulnerabilities catalog pull into CVE threat indicators (admin-auth)", + false, + ), ] } diff --git a/docs/architecture.md b/docs/architecture.md index 89291bf5..33a62f83 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -45,7 +45,7 @@ flowchart LR - **WAF**: Coraza/OWASP CRS audit JSON/NDJSON ingest is available at `POST /api/waf/coraza/audit` (admin token). Interrupted transactions and CRS rule messages become `SecurityEvent` rows and feed gateway enforcement (DNSBL + `client_ip`/`path` threat indicators) so subsequent gateway decisions block matching clients. In-process Coraza embedding remains a follow-up — do not replace CRS with hand-rolled rules. - **IDS**: Suricata EVE JSON/NDJSON ingest is available at `POST /api/ids/suricata/eve` (admin token). Alert records become `SecurityEvent` rows for SOC export/KPI; full route correlation and live EVE tailing remain follow-ups. -- **Threat Intelligence**: STIX 2.x indicator/bundle ingest is available at `POST /api/threat-intel/stix` (admin token), MISP Event/attribute JSON ingest at `POST /api/threat-intel/misp` (admin token), TAXII 2.1 collection poll at `POST /api/threat-intel/taxii/poll` (admin token; Basic/Bearer optional), and OpenCTI observable/indicator export ingest at `POST /api/threat-intel/opencti` (admin token). All update `ThreatIndicator` / `DnsblEntry` plus feed freshness. Live MISP REST pull and live OpenCTI GraphQL pull remain follow-ups. +- **Threat Intelligence**: STIX 2.x indicator/bundle ingest is available at `POST /api/threat-intel/stix` (admin token), MISP Event/attribute JSON ingest at `POST /api/threat-intel/misp` (admin token), TAXII 2.1 collection poll at `POST /api/threat-intel/taxii/poll` (admin token; Basic/Bearer optional), OpenCTI observable/indicator export ingest at `POST /api/threat-intel/opencti` (admin token), and a CISA Known Exploited Vulnerabilities (KEV) catalog pull at `POST /api/threat-intel/cisa-kev` (admin token; fetches the official catalog and upserts one `cve` threat indicator per entry, severity escalated when CISA has tied the CVE to a known ransomware campaign). All update `ThreatIndicator` / `DnsblEntry` plus feed freshness. Live MISP REST pull and live OpenCTI GraphQL pull remain follow-ups. - **DNSBL Serving**: Hickory DNS should serve authoritative DNSBL responses directly after zone export semantics stabilize. - **AI SOC**: AI triage should summarize events, map likely ATT&CK tactics, and recommend actions. Enforcement-changing recommendations require human approval. diff --git a/docs/security/compliance-mapping.md b/docs/security/compliance-mapping.md index 8251d794..b956bad7 100644 --- a/docs/security/compliance-mapping.md +++ b/docs/security/compliance-mapping.md @@ -11,7 +11,7 @@ This document maps the commercial baseline to common enterprise security review | Change Control | Route-scoped monitor/block modes | Approval workflow and rollback attestations | | Availability | Health endpoint, Kubernetes probes | HA storage, multi-replica state backend | | Incident Response | Operations runbook and support bundle | On-call process, SLA/SLO reporting | -| Threat Intelligence | Feed import API, STIX/MISP/OpenCTI document ingest, TAXII 2.1 poll, feed status | Signed feeds, live MISP REST / OpenCTI GraphQL pull | +| Threat Intelligence | Feed import API, STIX/MISP/OpenCTI document ingest, TAXII 2.1 poll, CISA KEV catalog pull, feed status | Signed feeds, live MISP REST / OpenCTI GraphQL pull | | DNSBL | Zone export and response-code validation | Authoritative DNS service and publication controls | | AI Governance | Human approval boundary documented | Model evals, prompt audit, recommendation traceability | diff --git a/src/kev_import.rs b/src/kev_import.rs new file mode 100644 index 00000000..3c6e1e68 --- /dev/null +++ b/src/kev_import.rs @@ -0,0 +1,223 @@ +//! CISA Known Exploited Vulnerabilities (KEV) catalog import adapter. +//! +//! Parses the CISA KEV catalog JSON +//! () +//! into gateway [`ThreatIndicator`] rows. This is a proven federal +//! authoritative-source boundary — not a hand-rolled detection engine. +//! +//! The catalog is CVE-centric: entries carry no IP/domain/URL/hash +//! observable, so `dnsbl` stays empty. It is kept on [`KevImportMaterial`] +//! only for parity with the shared [`waf_ids_core::ThreatFeedImport`] shape +//! every adapter in this family produces. + +use waf_ids_core::{DnsblEntry, Severity, ThreatIndicator}; + +/// Parsed KEV import ready for the existing threat-feed upsert path. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct KevImportMaterial { + pub threats: Vec, + pub dnsbl: Vec, + pub skipped_entries: usize, +} + +/// Extract CVE indicators from a CISA KEV catalog JSON document. +/// +/// Accepts the real catalog shape (`{"vulnerabilities": [...], ...}`) or a +/// bare JSON array of vulnerability entries. +pub fn kev_material_from_value( + value: &serde_json::Value, + source: &str, + ttl_seconds: u64, +) -> Result { + let entries = match value { + serde_json::Value::Object(_) => value + .get("vulnerabilities") + .and_then(|v| v.as_array()) + .ok_or("KEV document must have a \"vulnerabilities\" array")?, + serde_json::Value::Array(items) => items, + _ => { + return Err( + "KEV document must be a catalog object or an array of vulnerability entries" + .to_string(), + ); + } + }; + if entries.is_empty() { + return Err("KEV catalog contained no vulnerability entries".to_string()); + } + + let mut threats = Vec::new(); + let mut skipped_entries = 0usize; + for entry in entries { + match kev_entry_outcome(entry, source, ttl_seconds) { + KevEntryOutcome::Mapped(threat) => threats.push(threat), + KevEntryOutcome::Skipped => skipped_entries += 1, + } + } + + if threats.is_empty() { + return Err("no KEV entries carried a usable cveID".to_string()); + } + + Ok(KevImportMaterial { + threats, + dnsbl: Vec::new(), + skipped_entries, + }) +} + +/// Parse a CISA KEV catalog JSON body string. +pub fn parse_kev_document( + body: &str, + source: &str, + ttl_seconds: u64, +) -> Result { + let trimmed = body.trim(); + if trimmed.is_empty() { + return Err("empty KEV catalog body".to_string()); + } + let value: serde_json::Value = + serde_json::from_str(trimmed).map_err(|error| format!("invalid KEV JSON: {error}"))?; + kev_material_from_value(&value, source, ttl_seconds) +} + +enum KevEntryOutcome { + Mapped(ThreatIndicator), + Skipped, +} + +fn kev_entry_outcome(entry: &serde_json::Value, source: &str, ttl_seconds: u64) -> KevEntryOutcome { + let Some(cve_id) = entry + .get("cveID") + .and_then(|v| v.as_str()) + .map(str::trim) + .filter(|s| !s.is_empty()) + else { + return KevEntryOutcome::Skipped; + }; + + // CISA's own inclusion criteria (confirmed active exploitation in the + // wild) already implies high severity; entries CISA has additionally + // tied to a known ransomware campaign are escalated to critical. + let known_ransomware = entry + .get("knownRansomwareCampaignUse") + .and_then(|v| v.as_str()) + .is_some_and(|s| s.eq_ignore_ascii_case("known")); + let severity = if known_ransomware { + Severity::Critical + } else { + Severity::High + }; + + KevEntryOutcome::Mapped(ThreatIndicator { + value: cve_id.to_ascii_uppercase(), + indicator_type: "cve".to_string(), + severity, + source: source.to_string(), + ttl_seconds, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn sample_catalog() -> &'static str { + r#"{ + "title": "CISA Catalog of Known Exploited Vulnerabilities", + "catalogVersion": "2026.08.27", + "dateReleased": "2026-08-27T17:00:36.6632Z", + "count": 2, + "vulnerabilities": [ + { + "cveID": "cve-2023-49105", + "vendorProject": "ownCloud", + "product": "ownCloud", + "vulnerabilityName": "ownCloud Improper Authentication Vulnerability", + "dateAdded": "2026-08-27", + "shortDescription": "ownCloud contains an improper authentication vulnerability.", + "requiredAction": "Apply mitigations in accordance with vendor instructions.", + "dueDate": "2026-08-30", + "knownRansomwareCampaignUse": "Unknown", + "notes": "https://owncloud.org/security", + "cwes": ["CWE-287"] + }, + { + "cveID": "CVE-2021-44228", + "vendorProject": "Apache", + "product": "Log4j2", + "vulnerabilityName": "Apache Log4j2 Remote Code Execution Vulnerability", + "dateAdded": "2021-12-10", + "shortDescription": "Apache Log4j2 JNDI features do not protect against attacker controlled LDAP.", + "requiredAction": "Apply mitigations in accordance with vendor instructions.", + "dueDate": "2021-12-24", + "knownRansomwareCampaignUse": "Known", + "notes": "", + "cwes": ["CWE-917", "CWE-400"] + } + ] + }"# + } + + #[test] + fn maps_catalog_entries_and_escalates_ransomware_severity() { + let material = parse_kev_document(sample_catalog(), "feed:cisa-kev", 86_400).unwrap(); + assert_eq!(material.threats.len(), 2); + assert!(material.dnsbl.is_empty()); + assert_eq!(material.skipped_entries, 0); + + let owncloud = material + .threats + .iter() + .find(|t| t.value == "CVE-2023-49105") + .expect("owncloud CVE normalized to uppercase"); + assert_eq!(owncloud.indicator_type, "cve"); + assert_eq!(owncloud.severity, Severity::High); + assert_eq!(owncloud.source, "feed:cisa-kev"); + assert_eq!(owncloud.ttl_seconds, 86_400); + + let log4j = material + .threats + .iter() + .find(|t| t.value == "CVE-2021-44228") + .expect("log4j CVE present"); + assert_eq!(log4j.severity, Severity::Critical); + } + + #[test] + fn accepts_bare_array_of_entries() { + let raw = r#"[{"cveID":"CVE-2024-0001","knownRansomwareCampaignUse":"Unknown"}]"#; + let material = parse_kev_document(raw, "feed:cisa-kev", 3600).unwrap(); + assert_eq!(material.threats.len(), 1); + assert_eq!(material.threats[0].value, "CVE-2024-0001"); + } + + #[test] + fn skips_entries_missing_cve_id() { + let raw = r#"{"vulnerabilities": [ + {"vendorProject": "NoId Inc", "knownRansomwareCampaignUse": "Unknown"}, + {"cveID": "CVE-2024-9999", "knownRansomwareCampaignUse": "Unknown"} + ]}"#; + let material = parse_kev_document(raw, "feed:cisa-kev", 3600).unwrap(); + assert_eq!(material.threats.len(), 1); + assert_eq!(material.skipped_entries, 1); + } + + #[test] + fn rejects_empty_and_non_kev_documents() { + assert!(parse_kev_document("", "s", 60).is_err()); + assert!(parse_kev_document("not-json", "s", 60).is_err()); + assert!(parse_kev_document(r#"{"foo":1}"#, "s", 60).is_err()); + assert!(parse_kev_document(r#"{"vulnerabilities": []}"#, "s", 60).is_err()); + assert!( + parse_kev_document(r#"{"vulnerabilities": [{"vendorProject":"x"}]}"#, "s", 60).is_err() + ); + } + + #[test] + fn parse_never_panics_on_arbitrary_text() { + for sample in ["", "{", "[]", "null", "\0", "{\"vulnerabilities\":[]}"] { + let _ = parse_kev_document(sample, "s", 60); + } + } +} diff --git a/src/lib.rs b/src/lib.rs index 8f54751d..9de4bdff 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -37,6 +37,7 @@ pub use waf_ids_core::{ mod coraza_audit; mod credentials; +mod kev_import; mod misp_import; mod opencti_import; mod stix_import; @@ -424,6 +425,29 @@ fn phishing_database_default_severity() -> Severity { Severity::High } +const KEV_DEFAULT_FEED_ID: &str = "cisa-kev"; +const KEV_DEFAULT_SOURCE: &str = "feed:cisa-kev"; +const KEV_DEFAULT_URL: &str = + "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json"; +const KEV_DEFAULT_TTL_SECONDS: u64 = 86_400; +const KEV_ALLOWED_HOSTS: &[&str] = &["www.cisa.gov"]; + +fn kev_default_feed_id() -> String { + KEV_DEFAULT_FEED_ID.to_string() +} + +fn kev_default_source() -> String { + KEV_DEFAULT_SOURCE.to_string() +} + +fn kev_default_url() -> String { + KEV_DEFAULT_URL.to_string() +} + +fn kev_default_ttl_seconds() -> u64 { + KEV_DEFAULT_TTL_SECONDS +} + fn default_true() -> bool { true } @@ -473,6 +497,7 @@ pub fn build_app(state: AppState) -> Router { .route("/api/threat-intel/misp", post(import_misp_document)) .route("/api/threat-intel/taxii/poll", post(poll_taxii_collection)) .route("/api/threat-intel/opencti", post(import_opencti_document)) + .route("/api/threat-intel/cisa-kev", post(import_kev_feed)) .route("/api/clearfolio/config", get(clearfolio_config)) .route("/api/clearfolio/documents/{kind}", post(clearfolio_submit)) .route("/api/clearfolio/jobs/{job_id}", get(clearfolio_status)) @@ -727,6 +752,29 @@ struct PhishingDatabaseImportRequest { allow_non_default_hosts: bool, } +#[derive(Debug, Clone, Deserialize, Serialize)] +struct KevImportRequest { + #[serde(default = "kev_default_feed_id")] + feed_id: String, + #[serde(default = "kev_default_source")] + source: String, + #[serde(default = "kev_default_url")] + kev_url: String, + #[serde(default = "kev_default_ttl_seconds")] + ttl_seconds: u64, + #[serde(default)] + allow_non_default_hosts: bool, +} + +#[derive(Debug, Serialize)] +struct KevImportResult { + feed_id: String, + upserted_threats: usize, + upserted_dnsbl: usize, + skipped_entries: usize, + last_updated_unix: u64, +} + #[derive(Serialize)] struct SocAnalyzeResponse { event_id: u64, @@ -1476,7 +1524,7 @@ async fn poll_taxii_collection( Ok(url) => url, Err(message) => return error(StatusCode::BAD_REQUEST, message), }; - if let Err(message) = validate_http_url(&base_url, /* allow_non_default_hosts */ true) { + if let Err(message) = validate_http_url(&base_url, /* allow_non_default_hosts */ true, &[]) { return error( StatusCode::BAD_REQUEST, format!("invalid TAXII objects URL: {message}"), @@ -2007,18 +2055,108 @@ fn validate_phishing_database_import_request( if request.domain_limit == 0 { return Err("domain_limit must be greater than zero when import_domains is enabled"); } - validate_http_url(&request.domain_url, request.allow_non_default_hosts)?; + validate_http_url( + &request.domain_url, + request.allow_non_default_hosts, + PHISHING_DATABASE_ALLOWED_HOSTS, + )?; } if request.import_ips { if request.ip_limit == 0 { return Err("ip_limit must be greater than zero when import_ips is enabled"); } - validate_http_url(&request.ip_url, request.allow_non_default_hosts)?; + validate_http_url( + &request.ip_url, + request.allow_non_default_hosts, + PHISHING_DATABASE_ALLOWED_HOSTS, + )?; } Ok(()) } -fn validate_http_url(value: &str, allow_non_default_hosts: bool) -> Result<(), &'static str> { +fn validate_kev_import_request(request: &KevImportRequest) -> Result<(), &'static str> { + if request.feed_id.trim().is_empty() { + return Err("feed_id is required"); + } + if request.source.trim().is_empty() { + return Err("source is required"); + } + if request.ttl_seconds == 0 { + return Err("ttl_seconds must be greater than zero"); + } + validate_http_url( + &request.kev_url, + request.allow_non_default_hosts, + KEV_ALLOWED_HOSTS, + ) +} + +async fn import_kev_feed( + State(state): State, + headers: HeaderMap, + Json(request): Json, +) -> Response { + if !admin_authorized(&state, &headers) { + return error(StatusCode::UNAUTHORIZED, "missing or invalid X-Admin-Token"); + } + if let Err(message) = validate_kev_import_request(&request) { + return error(StatusCode::BAD_REQUEST, message); + } + + let body_text = match fetch_text_feed(&state, &request.kev_url).await { + Ok(text) => text, + Err(message) => return error(StatusCode::BAD_GATEWAY, message), + }; + let material = match kev_import::parse_kev_document( + &body_text, + request.source.trim(), + request.ttl_seconds, + ) { + Ok(material) => material, + Err(message) => { + return error( + StatusCode::BAD_GATEWAY, + format!("invalid fetched KEV catalog: {message}"), + ); + } + }; + + let actor = audit_actor(&state, &headers); + let feed = ThreatFeedImport { + feed_id: request.feed_id.trim().to_string(), + source: request.source.trim().to_string(), + ttl_seconds: request.ttl_seconds, + threats: material.threats, + dnsbl: material.dnsbl, + }; + if let Err(message) = validate_threat_feed_import(&feed) { + return error( + StatusCode::BAD_GATEWAY, + format!("invalid fetched feed data: {message}"), + ); + } + let skipped_entries = material.skipped_entries; + match apply_threat_feed_import(&state, actor, "import_kev_feed", feed).await { + Ok(result) => ( + StatusCode::CREATED, + Json(KevImportResult { + feed_id: result.feed_id, + upserted_threats: result.upserted_threats, + upserted_dnsbl: result.upserted_dnsbl, + skipped_entries, + last_updated_unix: result.last_updated_unix, + }), + ) + .into_response(), + Err(message) => error(StatusCode::INTERNAL_SERVER_ERROR, message), + } +} + +fn validate_http_url( + value: &str, + allow_non_default_hosts: bool, + allowed_hosts: &[&str], +) -> Result<(), &'static str> { let parsed = reqwest::Url::parse(value).map_err(|_| "feed URL must be an absolute URL")?; let host = parsed.host_str().ok_or("feed URL host is required")?; match parsed.scheme() { @@ -2028,7 +2166,7 @@ fn validate_http_url(value: &str, allow_non_default_hosts: bool) -> Result<(), & _ => return Err("feed URL scheme must be http or https"), } if !allow_non_default_hosts - && !PHISHING_DATABASE_ALLOWED_HOSTS + && !allowed_hosts .iter() .any(|allowed| host.eq_ignore_ascii_case(allowed)) { @@ -2501,7 +2639,7 @@ async fn apply_threat_feed_import( async fn fetch_text_feed(state: &AppState, url: &str) -> Result { use futures_util::StreamExt; - validate_http_url(url, /* allow_non_default_hosts */ true) + validate_http_url(url, /* allow_non_default_hosts */ true, &[]) .map_err(|message| format!("invalid feed URL {url}: {message}"))?; let response = state .feed_http @@ -2788,6 +2926,9 @@ input,select{font:inherit;min-height:44px;padding:0 12px;border:1px solid var(--

OpenCTI threat intelligence

POST admin-authenticated OpenCTI GraphQL/list export JSON to /api/threat-intel/opencti (optional query: feed_id, source, ttl_seconds). Maps IPv4/IPv6, Domain-Name, Url, file hashes, and STIX indicators into threats/DNSBL. Live OpenCTI GraphQL pull is a follow-up.

+

CISA KEV catalog

+

POST admin-authenticated JSON to /api/threat-intel/cisa-kev with optional feed_id, source, kev_url (defaults to the official CISA Known Exploited Vulnerabilities catalog), and ttl_seconds. Fetches the catalog and upserts a cve threat indicator per entry (severity escalated to critical when CISA has tied the CVE to a known ransomware campaign). Fetch is restricted to the default CISA host unless allow_non_default_hosts is set.

+

CISA KEV catalog

-

POST admin-authenticated JSON to /api/threat-intel/cisa-kev with optional feed_id, source, kev_url (defaults to the official CISA Known Exploited Vulnerabilities catalog), and ttl_seconds. Fetches the catalog and upserts a cve threat indicator per entry (severity escalated to critical when CISA has tied the CVE to a known ransomware campaign). Fetch is restricted to the default CISA host unless allow_non_default_hosts is set.

+

POST admin-authenticated JSON to /api/threat-intel/cisa-kev with optional feed_id, source, and ttl_seconds. Fetches the deployment-configured CISA Known Exploited Vulnerabilities catalog URL (server-side config only, not part of this request) and upserts a cve threat indicator per entry (severity escalated to critical when CISA has tied the CVE to a known ransomware campaign).