P7b-1: match customer threat-intel lists and rank matched findings

ARCHITECTURE.md

@@ -245,6 +245,24 @@ beside the configured ones. Everything is cost-gated: Terraform creates
 nothing without `flow_logs_bucket`, the CronJob ships suspended, and a
 deployment without the table simply has no observed traffic.
 
+## Threat-intelligence matching
+
+Customer-supplied intelligence arrives as plain S3 lists (one indicator per
+line): `threat-intel-import` validates them (IPs v4/v6, 12-digit account ids),
+caps each list, and writes `ThreatIndicators` with a TTL — freshness is the
+TTL, so an indicator dropped from a list drains itself. Nothing here calls the
+internet: AWS's own managed reputation lists need no import because GuardDuty
+applies them and the findings they drive already feed RULE-018/RULE-029. The
+rule runner joins the store against the facts rules already match: an anomaly's
+`source_ip` and a trusted `ExternalAccount`'s account id. A match adds a flat
+**+10 threat-intelligence factor** to the risk score (a fact about the path,
+not a proportional strength; recorded as `threatIntelMatches` on the issue and
+labelled on the score card) and matches clear on refresh once the indicator is
+gone. Matched issues do **not** auto-ticket until `/rules/quality` shows the
+measured precision P6 requires; the local
+`scripts/risk-plan` gate is unaffected (it has no store to read, so it never
+claims a match).
+
 ## Compliance reconciliation and the trust center
 
 The compliance job evaluates the CIS AWS Foundations v3.0.0 and Khalifa

CHANGELOG.md

@@ -2,6 +2,17 @@
 
 All notable changes to Khalifa.
 
+## [Unreleased] - Threat-intelligence matching (P7b-1)
+
+### Added
+
+- **Customer threat-intel matching**: `threat-intel-import` (runtime command; CronJob **suspended by default**, running as the rule-runner service account) imports plain S3 lists — one indicator per line, `#` comments, IPs (v4/v6) and 12-digit account ids — into a `ThreatIndicators` table with a TTL (`threat_intel_ttl_days`, default 90): freshness is the TTL, so a dropped indicator expires by itself, and invalid lines are skipped and counted in the log. Nothing calls the internet — the lists live in an account-owned bucket, and AWS's own managed reputation lists need no import because GuardDuty applies them (the findings they drive already feed RULE-018/RULE-029). Terraform creates the table, grants the rule-runner role read/write on it, and `s3:GetObject` on `threat_intel_list_bucket` when set; the list URIs, table name, and TTL flow through the chart config.
+- **Matches change ranking**: the rule runner joins the store against the facts rules already match — an anomaly's `source_ip` and a trusted `ExternalAccount`'s account id — and a match adds a flat **+10 `threatIntelContribution`** to the risk score (additive like exploitability; a fact about the path, not a proportional strength). The issue records `threatIntelMatches` (type, value, source) and the issue page renders a "Threat intelligence" card; a refresh clears matches whose indicator is gone. **No auto-ticketing** for matched issues until `/rules/quality` shows the measured precision P6 requires, and the offline `risk:plan` gate is unaffected (no store to read, so it never claims a match). The lookup is failure-tolerant: a missing table or grant scores without it and logs a warning.
+
+### Notes
+
+- P7b-2 (TTP mapping to MITRE ATT&CK and graph-only adversary-emulation scenarios) remains open — it depends on the curated mapping table this phase deliberately left out; and the importer does not fetch external feeds (commercial/open IoC subscriptions would add the deployment's first outbound-internet component and licensing constraints on storing indicators).
+
 ## [Unreleased] - Observed reachability (P7a)
 
 ### Added

OPERATIONAL.md

@@ -13,9 +13,10 @@
 7. **compliance** - Compliance CronJob
 8. **cloudtrail-analyzer** - CloudTrail usage CronJob (Athena → AccessAnalyzerCache; optional anomaly nodes via graph-writer)
 9. **flow-log-analyzer** - VPC flow-log CronJob (Athena → TrafficObservations; **suspended by default**, reuses the cloudtrail-analyzer IRSA role)
-10. **dspm-scanner** - CronJob: secret scanning of S3 objects and Lambda environment variables (Macie primary, regex fallback; findings written via graph-writer)
-11. **policy-evaluator / least-privilege-sweep** - Lambdas: effective permissions, escalation paths, rightsizing
-12. **IRSA roles** - IAM roles for pod authentication to Neptune/DynamoDB/SQS
+10. **threat-intel-import** - CronJob: imports customer threat-intel lists from S3 into ThreatIndicators (**suspended by default**, runs as the rule-runner service account)
+11. **dspm-scanner** - CronJob: secret scanning of S3 objects and Lambda environment variables (Macie primary, regex fallback; findings written via graph-writer)
+12. **policy-evaluator / least-privilege-sweep** - Lambdas: effective permissions, escalation paths, rightsizing
+13. **IRSA roles** - IAM roles for pod authentication to Neptune/DynamoDB/SQS
 
 ### Network Flow
 
@@ -201,6 +202,29 @@ kubectl create job --from=cronjob/khalifa-rule-runner rule-runner-manual -n secu
    the collector stops seeing fresh rows (a stale row still marks an instance
    until it expires — acceptable, traffic was real).
 
+### Runbook: Importing customer threat-intel lists
+
+1. Put plain lists in an account-owned S3 bucket: one indicator per line,
+   `#` comments and blanks ignored. IPs (v4/v6) in one object, 12-digit
+   account ids in another.
+2. In `khalifa-data`: set `threat_intel_list_bucket` (grants the rule-runner
+   role `s3:GetObject` on it) and `threat_intel_ip_list_s3_uri` /
+   `threat_intel_account_list_s3_uri` (flow into the chart's config), plus
+   `threat_intel_ttl_days` (default 90). Apply.
+3. Set `threatIntelImport.image` (the rule-runner digest) and unsuspend or
+   run it once:
+   `kubectl create job --from=cronjob/khalifa-threat-intel-import threat-intel-manual -n security-graph`.
+   An unreadable list fails the job (exit non-zero) so the failure is visible;
+   invalid lines are skipped and counted in the log.
+4. Verify: `ThreatIndicators` gains rows with a TTL; the next rule run scores
+   matching anomaly IPs / trusted external accounts with the +10
+   threat-intelligence factor and records the match on the issue.
+5. Freshness is the TTL: an indicator dropped from the list expires by itself
+   up to `threat_intel_ttl_days` after it was last imported. AWS's own managed
+   reputation lists need no import (GuardDuty applies them; RULE-018/RULE-029
+   already consume the findings). Auto-ticketing stays off for matched issues
+   until `/rules/quality` shows measured precision.
+
 ## I. Graph Query Profiling
 
 Neptune serves id lookups from the index; a property filter on a whole label

README.md

@@ -29,7 +29,7 @@ One member role. Evaluation stays in-account.
 | **CIEM**            | IAM decomposition, effective permissions, escalation paths (assume chains and `iam:PassRole` to compute, RULE-034), CloudTrail-grounded rightsizing.                                                                                                                                                                                                                                                         |
 | **Least privilege** | Sweep of roles and users. Deployable policy recommendations.                                                                                                                                                                                                                                                                                                                                                |
 | **Secrets**         | Secrets Manager and SSM parameter inventory with who-can-read analysis from identity and resource policies, rotation and access age, sharing detection (RULE-010, RULE-032, RULE-033), and admin users holding active access keys (RULE-031).                                                                                                                                                                  |
-| **Risk**            | Scheduled rules and attack-path traversals on the live graph (34 rules), with an exploitability factor that raises scores for paths carrying Amazon Inspector findings, an explainable likelihood on every path, a per-fix simulation that counts what each edge removal would clear, and a per-rule quality report (suppression vs resolution) that calibrates which rules to trust.                                                                                                                      |
+| **Risk**            | Scheduled rules and attack-path traversals on the live graph (34 rules), with an exploitability factor that raises scores for paths carrying Amazon Inspector findings, an explainable likelihood on every path, a per-fix simulation that counts what each edge removal would clear, matching against customer-supplied threat-intel lists (additive score bonus, TTL-fresh), and a per-rule quality report (suppression vs resolution) that calibrates which rules to trust.                                                                                                                      |
 | **Detection**       | GuardDuty detector coverage per region (including `MISSING`), Security Hub findings joined to internet-exposed resources (RULE-017, RULE-018), CloudTrail anomaly signals: root activity, secret reads from IPs outside each principal's own 30-day baseline, AccessDenied deviations from each principal's own daily baseline (RULE-020 to RULE-022), canary credential tripwires (RULE-023), secret scanning of Lambda environment variables plus IMDSv1-with-role detection (RULE-024), and Amazon Inspector CVEs on internet-exposed workloads (RULE-003). |
 | **Exposure**        | Public EKS control planes (RULE-019) plus public S3/RDS and world-open security groups, derived from collected endpoint and network config, with optional observed traffic from VPC flow logs confirming where traffic actually goes.                                                                                                                                                                                                                                                                 |
 | **Compliance**      | CIS AWS Foundations v3.0 and the Khalifa baseline, per-control evidence, mapped SOC 2 / ISO 27001 / PCI DSS / NIST 800-53 requirements, and CIS reconciliation against Security Hub before anything is publishable.                                                                                                                                                                                          |
@@ -158,7 +158,11 @@ EKS uses [`deploy/helm/khalifa/values.yaml`](deploy/helm/khalifa/values.yaml). T
 | `NEPTUNE_ENDPOINT` (auth)         | Graph access uses SigV4 from the execution role; no Neptune auth secret is read   | —                               |
 | `ISSUES_TABLE`                    | DynamoDB issues table                                                             | `SecurityIssues`                |
 | `ACCESS_ANALYZER_TABLE`           | CloudTrail usage cache                                                            | `AccessAnalyzerCache`           |
-| `TRAFFIC_OBSERVATIONS_TABLE`      | Observed-traffic cache for the EC2 collector (amber: empty disables the read)     | `TrafficObservations`           |
+| `TRAFFIC_OBSERVATIONS_TABLE`      | Observed-traffic cache for the EC2 collector (empty disables the read)            | `TrafficObservations`           |
+| `THREAT_INDICATORS_TABLE`         | Threat-intel indicators read by the rule runner                                   | `ThreatIndicators`              |
+| `THREAT_INTEL_IP_LIST_S3_URI`     | s3://bucket/key of the newline-separated IP indicator list (empty disables)       | —                               |
+| `THREAT_INTEL_ACCOUNT_LIST_S3_URI`| s3://bucket/key of the newline-separated account-id list (empty disables)         | —                               |
+| `THREAT_INTEL_TTL_DAYS`           | Days an imported indicator stays valid (DynamoDB TTL)                             | `90`                            |
 | `ATHENA_DATABASE`                 | Glue database for CloudTrail                                                      | `khalifa_cloudtrail_db`         |
 | `ATHENA_WORKGROUP`                | Athena workgroup                                                                  | `khalifa-cloudtrail-analysis`   |
 | `CLOUDTRAIL_S3_LOCATION`          | CloudTrail S3 prefix                                                              | `s3://cloudtrail-logs/AWSLogs/` |
@@ -272,7 +276,7 @@ reported as certified.
 
 | Path                                                                                                       | Contents                                                                                           |
 | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
-| [`lambdas/`](lambdas)                                                                                      | Collector, incremental, graph-writer, policy-evaluator, CloudTrail, flow-log, least-privilege, DSPM, runtime |
+| [`lambdas/`](lambdas)                                                                                      | Collector, incremental, graph-writer, policy-evaluator, CloudTrail, flow-log, threat-intel import, least-privilege, DSPM, runtime |
 | [`packages/risk-engine`](packages/risk-engine)                                                             | Rules, scoring, compliance                                                                         |
 | [`api-service/`](api-service) · [`ui/`](ui)                                                                | REST API and Next.js UI                                                                            |
 | [`deploy/`](deploy)                                                                                        | Helm chart, Terraform modules, fixtures                                                            |

deploy/helm/khalifa/templates/configmap.yaml

@@ -16,6 +16,10 @@ data:
   POSTURE_TRENDS_TABLE: {{ .Values.config.postureTrendsTable | quote }}
   ACCESS_ANALYZER_TABLE: {{ .Values.config.accessAnalyzerTable | quote }}
   TRAFFIC_OBSERVATIONS_TABLE: {{ .Values.config.trafficObservationsTable | quote }}
+  THREAT_INDICATORS_TABLE: {{ .Values.config.threatIndicatorsTable | quote }}
+  THREAT_INTEL_IP_LIST_S3_URI: {{ .Values.config.threatIntelIpListS3Uri | quote }}
+  THREAT_INTEL_ACCOUNT_LIST_S3_URI: {{ .Values.config.threatIntelAccountListS3Uri | quote }}
+  THREAT_INTEL_TTL_DAYS: {{ .Values.config.threatIntelTtlDays | quote }}
   LEAST_PRIVILEGE_TABLE: {{ .Values.config.leastPrivilegeTable | quote }}
   COGNITO_USER_POOL_ID: {{ .Values.config.cognitoUserPoolId | quote }}
   COGNITO_CLIENT_ID: {{ .Values.config.cognitoClientId | quote }}

deploy/helm/khalifa/templates/networkpolicies.yaml

@@ -1,5 +1,5 @@
 {{- if .Values.networkPolicies.enabled }}
-{{- range $component := list "api" "ui" "rule-runner" "org-collector" "incremental" "graph-writer" "compliance" "cloudtrail-analyzer" "flow-log-analyzer" "dspm-scanner" "policy-evaluator" "least-privilege-sweep" }}
+{{- range $component := list "api" "ui" "rule-runner" "org-collector" "incremental" "graph-writer" "compliance" "cloudtrail-analyzer" "flow-log-analyzer" "threat-intel-import" "dspm-scanner" "policy-evaluator" "least-privilege-sweep" }}
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:

deploy/helm/khalifa/templates/threat-intel-import.yaml

@@ -0,0 +1,70 @@
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: {{ include "khalifa.fullname" . }}-threat-intel-import
+  labels:
+    {{- include "khalifa.labels" . | nindent 4 }}
+    app.kubernetes.io/component: threat-intel-import
+spec:
+  schedule: {{ .Values.threatIntelImport.schedule | quote }}
+  {{- with .Values.threatIntelImport.timeZone }}
+  timeZone: {{ . | quote }}
+  {{- end }}
+  suspend: {{ .Values.threatIntelImport.suspend }}
+  concurrencyPolicy: {{ .Values.threatIntelImport.concurrencyPolicy }}
+  successfulJobsHistoryLimit: {{ .Values.threatIntelImport.successfulJobsHistoryLimit }}
+  failedJobsHistoryLimit: {{ .Values.threatIntelImport.failedJobsHistoryLimit }}
+  startingDeadlineSeconds: {{ .Values.threatIntelImport.startingDeadlineSeconds }}
+  jobTemplate:
+    metadata:
+      labels:
+        {{- include "khalifa.selectorLabels" (dict "root" . "component" "threat-intel-import") | nindent 8 }}
+    spec:
+      activeDeadlineSeconds: {{ .Values.threatIntelImport.activeDeadlineSeconds }}
+      ttlSecondsAfterFinished: {{ .Values.threatIntelImport.ttlSecondsAfterFinished }}
+      backoffLimit: {{ .Values.threatIntelImport.backoffLimit }}
+      template:
+        metadata:
+          labels:
+            {{- include "khalifa.selectorLabels" (dict "root" . "component" "threat-intel-import") | nindent 12 }}
+            {{- with .Values.threatIntelImport.podLabels }}
+            {{- toYaml . | nindent 12 }}
+            {{- end }}
+          {{- with .Values.threatIntelImport.podAnnotations }}
+          annotations:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
+        spec:
+          # The rule-runner service account owns the ThreatIndicators table:
+          # the runner reads it and this import writes it.
+          serviceAccountName: {{ include "khalifa.serviceAccountName" (dict "root" . "key" "ruleRunner" "suffix" "rule-runner") }}
+          automountServiceAccountToken: {{ .Values.serviceAccounts.ruleRunner.automountServiceAccountToken }}
+          restartPolicy: Never
+          {{- with .Values.imagePullSecrets }}
+          imagePullSecrets:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
+          securityContext:
+            {{- include "khalifa.podSecurityContext" . | nindent 12 }}
+          {{- if .Values.topologySpread.enabled }}
+          topologySpreadConstraints:
+            {{- include "khalifa.topologySpreadConstraints" (dict "root" . "component" "threat-intel-import") | nindent 12 }}
+          {{- end }}
+          containers:
+            - name: threat-intel-import
+              image: {{ required "threatIntelImport.image must be an immutable repository@sha256 digest" .Values.threatIntelImport.image | quote }}
+              imagePullPolicy: IfNotPresent
+              args: ["threat-intel-import"]
+              securityContext:
+                {{- include "khalifa.containerSecurityContext" . | nindent 16 }}
+              envFrom:
+                - configMapRef:
+                    name: {{ include "khalifa.fullname" . }}-config
+              resources:
+                {{- toYaml .Values.threatIntelImport.resources | nindent 16 }}
+              volumeMounts:
+                - name: tmp
+                  mountPath: /tmp
+          volumes:
+            - name: tmp
+              emptyDir: {}