Repository navigation
P7a: observed reachability from VPC flow logs - #36
Merged
Merged
Conversation
Predictive security roadmap P7, observed half (the user chose P7a over the threat-intel half, which stays deferred on a provider decision): - New lambdas/flow-log-analyzer: Athena over a vpc_flow_logs Glue table (partition projection over account/region/day; the standard v5 fields with the account id taken from the partition via RegexSerDe's output format) aggregating ACCEPT flows per network interface within a bounded window (FLOW_LOG_DAYS, default 7, max 31; at most 5000 interfaces per run, truncation logged) into TrafficObservations in DynamoDB — one row per interface with packet/byte/flow counts, first/ last seen, and a 30-day TTL. Wired as the flow-log-analyzer runtime command, a suspended-by-default CronJob (reusing the cloudtrail- analyzer IRSA role and Athena database/workgroup/results bucket), a NetworkPolicy egress component, and chart values (flowLogAnalyzer.*, config.trafficObservationsTable, cronJob.days). - Terraform: the observations table (always; TTL, deletion protection, CMK) and, only when flow_logs_bucket is set alongside the CloudTrail database it reuses, the flow-logs Glue table plus the analyzer role's writes/reads (table, Glue, bucket, optional KMS decrypt); the hub collector roles gain dynamodb BatchGetItem/GetItem on the table. - Collector: the EC2 stage reads the observations with the hub's own credentials (explicitly not the member's) and marks instances traffic_observed + traffic_observed_at. The read is tolerant — absent table, denied grant, or an analyzer that never ran all mean "none" and never fail a collection; bounded at 500 lookups per run. - Prediction and evidence: path likelihood reachability is observed AND configured — traffic observed on a workload pins the factor to 1 even when lambda_exposure_restricted/collection_restricted previously halved it — and the issue page lists an "Observed traffic" fact on instances. - Docs: README (Exposure row, env, repo map), ARCHITECTURE (new Observed reachability section + traffic_observed property), OPERATIONAL (component list + enabling flow logs runbook), CHANGELOG. Cost-gated by construction: Terraform creates nothing without flow_logs_bucket, the CronJob ships suspended, and every consumer degrades to "no observed traffic". Why this is safe: read-only everywhere except the new analyzer's own DynamoDB rows and graph facts derived from them; member credentials never cross into hub data; failed reads fail soft; and no severity, issue, or published claim changes shape.
The new workspace package.json was not in package-lock.json, so every CI job's `npm ci` failed with "Missing: flow-log-analyzer@1.0.0 from lock file" before running anything. Refreshed with the CI-pinned npm; the diff is only the new workspace entry (its dependencies already exist in the tree via cloudtrail-analyzer), and the scoped npm ci dry-run passes.
`input.regex` and `output.format` as bare object keys are HCL traversals, not literal keys, so terraform validate reported "Reference to undeclared resource". Quoted, they are the literal serde parameters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Predictive security roadmap P7a (observed reachability) — the observed half of P7, chosen over the threat-intel half (which stays deferred on a provider/format decision). 34 files, +1496/−33.
Analyzer — new
lambdas/flow-log-analyzervpc_flow_logsGlue table: partition projection account/region/day, the standard v5 default fields with the account id taken from the partition (RegexSerDeoutput.formatskips the file's account-id field)FLOW_LOG_DAYS, default 7, max 31; ≤5000 interfaces/run with logged truncation) intoTrafficObservations— one row per interface, counts + first/last seen, 30-day TTLflow-log-analyzerruntime command, a suspended-by-default CronJob reusing the cloudtrail-analyzer IRSA role, a NetworkPolicy egress component, and chart values (flowLogAnalyzer.*,config.trafficObservationsTable,cronJob.days)Terraform
TrafficObservationstable (TTL, deletion protection, CMK) always; the flow-logs Glue table and the analyzer role's permissions (DDB writes, Glue, bucket, optional KMS decrypt) only whenflow_logs_bucketis set alongside the CloudTrail database it reusesdynamodb:BatchGetItem/GetItemon the observations tableCollector + prediction
traffic_observed+ timestamp; tolerant: absent table/grant/never-run analyzer mean "none", bounded at 500 lookupsDocs: README, ARCHITECTURE (new section + property), OPERATIONAL (component list + enabling runbook with cost notes), CHANGELOG.
Deferred (CHANGELOG notes): P7b threat-intel enrichment/IOC feeds + adversary emulation; source-ASN/time-of-day baselines from P4; CDK doesn't create the table (the tolerant read covers it).
Why this is safe
flow_logs_bucket→ no Glue table, no role changes; CronJob ships suspended.Test plan
npx jest --runInBand— 18 projects / 1753 tests passed (new: analyzer queries/rows/retry/cap, collector lookup + instance fact, likelihood observed, runtime command validation/dispatch)npm run typecheck; full workspace lint;npm run format:checknpm run ci:build;npm run package:lambdas && npm run smoke:lambdasterraform fmt -check; helm/testdata values kept consistent (flowLogAnalyzer.imagedigest added to testdata); Deployment modules CI validates the schema + render