diff --git a/docs/adr/2026-09-05-outbound-site-reputation-engine.md b/docs/adr/2026-09-05-outbound-site-reputation-engine.md new file mode 100644 index 0000000..602c2b8 --- /dev/null +++ b/docs/adr/2026-09-05-outbound-site-reputation-engine.md @@ -0,0 +1,51 @@ +# ADR-2026-09-05-OUTBOUND-REPUTATION: Wardnet owns outbound site security reputation + +- Date: 2026-09-05 +- Status: Proposed for architectural review; not an implemented capability. +- Scope: Wardnet security policy, threat evidence, and SOC accountability for outbound destinations. +- Baseline: `main@a52ccd0a24a727d9349bb32def7713882d8cad1e`. +- Product request: prevent internal users, services, and agents from contacting external services with adverse security reputation. + +## Context + +Wardnet already owns WAF/IDS/SOC gateway decisions, threat-intelligence ingestion, and DNSBL publishing. Its current `/gateway/{path}` evaluates an incoming request before forwarding to a configured upstream. That is not a company-wide outbound website reputation enforcement system. The current `ThreatIndicator` and `SecurityEvent` models also lack the complete destination, evidence-lifecycle, tenant, and policy-version contract proposed here. + +The objection to using Wardnet as a generic website-reputation utility is valid when it means putting crawler rankings, extraction difficulty, or an unrelated HTTP client into the WAF. It does not exclude a distinct security responsibility: deciding whether an identified internal workload may contact an external destination under current threat intelligence and organizational policy. This decision complements ingress WAF and IDS observations rather than reinterpreting them. + +Current owner evidence in [Wardnet #136](https://github.com/ContextualWisdomLab/wardnet/pull/136), [#115](https://github.com/ContextualWisdomLab/wardnet/pull/115), and [EgressWeave #237](https://github.com/ContextualWisdomLab/EgressWeave/issues/237) assigns reusable outbound HTTP destination/address, DNS, redirect, proxy, TLS, and resource authorization to EgressWeave. This ADR does not reverse that boundary. The EgressWeave GitHub Releases listing returned no releases during the 2026-09-06 review; an immutable compatible Rust-consumer contract is an integration prerequisite, not a reason to defer Wardnet's domain design or offline implementation. + +## Decision + +Add **Outbound Site Reputation** as a bounded security capability owned by Wardnet. Implement its future deterministic policy and evidence core in a separate Rust workspace crate, with Wardnet application adapters and a versioned evaluation surface. Keep it independently testable without network access, an LLM, or a sibling repository checkout. + +Wardnet owns destination maliciousness assessment, evidence admission and lifecycle, tenant/workload/purpose policy, scoped business exceptions, explanatory decisions, and SOC records. EgressWeave owns safe transport and connection authority. A policy enforcement point composes the two; neither product's allow decision can override the other's denial. Protect-mode forwarding requires authenticated context, an explicit Wardnet allow, a valid EgressWeave authorization bound to the actual connection, a current applicable evidence state with healthy required-source state, and the required audit reservation. A no-match `unknown` destination therefore does not require a fabricated matching indicator: it may receive a Wardnet allow only through an exact-scope valid business authorization while required evidence authorities remain healthy, after which transport and audit gates still apply. + +A reputation verdict is not proof that traffic was intercepted or blocked. Coverage and actual enforcement outcomes must be recorded separately. Company-wide protection requires an enforced network path; a voluntary SDK integration, DNS feed, or the present reverse-proxy endpoint alone is insufficient. + +The normative requirements, threat cases, and delivery boundaries are in the [design](../superpowers/specs/2026-09-05-outbound-site-reputation-design.md); the [implementation plan](../superpowers/plans/2026-09-05-outbound-site-reputation.md) divides independently testable slices. + +## Alternatives considered + +| Alternative | Benefit | Rejection or consequence | +| --- | --- | --- | +| Extend the existing WAF score/DNSBL matcher into a universal site score | Minimal apparent code change | Conflates request payload, source IP, destination identity, and evidence lifetimes. An IP DNSBL is not a domain/URL reputation model. | +| Put maliciousness policy inside EgressWeave, or introduce a new mandatory general reputation service | Central-looking interface | Moves SOC/security authority into the transport owner or adds a premature independent deployment. External intelligence providers remain useful inputs, not owners of Wardnet business policy. | +| Separate Wardnet reputation core with versioned transport integration | Explicit security responsibility, deterministic testing, reusable standalone Rust boundary | Selected. Requires disciplined schemas, interception coverage, and an immutable compatible transport contract before production integration. | + +## Safety and non-goals + +Unknown is not safe. Expired, deleted, revoked, unverifiable, or out-of-scope evidence cannot silently authorize traffic. Freshness and source health remain distinct from maliciousness. Correlated feeds are not independent votes; provider confidence is not a calibrated probability. Do not inherit `BLOCK_SCORE` or invent a weighted machine-learning score for this feature. + +Business exceptions are authenticated, exact-scope, expiring, audited grants for unknown destinations; they do not erase evidence or override hard threat or transport denials. No raw URLs containing credentials, request bodies, or tokens are exported to providers or ordinary logs. + +This capability does not own crawling, stealth, CAPTCHA solving, website popularity, content credibility, general HTTP transport, malware detonation, LLM routing, DLP, or a full secure-web-gateway product. CAPTCHA, robots rules, HTTP 403/429, domain novelty, and geography alone are not maliciousness evidence. Optional future analysis must use its canonical owner and cannot automatically widen enforcement authority. + +## Consequences and adoption + +The design adds a deliberate security bounded context without duplicating EgressWeave. It creates operating costs: feed licensing and freshness, false-positive handling, policy rollout, audit durability, and deployment coverage. Opaque HTTPS supports only the identities actually observed; full URL protection cannot be advertised without URL visibility. + +This documentation PR changes no runtime, deployment, dependency, or workflow. Its ADR remains Proposed until reviewed. Offline domain work can proceed independently; live enforcement is gated on the versioned owner contract and the plan's security acceptance. Existing preservation PRs are neither merged nor superseded by this record. PR #130 remains the sole writer of `docs/product-technical-gap-baseline.md`; this change does not create a competing ledger. + +## Evidence + +See [research and source traceability](../papers/outbound-site-reputation-sources.md). Protective DNS practice supports the security use case; DNS reputation research motivates evidence and temporal evaluation, not a claim that Wardnet already reproduces a published detector. Repository observations above describe the pinned baseline and inspected open work, not shipped future features. diff --git a/docs/papers/outbound-site-reputation-sources.md b/docs/papers/outbound-site-reputation-sources.md new file mode 100644 index 0000000..58a4c2b --- /dev/null +++ b/docs/papers/outbound-site-reputation-sources.md @@ -0,0 +1,68 @@ +# Outbound site reputation: research and source traceability + +Reviewed 2026-09-06. This register supports the [ADR](../adr/2026-09-05-outbound-site-reputation-engine.md) and [design](../superpowers/specs/2026-09-05-outbound-site-reputation-design.md). Engineering requirements and benchmark targets are Wardnet proposals, not claims that the cited systems or this repository already implement them. + +## R1. DNS reputation research + +Antonakakis, M., Perdisci, R., Dagon, D., Lee, W., & Feamster, N. (2010). Building a dynamic reputation system for DNS. In *19th USENIX Security Symposium*. USENIX Association. https://www.usenix.org/conference/usenixsecurity10/building-dynamic-reputation-system-dns + +Author-group overview: https://astrolavos.gatech.edu/2010/08/11/Building_a_Dynamic_Reputation_System_for_DNS/ + +**Application:** Notos is relevant evidence that DNS reputation is a security-analysis problem, not a crawler-quality metric. Wardnet therefore keeps temporal evidence and separates address observations from domain assessment. **Limit:** This proposal does not reproduce Notos, adopt its trained model, or transfer its reported accuracy to Wardnet. Source-family and time-separated evaluation are project safeguards, not assertions that a feed match achieves those research results. + +## R2. Passive DNS analysis + +Bilge, L., Kirda, E., Kruegel, C., & Balduzzi, M. (2011). EXPOSURE: Finding malicious domains using passive DNS analysis. In *Network and Distributed System Security Symposium*. Internet Society. https://www.ndss-symposium.org/ndss2011/exposure-finding-malicious-domains-using-passive-dns-analysis/ + +Author-institution record: https://www.eurecom.fr/en/publication/3281 + +**Application:** The work concerns malicious-domain identification from passive DNS behavior. It motivates retaining observation provenance and evaluating detection over time rather than declaring every unseen domain benign. **Limit:** Passive-DNS analytics and a learned detector are not required for the first deterministic, proven-feed-based Wardnet implementation. No published detection rate is a product acceptance result. + +## R3. Threat-intelligence semantics + +Jordan, B., Piazza, R., & Darley, T. (Eds.). (2021, June 10). *STIX version 2.1* (OASIS Standard). OASIS Open. https://docs.oasis-open.org/cti/stix/v2.1/os/stix-v2.1-os.html + +**Application:** Sections on common properties, versioning, indicators, and markings distinguish creator confidence, validity, revocation, and distribution restrictions. Wardnet must preserve these dimensions instead of reducing all imported data to a permanent numeric score. Missing confidence is not a safety statement. **Limit:** A STIX-shaped payload is not automatically authentic, current, trustworthy, or authorized for a particular tenant; source admission remains necessary. Only supported patterns become enforcement material. + +## R4. Protective DNS operational guidance + +National Cyber Security Centre. (n.d.). *Protective DNS for the private sector*. Retrieved September 5, 2026, from https://www.ncsc.gov.uk/guidance/protective-dns-for-private-sector + +**Application:** Protective DNS addresses access to malicious destinations and provides operational security evidence. This supports Wardnet's outbound-security use case and the need to connect enforcement with monitoring and false-positive handling. **Limit:** DNS filtering is one control, not proof of complete traffic interception or visibility into encrypted URL paths. The proposed PEP and deployment acceptance must establish their own coverage. + +## R5. Malware URL intelligence + +abuse.ch. (n.d.). *URLhaus API*. Retrieved September 5, 2026, from https://urlhaus.abuse.ch/api/ + +**Application:** A proven malware-URL source is a candidate input. A URL observation and a derived hostname block have different scopes; the adapter must retain that distinction and source notices. **Limit:** Eligibility, current API authentication, polling bounds and commercial use must be checked before enabling a provider. Community access is not an unlimited commercial redistribution grant. Do not send private per-request URLs to an external provider in v1. + +## R6. IOC lifecycle and provider access + +abuse.ch. (n.d.). *ThreatFox API*. Retrieved September 5, 2026, from https://threatfox.abuse.ch/api/ + +**Application:** ThreatFox documents authenticated access and expiration of older indicators, illustrating why an imported IP must not remain a timeless verdict. Preserve source status and provenance rather than extending validity at every refresh. **Limit:** Provider-specific expiration is not a universal TTL for every source. Use the current access/usage contract, maintain TLS verification, and obtain any required commercial entitlement before production ingestion. + +## R7. Credential-bearing transport confidentiality + +MITRE. (2026). *CWE-319: Cleartext transmission of sensitive information* (CWE Version 4.20). https://cwe.mitre.org/data/definitions/319.html + +**Application:** CWE-319 identifies transmission of sensitive/security-critical data over cleartext channels as a confidentiality/integrity weakness and recommends reliable confidentiality-protecting cryptographic protocols for transmission. Wardnet therefore requires the released EgressWeave/PEP composition to prove authenticated encrypted transport before a credential-bearing hop; an initial HTTP request, HTTPS-to-HTTP downgrade, redirect, or retry must strip credentials before the unsafe hop or be rejected. **Limit:** Wardnet does not implement a second TLS stack or infer transport security from scheme text alone. The transport owner must supply the executable peer-bound proof, and deployment tests must observe zero credential bytes on cleartext paths. + +## Repository evidence and ownership + +Inspected protected Wardnet commit: `a52ccd0a24a727d9349bb32def7713882d8cad1e`. + +- [AGENTS.md](https://github.com/ContextualWisdomLab/wardnet/blob/a52ccd0a24a727d9349bb32def7713882d8cad1e/AGENTS.md): Rust-first, proven engines, configuration, research and governance constraints. +- [Architecture](https://github.com/ContextualWisdomLab/wardnet/blob/a52ccd0a24a727d9349bb32def7713882d8cad1e/docs/architecture.md) and [core models](https://github.com/ContextualWisdomLab/wardnet/blob/a52ccd0a24a727d9349bb32def7713882d8cad1e/crates/waf-ids-core/src/lib.rs): actual gateway, ingestion, DNSBL and event baseline. +- [Wardnet #136](https://github.com/ContextualWisdomLab/wardnet/pull/136) and [#115](https://github.com/ContextualWisdomLab/wardnet/pull/115): preserve consumer/feed evidence without a duplicate transport-policy owner. +- [EgressWeave #237](https://github.com/ContextualWisdomLab/EgressWeave/issues/237): immutable Rust-compatible transport authorization contract. The [GitHub Releases listing](https://api.github.com/repos/ContextualWisdomLab/EgressWeave/releases?per_page=1) returned an empty array at review time; this observation is not a claim about every possible package registry. +- [Wardnet #167](https://github.com/ContextualWisdomLab/wardnet/pull/167) and [#170](https://github.com/ContextualWisdomLab/wardnet/pull/170): ongoing MISP lifecycle and source-severity repairs, not presumed protected behavior. +- [Wardnet #130](https://github.com/ContextualWisdomLab/wardnet/pull/130): sole product-gap ledger writer. The design PR adds separate documents and does not edit that ledger. + +Open PR/issue descriptions are dependency and ownership evidence as inspected on 2026-09-06, not immutable API releases or proof of shipped implementation. Re-read them before implementation. + +## Research artifact and redistribution decision + +No third-party PDF is committed in this documentation slice. USENIX makes the Notos paper openly accessible, but the reviewed page did not establish an explicit public-repository redistribution grant. The EXPOSURE institutional [copyright notice](https://www.eurecom.fr/en/publication/3281/copyright) permits personal use; that is not treated as permission to redistribute the full paper in this repository. Follow AGENTS.md's cite/link/original-summary fallback rather than infer rights from download availability. + +The STIX standard and CWE entry are cited in their normative/authoritative web forms; they are standards/weakness taxonomies, not academic-paper substitutes. Provider datasets, API examples and malicious samples are not vendored. Source-specific licensing, attribution, access controls and distribution markings must be recorded by any future adapter. This register contains original summaries and bibliographic references, not copied papers or datasets. diff --git a/docs/superpowers/plans/2026-09-05-outbound-site-reputation.md b/docs/superpowers/plans/2026-09-05-outbound-site-reputation.md new file mode 100644 index 0000000..540ae80 --- /dev/null +++ b/docs/superpowers/plans/2026-09-05-outbound-site-reputation.md @@ -0,0 +1,125 @@ +# Outbound Site Reputation Implementation Plan + +> **For agentic workers:** Execute one bounded task at a time with the executing-plans or subagent-driven-development workflow. Record a failing test before its causal implementation; retain exact-head review and validation evidence. Unchecked items below are planned work, not completed code. + +**Goal:** Make outbound destination security reputation an explainable Wardnet admission capability. +**Architecture:** A pure Rust reputation core consumes immutable canonical evidence. Wardnet adapters own authenticated policy/SOC use cases; a released EgressWeave boundary owns actual safe transport. +**Tech Stack:** Rust workspace, versioned serialized contracts, existing Wardnet management and credential boundaries. No dependency is added by this documentation PR. +**Spec:** [Product and technical design](../specs/2026-09-05-outbound-site-reputation-design.md). + +## Global constraints + +Use the [ADR](../../adr/2026-09-05-outbound-site-reputation-engine.md) and [source register](../../papers/outbound-site-reputation-sources.md). Start each implementation slice from then-current protected `main`, not a preservation branch. Proposed paths below must be reconciled with protected renames before creation. + +No duplicated DNS/redirect/proxy/TLS/resource authority, mutable sibling dependency, raw runtime environment read, WAF-score reuse, or unreviewed detector. No engine implementation, tests, API, or deployment is supplied by this documentation PR. No code coverage or production readiness is claimed. + +Tasks 1-3 can develop offline before EgressWeave releases its Rust-consumer boundary. Task 5 and protect deployment require [EgressWeave #237](https://github.com/ContextualWisdomLab/EgressWeave/issues/237) or an immutable compatible successor. Missing contract support must not be replaced with local `reqwest` authorization logic. Configuration/authentication/durable-state prerequisites remain their canonical Wardnet owner lanes (#140, #155, #80/#81); adopt protected implementations rather than copying open PR code. + +## Task 1: Versioned domain contracts and fixture harness + +**Create:** `crates/wardnet-reputation-core/{Cargo.toml,src/lib.rs,src/model.rs,tests/contract.rs}` and `tests/fixtures/reputation/v1/`. +**Modify:** root `Cargo.toml` workspace membership and lockfile only as required. +**Consumes:** reviewed spec, canonical offline fixtures, injected evaluation time. +**Produces:** `DestinationContextV1`, `EvidenceRecordV1`, `SourcePolicyV1`, `PolicySnapshotV1`, `DecisionEnvelopeV1`, and typed validation errors. No network-facing URL parser. + +- [ ] Add RED serialization/validation tests rejecting wrong direction, unknown schema, blank workload/purpose, missing required-source policy, invalid time ordering, and ambiguous subject scope. Assert round-trip stability on a synthetic exact-host fixture. +- [ ] Run `cargo test -p wardnet-reputation-core --test contract`; record the missing-contract failures before adding production models. +- [ ] Implement closed enums, bounded fields, and fallible constructors with injected time. Keep identity verification outside caller-controlled deserialization. +- [ ] Repeat the focused test, then all workspace gates below. Commit only this contract/harness slice. + +Fixture convention: every case includes `case_id`, fixed `now_unix`, canonical profile ID, authenticated context, source policy, evidence, expected assessment/action/reason, and expected visibility. Use reserved synthetic domains and test networks only. The integration harness must explicitly allow its loopback fixtures; no live malicious service is contacted. + +## Task 2: Source lifecycle and immutable evidence snapshots + +**Create:** core `src/evidence.rs`, `tests/evidence_lifecycle.rs`; adapter `src/reputation/{mod.rs,ingest.rs,snapshot.rs}`. +**Consumes:** Task 1 contracts and authenticated source envelopes. +**Produces:** validated source batches, monotonic source versions/tombstones, atomic `EvidenceSnapshotV1` generations with required-source health. + +- [ ] Add RED cases for reimport/304 rejuvenation, expiry during cache lifetime, old-version replay after revocation, incomplete pagination, malformed/empty failure response, and failed replacement preserving valid last-known-good records. +- [ ] Add distinct tests for a complete authenticated empty snapshot versus an incomplete delta. Require atomic rejection rather than partially publishing a malformed batch. +- [ ] Implement source-specific admission, validity and lineage rules. Preserve confidence/markings and provenance; reject unsupported indicator syntax. Confirm MISP affirmative `to_ids` plus attribute/object lifecycle and source severity against protected successors to #167/#170 before enabling that adapter. +- [ ] Run focused lifecycle tests and workspace gates. Commit only evidence lifecycle and its adapters. Network refresh remains disabled until Task 5; use imported authenticated fixtures meanwhile. + +## Task 3: Matching, policy lattice, and bounded cache + +**Create:** core `src/{matching.rs,policy.rs}`, `tests/{matching.rs,policy.rs}`; adapter `src/reputation/cache.rs`. +**Consumes:** validated canonical context, policy, evidence snapshot, injected time. +**Produces:** deterministic `evaluate(context, policy, evidence, now) -> DecisionEnvelopeV1`; cache entries cannot bypass revalidation. + +- [ ] Write RED table-driven tests for the acceptance matrix below. In particular, no-match is `unknown`; without scoped authorization protect denies, while the same no-match case with an exact-scope valid business authorization and healthy required sources may produce the Wardnet allow. Business allow cannot defeat hard deny, and duplicate syndicated evidence cannot change a decision by arithmetic accumulation. +- [ ] Implement exact subject matching, explicit subdomain scope, source eligibility, stable reasons, and deny precedence. Keep maliciousness assessment, required-source health, and policy reason separate so a higher-precedence authority failure can explain a deny without rewriting an existing adverse assessment. No substring or registrable-domain widening; no `BLOCK_SCORE` reuse. +- [ ] Implement tenant/context/revision-bound cache keys, finite cardinality, minimum expiry, and invalidation. Add property tests: unrelated tenant evidence never changes a result; record permutation preserves a decision; adding a duplicate never raises confidence. +- [ ] Run focused tests and workspace gates, then commit. Add libFuzzer targets and stable property mirrors for the new untrusted structures in a Wardnet-owned fuzz change; do not copy central workflows. + +## Task 4: Authenticated evaluation and accountable policy administration + +**Create:** `src/reputation/{api.rs,application.rs,audit.rs}` and `tests/reputation_api.rs`. +**Modify:** `src/lib.rs` only for module/router wiring after the core boundary stabilizes. +**Consumes:** Task 3 evaluator, protected identity/configuration/storage capabilities. +**Produces:** the proposed `/api/v1/egress/reputation/*` surfaces, revisioned policy/exception state, immutable decision records, and separate enforcement-outcome correlation. + +- [ ] Add RED tests: forged tenant/workload, evaluator attempting policy mutation, stale If-Match, expired exception, secret-bearing input redaction, and HTTP 200 with deny not yielding a grant. +- [ ] Test 400/401/403/429/503 paths and audit-capacity exhaustion: none may produce protected authorization. A pure evaluation result cannot self-assert actual enforcement. +- [ ] Implement role-scoped handlers and authenticated, audience-bound grant handling at the adapter. Require durable audit reservation before protected allow. Reuse the canonical registry/outbox; do not substitute process-local state for production evidence. +- [ ] Run focused HTTP tests and workspace gates, document operator procedures, and commit this service slice. Keep live protect capability disabled. + +## Task 5: Released transport contract and real enforcement + +**Create:** `src/reputation/egress_acl.rs`, `tests/reputation_egress_contract.rs`, and `tests/reputation_enforcement.rs`. +**Consumes:** an immutable compatible EgressWeave release, Task 4 decisions, authenticated PEP context. +**Produces:** an owner-backed pre-connect/pre-send gate with actual-peer and per-hop correlation. The exact foreign API is selected by its owner, not invented in this plan. + +- [ ] Verify release/schema/artifact identity, provenance, compatibility, end-to-end deadline, peer-bound execution guarantees, and authenticated encrypted transport semantics for credential-bearing hops. If any is absent, leave integration unavailable; continue offline work only. +- [ ] Port the useful hostile requirements from #136 as black-box consumer vectors, not its local transport implementation. Preserve Wardnet-owned source refresh behavior from #115 through the released ACL, not its direct client. +- [ ] Start controlled DNS/HTTP fixtures and record RED tests for rebinding, mixed peers, malicious second redirect, ambient proxy influence, pool/coalesced-origin reuse, stale/replayed grants, and stalled DNS exceeding the whole operation deadline. +- [ ] Add credential-path RED cases for an initial `http://` target, HTTPS-to-HTTP downgrade, redirected credential-bearing request, and retry to a target whose authenticated encryption cannot be established. The required result is credential stripping before that hop or rejection with zero cleartext credential bytes observed by the fixture; successful cleartext fallback is never acceptable. +- [ ] Implement the thin ACL and PEP composition. Require zero denied-destination connections/payload hits as appropriate, not merely a deny response after transmission. The PEP consumes the released transport proof; it does not create a second TLS policy implementation. +- [ ] Verify 60-second maximum protected tunnel lease and denial propagation, transport outage fail-closed behavior, and incompatible release rejection. Run all gates and commit without deploying automatically. + +## Task 6: Operations, coverage, and controlled activation + +**Create:** `docs/runbooks/outbound-site-reputation.md`, `tests/reputation_rollout.rs`, and reproducible load/replay fixtures under `tests/fixtures/reputation/`. +**Consumes:** Tasks 1-5, deployment egress inventory, reviewed benign/malicious evaluation corpora. +**Produces:** measured protection scope, SLO evidence, rollback procedure, and audited activation decision. + +- [ ] Add RED acceptance for audit/outbox saturation, feed outage/recovery, policy rollback without tombstone resurrection, and secret-free exports. +- [ ] Test opaque CONNECT URL-inspection refusal, direct exits, unauthorized DNS/DoH/DoT, QUIC and proxy bypasses against the actual deployment. Unsupported paths remain uncovered or blocked, never implicitly protected. +- [ ] Measure the spec's one-million-indicator/1,000-evaluation-per-second benchmark with hardware, memory and p99 recorded. Separately report detection/false-positive/unknown rates and actual PEP enforcement coverage; shadow events do not count as prevention. +- [ ] Execute offline replay, monitor, and a small protect canary. Expand only after analyst-reviewed false positives, exact-head security evidence, denied-target zero-hit tests, outage drills, and rollback verification. +- [ ] Hand the final feature/evidence links to the existing #130 product-gap ledger owner without editing its path in a competing lane. + +## Acceptance matrix + +| ID | Hostile or realistic input | Required observation | +| --- | --- | --- | +| REP-01 | Active eligible C2/phishing indicator plus business allow | Hard deny; zero denied-target payload hits | +| REP-02 | No eligible match, with and without exact-scope valid business authorization | Assessment remains `unknown`; protect denies without authorization, while exact authorization plus healthy required sources may yield the Wardnet allow subject to independent transport and audit gates | +| REP-03 | Expired/revoked evidence reimported or HTTP 304 received | No renewed eligibility; required-source failure cannot allow | +| REP-04 | MISP invalid/missing `to_ids` or deleted enclosing Object | No admitted enforcement indicator | +| REP-05 | Older source version after tombstone; truncated replacement | No resurrection or partial snapshot publication | +| REP-06 | Similar suffix, sibling shared host, CNAME alias | Exact scope respected; no blanket host condemnation | +| REP-07 | Cross-tenant cache or forged context/grant | Deny; no foreign evidence disclosure | +| REP-08 | Good first hop redirects to malicious second hop | No second-hop payload; separately correlated decisions | +| REP-09 | DNS peer changes, mixed A/AAAA, mapped address alias | Actual peer bound; no unchecked fallback | +| REP-10 | Policy/evidence update during cached or pooled request | Revalidation; no stale positive authorization | +| REP-11 | CONNECT without URL visibility | No full-URL claim; URL-required profile denies | +| REP-12 | Evaluator/transport/audit outage or exhausted bound | Stable fail-closed reason; no grant | +| REP-13 | 200 deny envelope, replayed nonce, wrong audience | No authorization based on status or copied receipt | +| REP-14 | Optional feed outage with healthy required sources | Explicit degraded health; policy still deterministic | +| REP-15 | Direct/alternate egress path bypass attempt | Network block or explicit uncovered classification | +| REP-16 | Secret-bearing URL, cookie, source credential | No secret in logs, exports, error text or provider query | +| REP-17 | Newly denied persistent connection | Terminated within supported 60-second lease bound | +| REP-18 | Duplicate/permuted syndicated evidence | No manufactured independent votes or probability | +| REP-19 | Credential-bearing initial HTTP, HTTPS-to-HTTP downgrade, redirect, or retry without authenticated encryption | Strip credentials before the unsafe hop or reject; fixture observes zero credential bytes on cleartext transport | + +## Verification and merge discipline + +Every code slice must record focused RED then GREEN results and run: + +```bash +cargo fmt --check +cargo test --locked --workspace +cargo clippy --locked --workspace --all-targets -- -D warnings +``` + +Add the affected stable property tests and libFuzzer evidence, source/dependency/security scans, and deployment conformance where applicable. Re-read the exact PR head, protected base, reviews, unresolved threads, and required checks before any merge. Queued, skipped, stale, or absent evidence is not passing. Do not force-push, self-approve, weaken gates, merge a preservation branch wholesale, or consume a mutable foreign dependency. diff --git a/docs/superpowers/specs/2026-09-05-outbound-site-reputation-design.md b/docs/superpowers/specs/2026-09-05-outbound-site-reputation-design.md new file mode 100644 index 0000000..3c78d10 --- /dev/null +++ b/docs/superpowers/specs/2026-09-05-outbound-site-reputation-design.md @@ -0,0 +1,166 @@ +# Outbound Site Reputation: product and technical design + +**Status:** Proposed; documentation only. **Date:** 2026-09-05. +**Decision:** [ADR](../../adr/2026-09-05-outbound-site-reputation-engine.md). +**Delivery:** [implementation plan](../plans/2026-09-05-outbound-site-reputation.md). +**Sources:** [research register](../../papers/outbound-site-reputation-sources.md). + +## 1. Product responsibility and baseline + +Prevent internal users, services, automation, and agents from contacting external destinations with adverse security reputation, while giving operators an explainable reason and a controlled process for legitimate unknown destinations. The product question is: **may this workload contact this destination for this purpose now?** It is not whether a website is popular, credible, easy to scrape, or friendly to bots. + +At `main@a52ccd0a24a727d9349bb32def7713882d8cad1e`, Wardnet has a configured-upstream gateway, threat imports, an IP-oriented DNSBL export, request scoring, and SOC events. It does not yet implement this outbound contract or intercept all company traffic. `ThreatIndicator` carries value/type/severity/source/TTL; `SecurityEvent` carries a client IP, route, action, score, and path. Neither is sufficient evidence for the proposed versioned destination decision. The existing inbound `score_request` and its `BLOCK_SCORE` remain unchanged. + +Initial scope is domain, observable URL, and address security intelligence for HTTP(S) egress. Consumers include employee applications and internal API/LLM/tool workloads, but no application-specific scraping logic belongs in the engine. Full forward-proxy deployment, TLS interception, generic DLP, active crawling, and sandbox execution are not silently included in this first capability. + +## 2. Owners and proposed components + +| Owner/component | Responsibility | Explicit exclusion | +| --- | --- | --- | +| Proposed `crates/wardnet-reputation-core/` | Pure Rust evidence eligibility, canonical-subject matching, policy evaluation, explainable result | HTTP, DNS resolution, async runtime, environment, database, LLM | +| Proposed `src/reputation/` | Authenticated use cases, source adapters, immutable snapshots, policy administration, SOC projection | A second transport-policy implementation or copied sibling code | +| EgressWeave released port | URL/address authority, DNS/peer binding, redirects, proxy/TLS/trust, connection/resource limits | Wardnet maliciousness or business-exception authority | +| Controlled policy enforcement point (PEP) | Bind authenticated request to both decisions, prevent unauthorized connect/send, report actual outcome | Caller-supplied allow headers or unchecked secondary connections | +| Existing canonical supporting owners | Runtime configuration, durable storage/outbox, optional sandbox and orchestration | Reimplementation inside reputation modules | + +These are proposed paths, not existing files. Keep the existing `waf-ids-core` name and public behavior; the rename and other foundation PRs are separate work. Wardnet remains independently deployable. Versioned public contracts, not cross-service SQL or mutable source checkouts, connect bounded contexts. + +## 3. Admission and connection flow + +```text +Authenticated workload -> controlled PEP -> Wardnet local reputation assessment + -> EgressWeave transport/peer authorization + -> validate matching, fresh authorities and reserve audit + -> connect/send through owner-approved execution boundary + -> record enforcement outcome and SOC correlation +Reviewed intelligence -> bounded import -> validated atomic evidence snapshot +Reviewed policy/exception -> immutable policy revision -> cache invalidation +``` + +For a **protect** profile: + +```text +permit = authenticated_context + AND wardnet_action_is_allow + AND egress_transport_is_allow + AND current_applicable_evidence_state + AND healthy_required_source_state + AND same_target_context_and_current_authorities + AND required_audit_reservation_succeeded +``` + +`current_applicable_evidence_state` does not mean that every allowed destination has a matching threat indicator. A no-match destination remains `unknown`; it may receive the Wardnet allow only when an exact-scope valid business authorization applies and all required evidence authorities are healthy. The authorization changes policy action, not maliciousness assessment. This is the same contract used by the ADR and REP-02 acceptance case. + +The diagram is a logical composition, not permission for unchecked DNS between evaluation and connect. Domain and observable-URL evidence can be checked before resolution; address evidence must be checked against each actual candidate peer before it can be used. The released integration must support an authorize-and-connect operation or an equivalently strong peer-bound execution contract. A preflight URL check followed by an ordinary independent HTTP client is unacceptable. + +Bind evaluation and execution to tenant, authenticated workload, purpose, operation nonce, canonical target/profile, policy revision, evidence generation, transport-contract version, connection peer where applicable, and expiry. The PEP must verify authenticity, audience and binding of out-of-process receipts; a digest alone does not authenticate a receipt. Missing or unsupported contract versions, replay, target substitution, or peer mismatch deny protected traffic. + +Re-evaluate every redirect and retry that changes connection or authority. Connection pooling and HTTP/2 coalescing must not grant a different origin prior approval. Credentials are never forwarded to a new origin merely because the first hop was allowed. Source changes invalidate positive caches. Initial protected tunnel leases are bounded to 60 seconds; renewal rechecks both authorities. A deployment must terminate newly denied active leases within that bound or report that persistent-flow protection is unsupported. The bound is a proposed requirement, not a measured capability. + +Every hop that carries an authorization header, cookie, client credential, tenant-scoped secret, or equivalent security-sensitive credential must use authenticated encrypted transport under the released transport-owner contract. This applies to the initial request, redirects, retries, proxy hops, and protocol transitions. Plain `http://`, an HTTPS-to-HTTP downgrade, a redirect/retry target whose authenticated encryption cannot be established, or a peer/authority mismatch must cause the credential to be stripped before that hop or the request to be rejected; a credential-bearing cleartext fallback is never permitted. The PEP must not reinterpret TLS itself, but it must require and verify the EgressWeave receipt/operation contract that proves this property before releasing credential-bearing payload. This addresses the cleartext-sensitive-data weakness class described by CWE-319 [R7]. + +## 4. Destination and evidence contracts + +### Destination identity + +`DestinationContextV1` contains `direction=outbound`, authenticated tenant/workload, registered purpose, operation ID, observation capabilities, and a canonical target descriptor. The descriptor distinguishes exact host, exact observable URL, and actual address/port. It includes scheme/port where relevant, canonicalization profile/version, and connection-binding evidence when available. + +EgressWeave remains authoritative for executable outbound URL, address, and DNS interpretation. Wardnet matches its canonical descriptors; feed mappers must demonstrate compatible representation using released codecs or conformance vectors. Offline core tests can consume fixed canonical fixtures before that release. They must not become an independent network-authorizing parser. + +Host matching uses exact names or explicitly granted dot-boundary subdomain scope. No substring matching, implicit registrable-domain widening, or public-suffix wildcard is allowed. Shared hosting/CDN address matches do not automatically condemn every hosted domain. An actual peer-address threat may deny that connection without rewriting host reputation. CNAME observations preserve alias and source scope rather than collapsing distinct authorities. + +URL matching retains path case and relevant query semantics; log minimization is a separate operation. No double decoding or query stripping to manufacture a different matching identity. Credentials/fragments and ambiguous executable targets are handled by the transport owner. Only visible URL components may support URL-level assessment. IPv4/IPv6 aliases must use the same owner-approved identity; disagreement is an integration error, not permission to retry through another parser. + +### Evidence and source eligibility + +`EvidenceRecordV1` carries source ID, producer record ID/version, original source family, subject and explicit scope, security category, producer severity and optional confidence, observed/received times, validity interval, revocation/deletion/admission state, tenant/marking restrictions, licensing reference, and provenance references. Missing confidence remains unspecified, not zero risk or full confidence. Confidence and severity are not calibrated probabilities [R3]. + +`SourcePolicyV1` records allowed purposes/tenants, permitted observable kinds, whether that source may contribute to enforcement, required health/freshness, maximum evidence age, supported lifecycle grammar, polling limits, license and credential references. An authenticated transport plus a body hash is provenance evidence; the hash alone is not proof of producer authenticity. + +Initially consume reviewed operator evidence and proven STIX/MISP/OpenCTI or source-specific feeds [R3, R5, R6]. Each adapter retains original semantics. Existing lossy `ThreatIndicator` rows are not automatically promoted: ambiguous history remains non-authoritative until validated original evidence or a reviewed replacement is available. MISP adapters require affirmative recognized `to_ids` and active attribute/enclosing-object lifecycle; the open #167/#170 repairs are not assumed shipped. CVE/KEV membership alone is not a site-maliciousness assertion. + +Only supported indicator grammars become enforcement material. Unsupported patterns, conflicting lifecycle fields, malformed validity, or unauthorized source scope cannot be guessed into an active indicator. Preserve diagnostic counts without logging hostile payloads. + +## 5. Freshness, snapshots, and cache behavior + +Compute eligibility at evaluation time from producer validity and source-policy age limits. Received/import time must not replace last-observed validity. Reimport, conditional HTTP 304, retry, and process restart never rejuvenate expired evidence. Source-poll freshness and individual IOC validity are separate facts. + +Validate a complete, bounded source result before atomically publishing a new generation. Full-snapshot replacement requires verified completeness, including pagination. Delta feeds require a consistent cursor and explicit tombstones; absence from a delta is not withdrawal. A successful authenticated empty snapshot differs from an empty body caused by failure. Keep only still-valid last-known-good evidence on refresh failure, without extending TTL. Older source versions cannot resurrect withdrawals; STIX revoked-object semantics remain source-faithful [R3]. + +Reads use immutable snapshots. Publish/invalidate atomically and retain revision identities for replay. No provider request, DNS lookup, crawler, or LLM runs inside the pure evaluation path. Evidence eligibility and tenant markings are checked again after a cache hit. + +Decision-cache keys include tenant, workload, purpose, direction, full canonical target and observation scope, policy revision, evidence generation, contract/profile versions, and applicable peer binding. Expiry is the minimum of policy/exception/evidence/lease validity. Required-source expiry invalidates authorization even without a new generation. Both new threats and legitimate delisting must propagate; never retain an allow or denial indefinitely. Cache cardinality is bounded, not controlled by arbitrary host labels. + +## 6. Deterministic reputation and policy + +Keep three distinct dimensions: + +- Assessment: `known_malicious`, `suspicious`, or `unknown`. +- Evidence health: `fresh`, `degraded`, `expired`, or `unavailable`, evaluated against required sources. `degraded` means all required authorities remain healthy while only optional authority is degraded; required-source expiry is `expired`, and missing/unverifiable required authority is `unavailable`. +- Policy result: `allow` or `deny`, with stable reason codes; monitor produces a separate shadow result. + +The following table is normative for protect-mode evaluation, `DecisionEnvelopeV1`, and the PEP. Higher rows take precedence and the retained assessment is not rewritten merely to explain a higher-precedence authority failure. + +| Contract/assessment and evidence health | Exact-scope business authorization | Wardnet action | Required reason | PEP consequence | +| --- | --- | --- | --- | --- | +| Invalid identity, schema, contract, binding, or required visibility | Any | `deny` | `invalid_contract` or the corresponding fail-closed contract/visibility error | No Wardnet grant; do not call an allow an invalid-contract result | +| Any assessment with required evidence `expired` or `unavailable` | Any | `deny` | `required_authority_unavailable` | No Wardnet grant; business authorization cannot mask required-authority failure | +| `known_malicious` with `fresh` or optional-only `degraded` health | Any | `deny` | `known_malicious` | Hard deny | +| `suspicious` with `fresh` or optional-only `degraded` health | Any | `deny` | `suspicious` | Deny in the initial protect profile | +| `unknown` with `fresh` or optional-only `degraded` health | Valid and exact-scope | `allow` | `business_authorization` | May continue to independent EgressWeave and audit gates; this is not transport authorization | +| `unknown` with `fresh` or optional-only `degraded` health | Missing, expired, revoked, or out-of-scope | `deny` | `unknown_destination` | No Wardnet grant | + +`DecisionEnvelopeV1::validate()` must reject every cross-field combination that contradicts this table. In particular, `unknown_destination`, `invalid_contract`, `known_malicious`, `suspicious`, and `required_authority_unavailable` are denial reasons and cannot validate with `action=allow`; `business_authorization` is the only v1 allow reason and is valid only for an `unknown` assessment while required evidence health is `fresh` or optional-only `degraded`. Required evidence `expired` or `unavailable` always denies and uses `required_authority_unavailable`, even if an adverse assessment is retained for SOC explanation. The PEP consumes only a table-consistent Wardnet result and then independently requires EgressWeave allow, current binding, and durable audit reservation. + +A current eligible match from a reviewed enforcement-capable source can establish `known_malicious`. Non-enforcement evidence may establish `suspicious`; no active eligible match means `unknown`, not safe. Dedupe syndicated feeds by source lineage. Do not add duplicate severities or invent a weighted score. Domain age, registration novelty, geography, popularity, anti-bot behavior, and isolated HTTP failures are not hard-deny evidence. + +| Protect-mode condition, in precedence order | Result | +| --- | --- | +| Invalid identity/contract/binding, unmet visibility, or unavailable required authority | Deny; explain authority/visibility failure, not invented maliciousness | +| Explicit non-overridable policy deny or eligible hard-threat match | Deny, even with a business allow | +| Required evidence expired/unavailable | Deny; a business exception cannot conceal missing authority | +| Suspicious evidence | Deny in the initial protect profile; analyst review does not auto-clear it | +| Unknown with valid exact-scope business authorization and healthy required evidence | Allow subject to transport and audit gates | +| Unknown without that authorization | Deny with `unknown_destination` | + +A policy declares its nonempty required-source set and acceptable maximum ages; new protect configuration is invalid without them. Optional-source failure may report `degraded` without violating required-source health. Threat-deny and transport-deny cannot be overridden by normal exceptions. + +A business authorization binds exact origin, tenant, workload, purpose, approver, ticket/reason, and validity. No global wildcard. Changes use optimistic concurrency and an audit record; expiry/revocation invalidates cached decisions. An exception changes authorization, not assessment. Malware-research access belongs to a separately authorized isolated workflow, outside the ordinary protect profile. + +Monitor mode records `would_allow`/`would_deny` and `enforced=false`; it does not issue a protect authorization. Existing controls still apply, but monitored traffic is explicitly not claimed reputation-protected. Mode changes and rollback are reviewed policy changes, not hidden runtime fail-open switches. + +## 7. Proposed service and audit surfaces + +These routes do not exist on the baseline: + +| Proposed surface | Contract | +| --- | --- | +| `POST /api/v1/egress/reputation/evaluate` | Authenticated service evaluation; bounded input; immutable decision envelope | +| `GET /api/v1/egress/reputation/sources` | Source health, policy eligibility and freshness; no credentials | +| `GET/PUT /api/v1/egress/reputation/policies/{id}` | Tenant-scoped read/admin mutation with revision/If-Match | +| `POST/DELETE /api/v1/egress/reputation/exceptions/{id}` | Expiring exact-scope business authorization and revocation | +| `GET /api/v1/egress/reputation/decisions/{id}` | Authorized explanation with marking-aware evidence visibility | + +Authenticate service identity through a deployment-approved workload identity/mTLS boundary; derive tenant/workload from verified identity, never trust caller JSON claims. Evaluation credentials cannot administer sources, policy, or exceptions. Operators use reader, analyst, or policy-admin capabilities; analysis does not imply policy-write permission. Runtime configuration and secrets come from the canonical registry, not new raw environment reads. + +`DecisionEnvelopeV1` contains schema/version, evaluation ID, target/context binding, assessment, policy action/reasons, freshness, evaluation/expiry times, policy/evidence identities, bounded evidence references, and observation scope. Any grant artifact is authenticated and audience-bound by the service adapter, not manufactured by the pure core. A successful HTTP 200 may contain **deny**; consumers must validate the envelope, not the status alone. Invalid input/version is 400; unauthorized is 401/403; overload is 429; unavailable evaluation is 503. All error paths return no grant. + +Create a separate versioned `OutboundReputationEventV1` with requester and destination distinct, decision ID, revisions, evidence references, and PEP outcome. Do not store the destination in legacy `client_ip` or project a fabricated WAF score. Link a minimized summary into existing SOC events; delivery uses the canonical durable outbox boundary, not a new network exporter. + +For protect-mode allow, reserve a durable audit slot before payload release. An unavailable/full audit path denies with `audit_unavailable`; do not claim durability from an in-memory queue. Retain detailed evidence under configurable tenant retention and deletion policies. Never log raw query strings, bodies, Authorization/Cookie values, or provider credentials. Sensitive host/URL evidence is restricted; ordinary telemetry uses opaque identifiers and low-cardinality reason labels. External providers receive no per-request URLs in v1. + +## 8. Interception and visibility acceptance + +Company-wide claims require network enforcement: enrolled traffic must traverse the PEP, and alternative direct exits must be restricted and tested. A service SDK is sufficient only for that controlled workload, not the whole organization. Protected DNS can complement the design [R4], but existing IP DNSBL export is not domain RPZ enforcement. + +For opaque HTTPS CONNECT, only the observed host/port and peer are assessed. Do not claim path-level inspection. A profile requiring full URL evidence denies insufficient visibility or requires an independently approved inspection/enrolled-client architecture. ECH, unauthorized encrypted DNS, QUIC, direct IP access, external proxies, and alternate network paths must be blocked by the deployment boundary or explicitly marked outside coverage. This PR does not assert those controls already exist. + +## 9. Limits, verification, and rollout + +Initial proposed limits are a 64 KiB evaluation envelope, an 8 KiB observable URL, 32 returned evidence references, and 100,000 cache entries per instance. Source imports have separately configured compressed/decompressed byte and record limits. Explanation truncation reports the total match count; it must never truncate the security decision itself. Bound candidate peers, source work, and deadlines under the released owner contract. Rejection is preferable to partial authorization. + +Performance acceptance is future measured evidence: on a documented 4-vCPU/8-GiB test host, one million synthetic indicators and 1,000 evaluations/second should meet local-core p99 <= 5 ms, with all snapshot/cache memory measured. This is a proposed benchmark gate, not a current performance claim. DNS/transport/provider costs are reported separately. No production enablement based solely on a microbenchmark. + +Required hostile cases and task commands are in the plan. Measure false-positive rate on a reviewed benign corpus, detection on a labeled malicious corpus, unknown rate, required-source availability, revision propagation, and actual enforced coverage. Hold out time/source-family cohorts to reduce evaluation leakage [R1, R2]. IOC match rate is not global detection recall; shadow denials are not prevented connections. + +Roll out offline replay, monitor, a small protect canary, then approved workload cohorts. Require zero upstream hits for deterministic deny cases, reviewed false-positive handling, outage/recovery drills, bounded active-flow revocation, and bypass-path tests before expanding. Roll back to a compatible reviewed policy and transport release without resurrecting withdrawn intelligence or disabling existing security controls. Keep feature capability disabled by default until the applicable deployment gates pass.