Real atomics, real detections, and four new assurance gates

.github/workflows/ci.yml

@@ -11,28 +11,77 @@ jobs:
       - uses: actions/setup-go@v5
         with:
           go-version: "1.22"
+
+      # The engine resolves every technique's command from the pinned Atomic Red
+      # Team tree. Without it the atomic, replay and precision tests cannot run,
+      # so the fetch is part of the build, not an optional step.
+      - name: vendor atomic red team (pinned)
+        run: make atomics
+
       - name: build
         run: go build ./cmd/... ./internal/...
       - name: vet
         run: go vet ./cmd/... ./internal/...
+      - name: gofmt
+        run: |
+          unformatted=$(gofmt -l cmd internal)
+          if [ -n "$unformatted" ]; then
+            echo "not gofmt-clean:"; echo "$unformatted"; exit 1
+          fi
+
+      # ./cmd/... is included deliberately: the atomic-resolution, attribution,
+      # replay and diff guards live there, and running only ./internal/... would
+      # skip every one of them.
       - name: test
-        run: go test ./internal/... -count=1
+        run: go test ./cmd/... ./internal/... -count=1
+
+      # Runs the real matcher over captured-format telemetry with no lab. This
+      # is the check that fixtures alone cannot provide.
+      - name: replay sample dataset
+        run: go run ./cmd/purpleloop replay testdata/sample-run > /dev/null
+
+      # Coverage without a false-positive rate is half a metric. Exits non-zero
+      # if any rule fires on the benign administrator workload.
+      - name: precision (false-positive gate)
+        run: go run ./cmd/purpleloop precision
+
   detections:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - name: sigma lint
+      - name: sigma lint (repo linter)
         run: |
           pip install pyyaml
           python3 scripts/lint-sigma.py
+
+      # pySigma enforces spec rules the hand-rolled Go parser tolerates. It
+      # caught a non-UUID rule identifier the Go parser accepted silently.
+      #
+      # The two excluded validators are SigmaHQ house conventions for their own
+      # repository, not parts of the Sigma specification, and both conflict with
+      # deliberate choices here: rules are named for the technique they detect
+      # (T1082.yml), and the pipeline canary is a positive control rather than an
+      # ATT&CK detection, so it carries a purpleloop.* tag.
+      - name: sigma check (pySigma spec validation)
+        run: |
+          pip install sigma-cli
+          sigma check --fail-on-issues \
+            -x filename_length -x namespace_tag \
+            detections/linux detections/windows detections/canary
+
       - name: fixture regression
         run: python3 scripts/regression-test.py
+
   secrets:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
+      # gitleaks-action requires a token to scan pull requests since v2. Without
+      # it the job fails on every PR for a configuration reason rather than a
+      # finding — which trains everyone to ignore a red secrets check.
       - name: gitleaks
         uses: gitleaks/gitleaks-action@v2
-  # Phase 3 adds a `detections` job: sigma lint + regression vs fixtures.
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -22,3 +22,6 @@ reports/
 # os / editor
 .DS_Store
 *.swp
+
+# local go test cache (see review remediation)
+.gotmp/

CHANGELOG.md

@@ -2,6 +2,171 @@
 All notable changes to this project follow Keep a Changelog and Semantic Versioning.
 
 ## [Unreleased]
+### Fixed (2026-08 review — the engine now executes the atomics it claims to)
+- **Every technique ran the same hardcoded `id; whoami`.** `atomic_ids` was parsed from plan
+  YAML, carried through emulation stages and the arbiter feed and written into the proof
+  chain — but never resolved to a command. A ten-technique campaign ran one command ten times
+  and labelled the results T1082, T1033, T1518… so every verdict but `T1059.004`'s was about a
+  command nobody asked for. The README's claim that Execute "runs Atomic Red Team tests" was
+  not true of the code. New `internal/atomic` package resolves each `atomic_id` against the
+  pinned ART tree; resolution is strict (`override → vendored ART → error`) and there is no
+  default command.
+- **Five of ten techniques in `plans/discovery.yml` named Windows atomics** while the campaign
+  targets a Linux victim (`T1082-1`, `T1033-1`, `T1007-1`, `T1016-1`, `T1049-1` are all
+  `command_prompt` tests). Invisible while everything ran `id; whoami`. Plans and emulation
+  chains now name the correct Linux tests, and a platform mismatch is a loud `ERROR`.
+- **Containment breach in the pinned atomics.** `T1059.004-1` writes a script that pings
+  `8.8.8.8` — traffic leaving the isolated lab network, against the project's own rule. Its
+  `host` input is pinned to loopback via an override that keeps the upstream command.
+- **`ERROR` was counted as `missed`** in the HTML headline, and `SKIPPED_PREREQ`/`ERROR` were
+  inside the dashboard's coverage denominator. A harness failure is not a detection gap;
+  coverage is now measured only over techniques that actually exercised a detection.
+- **Atomic Red Team was recorded, not pinned.** `lab-fetch.sh` cloned HEAD then overwrote the
+  pin file with whatever it got, so a re-clone could silently change every command the engine
+  runs. `scripts/fetch-atomics.sh` checks out the recorded commit and verifies HEAD matches.
+- Dashboard no longer hardcodes `campaign: "discovery"` and `build: "v1.3.0"`; both come from
+  the run. `gap.why` is populated from the verdict's note instead of always being empty.
+- `make help` printed "Makefile" in place of every target name (`grep` needed `-h`).
+
+### Fixed (2026-08 — the detections had the same defect as the engine)
+- **Nine of the ten Linux Sigma rules were the identical rule.** Every one matched
+  `Image endswith /id or /whoami`, differing only in title, because they were written against
+  an engine that executed `id; whoami` for every technique. All nine shared one byte-identical
+  positive fixture, so the "10 rules, 20 fixtures, CI enforced" detection-as-code claim was
+  testing one rule nine times against one event. Each technique now has a detection for what
+  its own atomic actually does, with distinct fixtures whose negatives include a *neighbouring*
+  technique's activity — so a rule that is secretly a copy fails its own negative test.
+- **Two false positives found by the new precision gate and fixed**: `T1087.001` fired on
+  `getent hosts localhost` (a DNS lookup, not account enumeration) and `T1135` fired on a bare
+  `mount`. Both rules keyed on a tool name rather than what the tool was being asked to do.
+- **`win_proc_create.yml` had a non-UUID identifier** (`win-proc-create-001`), a Sigma spec
+  violation the hand-rolled Go parser accepted silently. Found by adding `sigma check`.
+- **The Windows path was unreachable from `run`.** `runTasks` hardcoded a Linux target, rules
+  directory and executor; `SSHExecutor` and the Windows rule existed only inside the standalone
+  `canary` subcommand, which also hardcoded the IP `192.168.88.13`. Targets are now declared in
+  `lab/targets.yml`, techniques are dispatched by the platform their rule-map entry names, and
+  the canary runs once per platform in use — a healthy Linux canary says nothing about whether
+  Windows telemetry is flowing.
+- **`history.json` stored only a coverage percentage**, so no run could be compared with
+  another. It now carries per-technique verdicts, which is what `purpleloop diff` needs.
+
+### Fixed (2026-08 — lab reproducibility)
+- **Archive logging did not survive a restart.** The collector reads
+  `archives.json`, which the manager only writes when `<logall_json>` is `yes`; the image ships
+  it as `no`. It had been enabled by hand *inside the running container*, which cannot work:
+  the manager bind-mounts `/wazuh-config-mount/etc/ossec.conf` from the host and its entrypoint
+  copies that over `/var/ossec/etc/ossec.conf` on every start, so the edit is wiped by the very
+  restart needed to apply it. The stack then comes up healthy, the agent enrolls, and every
+  technique reports `NO_TELEMETRY` — which reads as a detection problem and is not one.
+  `scripts/enable-archives.sh` (`make archives`) patches the host-side source of the config
+  mount and then *verifies* the setting is live inside the container.
+
+### Fixed (2026-08 — victim telemetry honesty)
+- **The victim entrypoint hid a total telemetry failure.** It ran
+  `service auditd start || auditd || true`, so every failure was swallowed: the container came
+  up looking healthy with no process-creation source at all, and the pipeline then reported
+  `NO_TELEMETRY` for every technique — which reads as a detection problem and is not one. The
+  entrypoint now probes the kernel audit subsystem, loads `execve` rules when it is usable, and
+  otherwise says loudly that this victim has no process-creation telemetry and that runs will
+  be `INCONCLUSIVE` as a result.
+- **`--settle-timeout` / `--settle-poll` are configurable.** The 60s default is below the
+  victim's 360s `<localfile>` collection interval, so even command-output telemetry could not
+  land inside a technique's window. Documented on the flag itself.
+- Verified and documented in `docs/PITFALLS.md` what actually blocks Linux process-creation
+  telemetry: auditd is impossible on a host whose kernel exposes no audit sysctls (reproduced
+  with `--privileged --network host`), and Sysmon for Linux only runs under systemd (its
+  `-service` mode still shells out to `systemctl`). Closing the gap needs a systemd-based
+  victim — a deliberate change to the lab's security posture, not a config tweak.
+
+### Added (2026-08 — Linux-Sysmon readiness)
+- Normalizer handles a `data.sysmon` path: Linux-Sysmon `EventID 1` is treated as genuine
+  process creation, any other Sysmon event ID is low fidelity and can never satisfy a
+  `process_creation` rule. Events are deliberately not routed through `data.win.*`, which is
+  Wazuh's Windows eventchannel convention. Tested in `internal/evaluator/sysmon_linux_test.go`,
+  including that a non-creation event with identical fields does not match.
+- Documented in `docs/PITFALLS.md` why the Linux victim is a container without process-creation
+  telemetry rather than a container running Sysmon: Sysmon for Linux works, but its eBPF is
+  kernel-global and the event carries no container identity, so scoping it either leaks host
+  processes (PID reuse) or drops the atomics' own short-lived ones. Both manufacture false
+  verdicts. The fix is a Linux VM, as the Windows victim already is.
+
+### Added (2026-08)
+- `purpleloop precision` and `internal/precision`: measures the false-positive rate against a
+  benign administrator workload (`emulation/benign-baseline.yml`, 26 commands across 7
+  categories) including adjacent cases — benign uses of the binaries the rules watch. Exits
+  non-zero on any false positive, so it gates CI.
+- `purpleloop replay` and `internal/capture`: every real run writes
+  `reports/runs/<id>/events.jsonl`; replay re-runs the real evaluator over it with no lab. CI
+  replays `testdata/sample-run` on every push, so rule edits are checked against telemetry
+  rather than only against fixtures written to match them. The capture writer refuses synthetic
+  runs, whose "telemetry" is fabricated.
+- `purpleloop diff <a> <b> [--fail-on-regression]`: names the technique that regressed
+  (`T1082 DETECTED -> MISSED`) instead of only moving a percentage. Verdicts that mean "never
+  exercised" share a rank, so ERROR -> SKIPPED_PREREQ is a change, not a regression.
+- `collector.Settler`: polls until an expected rule matches or ingestion settles, replacing the
+  fixed `time.Sleep(10s)` per technique and its 10-minute query window.
+- Attribution labels on every verdict (`window_and_host_scoped`, `window_scoped`,
+  `window_overlap`, `unscoped`) and `detect_latency_ms`. Overlap is computed campaign-wide once
+  every window is known. This changes no verdict; it states the quality of the evidence behind
+  one.
+- `mappings/attack_rule_map.json` with `expected_rules[]` — a technique is DETECTED when ANY
+  expected rule matches. Replaces the technique->rule map hardcoded in `cmd/purpleloop`.
+- `untested_rules` accounting: every rule is mapped to a technique or declared untested with a
+  reason, enforced by a CI lint, and reported per run.
+- `collection_gaps` in the dashboard output, naming the technique and rules behind each
+  `NO_TELEMETRY`, for handoff to the sibling detection-decay repo.
+- `lab/targets.yml` + `internal/lab`, `plans/windows-discovery.yml`, `plans/cross-platform.yml`.
+- `make archives` / `scripts/enable-archives.sh`.
+- `sigma check` (pySigma) in CI, and a `gofmt` gate. CI now runs `./cmd/...` too — the
+  atomic-resolution, attribution, replay and diff guards live there and were being skipped.
+- Regression guards for both halves of the defect: no two techniques in a campaign may resolve
+  to the same command, and no two rules may share detection logic.
+
+### Added (2026-08, atomics)
+- `internal/atomic`: ART loader with input-argument interpolation, prerequisite parsing,
+  platform support checks and lab overrides.
+- `SKIPPED_PREREQ` verdict — an atomic whose own prerequisites are unmet never ran, and is no
+  longer reported as a detection miss.
+- `mappings/atomic-overrides.yml`: two override kinds — pinned inputs (keeps the upstream
+  command, used for lab containment) and replaced commands (only where no usable upstream test
+  exists). Every entry requires a `reason`, which the loader enforces and the report shows.
+- Proof chains carry atomic provenance (`name`, `guid`, `source`, `override_reason`) and a
+  `note` explaining any non-coverage verdict. Campaign results record the ART commit.
+- `make atomics` target and `scripts/fetch-atomics.sh`.
+- Regression guard: a test fails if any two techniques in the shipped campaign resolve to the
+  same command — the defect above, made unshippable.
+
+### Fixed (2026-07 code review — trust contracts enforced in code, not just docs)
+- **Canary gating now runs in the pipeline.** Every `run` path executes the positive
+  control against the same Linux/Docker executor+collector the campaign uses; if it does
+  not fire, the run is marked `INCONCLUSIVE` and its coverage is not presented or published.
+  Previously the canary ran only in the standalone `canary` subcommand and no `run` was ever
+  gated.
+- **Dashboard canary status is the real result**, never hardcoded `healthy:true`.
+- **Dry/synthetic runs are marked non-evidentiary end to end** (`CampaignResult.Synthetic`):
+  loud banner in the HTML/Navigator output, and such runs are never appended to `history.json`
+  or published to `docs/data/coverage.json`.
+- **Partial container config is rejected** — a real run requires both `--victim-container`
+  and `--manager-container` (or `--dry-run`), so a real atomic is never paired with a
+  synthetic collector.
+- **Fidelity gate corrected**: only genuine process-creation events (Sysmon 1 / Security 4688,
+  or the image+commandLine signature) are treated as process telemetry. Enumeration/network
+  events (e.g. 4798 `callerProcessName`) no longer credit a process-creation detection.
+- **Unsupported Sigma modifiers** (base64, cidr, windash, …) now yield `INCONCLUSIVE`, not a
+  silent `MISSED`.
+- **Sigma matcher correctness**: wildcards compile to anchored regexps (`?` supported,
+  `*\svchost.exe` no longer matches `…svchost.exe.malware`, metacharacters escaped);
+  `N of prefix_*` / `all of prefix_*` expand by glob and no longer drop the trailing
+  `and not filter`; `Field: null` matches absent fields; `not(...)` (no space) parses; `of
+  them` excludes filter/falsepositive identifiers.
+- **Evidence window** pads only the end (ingestion lag), not the start — a query can no
+  longer reach back and credit an earlier run's events (false `DETECTED`).
+- **Collector date pre-filter** widened ±1 day so a non-UTC Wazuh timestamp near a day
+  boundary is not dropped before the precise instant check.
+- Removed the hardcoded `admin:SecretPassword` from `scripts/verify-lab.sh` (now sourced from
+  env / the gitignored secrets file, matching the manager check). Removed dead
+  `BaseURL`/`User`/`Pass` collector fields and the unused `canary.Check` dry-run parameter.
+
 ### Added
 - Evidence fidelity: normalizer tags each event's source; `process_creation` rules now
   only accept genuine process-creation telemetry (Sysmon/EventChannel eventdata, auditd

Makefile

@@ -6,10 +6,10 @@ export
 TECHNIQUE ?= T1059.004
 LAB_DIR := lab/wazuh-docker/single-node
 
-.PHONY: help host-prep lab-fetch lab-up lab-down verify run build vet reset snapshot restore
+.PHONY: help host-prep lab-fetch atomics lab-up lab-down archives verify run build vet reset snapshot restore
 
 help: ## list targets
-	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
+	@grep -hE '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
 	  awk 'BEGIN{FS=":.*?## "}{printf "  \033[36m%-12s\033[0m %s\n", $$1, $$2}'
 
 host-prep: ## one-time host tweak (vm.max_map_count) + tooling check
@@ -18,9 +18,15 @@ host-prep: ## one-time host tweak (vm.max_map_count) + tooling check
 lab-fetch: ## pull pinned Wazuh single-node + Atomic Red Team, install override
 	@bash scripts/lab-fetch.sh
 
+atomics: ## vendor Atomic Red Team at the pinned commit (required before any run)
+	@bash scripts/fetch-atomics.sh
+
 lab-up: ## bring the merged Wazuh stack + victim up
 	@cd $(LAB_DIR) && docker compose up -d
-	@echo "give the indexer ~60-90s, then: make verify"
+	@echo "give the indexer ~60-90s, then: make archives && make verify"
+
+archives: ## enable Wazuh archive logging (the collector reads archives.json)
+	@bash scripts/enable-archives.sh
 
 lab-down: ## stop the stack (keeps data)
 	@cd $(LAB_DIR) && docker compose down
@@ -32,7 +38,7 @@ verify: ## binary health check + canary — must pass before proceeding
 canary: ## run the pipeline positive control
 	@go run ./cmd/purpleloop canary
 
-build: ## compile everything
+build: ## compile everything (scoped: the vendored ART tree is not Go source)
 	@go build ./cmd/... ./internal/...
 
 vet: ## static checks