feat(v0.3.0): metadata namespace redesign (BREAKING)

CHANGELOG.md

@@ -1,5 +1,92 @@
 # Changelog
 
+## v0.3.0 — Metadata Namespace Redesign (BREAKING)
+
+### Breaking changes
+
+- Top-level `ads:` block is removed. Move it under `metadata.ads:`.
+- Top-level flat `metadata:` with maturity fields at root is removed. Nest
+  fields under `metadata.maturity:`.
+- All seven resource types now share a reserved top-level `metadata:`
+  namespace that is stripped from API payloads and content hashes. Third-party
+  frameworks can add sub-namespaces (e.g. `metadata.<team_name>:`) with a
+  guarantee they will never leak to the CrowdStrike/Humio API.
+
+### Migration (before → after)
+
+Before:
+
+```yaml
+resource_id: my_detection
+name: "My Detection"
+ads:
+  goal: "Detect something"
+  mitre_attack: ["TA0011:T1090.003"]
+metadata:
+  created: "2026-04-14"
+  last_tuned: "2026-04-16"
+  tune_count: 2
+  confidence: high
+severity: 50
+search: { ... }
+```
+
+After:
+
+```yaml
+resource_id: my_detection
+name: "My Detection"
+metadata:
+  maturity:
+    created: "2026-04-14"
+    last_tuned: "2026-04-16"
+    tune_count: 2
+    confidence: high
+  ads:
+    goal: "Detect something"
+    mitre_attack: ["TA0011:T1090.003"]
+severity: 50
+search: { ... }
+```
+
+Migration rules (mechanical):
+
+- Top-level `ads:` → `metadata.ads:`.
+- Top-level flat `metadata:` with `created` / `last_tuned` / `tune_count` /
+  `confidence` at root → `metadata.maturity:` with the same fields inside.
+- If both were present, merge into a single `metadata:` with `maturity:` and
+  `ads:` sub-namespaces.
+
+No `talonctl migrate` command ships — the corpus is small enough to hand-edit
+or rewrite with an editor or assistant.
+
+### Fixed
+
+- Dashboard apply no longer leaks `_template_path` into the Humio schema
+  validation payload (closes issue #7). The stripping fix is shared across
+  all resource types via `core/template_sanitizer` (shipped in v0.2.1) and
+  is now exercised by every provider's payload/hash path.
+- Reference templates `examples/resources/lookup_file.yaml`,
+  `rtr_put_file.yaml`, and `workflow.yaml` had pre-existing schema bugs
+  (missing required fields, wrong field names, broken file paths). These
+  are fixed and now exercised by a new parity test.
+
+### Added
+
+- `metadata.maturity:` validation now runs on every resource type, not just
+  detections. Schema: `created` (YYYY-MM-DD), `last_tuned` (YYYY-MM-DD or
+  null), `tune_count` (non-negative int), `confidence` (low/medium/high/
+  validated). All four fields optional.
+- `metadata.ads:` remains detection-only. Non-detection providers reject it
+  with a clear error pointing at the resource type.
+- Third-party frameworks may add arbitrary sub-namespaces under `metadata.`
+  (e.g. `metadata.acme_corp:`). These are stripped from API payloads and
+  ignored by talonctl; users supply their own validators if needed.
+- `tests/unit/test_examples.py` parity test ensures reference YAMLs under
+  `examples/resources/` do not drift out of spec.
+- `tests/unit/test_old_shape_rejection.py` locks every provider's migration
+  pointer error string so a silent refactor can never drop it.
+
 ## v0.2.1 — 2026-04-16
 
 ### Fixed

examples/resources/README.md

@@ -27,4 +27,39 @@ python scripts/resource_deploy.py plan --resources=detection
 - **`name`** — display name in Falcon console, can be updated freely
 - **`_search_domain`** — which NGSIEM search domain the resource belongs to
 
+## The `metadata:` namespace
+
+Every resource type (detection, saved_search, dashboard, workflow, lookup_file,
+rtr_script, rtr_put_file) supports an optional top-level `metadata:` block that
+is **always stripped from the API payload and content hash**. Editing anything
+under `metadata:` produces zero output in `talonctl plan`.
+
+Two sub-namespaces are validated out-of-the-box:
+
+- **`metadata.maturity:`** (universal) — track created date, last tuned date,
+  tune count, and confidence level. Optional fields:
+  - `created`: ISO date `YYYY-MM-DD`
+  - `last_tuned`: ISO date `YYYY-MM-DD` or `null` (never tuned)
+  - `tune_count`: non-negative integer
+  - `confidence`: one of `low`, `medium`, `high`, `validated`
+
+- **`metadata.ads:`** (detection-only) — Palantir Alerting & Detection Strategy
+  documentation. Schema: see `examples/resources/detection.yaml` for the
+  annotated reference.
+
+Third-party frameworks may add their own sub-namespaces under `metadata.`:
+
+```yaml
+metadata:
+  maturity:
+    created: "2026-04-16"
+  my_framework:
+    anything: true
+    custom_refs: ["path/to/file.md#anchor"]
+```
+
+Talonctl does not validate third-party sub-namespaces — users supply their own
+validators if needed. The guarantee talonctl makes is that `metadata.<anything>`
+never leaks to the API and never affects the content hash.
+
 See the main README and CLAUDE.md for full documentation.

examples/resources/_assets/sysmonconfig-export.xml

@@ -0,0 +1,5 @@
+<!-- Stub Sysmon config for example reference. Real configs available from
+     https://github.com/SwiftOnSecurity/sysmon-config (MIT-licensed) or similar. -->
+<Sysmon schemaversion="4.90">
+  <EventFiltering/>
+</Sysmon>

examples/resources/_assets/tor_exit_nodes.csv

@@ -0,0 +1,3 @@
+ip
+203.0.113.1
+203.0.113.2

examples/resources/detection.yaml

@@ -2,16 +2,19 @@
 # Detects TOR traffic from internal networks to the internet.
 # This is a real, deployable detection — 100% generic, no tenant-specific content.
 #
-# Optional top-level blocks demonstrated below:
-#   - metadata: detection maturity tracking (created, last_tuned, tune_count, confidence).
-#     All four fields optional when the block is present. Ignored by the CrowdStrike
-#     API and excluded from compute_content_hash — editing any metadata field produces
-#     zero output in `talonctl plan`.
-#   - ads: Alerting & Detection Strategy documentation. Optional; when present, `goal`
-#     is required. The fields `false_positives`, `response`, and `validation` each
-#     accept either inline content or a `{path, label?}` ref dict pointing at a
-#     knowledge file (e.g. knowledge/patterns/<platform>.md#anchor). Both the inline
-#     and ref-dict forms are shown below.
+# The top-level `metadata:` block (new in v0.3.0) is reserved, stripped from API
+# payloads, and excluded from compute_content_hash. Editing anything under
+# `metadata:` produces zero output in `talonctl plan`. Two sub-namespaces are
+# validated out-of-the-box:
+#   - metadata.maturity: detection maturity tracking — created, last_tuned,
+#     tune_count, confidence. All four fields optional.
+#   - metadata.ads: Alerting & Detection Strategy documentation (detection-only).
+#     When present, `goal` is required. The fields `false_positives`, `response`,
+#     and `validation` each accept either inline content or a `{path, label?}` ref
+#     dict pointing at a knowledge file (e.g. knowledge/patterns/<platform>.md#anchor).
+#     Both the inline and ref-dict forms are shown below.
+# Third-party frameworks may add their own sub-namespaces under `metadata.` —
+# talonctl will strip them from the API and ignore them.
 
 resource_id: generic___network___tor_traffic_to_the_internet
 name: Generic - Network - TOR Traffic to the Internet
@@ -22,48 +25,49 @@ severity: 50
 status: active
 mitre_attack: ["TA0011:T1090.003"]
 metadata:
-  created: "2026-04-14"
-  last_tuned: "2026-04-16"
-  tune_count: 2
-  confidence: high
-ads:
-  goal: >
-    Detect traffic associated with TOR anonymization network from internal
-    networks, which may indicate an adversary attempting to evade network
-    monitoring or exfiltrate data through encrypted channels.
-  mitre_attack:
-    - "Command and Control / Proxy: Multi-hop Proxy (T1090.003)"
-  strategy_abstract: >
-    Monitors network flow logs for TCP connections to known TOR ports
-    (9001, 9030, 9040, 9050, 9051, 9150) from internal RFC1918 addresses,
-    plus port 443 with TOR application identification.
-  technical_context: >
-    Requires network flow logs from supported vendors (Akamai, AWS, Cisco,
-    Cloudflare, Corelight, etc.) with destination port and application fields.
-    Uses CIDR matching to identify internal-to-external traffic only.
-  blind_spots:
-    - "TOR over non-standard ports not covered"
-    - "TOR bridges using obfs4 transport may not be identified"
-    - "Vendors not in the filter list are excluded"
-  false_positives:
-    - pattern: "Security tool TOR exit node scanning"
-      characteristics: "Automated security scanner source IPs"
-      tuning: "Add scanner IPs to exclusion list"
-      status: "open"
-    # Ref-form alternative — points at the canonical pattern doc in the user
-    # project's knowledge/ directory. Useful when the same FP pattern applies
-    # across multiple detections and lives in a shared pattern file.
-    - path: "knowledge/patterns/network.md#tor-bridges-obfs4"
-      label: "TOR bridge traffic via obfs4 transport"
-  validation:
-    - "Generate TOR traffic from an internal IP to port 9050"
-    - "Verify alert fires with correct source/destination fields"
-  priority_rationale: >
-    Medium severity (50) — TOR usage is suspicious but may have legitimate
-    privacy use cases. Requires analyst investigation to determine intent.
-  response: "Investigate user, check for data exfiltration, review browsing history"
-  ads_created: "2026-04-14"
-  ads_author: "talonctl example"
+  maturity:
+    created: "2026-04-14"
+    last_tuned: "2026-04-16"
+    tune_count: 2
+    confidence: high
+  ads:
+    goal: >
+      Detect traffic associated with TOR anonymization network from internal
+      networks, which may indicate an adversary attempting to evade network
+      monitoring or exfiltrate data through encrypted channels.
+    mitre_attack:
+      - "Command and Control / Proxy: Multi-hop Proxy (T1090.003)"
+    strategy_abstract: >
+      Monitors network flow logs for TCP connections to known TOR ports
+      (9001, 9030, 9040, 9050, 9051, 9150) from internal RFC1918 addresses,
+      plus port 443 with TOR application identification.
+    technical_context: >
+      Requires network flow logs from supported vendors (Akamai, AWS, Cisco,
+      Cloudflare, Corelight, etc.) with destination port and application fields.
+      Uses CIDR matching to identify internal-to-external traffic only.
+    blind_spots:
+      - "TOR over non-standard ports not covered"
+      - "TOR bridges using obfs4 transport may not be identified"
+      - "Vendors not in the filter list are excluded"
+    false_positives:
+      - pattern: "Security tool TOR exit node scanning"
+        characteristics: "Automated security scanner source IPs"
+        tuning: "Add scanner IPs to exclusion list"
+        status: "open"
+      # Ref-form alternative — points at the canonical pattern doc in the user
+      # project's knowledge/ directory. Useful when the same FP pattern applies
+      # across multiple detections and lives in a shared pattern file.
+      - path: "knowledge/patterns/network.md#tor-bridges-obfs4"
+        label: "TOR bridge traffic via obfs4 transport"
+    validation:
+      - "Generate TOR traffic from an internal IP to port 9050"
+      - "Verify alert fires with correct source/destination fields"
+    priority_rationale: >
+      Medium severity (50) — TOR usage is suspicious but may have legitimate
+      privacy use cases. Requires analyst investigation to determine intent.
+    response: "Investigate user, check for data exfiltration, review browsing history"
+    ads_created: "2026-04-14"
+    ads_author: "talonctl example"
 search:
   filter: |
     #repo!=xdr_*

examples/resources/lookup_file.yaml

@@ -12,5 +12,5 @@ description: |
     | match(file="tor_exit_nodes.csv", field=RemoteAddressIP4, column="ip", strict=false)
   Source: Query-Hub (open-source review)
 format: csv
-source: resources/lookup_files/crowdstrike/tor_exit_nodes.csv
+source: examples/resources/_assets/tor_exit_nodes.csv
 _search_domain: all

examples/resources/rtr_put_file.yaml

@@ -9,5 +9,4 @@ name: sysmonconfig-export.xml
 description: |
   Sysmon configuration file for enhanced endpoint telemetry.
   Pushed to endpoints during IR setup or baseline hardening.
-source: resources/rtr_put_files/sysmonconfig-export.xml
-file_type: config
+file_path: _assets/sysmonconfig-export.xml

examples/resources/workflow.yaml

@@ -10,20 +10,20 @@ description: |
   Sends a notification when a critical-severity detection fires.
   Trigger: detection with severity >= 70.
   Action: Posts to a webhook endpoint.
-definition:
-  # The definition field contains the Falcon Fusion workflow JSON.
-  # Export an existing workflow from the Falcon console to get the full structure,
-  # or use the fusion-workflows skill to build one from scratch.
-  #
-  # Example structure (simplified):
-  # {
-  #   "trigger": { "type": "detection", "conditions": { "severity": { "gte": 70 } } },
-  #   "actions": [
-  #     { "type": "webhook", "config": { "url": "https://hooks.example.com/alerts" } }
-  #   ]
-  # }
+enabled: true
 trigger:
-  type: detection
+  event: Investigatable/NGSIEM
+  type: Signal
+  conditions:
+    severity:
+      gte: 70
+actions:
+  webhook_notify:
+    id: generic_webhook
+    name: Post to webhook
+    properties:
+      url: https://hooks.example.com/alerts
+conditions: {}
 tags:
   - notification
   - critical