From e54c07fcc35422fde237acb2693bc77eb8491a9b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?FabioLeit=C3=A3o?= Date: Tue, 30 Jun 2026 02:13:38 -0300 Subject: [PATCH 1/8] =?UTF-8?q?docs(plans):=20close=20#1069=20audit=20?= =?UTF-8?q?=E2=80=94=20subject=20category=20axis=20plan=20+=20#1072?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Promote Bearer-class titular gap (Frente 1) to PLAN_SUBJECT_CATEGORY_AXIS (candidate v1.9.0); PLANS_TODO + PLANS_HUB sync. Frente 2 (--report multi-lens) deferred. Audit-only — no product code. --- docs/plans/PLANS_HUB.md | 1 + docs/plans/PLANS_TODO.md | 2 + docs/plans/PLAN_SUBJECT_CATEGORY_AXIS.md | 139 +++++++++++++++++++++++ 3 files changed, 142 insertions(+) create mode 100644 docs/plans/PLAN_SUBJECT_CATEGORY_AXIS.md diff --git a/docs/plans/PLANS_HUB.md b/docs/plans/PLANS_HUB.md index 078ce251..3b156691 100644 --- a/docs/plans/PLANS_HUB.md +++ b/docs/plans/PLANS_HUB.md @@ -111,6 +111,7 @@ Do **not** edit the table manually; refresh with `python scripts/plans_hub_sync. | **Open** | [PLAN_SELENIUM_QA_TEST_SUITE.md](PLAN_SELENIUM_QA_TEST_SUITE.md) | Plan: Selenium-based QA test suite (on-demand, report and recommendations) | **Status:** Deferred **Date:** 2026-03-15 **Authors:** Fabio Leitao **Priority:** H3 | — | | **Open** | [PLAN_SELF_UPGRADE_AND_VERSION_CHECK.md](PLAN_SELF_UPGRADE_AND_VERSION_CHECK.md) | Plan: Version check and self-upgrade (with container-aware behaviour) | **Status:** Deferred **Date:** 2026-03-15 **Authors:** Fabio Leitao **Priority:** H3 | — | | **Open** | [PLAN_STRICTO_SENSU_RESEARCH_PATH.md](PLAN_STRICTO_SENSU_RESEARCH_PATH.md) | Plan: Stricto sensu research path (M.Sc. and PhD) on top of Data Boar | M.Sc./PhD research lines using Data Boar as experimental platform—detection, evidence, methodology; revisit when programs/advisors near. | — | +| **Open** | [PLAN_SUBJECT_CATEGORY_AXIS.md](PLAN_SUBJECT_CATEGORY_AXIS.md) | Plan: Optional subject category axis (data-subject / titular category) | Optional configurable subject-category axis (Bearer-style subject_mapping) complementing norm_tag; audit #1069 gap; candidate v1.9.0. | [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) [PLAN_COMPLIANCE_EXPANSION_GLOBAL_JURISDICTIONS.md](PLAN_COMPLIANCE_EXPANSION_GLOBAL_JURISDICTIONS.md) [PLAN_ADDITIONAL_DETECTION_TECHNIQUES_AND_FN_REDUCTION.md](PLAN_ADDITIONAL_DETECTION_TECHNIQUES_AND_FN_REDUCTION.md) | | **Open** | [PLAN_SYNTHETIC_DATA_AND_CONFIDENCE_VALIDATION.md](PLAN_SYNTHETIC_DATA_AND_CONFIDENCE_VALIDATION.md) | Plan: Synthetic and true-like data sources, confidence scoring, and operator guidance | **Status:** Deferred **Date:** 2026-03-15 **Authors:** Fabio Leitao **Priority:** H3 **Depends on:** ADR-0007 | — | | **Open** | [PLAN_SYNTHETIC_DATA_LAB.pt_BR.md](PLAN_SYNTHETIC_DATA_LAB.pt_BR.md) | PLAN: Laboratório de Dados Sintéticos para Testes do Data Boar | **Status:** Pending **Date:** 2026-04-04 **Authors:** Fabio Leitao **Priority:** H3 **Depends on:** ADR-0007 | — | | **Open** | [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) | Plan: Taxonomy axes (single map of orthogonal priorities) | Canonical map of H/U/A/P/G/S/M and CodeQL axes; links to operational sources | — | diff --git a/docs/plans/PLANS_TODO.md b/docs/plans/PLANS_TODO.md index 9e65c364..7c7e9f23 100644 --- a/docs/plans/PLANS_TODO.md +++ b/docs/plans/PLANS_TODO.md @@ -89,6 +89,7 @@ When **partners** or **buyers** anchor on a vertical (MSP, insurance, RPO, real - **Sprint mirror:** [SPRINTS_AND_MILESTONES.md](SPRINTS_AND_MILESTONES.md) §3 + **M-RICH** when milestones change. - **Gemini / LLM doc triage (optional):** After a public-bundle review, use [PLAN_GEMINI_FEEDBACK_TRIAGE.md](PLAN_GEMINI_FEEDBACK_TRIAGE.md) for non-authoritative optional to-dos and promotion gates—does not override CI, pytest, or agreed sequencing until promoted here or in an issue. - **Taxonomy axes (G0-G3 + hub):** [PLAN_G_TIER.md](PLAN_G_TIER.md) ([pt-BR](PLAN_G_TIER.pt_BR.md)) formalizes **gravity** for findings; [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) maps orthogonal axes; [ADR-0055](../adr/ADR-0055-orthogonal-priority-axes-anti-collision-contract.md) records the anti-collision contract. **`scripts/inv-adr.ps1`** regenerates **`docs/adr/INVENTORY.txt`**. +- **Subject category axis (audit → plan, candidate v1.9.0):** Bearer-class **titular/subject** dimension is **absent** today (`norm_tag` = data type only) — RO audit **[#1069](https://github.com/FabioLeitao/data-boar/issues/1069)** closed **2026-06-30**; implementation **[#1072](https://github.com/FabioLeitao/data-boar/issues/1072)** + [PLAN_SUBJECT_CATEGORY_AXIS.md](PLAN_SUBJECT_CATEGORY_AXIS.md). Strategic taxonomy eval **[#1071](https://github.com/FabioLeitao/data-boar/issues/1071)** (Fideslang) informs naming/interop in Phase 6 — **not** a runtime dependency. **Frente 2** (`--report` multi-lens CLI) **deferred** (enhancement, not urgent). - **Plan checkbox discipline:** When code ships for a named plan slice, update **`PLAN_*.md`** checkboxes in the **same PR** — see **`AGENTS.md`** (*Plan checkbox discipline*); **`plans-status-pl-sync.mdc`** stays situational (plan globs only). - **Licensing enforcement (open issues — PMO index):** Cluster **#704** [P0] (Maestro + JWT on four lab hosts) → **#719** [P1] **`[U1]`** (silent bypass via `DATA_BOAR_ENV=development` / `DEBUG=1` — CRITICAL log + Docker doc) → **#708–#722** (issuer, trial/grace/revocation ADRs). Table below under **`[H0][U1]` Licensing enforcement**; **A6** `license-smoke` should gain a **#719** regression when the fix lands. **After** Wave 1 **#656** U0/U1 slices and **#406** release gate — not a substitute for **#606** [P0] plugin hooks unless operator reprioritizes. - **LAB-OP Ansible + host hygiene ([#756](https://github.com/FabioLeitao/data-boar/issues/756)):** **`[H0][U1]`** one lab node's disk at **~90%** (may block completão / bare mirror on that node) · **`[H0][U2]`** install **`bw`** CLI via the lab Ansible playbook (idempotent). Issue states **no release-gate blocker** — operator-paced; see **LAB-OP** paragraph below and **M-PILOT-READY** completão row. @@ -164,6 +165,7 @@ Post–PR **#118**: clarified **`private-layout`** vs **`docs/private/homelab`** | Scope import from exports (inventory bootstrap) | Optional: canonical schema; merge-safe config fragments | ✅ Done (v1 CSV + GLPI on `main`; further vendor adapters backlog) | Offline exports (e.g. Live Optics, GLPI, Nagios/Cacti-class, Prometheus/Datadog/Dynatrace **exports** where available) → Data Boar target hints; early scan rounds for consultants/customer teams. Complements PLAN_OPT_IN_NETWORK_PORT_SERVICE_HINTS. See [PLAN_SCOPE_IMPORT_FROM_EXPORTS.md](completed/PLAN_SCOPE_IMPORT_FROM_EXPORTS.md). | | Additional data soup formats | Optional: Compressed, content-type | None | **POC tiers on `main`:** **Tier 3** rich media + **Tier 1** + **opt-in stego hints** (`[dataformats]` optional). **Tier 3b** + **Tier 4** backlog. See PLAN_ADDITIONAL_DATA_SOUP_FORMATS + **Integration / active threads**. | | Additional detection techniques & FN reduction | Optional: Synthetic data (for validation) | None | Additive: optional engines (fuzzy, stemming, format hint, embedding prototype); config thresholds; “suggested review”; reduce false negatives. See PLAN_ADDITIONAL_DETECTION_TECHNIQUES_AND_FN_REDUCTION. | +| Subject category axis (titular / data-subject) | Optional: PLAN_TAXONOMY_AXES; eval [#1071](https://github.com/FabioLeitao/data-boar/issues/1071) for naming/interop | None | **[H2]** **candidate v1.9.0:** optional `subject_mapping` rules (Bearer-style) complementing `norm_tag`; additive finding fields; default off. Audit [#1069](https://github.com/FabioLeitao/data-boar/issues/1069) closed **2026-06-30**; impl **[#1072](https://github.com/FabioLeitao/data-boar/issues/1072)**. See [PLAN_SUBJECT_CATEGORY_AXIS.md](PLAN_SUBJECT_CATEGORY_AXIS.md). | | Home lab validation (production-readiness) | Optional: –1/–1b maintenance in acceptable state | None | Manual second-machine smoke per [HOMELAB_VALIDATION.md](../ops/HOMELAB_VALIDATION.md); proves deploy + ≥1 connector path before demo/customer confidence; low token. | | Lab firewall + L3 + observability sequencing | UniFi/DHCP baseline optional before heavy stacks; [PLAN_LAB_OP_OBSERVABILITY_STACK.md](PLAN_LAB_OP_OBSERVABILITY_STACK.md) phases A–E | None | **Spine:** [PLAN_LAB_FIREWALL_ACCESS_AND_OBSERVABILITY.md](PLAN_LAB_FIREWALL_ACCESS_AND_OBSERVABILITY.md) orders phases **0–5** (firewall → assistant access → posture → optional metrics → syslog/Loki → optional Wazuh). Complements –1L; does not replace pytest/CI. | | Lab-OP CAPEX/OPEX & procurement spine | Optional: latest lab-op inventory and private shopping master are updated | None | Single tracked decision spine for priorities and justification across hardware/storage/network/power/HVAC/software/training; no prices in tracked docs. See [PLAN_LAB_OP_CAPEX_OPEX_AND_PROCUREMENT.md](PLAN_LAB_OP_CAPEX_OPEX_AND_PROCUREMENT.md). | diff --git a/docs/plans/PLAN_SUBJECT_CATEGORY_AXIS.md b/docs/plans/PLAN_SUBJECT_CATEGORY_AXIS.md new file mode 100644 index 00000000..6fe88537 --- /dev/null +++ b/docs/plans/PLAN_SUBJECT_CATEGORY_AXIS.md @@ -0,0 +1,139 @@ +# Plan: Optional subject category axis (data-subject / titular category) + + + + +**Status:** Pending +**Date:** 2026-06-30 +**Authors:** Fabio Leitao +**Priority:** H2 · candidate **v1.9.0** +**Tags:** detection, taxonomy, LGPD, norm_tag, subject-mapping, DPO-facing +**GitHub audit:** [#1069](https://github.com/FabioLeitao/data-boar/issues/1069) (closed — gap confirmed, no code) +**GitHub implementation:** [#1072](https://github.com/FabioLeitao/data-boar/issues/1072) +**Related evaluation:** [#1071](https://github.com/FabioLeitao/data-boar/issues/1071) (Fideslang / IAB Tech Lab taxonomy — strategic input, not a runtime dependency) + +**Synced with:** [PLANS_TODO.md](PLANS_TODO.md), [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) + +--- + +## Motivation + +External privacy-SAST tooling (Bearer-class) separates **data type** (phone, email, national ID) from **data-subject category** (employee, customer, patient, minor) via operator-configurable mapping files. Data Boar today classifies findings primarily on **type** (`norm_tag`) and **sensitivity** — not on **whose** data the column or field likely represents in context. + +**Audit conclusion ([#1069](https://github.com/FabioLeitao/data-boar/issues/1069), RO-verified):** `SensitivityDetector.analyze()` returns `(sensitivity_level, pattern_detected, norm_tag, confidence)` with **no titular/subject dimension**. Table/schema/column context does not feed a subject category. The `data_subject` field in DSAR export is the **requester** of a DSAR (GDPR Art. 15 sense), not a finding-classification axis. + +**Why it matters (LGPD-oriented example, not legal advice):** The same `norm_tag` (e.g. phone) in an HR table vs a customer table vs a health context can imply **different compliance narratives** (employment law, consumer relationship, health-sector rules, **Art. 14** minors). Today both phones receive **identical severity** when only type is scored — a gap a technical DPO/CISO reviewer can defend. + +This plan adds an **optional**, **configurable** subject-category axis **alongside** `norm_tag`, without breaking existing finding schema or reports for operators who leave it disabled. + +--- + +## Relationship to other plans + +| Plan / issue | Relationship | +| ------------ | ------------ | +| [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) | New **orthogonal axis** (subject category vs type vs gravity **G**); document in taxonomy map when shipped. | +| [#1071](https://github.com/FabioLeitao/data-boar/issues/1071) | Strategic eval: Fideslang **Data Subject** / **Data Use** interop vs in-house hierarchy — **input before Phase 2 naming**, not a platform dependency. | +| [PLAN_ADDITIONAL_DETECTION_TECHNIQUES_AND_FN_REDUCTION.md](PLAN_ADDITIONAL_DETECTION_TECHNIQUES_AND_FN_REDUCTION.md) | Complementary; subject axis does not replace FN reduction engines. | +| [PLAN_ACTION_PLAN_GENERATOR_POST_SCAN.md](PLAN_ACTION_PLAN_GENERATOR_POST_SCAN.md) | Future APG tiers may reference subject category in suggested actions (later phase). | +| **#1069 Frente 2** (`--report=security\|privacy\|dataflow`) | **Out of scope** — Markdown-centric `data-boar-report` is an **enhancement**, not this plan; defer unless operator opens a reporter plan. | + +--- + +## Design principles + +1. **Default off:** No subject category unless `subject_mapping` (name TBD) is enabled and configured. +2. **Additive schema:** New fields are **optional** on findings / SQLite / export paths; absent fields mean legacy behaviour. +3. **Complement `norm_tag`:** Never replace or overload `norm_tag`; subject category is a **second axis**. +4. **Configurable, not hard-coded law:** Categories are **operator-defined labels** mapped from schema/table/column patterns — product does not emit legal conclusions. +5. **Evidence, not verdict:** Store **which rule matched** (pattern id, source table name redacted per policy) for audit trail; label output **heuristic**. +6. **No Bearer / Go dependency:** Inspiration only; YAML/JSON config native to Data Boar loader conventions. + +--- + +## Proposed config sketch (draft — not validated by loader) + +```yaml +# Optional root block — entire section omitted = feature disabled. +subject_mapping: + enabled: false + # Ordered rules: first match wins (deterministic). + rules: + - id: hr_employee_context + match: + schema_regex: "(?i)^(hr|rh|payroll|folha)" + table_regex: "(?i)(employee|funcionario|staff)" + subject_category: employee + - id: crm_customer_context + match: + table_regex: "(?i)(customer|cliente|account)" + subject_category: customer + - id: health_patient_context + match: + schema_regex: "(?i)(clinical|paciente|ehr)" + subject_category: patient + default_subject_category: unknown # or omit column when no rule matches +``` + +**Future:** optional `column_regex`, connector tags, or `norm_tag` conjunction — only after Phase 1 proves value. + +--- + +## Proposed finding fields (additive) + +| Field | Type | When present | +| ----- | ---- | ------------ | +| `subject_category` | string (operator label) | Rule matched and feature enabled | +| `subject_category_rule_id` | string | Same | +| `subject_category_confidence` | float 0–1 optional | If heuristic tiering is added | + +Existing consumers that ignore unknown keys continue to work. Excel/Markdown report columns: **opt-in** new sheet rows or columns with disclaimer row. + +--- + +## Phases + +| Phase | Deliverable | Status | +| ----- | ----------- | ------ | +| 0 | **This plan** + PLANS_TODO row + impl issue (audit closure **#1069**) | ✅ Done | +| 1 | Config schema + loader validation; unit tests for rule matching on synthetic table names | ⬜ Pending | +| 2 | Wire `SensitivityDetector` / scan pipeline to attach optional `subject_category*` on findings when enabled | ⬜ Pending | +| 3 | SQLite persistence + backward-compatible migration (`_ensure_*` if new columns) | ⬜ Pending | +| 4 | Report / Excel / optional JSON export surfaces + USAGE disclaimer (heuristic, not legal advice) | ⬜ Pending | +| 5 | Dashboard opt-in column (if applicable) + operator-help manifest sync | ⬜ Pending | +| 6 | Reconcile naming with **#1071** eval (Fideslang interop field vs internal enum) — doc + config only unless operator chooses external ids | ⬜ Pending | + +--- + +## Non-goals + +- **Data Use / purpose** axis (finalidade LGPD Art. 6/7) — see **#1071**; separate future plan if pursued. +- **Automatic legal regime mapping** (CLT vs CFM vs Art. 14) — documentation may **cite** examples; product emits **category label** only. +- **Reporter multi-lens CLI** (`--report=privacy|security|…`) — tracked as defer from **#1069** Frente 2. +- Adopting **ethyca/fides** platform or runtime dependency. + +--- + +## Risks + +| Risk | Mitigation | +| ---- | ---------- | +| False subject assignment from weak table names | Default off; require explicit rules; `unknown` bucket; DPO review copy | +| Schema creep | Optional fields only; migration tests for legacy DBs | +| Overclaim in marketing | Same disclaimer pattern as `jurisdiction_hints` (ADR 0026 class) | + +--- + +## Acceptance (implementation issue) + +- [ ] Feature disabled by default; enabling without rules fails closed or no-ops per loader policy +- [ ] Golden-path scan on fixture DB shows distinct `subject_category` for HR vs CRM tables with same `norm_tag` +- [ ] Existing tests / exports unchanged when block omitted +- [ ] USAGE + pt-BR sync; no legal-advice wording +- [ ] Plan phase table updated in same PR as code + +--- + +## Audit trail + +- **2026-06-30:** RO audit on **#1069** confirmed Frente 1 gap; operator promoted to this plan (candidate **v1.9.0**). Frente 2 deferred (Markdown-centric reporter). From a5d11258ed2f5238b07ecde9e78154f71596e110 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?FabioLeit=C3=A3o?= Date: Tue, 30 Jun 2026 02:16:44 -0300 Subject: [PATCH 2/8] =?UTF-8?q?docs(plans):=20close=20#1071=20eval=20?= =?UTF-8?q?=E2=80=94=20hybrid=20taxonomy=20trilogy=20+=20#1074=E2=80=93#10?= =?UTF-8?q?76?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Approve B-internal + Fideslang export-adapter hybrid: three PLAN_* slices (norm_tag hierarchy + Data-Subject, Data-Use, export adapter). Supersede #1072 narrow track. Evaluation-only — no product code or dependencies. --- docs/plans/PLANS_HUB.md | 3 + docs/plans/PLANS_TODO.md | 7 +- docs/plans/PLAN_DATA_USE_AXIS.md | 88 ++++++++++++++ docs/plans/PLAN_FIDESLANG_EXPORT_ADAPTER.md | 82 +++++++++++++ ...LAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md | 111 ++++++++++++++++++ docs/plans/PLAN_SUBJECT_CATEGORY_AXIS.md | 4 +- 6 files changed, 291 insertions(+), 4 deletions(-) create mode 100644 docs/plans/PLAN_DATA_USE_AXIS.md create mode 100644 docs/plans/PLAN_FIDESLANG_EXPORT_ADAPTER.md create mode 100644 docs/plans/PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md diff --git a/docs/plans/PLANS_HUB.md b/docs/plans/PLANS_HUB.md index 3b156691..1f743f98 100644 --- a/docs/plans/PLANS_HUB.md +++ b/docs/plans/PLANS_HUB.md @@ -58,11 +58,13 @@ Do **not** edit the table manually; refresh with `python scripts/plans_hub_sync. | **Open** | [PLAN_DASHBOARD_REPORTS_ACCESS_CONTROL.md](PLAN_DASHBOARD_REPORTS_ACCESS_CONTROL.md) | Plan: Dashboard / reports access control (roles & permissions) | Single place for **D-WEB** — what exists on `main` today, how middleware runs, and **target** hooks for Phase 1 (session + Bitwarden Passwordless.dev) and Phase 3 (SSO). **No code changes** in this subsection; re-verify | — | | **Open** | [PLAN_DATABRICKS_UNITY_LAKEHOUSE_SCOPE_AND_SCAN.md](PLAN_DATABRICKS_UNITY_LAKEHOUSE_SCOPE_AND_SCAN.md) | Plan: Databricks SQL, Unity Catalog, and lakehouse scope (medium-term) | Medium-term Pro/Ent-shaped backlog: (1) lakehouse tables as scan targets via Databricks SQL; (2) UC or curated tables as source of target list / scope metadata—aligned with CSV scope import and data soup formats. | [PLAN_SCOPE_IMPORT_FROM_EXPORTS.md](completed/PLAN_SCOPE_IMPORT_FROM_EXPORTS.md) [PLAN_ADDITIONAL_DATA_SOUP_FORMATS.md](completed/PLAN_ADDITIONAL_DATA_SOUP_FORMATS.md) [PLAN_ENTERPRISE_HR_SST_ERP_CONNECTORS.md](PLAN_ENTERPRISE_HR_SST_ERP_CONNECTORS.md) [PLAN_OBJECT_STORAGE_CLOUD_CONNECTORS.md](PLAN_OBJECT_STORAGE_CLOUD_CONNECTORS.md) [PLAN_FINDINGS_CORPORATE_REPOSITORY_EXPORT.md](PLAN_FINDINGS_CORPORATE_REPOSITORY_EXPORT.md) [ADDING_CONNECTORS.md](ADDING_CONNECTORS.md) | | **Open** | [PLAN_DATA_SOURCE_VERSIONS_AND_HARDENING.md](PLAN_DATA_SOURCE_VERSIONS_AND_HARDENING.md) | Plan: Data source version and protocol detection with CVE awareness and hardening guidance | **Status:** Deferred **Date:** 2026-03-15 **Authors:** Fabio Leitao **Priority:** H3 | — | +| **Open** | [PLAN_DATA_USE_AXIS.md](PLAN_DATA_USE_AXIS.md) | Plan: Optional Data-Use axis (purpose / finalidade) | Hybrid slice 2 (#1071): optional Data-Use (purpose) axis for LGPD Art.6/7-style finalidade; internal model, default off. | [PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md](PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md) [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) [PLAN_FIDESLANG_EXPORT_ADAPTER.md](PLAN_FIDESLANG_EXPORT_ADAPTER.md) | | **Open** | [PLAN_ENGINEERING_DOCTRINE_CONSOLIDATION.md](PLAN_ENGINEERING_DOCTRINE_CONSOLIDATION.md) | Plan — Engineering doctrine consolidation (Data Boar DNA) | **Status:** Active **Date:** 2026-04-27 **Authors:** Fabio Leitao **Priority:** H0 **Depends on:** ADR-0046, ADR-0047 | — | | **Open** | [PLAN_ENTERPRISE_HR_SST_ERP_CONNECTORS.md](PLAN_ENTERPRISE_HR_SST_ERP_CONNECTORS.md) | Plan: Enterprise back-office connectors (SST, HR, ERP, CRM, folha, helpdesk, URM) | **Status:** Deferred **Date:** 2026-03-21 **Authors:** Fabio Leitao **Priority:** H3 | — | | **Open** | [PLAN_EXTENDED_SENSITIVE_DISCOVERY_POSITIONING.md](PLAN_EXTENDED_SENSITIVE_DISCOVERY_POSITIONING.md) | Plan: Extended sensitive discovery positioning (clinical adjacency, IP, security artifacts) | Buyer/DPO positioning for clinical adjacency, IP, and security artifacts; wave 2 adds explicit entity taxonomy (SWIFT, IBAN, VIN, MAC, CVV, AWS keys, PIN) tied to #1056. | [PLAN_ADDITIONAL_DETECTION_TECHNIQUES_AND_FN_REDUCTION.md](PLAN_ADDITIONAL_DETECTION_TECHNIQUES_AND_FN_REDUCTION.md) [PLAN_YAML_PLUGIN_SYSTEM.md](PLAN_YAML_PLUGIN_SYSTEM.md) | | **Open** | [PLAN_FERPA_AND_EDTECH_COMPLIANCE.md](PLAN_FERPA_AND_EDTECH_COMPLIANCE.md) | PLAN: FERPA and EdTech Compliance Vertical | Technical inventory support for FERPA (US education records) and broader EdTech compliance verticals. Covers pattern detection, YAML sample creation, and use-case alignment for K-12, higher education, and EAD/LMS platfor | [PLAN_YAML_PLUGIN_SYSTEM.md](PLAN_YAML_PLUGIN_SYSTEM.md) [PLAN_COMPLIANCE_EXPANSION_GLOBAL_JURISDICTIONS.md](PLAN_COMPLIANCE_EXPANSION_GLOBAL_JURISDICTIONS.md) | | **Open** | [PLAN_FERPA_AND_EDTECH_COMPLIANCE.pt_BR.md](PLAN_FERPA_AND_EDTECH_COMPLIANCE.pt_BR.md) | PLANO: Vertical de Conformidade FERPA e EdTech | Suporte de inventário técnico para FERPA (registros educacionais EUA) e verticais de conformidade EdTech. Cobre detecção de padrões, criação de amostra YAML e alinhamento de casos de uso para K-12, ensino superior e plat | [PLAN_YAML_PLUGIN_SYSTEM.md](PLAN_YAML_PLUGIN_SYSTEM.md) [PLAN_COMPLIANCE_EXPANSION_GLOBAL_JURISDICTIONS.md](PLAN_COMPLIANCE_EXPANSION_GLOBAL_JURISDICTIONS.md) | +| **Open** | [PLAN_FIDESLANG_EXPORT_ADAPTER.md](PLAN_FIDESLANG_EXPORT_ADAPTER.md) | Plan: Fideslang export adapter (optional output interop) | Hybrid slice 3 (#1071): optional lossy Fideslang data_category on export only; no runtime dep; closes SWOT catalog-tagging W-novo-1 when customer demands. | [PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md](PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md) [PLAN_DATA_USE_AXIS.md](PLAN_DATA_USE_AXIS.md) [PLAN_FINDINGS_CORPORATE_REPOSITORY_EXPORT.md](PLAN_FINDINGS_CORPORATE_REPOSITORY_EXPORT.md) | | **Open** | [PLAN_FINDINGS_CORPORATE_REPOSITORY_EXPORT.md](PLAN_FINDINGS_CORPORATE_REPOSITORY_EXPORT.md) | Plan: Corporate findings repository — export / sync beyond local SQLite (medium-term) | Medium-term Pro/Ent-shaped backlog: optional export or sync of scan findings (metadata model) from local SQLite to customer-chosen corporate stores—SQL, MongoDB, object/lake paths—without changing the core scan contract. | [PLAN_DATABRICKS_UNITY_LAKEHOUSE_SCOPE_AND_SCAN.md](PLAN_DATABRICKS_UNITY_LAKEHOUSE_SCOPE_AND_SCAN.md) [PLAN_OBJECT_STORAGE_CLOUD_CONNECTORS.md](PLAN_OBJECT_STORAGE_CLOUD_CONNECTORS.md) [PLAN_NOTIFICATIONS_OFFBAND_AND_SCAN_COMPLETE.md](PLAN_NOTIFICATIONS_OFFBAND_AND_SCAN_COMPLETE.md) [SECURITY.md](SECURITY.md) [REPORTS_AND_COMPLIANCE_OUTPUTS.md](REPORTS_AND_COMPLIANCE_OUTPUTS.md) | | **Open** | [PLAN_FORENSIC_SEARCH_AND_GEOGRAPHIC_HINTS.md](PLAN_FORENSIC_SEARCH_AND_GEOGRAPHIC_HINTS.md) | Plan: Forensic search and geographic breadcrumb hints (breach investigation + intra-Brazil jurisdiction) | Two adjacent product capabilities surfaced by commercial prospects: (1) targeted breach forensics scan for a specific identifier across all data sources; (2) intra-Brazil geographic breadcrumb detection for multi-jurisdi | [PLAN_EXTENDED_SENSITIVE_DISCOVERY_POSITIONING.md](PLAN_EXTENDED_SENSITIVE_DISCOVERY_POSITIONING.md) [PLAN_ADDITIONAL_DETECTION_TECHNIQUES_AND_FN_REDUCTION.md](PLAN_ADDITIONAL_DETECTION_TECHNIQUES_AND_FN_REDUCTION.md) | | **Open** | [PLAN_FOUNDER_CAREER_AND_BRANDING.pt_BR.md](PLAN_FOUNDER_CAREER_AND_BRANDING.pt_BR.md) | Untitled | Plano privado do operador - detalhes de carreira, branding e presença digital. Documento completo em docs/private/ (gitignored). | — | @@ -88,6 +90,7 @@ Do **not** edit the table manually; refresh with `python scripts/plans_hub_sync. | **Open** | [PLAN_MAINTAINER_ISSUE_BATCH_AND_PMO_SYNC.md](PLAN_MAINTAINER_ISSUE_BATCH_AND_PMO_SYNC.md) | Plan: Maintainer issue batch and PMO sync (GitHub ↔ docs/plans) | Maintainer workflow: batch open GitHub issues into thin PRs, promote durable work into PLAN_*.md + PLANS_TODO + PLANS_HUB sync so the repo index matches reality. | [PLAN_CLI_VALIDATE_DIFF_AND_DSAR_EXPORT.md](completed/PLAN_CLI_VALIDATE_DIFF_AND_DSAR_EXPORT.md) [PLAN_OPERATING_DOMAIN_CONTACTS_AND_ALIAS_POLICY.md](completed/PLAN_OPERATING_DOMAIN_CONTACTS_AND_ALIAS_POLICY.md) | | **Open** | [PLAN_MARKET_ALIGNMENT_AND_WABBIX_REVIEW_TIMING.md](PLAN_MARKET_ALIGNMENT_AND_WABBIX_REVIEW_TIMING.md) | Plan: Market alignment checkpoint and Corporate-Entity-C review timing | **Status:** Pending **Date:** 2026-03-25 **Authors:** Fabio Leitao **Priority:** H1 | — | | **Open** | [PLAN_NEXT_WAVE_PLATFORM_AND_GTM.md](PLAN_NEXT_WAVE_PLATFORM_AND_GTM.md) | Plan: Next wave after core trust foundations (platform + GTM) | **Status:** Pending **Date:** 2026-04-02 **Authors:** Fabio Leitao **Priority:** H1 | — | +| **Open** | [PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md](PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md) | Plan: Hierarchical norm_tag + configurable Data-Subject axis | Hybrid slice 1 (#1071): parent→child norm_tag inheritance preserving BR/LGPD vocabulary + optional Data-Subject axis; closes #1069 titular gap. | [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) [PLAN_SUBJECT_CATEGORY_AXIS.md](PLAN_SUBJECT_CATEGORY_AXIS.md) [PLAN_DATA_USE_AXIS.md](PLAN_DATA_USE_AXIS.md) [PLAN_FIDESLANG_EXPORT_ADAPTER.md](PLAN_FIDESLANG_EXPORT_ADAPTER.md) | | **Open** | [PLAN_NOTIFICATIONS_OFFBAND_AND_SCAN_COMPLETE.md](PLAN_NOTIFICATIONS_OFFBAND_AND_SCAN_COMPLETE.md) | Plan: Off-band notifications and scan-completion notifications | **Status:** Pending **Date:** 2026-03-15 **Authors:** Fabio Leitao **Priority:** H2 | — | | **Open** | [PLAN_OBJECT_STORAGE_CLOUD_CONNECTORS.md](PLAN_OBJECT_STORAGE_CLOUD_CONNECTORS.md) | Plan: Object storage connectors (S3-class, Azure Blob, GCS) | **Status:** Deferred **Date:** 2026-03-26 **Authors:** Fabio Leitao **Priority:** H2 | — | | **Open** | [PLAN_OPERATOR_API_KEY_FIRST_AUTH_UX.md](PLAN_OPERATOR_API_KEY_FIRST_AUTH_UX.md) | Plan: Operator API key–first auth UX (reduce JWT / manual Bearer toil) | **Status:** Pending **Date:** 2026-03-29 **Authors:** Fabio Leitao **Priority:** H2 | — | diff --git a/docs/plans/PLANS_TODO.md b/docs/plans/PLANS_TODO.md index 7c7e9f23..aee6f8d4 100644 --- a/docs/plans/PLANS_TODO.md +++ b/docs/plans/PLANS_TODO.md @@ -89,7 +89,7 @@ When **partners** or **buyers** anchor on a vertical (MSP, insurance, RPO, real - **Sprint mirror:** [SPRINTS_AND_MILESTONES.md](SPRINTS_AND_MILESTONES.md) §3 + **M-RICH** when milestones change. - **Gemini / LLM doc triage (optional):** After a public-bundle review, use [PLAN_GEMINI_FEEDBACK_TRIAGE.md](PLAN_GEMINI_FEEDBACK_TRIAGE.md) for non-authoritative optional to-dos and promotion gates—does not override CI, pytest, or agreed sequencing until promoted here or in an issue. - **Taxonomy axes (G0-G3 + hub):** [PLAN_G_TIER.md](PLAN_G_TIER.md) ([pt-BR](PLAN_G_TIER.pt_BR.md)) formalizes **gravity** for findings; [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) maps orthogonal axes; [ADR-0055](../adr/ADR-0055-orthogonal-priority-axes-anti-collision-contract.md) records the anti-collision contract. **`scripts/inv-adr.ps1`** regenerates **`docs/adr/INVENTORY.txt`**. -- **Subject category axis (audit → plan, candidate v1.9.0):** Bearer-class **titular/subject** dimension is **absent** today (`norm_tag` = data type only) — RO audit **[#1069](https://github.com/FabioLeitao/data-boar/issues/1069)** closed **2026-06-30**; implementation **[#1072](https://github.com/FabioLeitao/data-boar/issues/1072)** + [PLAN_SUBJECT_CATEGORY_AXIS.md](PLAN_SUBJECT_CATEGORY_AXIS.md). Strategic taxonomy eval **[#1071](https://github.com/FabioLeitao/data-boar/issues/1071)** (Fideslang) informs naming/interop in Phase 6 — **not** a runtime dependency. **Frente 2** (`--report` multi-lens CLI) **deferred** (enhancement, not urgent). +- **Hybrid taxonomy (#1071, Fideslang eval — closed 2026-06-30):** RO recommendation **B-interno + A-export-adapter** — three implementation slices (all **optional**, backward compatible): **(1)** [PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md](PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md) **[#1074](https://github.com/FabioLeitao/data-boar/issues/1074)** — hierarchical `norm_tag` + Data-Subject → closes **#1069** (supersedes [#1072](https://github.com/FabioLeitao/data-boar/issues/1072)); **(2)** [PLAN_DATA_USE_AXIS.md](PLAN_DATA_USE_AXIS.md) **[#1075](https://github.com/FabioLeitao/data-boar/issues/1075)**; **(3)** [PLAN_FIDESLANG_EXPORT_ADAPTER.md](PLAN_FIDESLANG_EXPORT_ADAPTER.md) **[#1076](https://github.com/FabioLeitao/data-boar/issues/1076)** → SWOT **W-novo-1** when customer-pull. Legacy Bearer-only plan: [PLAN_SUBJECT_CATEGORY_AXIS.md](PLAN_SUBJECT_CATEGORY_AXIS.md). - **Plan checkbox discipline:** When code ships for a named plan slice, update **`PLAN_*.md`** checkboxes in the **same PR** — see **`AGENTS.md`** (*Plan checkbox discipline*); **`plans-status-pl-sync.mdc`** stays situational (plan globs only). - **Licensing enforcement (open issues — PMO index):** Cluster **#704** [P0] (Maestro + JWT on four lab hosts) → **#719** [P1] **`[U1]`** (silent bypass via `DATA_BOAR_ENV=development` / `DEBUG=1` — CRITICAL log + Docker doc) → **#708–#722** (issuer, trial/grace/revocation ADRs). Table below under **`[H0][U1]` Licensing enforcement**; **A6** `license-smoke` should gain a **#719** regression when the fix lands. **After** Wave 1 **#656** U0/U1 slices and **#406** release gate — not a substitute for **#606** [P0] plugin hooks unless operator reprioritizes. - **LAB-OP Ansible + host hygiene ([#756](https://github.com/FabioLeitao/data-boar/issues/756)):** **`[H0][U1]`** one lab node's disk at **~90%** (may block completão / bare mirror on that node) · **`[H0][U2]`** install **`bw`** CLI via the lab Ansible playbook (idempotent). Issue states **no release-gate blocker** — operator-paced; see **LAB-OP** paragraph below and **M-PILOT-READY** completão row. @@ -165,7 +165,10 @@ Post–PR **#118**: clarified **`private-layout`** vs **`docs/private/homelab`** | Scope import from exports (inventory bootstrap) | Optional: canonical schema; merge-safe config fragments | ✅ Done (v1 CSV + GLPI on `main`; further vendor adapters backlog) | Offline exports (e.g. Live Optics, GLPI, Nagios/Cacti-class, Prometheus/Datadog/Dynatrace **exports** where available) → Data Boar target hints; early scan rounds for consultants/customer teams. Complements PLAN_OPT_IN_NETWORK_PORT_SERVICE_HINTS. See [PLAN_SCOPE_IMPORT_FROM_EXPORTS.md](completed/PLAN_SCOPE_IMPORT_FROM_EXPORTS.md). | | Additional data soup formats | Optional: Compressed, content-type | None | **POC tiers on `main`:** **Tier 3** rich media + **Tier 1** + **opt-in stego hints** (`[dataformats]` optional). **Tier 3b** + **Tier 4** backlog. See PLAN_ADDITIONAL_DATA_SOUP_FORMATS + **Integration / active threads**. | | Additional detection techniques & FN reduction | Optional: Synthetic data (for validation) | None | Additive: optional engines (fuzzy, stemming, format hint, embedding prototype); config thresholds; “suggested review”; reduce false negatives. See PLAN_ADDITIONAL_DETECTION_TECHNIQUES_AND_FN_REDUCTION. | -| Subject category axis (titular / data-subject) | Optional: PLAN_TAXONOMY_AXES; eval [#1071](https://github.com/FabioLeitao/data-boar/issues/1071) for naming/interop | None | **[H2]** **candidate v1.9.0:** optional `subject_mapping` rules (Bearer-style) complementing `norm_tag`; additive finding fields; default off. Audit [#1069](https://github.com/FabioLeitao/data-boar/issues/1069) closed **2026-06-30**; impl **[#1072](https://github.com/FabioLeitao/data-boar/issues/1072)**. See [PLAN_SUBJECT_CATEGORY_AXIS.md](PLAN_SUBJECT_CATEGORY_AXIS.md). | +| Subject category axis (titular / data-subject) | Optional: PLAN_TAXONOMY_AXES; eval [#1071](https://github.com/FabioLeitao/data-boar/issues/1071) for naming/interop | None | **[H2]** **superseded** by hybrid slice 1 — see [PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md](PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md). Legacy: [PLAN_SUBJECT_CATEGORY_AXIS.md](PLAN_SUBJECT_CATEGORY_AXIS.md) / [#1072](https://github.com/FabioLeitao/data-boar/issues/1072). | +| Hierarchical norm_tag + Data-Subject (hybrid #1071 slice 1) | Optional: PLAN_TAXONOMY_AXES; [#1069](https://github.com/FabioLeitao/data-boar/issues/1069) gap | None | **[H2]** **candidate v1.9.0:** BR-preserving `norm_tag` hierarchy + optional `data_subject` axis; default off. **[#1074](https://github.com/FabioLeitao/data-boar/issues/1074)**. Closes **#1069** titular dimension. | +| Data-Use axis (purpose / finalidade, hybrid slice 2) | Slice 1 loader stable (soft) | None | **[H2]** optional `data_use` axis (LGPD Art. 6/7 class labels); heuristic config; default off. **[#1075](https://github.com/FabioLeitao/data-boar/issues/1075)**. | +| Fideslang export adapter (hybrid slice 3) | Slice 1 internal taxonomy; customer-pull | None | **[H3]** optional lossy `data_category` on export only; **no** `fideslang`/`ethyca-fides` dep. Closes SWOT **W-novo-1** when shipped. **[#1076](https://github.com/FabioLeitao/data-boar/issues/1076)**. | | Home lab validation (production-readiness) | Optional: –1/–1b maintenance in acceptable state | None | Manual second-machine smoke per [HOMELAB_VALIDATION.md](../ops/HOMELAB_VALIDATION.md); proves deploy + ≥1 connector path before demo/customer confidence; low token. | | Lab firewall + L3 + observability sequencing | UniFi/DHCP baseline optional before heavy stacks; [PLAN_LAB_OP_OBSERVABILITY_STACK.md](PLAN_LAB_OP_OBSERVABILITY_STACK.md) phases A–E | None | **Spine:** [PLAN_LAB_FIREWALL_ACCESS_AND_OBSERVABILITY.md](PLAN_LAB_FIREWALL_ACCESS_AND_OBSERVABILITY.md) orders phases **0–5** (firewall → assistant access → posture → optional metrics → syslog/Loki → optional Wazuh). Complements –1L; does not replace pytest/CI. | | Lab-OP CAPEX/OPEX & procurement spine | Optional: latest lab-op inventory and private shopping master are updated | None | Single tracked decision spine for priorities and justification across hardware/storage/network/power/HVAC/software/training; no prices in tracked docs. See [PLAN_LAB_OP_CAPEX_OPEX_AND_PROCUREMENT.md](PLAN_LAB_OP_CAPEX_OPEX_AND_PROCUREMENT.md). | diff --git a/docs/plans/PLAN_DATA_USE_AXIS.md b/docs/plans/PLAN_DATA_USE_AXIS.md new file mode 100644 index 00000000..3c0d29ce --- /dev/null +++ b/docs/plans/PLAN_DATA_USE_AXIS.md @@ -0,0 +1,88 @@ +# Plan: Optional Data-Use axis (purpose / finalidade) + + + + +**Status:** Pending +**Date:** 2026-06-30 +**Authors:** Fabio Leitao +**Priority:** H2 · candidate **v1.9.x** (slice **2 of 3** — hybrid #1071) +**Tags:** taxonomy, data-use, purpose, finalidade, LGPD, DPO-facing +**GitHub evaluation:** [#1071](https://github.com/FabioLeitao/data-boar/issues/1071) (closed — hybrid approved) +**GitHub implementation:** [#1075](https://github.com/FabioLeitao/data-boar/issues/1075) +**Depends on (soft):** [PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md](PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md) slice 1 shipped or parallel only if shared config loader is stable + +**Synced with:** [PLANS_TODO.md](PLANS_TODO.md), [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) + +--- + +## Motivation + +**Evaluation [#1071](https://github.com/FabioLeitao/data-boar/issues/1071):** Fideslang’s **Data Use** axis models **purpose / finalidade** (GDPR lawful basis class, LGPD Art. 6 bases and Art. 7 consent where applicable). Bearer and current Data Boar **do not** classify findings by **why** data is processed — only **what** (`norm_tag`) and (after slice 1) **whose** (Data-Subject). + +**Hybrid slice 2** adds an **optional**, **operator-configured** Data-Use label on findings — **heuristic**, derived from table/column naming, config overrides, or future connector metadata — **not** automated legal basis determination. + +--- + +## Design principles + +1. **Default off** — entire `data_use` config block omitted = no change to legacy output. +2. **Additive fields** — e.g. `data_use`, `data_use_rule_id`; absent = legacy consumers unaffected. +3. **Internal vocabulary first** — BR/LGPD-oriented purpose labels (e.g. `employment_admin`, `customer_contract`, `health_care`, `marketing_consent`) defined in config; not hard-coded statute text. +4. **No ethyca-fides / fideslang runtime** — naming may **align** with Fideslang Data Use ids in docs for slice 3 export mapping only. +5. **Product disclaimer** — technical mapping for inventory/RoPA **support**, not legal advice. + +--- + +## Config sketch (draft) + +```yaml +data_use: + enabled: false + rules: + - id: payroll_processing + match: + schema_regex: "(?i)^(hr|folha|payroll)" + use: employment_admin + - id: crm_sales + match: + table_regex: "(?i)(lead|prospect|crm)" + use: customer_relationship + default: unspecified +``` + +**Future:** link rules to `norm_tag` / `data_subject` conjunction; APG suggested actions in [PLAN_ACTION_PLAN_GENERATOR_POST_SCAN.md](PLAN_ACTION_PLAN_GENERATOR_POST_SCAN.md). + +--- + +## Phases + +| Phase | Deliverable | Status | +| ----- | ----------- | ------ | +| 0 | Plan + impl issue (#1071 slice 2) | ✅ Done | +| 1 | Config + loader validation + rule-matching tests | ⬜ Pending | +| 2 | Scan pipeline attaches optional `data_use*` on findings | ⬜ Pending | +| 3 | Persistence + export surfaces (opt-in) | ⬜ Pending | +| 4 | USAGE + pt-BR; taxonomy axes doc update | ⬜ Pending | + +--- + +## Non-goals + +- Automated DPIA or consent registry integration (future connectors). +- Replacing RoPA modules in customer GRC tools. +- Slice 3 Fideslang export (separate plan). + +--- + +## Acceptance + +- [ ] Feature off → byte-identical finding JSON vs baseline fixtures +- [ ] Feature on → distinct `data_use` for payroll vs CRM fixtures +- [ ] Disclaimers in USAGE; no “legal basis guaranteed” copy + +--- + +## Audit trail + +- **2026-06-30:** #1071 hybrid approved; slice 2 plan opened. diff --git a/docs/plans/PLAN_FIDESLANG_EXPORT_ADAPTER.md b/docs/plans/PLAN_FIDESLANG_EXPORT_ADAPTER.md new file mode 100644 index 00000000..0fc4afea --- /dev/null +++ b/docs/plans/PLAN_FIDESLANG_EXPORT_ADAPTER.md @@ -0,0 +1,82 @@ +# Plan: Fideslang export adapter (optional output interop) + + + + +**Status:** Pending +**Date:** 2026-06-30 +**Authors:** Fabio Leitao +**Priority:** H3 · **customer-pull** (slice **3 of 3** — hybrid #1071) +**Tags:** interop, fideslang, export, catalog-tagging, taxonomy, IAB +**GitHub evaluation:** [#1071](https://github.com/FabioLeitao/data-boar/issues/1071) (closed — hybrid approved) +**GitHub implementation:** [#1076](https://github.com/FabioLeitao/data-boar/issues/1076) +**SWOT:** closes **W-novo-1** (catalog-tagging output) when shipped +**Depends on:** Internal taxonomy from [PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md](PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md) (slice 1 minimum); [PLAN_DATA_USE_AXIS.md](PLAN_DATA_USE_AXIS.md) optional for richer mapping + +**Synced with:** [PLANS_TODO.md](PLANS_TODO.md) + +--- + +## Motivation + +**Evaluation [#1071](https://github.com/FabioLeitao/data-boar/issues/1071):** **Direction A** alone is **lossy** for BR tags (CPF → generic `user.government_id`). **Hybrid:** keep **full internal model** (Direction B) and add **border adapter** (Direction A) — optional `data_category` (and related fields) in **export JSON/CSV only**, mapped from internal `norm_tag` / hierarchy. + +**Credibility:** Fideslang (Ethyca → **IAB Tech Lab**) is an industry taxonomy reference — emitting compatible fields helps customers with Fides-class governance stacks **without** adopting the ethyca-fides **platform**. + +--- + +## Design principles + +1. **Export-only** — no change to core detection or SQLite required fields; adapter runs at export/report boundary. +2. **Default off** — `export.fideslang.enabled: false` or CLI flag `--fideslang-export` (exact UX TBD). +3. **Zero new PyPI deps** — mapping table in YAML/JSON in repo; **no** `fideslang` or `ethyca-fides` in `pyproject.toml`. +4. **Documented lossy mapping** — export manifest notes which BR tags collapse to generic Fideslang ids. +5. **Golden-standard + sauce** — internal tags remain authoritative; Fideslang fields are **secondary view**. + +--- + +## Mapping table (illustrative — not normative) + +| Internal (leaf) | Fideslang `data_category` (export) | Lossy? | +| --------------- | ---------------------------------- | ------ | +| `LGPD_CPF` | `user.government_id` | Yes | +| `LGPD_CNPJ` | `user.government_id` | Yes | +| `phone` | `user.contact.phone_number` | Partial | +| PEP (risk flag) | _(document as extension or custom; no false claim of native Fideslang PEP)_ | — | + +Full table lives in config `export/fideslang_mapping.yaml` (path TBD) with version pin to **documented** Fideslang release operator targets. + +--- + +## Phases + +| Phase | Deliverable | Status | +| ----- | ----------- | ------ | +| 0 | Plan + impl issue (#1071 slice 3) | ✅ Done | +| 1 | Mapping config + unit tests (internal tag → export field) | ⬜ Pending | +| 2 | CLI/report/JSON export hook; optional `data_category` column | ⬜ Pending | +| 3 | USAGE “interop” section + lossy-mapping appendix; pt-BR sync | ⬜ Pending | +| 4 | Customer-pull gate: ship when catalog/RoPA integration request exists | ⬜ Pending | + +--- + +## Non-goals + +- Importing Fideslang YAML as scan input (inbound). +- Running ethyca-fides DSR engine or Postgres catalog store. +- Claiming full Fideslang certification or IAB endorsement. + +--- + +## Acceptance + +- [ ] Adapter off → exports unchanged +- [ ] Adapter on → sample export includes `data_category` per mapping; lossy rows logged in export metadata +- [ ] No new dependencies in `pyproject.toml` / `uv.lock` +- [ ] SWOT **W-novo-1** marked addressed in plan close note when merged + +--- + +## Audit trail + +- **2026-06-30:** #1071 hybrid approved; slice 3 plan opened (customer-pull). diff --git a/docs/plans/PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md b/docs/plans/PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md new file mode 100644 index 00000000..d17c0aa4 --- /dev/null +++ b/docs/plans/PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md @@ -0,0 +1,111 @@ +# Plan: Hierarchical norm_tag + configurable Data-Subject axis + + + + +**Status:** Pending +**Date:** 2026-06-30 +**Authors:** Fabio Leitao +**Priority:** H2 · candidate **v1.9.0** (slice **1 of 3** — hybrid #1071) +**Tags:** detection, taxonomy, norm_tag, hierarchy, data-subject, LGPD, BR-vocabulary +**GitHub evaluation:** [#1071](https://github.com/FabioLeitao/data-boar/issues/1071) (closed — hybrid approved) +**GitHub audit (titular gap):** [#1069](https://github.com/FabioLeitao/data-boar/issues/1069) (closed) +**GitHub implementation:** [#1074](https://github.com/FabioLeitao/data-boar/issues/1074) +**Supersedes narrow track:** [#1072](https://github.com/FabioLeitao/data-boar/issues/1072) + [PLAN_SUBJECT_CATEGORY_AXIS.md](PLAN_SUBJECT_CATEGORY_AXIS.md) (Bearer-only subject mapping absorbed here) + +**Synced with:** [PLANS_TODO.md](PLANS_TODO.md), [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) + +--- + +## Motivation + +**Evaluation [#1071](https://github.com/FabioLeitao/data-boar/issues/1071)** (RO-verified, not re-done here): Data Boar’s **BR/LGPD vocabulary** (CPF, CNPJ, NIS/PIS, PEP, Mod-11 checks) **exceeds** generic industry taxonomies. **Hybrid decision:** internal model (**Direction B**) preserves depth; Fideslang interop stays **export-only** (slice 3). + +**Slice 1** delivers the **highest-value** gap closure from **#1069**: optional **Data-Subject** axis (employee / customer / patient / minor / …) **plus** **hierarchical `norm_tag`** (parent category → subcategory with inheritance), without lossy collapse of BR-specific tags. + +--- + +## Design principles + +1. **Default off** — no hierarchy or subject axis unless config enables it. +2. **Backward compatible** — flat `norm_tag` strings and legacy findings unchanged when disabled; optional additive fields only when enabled. +3. **Preserve BR moat** — internal tags stay granular (`LGPD_CPF`, `LGPD_CNPJ`, …); hierarchy is **composition**, not replacement by generic `user.government_id`. +4. **Orthogonal axes** — Data-Subject complements type (`norm_tag`); does not overload sensitivity or gravity **G**. +5. **Heuristic only** — no legal conclusions; DPO-facing disclaimers like `jurisdiction_hints` (ADR 0026 class). + +--- + +## Hierarchical norm_tag (internal) + +**Concept:** `norm_tag` may encode `parent.child` or explicit `parent` + `norm_tag_leaf` fields (implementation choice in Phase 1 design). + +| Parent (example) | Child examples (BR-preserved) | +| ---------------- | ----------------------------- | +| `government_id` | `LGPD_CPF`, `LGPD_CNPJ`, `NIS_PIS` | +| `contact` | `phone`, `email` | +| `financial` | `account_number`, `card_number` | + +**Inheritance rules (draft):** + +- Child inherits parent’s compliance-sample hooks and recommendation templates unless overridden. +- Reports may **roll up** to parent for executive view while retaining leaf in detail export. +- Existing flat tags remain valid leaf nodes (no forced migration). + +--- + +## Data-Subject axis (configurable) + +Same intent as Bearer `subject_mapping` and Fideslang **Data Subject** concept — **internal labels**, config-driven: + +```yaml +data_subject: + enabled: false + rules: + - id: hr_employee + match: + schema_regex: "(?i)^(hr|rh|payroll)" + subject: employee + - id: crm_customer + match: + table_regex: "(?i)(customer|cliente)" + subject: customer + default: unknown +``` + +**Additive finding fields (when enabled):** `data_subject`, `data_subject_rule_id` (names TBD in impl). + +--- + +## Phases + +| Phase | Deliverable | Status | +| ----- | ----------- | ------ | +| 0 | Plan + impl issue; supersede #1072 narrow track | ✅ Done | +| 1 | Config schema + loader; hierarchy resolution + subject rules unit tests | ⬜ Pending | +| 2 | Detector/scan pipeline: optional hierarchy fields + `data_subject*` on findings | ⬜ Pending | +| 3 | SQLite + `_ensure_*` migration; legacy DB bootstrap tests | ⬜ Pending | +| 4 | Report/Excel/JSON export (opt-in columns; rollup + leaf) | ⬜ Pending | +| 5 | USAGE + pt-BR; update [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) | ⬜ Pending | + +--- + +## Non-goals (this slice) + +- **Data-Use / purpose** — [PLAN_DATA_USE_AXIS.md](PLAN_DATA_USE_AXIS.md) (slice 2). +- **Fideslang `data_category` export** — [PLAN_FIDESLANG_EXPORT_ADAPTER.md](PLAN_FIDESLANG_EXPORT_ADAPTER.md) (slice 3). +- **ethyca-fides** platform or **`fideslang`** PyPI dependency. + +--- + +## Acceptance + +- [ ] Disabled config → identical behaviour to pre-change scans +- [ ] Enabled: HR vs CRM fixture shows same leaf `norm_tag`, different `data_subject` +- [ ] CPF/CNPJ remain distinct leaf tags under hierarchy (no lossy merge) +- [ ] Plan checkboxes updated in implementation PR + +--- + +## Audit trail + +- **2026-06-30:** #1071 hybrid approved; slice 1 plan opened; supersedes Bearer-only [PLAN_SUBJECT_CATEGORY_AXIS.md](PLAN_SUBJECT_CATEGORY_AXIS.md) / #1072. diff --git a/docs/plans/PLAN_SUBJECT_CATEGORY_AXIS.md b/docs/plans/PLAN_SUBJECT_CATEGORY_AXIS.md index 6fe88537..4aadfb78 100644 --- a/docs/plans/PLAN_SUBJECT_CATEGORY_AXIS.md +++ b/docs/plans/PLAN_SUBJECT_CATEGORY_AXIS.md @@ -3,13 +3,13 @@ -**Status:** Pending +**Status:** Pending — **superseded for implementation** by [PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md](PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md) ([#1071](https://github.com/FabioLeitao/data-boar/issues/1071) hybrid slice 1). Retained as Bearer-audit reference; do not implement #1072 separately. **Date:** 2026-06-30 **Authors:** Fabio Leitao **Priority:** H2 · candidate **v1.9.0** **Tags:** detection, taxonomy, LGPD, norm_tag, subject-mapping, DPO-facing **GitHub audit:** [#1069](https://github.com/FabioLeitao/data-boar/issues/1069) (closed — gap confirmed, no code) -**GitHub implementation:** [#1072](https://github.com/FabioLeitao/data-boar/issues/1072) +**GitHub implementation:** [#1072](https://github.com/FabioLeitao/data-boar/issues/1072) (closed) → **superseded by [#1074](https://github.com/FabioLeitao/data-boar/issues/1074)** **Related evaluation:** [#1071](https://github.com/FabioLeitao/data-boar/issues/1071) (Fideslang / IAB Tech Lab taxonomy — strategic input, not a runtime dependency) **Synced with:** [PLANS_TODO.md](PLANS_TODO.md), [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) From b5ca4b8c1be88247c1ab02149ded82bfe5067b87 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?FabioLeit=C3=A3o?= Date: Tue, 30 Jun 2026 03:26:22 -0300 Subject: [PATCH 3/8] fix(connectors): guard filesystem walk against symlink loops (#1080) Add iter_scan_files with directory inode de-duplication and skip re-scanning the same file inode via symlink aliases. Closes infinite walk / duplicate finding risk on circular directory symlinks. --- connectors/filesystem_connector.py | 91 ++++++++++++++++++++++++-- tests/test_filesystem_symlink_guard.py | 48 ++++++++++++++ 2 files changed, 135 insertions(+), 4 deletions(-) create mode 100644 tests/test_filesystem_symlink_guard.py diff --git a/connectors/filesystem_connector.py b/connectors/filesystem_connector.py index 67c942b4..ed30cda2 100644 --- a/connectors/filesystem_connector.py +++ b/connectors/filesystem_connector.py @@ -7,8 +7,10 @@ import hashlib import os +import stat import tempfile import zipfile +from collections.abc import Iterator from pathlib import Path from typing import Any @@ -45,6 +47,90 @@ def _file_content_fingerprint(raw_bytes: bytes | None) -> str | None: return hashlib.blake2s(raw_bytes, digest_size=8).hexdigest() +def _dir_inode_key(path: Path) -> tuple[int, int] | None: + """Return (st_dev, st_ino) for a directory, or None if stat fails.""" + try: + st = path.stat() + except OSError: + return None + if not stat.S_ISDIR(st.st_mode): + return None + return (st.st_dev, st.st_ino) + + +def _file_inode_key(path: Path) -> tuple[int, int] | None: + """Return (st_dev, st_ino) for a regular file (symlink not followed).""" + try: + st = path.stat(follow_symlinks=False) + except OSError: + return None + if not stat.S_ISREG(st.st_mode): + return None + return (st.st_dev, st.st_ino) + + +def iter_scan_files(root: Path, *, recursive: bool) -> Iterator[Path]: + """ + Yield files under *root* for filesystem scans. + + - Does not follow directory symlinks (avoids circular walk loops). + - Skips duplicate regular files reachable via multiple hard links / symlink aliases. + """ + root = root.resolve() + seen_dirs: set[tuple[int, int]] = set() + seen_files: set[tuple[int, int]] = set() + + def _walk_dir(directory: Path) -> Iterator[Path]: + dkey = _dir_inode_key(directory) + if dkey is None: + return + if dkey in seen_dirs: + return + seen_dirs.add(dkey) + try: + with os.scandir(directory) as it: + for entry in it: + try: + if entry.is_symlink(): + if entry.is_file(follow_symlinks=True): + fpath = Path(entry.path) + fkey = _file_inode_key(fpath) + if fkey is not None and fkey not in seen_files: + seen_files.add(fkey) + yield fpath + continue + if entry.is_file(follow_symlinks=False): + fpath = Path(entry.path) + fkey = _file_inode_key(fpath) + if fkey is not None and fkey not in seen_files: + seen_files.add(fkey) + yield fpath + elif recursive and entry.is_dir(follow_symlinks=False): + yield from _walk_dir(Path(entry.path)) + except OSError: + continue + except OSError: + return + + if recursive: + yield from _walk_dir(root) + else: + try: + with os.scandir(root) as it: + for entry in it: + try: + if entry.is_file(follow_symlinks=False): + fpath = Path(entry.path) + fkey = _file_inode_key(fpath) + if fkey is not None and fkey not in seen_files: + seen_files.add(fkey) + yield fpath + except OSError: + continue + except OSError: + return + + # Plain text and markup (read as text with errors=replace) _TEXT_EXTENSIONS = { ".txt", @@ -623,10 +709,7 @@ def run(self) -> None: ) except Exception: pass - pattern = "**/*" if recursive else "*" - for file_path in path.glob(pattern): - if not file_path.is_file(): - continue + for file_path in iter_scan_files(path, recursive=recursive): ext = file_path.suffix.lower() if ext not in self.extensions and not ( self.scan_compressed and ext in self.compressed_extensions diff --git a/tests/test_filesystem_symlink_guard.py b/tests/test_filesystem_symlink_guard.py new file mode 100644 index 00000000..29bfef1d --- /dev/null +++ b/tests/test_filesystem_symlink_guard.py @@ -0,0 +1,48 @@ +"""Filesystem connector: symlink loop guard and inode de-duplication.""" + +from __future__ import annotations + +import os +from pathlib import Path + +from connectors.filesystem_connector import iter_scan_files + + +def test_iter_scan_files_skips_circular_directory_symlinks( + tmp_path: Path, +) -> None: + root = tmp_path / "root" + sub = root / "sub" + sub.mkdir(parents=True) + # sub/loop -> .. creates a directory symlink cycle when walked naively. + os.symlink("..", sub / "loop") + + files = list(iter_scan_files(root, recursive=True)) + assert files == [] + + +def test_iter_scan_files_deduplicates_same_inode_via_symlink( + tmp_path: Path, +) -> None: + root = tmp_path / "root" + root.mkdir() + real = root / "data.txt" + real.write_text("ID 123.456.789-00", encoding="utf-8") + os.symlink(real, root / "alias.txt") + + files = list(iter_scan_files(root, recursive=True)) + assert len(files) == 1 + assert files[0].name == "data.txt" + + +def test_iter_scan_files_non_recursive_only_top_level(tmp_path: Path) -> None: + root = tmp_path / "root" + nested = root / "nested" + nested.mkdir(parents=True) + top = root / "top.txt" + deep = nested / "deep.txt" + top.write_text("a", encoding="utf-8") + deep.write_text("b", encoding="utf-8") + + files = {p.name for p in iter_scan_files(root, recursive=False)} + assert files == {"top.txt"} From 79080f65e2595da75d6b30b2b8134db1798308bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?FabioLeit=C3=A3o?= Date: Tue, 30 Jun 2026 03:29:23 -0300 Subject: [PATCH 4/8] =?UTF-8?q?docs(adr):=20close=20#1078/#1079=20research?= =?UTF-8?q?=20=E2=80=94=20benchmarks=20+=20OPA=20prototype?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-0076 OPA CI-lint (not runtime), ADR-0077 no client gitignore by design, ADR-0078 RegexSet-before-Vectorscan benchmark gate. Pin filesystem phase breakdown + refresh rust hotspot JSON. Isolated Rego prototype under research/. --- ...a-rego-ci-tier-drift-linter-not-runtime.md | 47 +++++ ...stem-scan-no-client-gitignore-by-design.md | 36 ++++ ...nchmark-gate-regexset-before-vectorscan.md | 50 +++++ docs/adr/INVENTORY.txt | 7 +- docs/adr/README.md | 3 + .../opa_tier_drift_prototype/README.md | 28 +++ .../manifest_snapshot.example.json | 10 + .../opa_tier_drift_prototype/tier_policy.rego | 44 +++++ .../tier_policy_test.rego | 22 +++ tests/benchmarks/README.md | 11 ++ .../filesystem_phase_breakdown.json | 26 +++ .../run_filesystem_phase_breakdown_bench.py | 179 ++++++++++++++++++ tests/benchmarks/rust_prefilter_hotspot.json | 10 +- 13 files changed, 466 insertions(+), 7 deletions(-) create mode 100644 docs/adr/ADR-0076-opa-rego-ci-tier-drift-linter-not-runtime.md create mode 100644 docs/adr/ADR-0077-filesystem-scan-no-client-gitignore-by-design.md create mode 100644 docs/adr/ADR-0078-multi-pattern-regex-benchmark-gate-regexset-before-vectorscan.md create mode 100644 research/licensing/opa_tier_drift_prototype/README.md create mode 100644 research/licensing/opa_tier_drift_prototype/manifest_snapshot.example.json create mode 100644 research/licensing/opa_tier_drift_prototype/tier_policy.rego create mode 100644 research/licensing/opa_tier_drift_prototype/tier_policy_test.rego create mode 100644 tests/benchmarks/filesystem_phase_breakdown.json create mode 100644 tests/benchmarks/run_filesystem_phase_breakdown_bench.py diff --git a/docs/adr/ADR-0076-opa-rego-ci-tier-drift-linter-not-runtime.md b/docs/adr/ADR-0076-opa-rego-ci-tier-drift-linter-not-runtime.md new file mode 100644 index 00000000..63fa0a4b --- /dev/null +++ b/docs/adr/ADR-0076-opa-rego-ci-tier-drift-linter-not-runtime.md @@ -0,0 +1,47 @@ +# ADR 0076 — OPA/Rego as CI drift linter for commercial tier enforcement (not runtime) + +- **Date (UTC):** 2026-06-30 +- **Authors:** Fabio Leitao +- **Deciders:** Fabio Leitao + +## Status + +Proposed + +### Status history + +- 2026-06-30 — Proposed (research closure GitHub #1079) + +## Context + +Tier boundaries are documented in [ADR 0027](ADR-0027-commercial-tier-boundaries-licensing-docs-and-future-jwt-claims.md) and implemented across **`core/licensing/tier_features.py`** (`FEATURE_TIER_MAP`), **`runtime_feature_tier.py`**, **`connector_registry.py`**, **`sampling_policy.py`**, and **`enterprise_surface_posture.py`**. Symptoms of **drift** between the documented ladder and code are tracked as **#854**, **#887**, and **#845**. + +External research (off-band, RO-verified #1079) evaluated **Open Policy Agent (OPA)** / **Rego** as a way to express the tier régua in one place. + +| Option | Pros | Cons | +| --- | --- | --- | +| **OPA daemon + Go binding at runtime** | Single policy engine in production | Breaks lightweight `pip install` open-core; new runtime dep; overlaps JWT/`LicenseGuard` | +| **Rego as CI/lint only (chosen direction)** | Catches drift before merge; no customer runtime cost | Requires exporting a machine-readable snapshot from Python for diff | +| **Status quo (dispersed `if tier`)** | No new tooling | Drift recurs (#854-class) | + +## Decision + +1. **Do not** add OPA or `ethyca-fides` as a **runtime** dependency or replace **`core/licensing/`** JWT/Ed25519 enforcement. +2. **Adopt as target architecture (when scheduled):** a **CI lint step** that: + - loads a **canonical Rego policy** mirroring [ADR 0027](ADR-0027-commercial-tier-boundaries-licensing-docs-and-future-jwt-claims.md) / `#853` ladder; + - compares against a **generated manifest** from `FEATURE_TIER_MAP` + connector registry metadata; + - **fails the build** on mismatch (missing connector row, tier downgrade, orphan feature). +3. **Prototype (isolated, not wired to CI yet):** `research/licensing/opa_tier_drift_prototype/` — sample Rego + README; proves expressiveness for connector tier cases (#854) without touching production gates. +4. **Implementation** of the CI job is **out of scope** for #1079; track via follow-up issue after operator prioritizes #854/#887/#845 cluster. + +## Consequences + +- **Positive:** Clear north star for tier drift prevention without bloating the runtime image. +- **Negative:** Two sources of truth until the exporter + lint land; Rego must be maintained when tiers change. +- **Watch:** When CI lint ships, extend `tests/test_claims_consistency.py` or add a dedicated gate — do not weaken existing connector fail-closed registry (#854). + +## References + +- GitHub **#1079** (evaluation), **#854**, **#887**, **#845** +- `research/licensing/opa_tier_drift_prototype/README.md` +- [ADR 0064](ADR-0064-license-enforcement-additive-model.md) — runtime enforcement stays additive diff --git a/docs/adr/ADR-0077-filesystem-scan-no-client-gitignore-by-design.md b/docs/adr/ADR-0077-filesystem-scan-no-client-gitignore-by-design.md new file mode 100644 index 00000000..6bb140ba --- /dev/null +++ b/docs/adr/ADR-0077-filesystem-scan-no-client-gitignore-by-design.md @@ -0,0 +1,36 @@ +# ADR 0077 — Filesystem scan does not honor client `.gitignore` (deliberate design) + +- **Date (UTC):** 2026-06-30 +- **Authors:** Fabio Leitao +- **Deciders:** Fabio Leitao + +## Status + +Accepted + +### Status history + +- 2026-06-30 — Accepted (audit closure GitHub #1080, RO-verified) + +## Context + +Research compared the filesystem connector to **ripgrep**'s `ignore` crate patterns (#1080). The connector has **no** `.gitignore` / `.boarignore` filtering today. + +For a **compliance-oriented inventory** scanner, honoring a repository's `.gitignore` would **hide paths the data owner chose to exclude from version control** — often exactly where sensitive exports, dumps, or local caches live. That conflicts with **defensive scanning** and **shadow-data** discovery goals ([PLAN_ADDITIONAL_DATA_SOUP_FORMATS.md](../plans/PLAN_ADDITIONAL_DATA_SOUP_FORMATS.md) narrative). + +## Decision + +1. **Default behaviour:** filesystem walks **do not** apply `.gitignore`, `.dockerignore`, or ad-hoc ignore globs unless a **future, explicit operator opt-in** is designed (separate plan/ADR). +2. **Optional future:** a clearly named config block (e.g. `file_scan.operator_exclude_globs`) may allow **operator-declared** skips for known noise paths — never silent inheritance of the client's VCS ignore rules. +3. **Document** in USAGE/SECURITY when operator docs next touch filesystem targets: scanning may traverse paths ignored by Git. + +## Consequences + +- **Positive:** Avoids false confidence ("green scan" because PII sat in gitignored folders). +- **Negative:** More files scanned; operators must scope `targets.path` and extensions deliberately. +- **Related fix (same issue, separate concern):** symlink loop guard and inode de-duplication — see `iter_scan_files` in `connectors/filesystem_connector.py` (#1080 🔴). + +## References + +- GitHub **#1080** (audit) +- [ADR 0051](ADR-0051-incremental-filesystem-scan-file-identity-fingerprint.md) — file identity metadata diff --git a/docs/adr/ADR-0078-multi-pattern-regex-benchmark-gate-regexset-before-vectorscan.md b/docs/adr/ADR-0078-multi-pattern-regex-benchmark-gate-regexset-before-vectorscan.md new file mode 100644 index 00000000..834a68de --- /dev/null +++ b/docs/adr/ADR-0078-multi-pattern-regex-benchmark-gate-regexset-before-vectorscan.md @@ -0,0 +1,50 @@ +# ADR 0078 — Multi-pattern regex acceleration gated by benchmark (RegexSet before Vectorscan) + +- **Date (UTC):** 2026-06-30 +- **Authors:** Fabio Leitao +- **Deciders:** Fabio Leitao + +## Status + +Proposed + +### Status history + +- 2026-06-30 — Proposed (research closure GitHub #1078) + +## Context + +Off-band research (#1078) proposed **Vectorscan** (Hyperscan fork) for SIMD multi-pattern matching as `norm_tag` patterns grow (#1056, #1074/#1075). RO verification noted: + +- `boar_fast_filter` uses **one Rust regex per call**; Python `re` in `core/detector.py` is not multi-pattern single-pass. +- **Vectorscan** adds C/FFI build complexity and **AVX2-biased** gains — tension with **no-AVX min-spec** lab hosts (#821). +- Crate **`regex::RegexSet`** (already in the dependency tree) offers portable multi-pattern matching **before** any external engine. + +**Benchmark gate (2026-06-30, primary Linux):** + +| Artifact | Finding | +| --- | --- | +| `rust_prefilter_hotspot_v1` | Rust `filter_batch` ~**4.7×** vs Python prefilter on 200k rows (pinned JSON refreshed) | +| `filesystem_phase_breakdown_v1` | On 2k local `.txt` files: **~99.7%** wall time in detect/matching vs **~0.1%** walk — walk parallelization **not** justified on this profile (#1080 🟡) | + +Matching **is** the hotspot for text-heavy local profiles, but **Rust prefilter already addresses the batch hot path**. Further wins should prefer **`RegexSet` in `boar_fast_filter`** (no new dep) before Vectorscan FFI. + +## Decision + +1. **No Vectorscan / Hyperscan dependency** on `main` until: + - `RegexSet` prototype on a **feature branch** fails to meet targets; **and** + - benchmark on a **representative mixed corpus** (PDF + text + connector path) still shows matching ≥ **10–15%** of end-to-end time. +2. **Next engineering step (when scheduled):** spike `RegexSet` in `rust/boar_fast_filter` with parity tests — **not** Vectorscan. +3. If Vectorscan is ever adopted: **Enterprise-tier optional**, **graceful degrade** to current path on no-AVX hosts — never hard dependency of open-core. +4. Pin benchmark artifacts under `tests/benchmarks/` when re-run; do not cite ADR-0007 for perf (synthetic corpus ADR — RO correction #1078). + +## Consequences + +- **Positive:** Avoids premature FFI/build blast radius; keeps min-spec story intact. +- **Negative:** Pattern count growth may still pressure detector until RegexSet ships. +- **Watch:** Re-run `filesystem_phase_breakdown` on SMB/NFS lab host before enabling parallel walk (#1080). + +## References + +- GitHub **#1078**, **#1080** (perf front) +- `tests/benchmarks/README.md`, `run_rust_prefilter_hotspot_bench.py`, `run_filesystem_phase_breakdown_bench.py` diff --git a/docs/adr/INVENTORY.txt b/docs/adr/INVENTORY.txt index acb31bd8..5b0e938b 100644 --- a/docs/adr/INVENTORY.txt +++ b/docs/adr/INVENTORY.txt @@ -1,5 +1,5 @@ # ADR Inventory - generated by scripts/inv-adr.ps1 -# GeneratedUtc: 2026-06-29T06:01:29Z +# GeneratedUtc: 2026-06-30T06:26:04Z # Algorithm: SHA256 # Format: NUMBER | STATUS | FILE_HASH | FILENAME | TITLE | RATIFIED_BY # RATIFIED_BY: - = N/A; PENDING = ratification line with signature PENDING; @op = stamped @@ -77,5 +77,8 @@ 0072 | Accepted | 0BE6A85E1154BC9951A0228B5528993471FA37760B1B3B507ABC952AD0045E8C | ADR-0072-commit-gate-vs-release-gate-distinct-criteria.md | Commit Gate vs Release Gate: distinct criteria | @FabioLeitao 0073 | Accepted | 8D9CDEFD451DDC943775FC4D5DAAE598CC1F433CE11702DE5E4357F40C6E3016 | ADR-0073-version-scheme-octet-maturity-and-roadmap.md | Version scheme: octet-maturity side-channel + release-line roadmap | - 0075 | Proposed | DFB13E4514D8BBC0E5A994FA6D6649C870AF672670D951DB938DBC1BCA13E456 | ADR-0075-plugin-auth-file-based-vs-bearer.md | Plugin authentication — file-based license vs Bearer per-request | - +0076 | Proposed | FD2760FE46D765A060352C48F4B99A892A40846817328691C0DCE29C34E6B25C | ADR-0076-opa-rego-ci-tier-drift-linter-not-runtime.md | OPA/Rego as CI drift linter for commercial tier enforcement (not runtime) | - +0077 | Accepted | F8A4C9D3B2BE2BEE3721C5AE09476F9CD066DB0B58E6B3174D4B783B123A1BAF | ADR-0077-filesystem-scan-no-client-gitignore-by-design.md | Filesystem scan does not honor client `.gitignore` (deliberate design) | - +0078 | Proposed | 5FEF0DCAC3DA292E33519F24FE6387584C04A850771958FF78B19D3EEB856990 | ADR-0078-multi-pattern-regex-benchmark-gate-regexset-before-vectorscan.md | Multi-pattern regex acceleration gated by benchmark (RegexSet before Vectorscan) | - # -# InventoryHash: 40AC1BC92ADF20B0B298BE8A70C11F7497E70E77D013424B12CFE2FD465F0C75 +# InventoryHash: 2AB38ED022D0CCC875EB2828F2D7D7DE863FF7795F3A177CDDA23C275D02691D diff --git a/docs/adr/README.md b/docs/adr/README.md index cb133fce..724b1827 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -93,6 +93,9 @@ Short, durable notes that capture **why** the project chose an approach—not on | 0072 | [Commit Gate vs Release Gate: distinct criteria](ADR-0072-commit-gate-vs-release-gate-distinct-criteria.md) | Accepted | | 0073 | [Version scheme: octet-maturity side-channel + release-line roadmap](ADR-0073-version-scheme-octet-maturity-and-roadmap.md) | Accepted | | 0075 | [Plugin authentication — file-based license vs Bearer](ADR-0075-plugin-auth-file-based-vs-bearer.md) | Proposed | +| 0076 | [OPA/Rego as CI tier drift linter (not runtime)](ADR-0076-opa-rego-ci-tier-drift-linter-not-runtime.md) | Proposed | +| 0077 | [Filesystem scan does not honor client `.gitignore`](ADR-0077-filesystem-scan-no-client-gitignore-by-design.md) | Accepted | +| 0078 | [Multi-pattern regex: RegexSet before Vectorscan (benchmark gate)](ADR-0078-multi-pattern-regex-benchmark-gate-regexset-before-vectorscan.md) | Proposed | ## Related docs diff --git a/research/licensing/opa_tier_drift_prototype/README.md b/research/licensing/opa_tier_drift_prototype/README.md new file mode 100644 index 00000000..22b38508 --- /dev/null +++ b/research/licensing/opa_tier_drift_prototype/README.md @@ -0,0 +1,28 @@ +# OPA tier drift — isolated Rego prototype (#1079) + +**Not wired to CI or runtime.** Demonstrates that connector/feature tier rules +from ADR-0027 / `FEATURE_TIER_MAP` can be expressed in Rego for a future +**drift linter** (ADR-0076). + +## Layout + +| File | Purpose | +| --- | --- | +| `tier_policy.rego` | Sample `min_tier` decisions for connector features | +| `tier_policy_test.rego` | `opa test` style cases (run manually with OPA CLI if installed) | +| `manifest_snapshot.example.json` | Illustrative export shape from Python registry | + +## Manual check (optional, operator machine) + +```bash +# Requires OPA CLI locally — NOT a repo dependency +opa test research/licensing/opa_tier_drift_prototype/ +``` + +## Out of scope + +- Modifying `core/licensing/*` runtime +- Adding OPA to `pyproject.toml` or Docker image +- Enforcing JWT / Ed25519 / `LicenseGuard` behaviour + +See [ADR-0076](../../../docs/adr/ADR-0076-opa-rego-ci-tier-drift-linter-not-runtime.md). diff --git a/research/licensing/opa_tier_drift_prototype/manifest_snapshot.example.json b/research/licensing/opa_tier_drift_prototype/manifest_snapshot.example.json new file mode 100644 index 00000000..af8d3677 --- /dev/null +++ b/research/licensing/opa_tier_drift_prototype/manifest_snapshot.example.json @@ -0,0 +1,10 @@ +{ + "generated_by": "illustrative — future exporter from core/licensing/tier_features.py", + "features": { + "connector_filesystem": "community", + "connector_postgresql": "community", + "connector_smb": "pro", + "connector_powerbi": "pro", + "report_pdf": "pro" + } +} diff --git a/research/licensing/opa_tier_drift_prototype/tier_policy.rego b/research/licensing/opa_tier_drift_prototype/tier_policy.rego new file mode 100644 index 00000000..0e510a91 --- /dev/null +++ b/research/licensing/opa_tier_drift_prototype/tier_policy.rego @@ -0,0 +1,44 @@ +package databoar.tier + +import rego.v1 + +# Illustrative ladder — mirrors FEATURE_TIER_MAP intent (#854 connector boundary). +tier_rank := { + "community": 1, + "pro": 2, + "enterprise": 3, + "open": 99, +} + +default min_tier := "community" + +min_tier := "pro" if { + input.feature == "connector_smb" +} + +min_tier := "pro" if { + input.feature == "connector_powerbi" +} + +min_tier := "community" if { + input.feature == "connector_filesystem" +} + +min_tier := "community" if { + input.feature == "connector_postgresql" +} + +# Symptom #887: custom/partner tiers must not bypass documented ladder without explicit rule. +deny[msg] if { + input.claimed_tier == "partner_custom" + not input.operator_override_documented + msg := "partner_custom tier requires documented operator override (#887)" +} + +# Symptom #854: connector without explicit mapping should fail closed in enforced mode. +deny[msg] if { + input.kind == "connector" + not input.feature_tier_map_has[input.feature] + input.enforcement_mode == "enforced" + msg := sprintf("connector '%s' missing FEATURE_TIER_MAP entry (#854)", [input.feature]) +} diff --git a/research/licensing/opa_tier_drift_prototype/tier_policy_test.rego b/research/licensing/opa_tier_drift_prototype/tier_policy_test.rego new file mode 100644 index 00000000..6a35b297 --- /dev/null +++ b/research/licensing/opa_tier_drift_prototype/tier_policy_test.rego @@ -0,0 +1,22 @@ +package databoar.tier_test + +import rego.v1 + +import data.databoar.tier + +test_smb_requires_pro if { + tier.min_tier with input as {"feature": "connector_smb"} == "pro" +} + +test_filesystem_community if { + tier.min_tier with input as {"feature": "connector_filesystem"} == "community" +} + +test_enforced_missing_connector_denied if { + count(tier.deny) > 0 with input as { + "kind": "connector", + "feature": "connector_unknown", + "feature_tier_map_has": {"connector_filesystem": true}, + "enforcement_mode": "enforced", + } +} diff --git a/tests/benchmarks/README.md b/tests/benchmarks/README.md index 7eeea16a..2ed42d55 100644 --- a/tests/benchmarks/README.md +++ b/tests/benchmarks/README.md @@ -7,6 +7,7 @@ Regenerate only with intent — commit artifact and prose together. | ------------ | ------ | ---------------- | ---------------------------- | | `official_pro_v1` | `run_official_bench.py` | OpenCore Python vs Pro+ `ProcessPoolExecutor` + `process_chunk_worker` on 200k seeded rows | Rust `filter_batch` hotspot; end-to-end scan | | `rust_prefilter_hotspot_v1` | `run_rust_prefilter_hotspot_bench.py` | `OpenCorePreFilter.filter_candidates` vs `FastFilter.filter_batch` on the same batch | Worker pool, ML/DL, connectors, Maestro gate (#1021) | +| `filesystem_phase_breakdown_v1` | `run_filesystem_phase_breakdown_bench.py` | Walk/glob vs read sample vs detect on synthetic `.txt` tree (local disk) | SMB/NFS, PDF extraction, parallel walk decisions (#1080) | ## Regenerate `rust_prefilter_hotspot_v1` @@ -32,3 +33,13 @@ uv run pytest tests/test_official_benchmark_200k_evidence.py -v The 200k artifact records Pro **slower** than OpenCore in that composite profile (~0.574×). Do **not** use it for the Rust prefilter ~11–13× headline — use `rust_prefilter_hotspot_v1`. + +## Regenerate `filesystem_phase_breakdown_v1` + +```bash +uv run python tests/benchmarks/run_filesystem_phase_breakdown_bench.py \ + --files 2000 --iterations 3 \ + --output tests/benchmarks/filesystem_phase_breakdown.json +``` + +Use results to gate parallel filesystem walk (#1080) and to complement `rust_prefilter_hotspot_v1` for #1078 (local txt profile only). diff --git a/tests/benchmarks/filesystem_phase_breakdown.json b/tests/benchmarks/filesystem_phase_breakdown.json new file mode 100644 index 00000000..6fab79ab --- /dev/null +++ b/tests/benchmarks/filesystem_phase_breakdown.json @@ -0,0 +1,26 @@ +{ + "benchmark_id": "filesystem_phase_breakdown_v1", + "generated_at_utc": "2026-06-30T06:23:35.772738+00:00", + "git_sha": "a5d11258", + "host": "Linux-6.12.94+deb13-amd64-x86_64-with-glibc2.41", + "file_count": 2000, + "iterations": 3, + "max_chars": 12000, + "seconds": { + "walk_glob": 0.006223199345792334, + "read_sample": 0.01712814497295767, + "detect_delta": 8.224179534707218, + "total_estimated": 8.247530879025968 + }, + "percent_of_total": { + "walk_glob": 0.08, + "read_sample": 0.21, + "detect_matching": 99.72 + }, + "gate_notes": { + "vectorscan_threshold_pct": 15, + "parallel_walk_threshold_pct": 15, + "matching_below_vectorscan_gate": false, + "walk_below_parallel_gate": true + } +} diff --git a/tests/benchmarks/run_filesystem_phase_breakdown_bench.py b/tests/benchmarks/run_filesystem_phase_breakdown_bench.py new file mode 100644 index 00000000..4a864a69 --- /dev/null +++ b/tests/benchmarks/run_filesystem_phase_breakdown_bench.py @@ -0,0 +1,179 @@ +#!/usr/bin/env python3 +""" +Filesystem scan phase breakdown (research benchmark for #1078 / #1080). + +Measures wall time for: directory enumeration (walk/glob) vs file read+extract sample +vs detector scan_file_content on a synthetic corpus of plain .txt files. + +Does NOT claim SMB/NFS latency — local ext4/tmpfs only. Use to gate parallel-walk +and vectorscan work: if read+detect dominate, regex SIMD is not the first lever. +""" + +from __future__ import annotations + +import argparse +import json +import platform +import statistics +import subprocess +import time +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +BENCHMARK_ID = "filesystem_phase_breakdown_v1" + + +def _git_sha_short() -> str | None: + try: + out = subprocess.check_output( + ["git", "rev-parse", "--short", "HEAD"], + stderr=subprocess.DEVNULL, + text=True, + ) + return out.strip() or None + except (OSError, subprocess.CalledProcessError): + return None + + +def _seed_corpus(root: Path, *, file_count: int) -> None: + root.mkdir(parents=True, exist_ok=True) + payload = ( + "Holder: sample_holder_a\nID: 123.456.789-00\nEmail: sample@example.com\n" + "Phone: (11) 90000-0000\n" + ) + for i in range(file_count): + (root / f"doc_{i:05d}.txt").write_text(payload, encoding="utf-8") + + +def _time_glob_walk(root: Path, iterations: int) -> float: + timings: list[float] = [] + for _ in range(iterations): + start = time.perf_counter() + count = 0 + for p in root.glob("**/*"): + if p.is_file(): + count += 1 + timings.append(time.perf_counter() - start) + assert count > 0 + return statistics.mean(timings) + + +def _time_read_samples(root: Path, iterations: int, max_chars: int) -> float: + from connectors.filesystem_connector import _read_text_sample + + files = [p for p in root.glob("**/*") if p.is_file()] + timings: list[float] = [] + for _ in range(iterations): + start = time.perf_counter() + for fp in files: + _read_text_sample(fp, ".txt", max_chars, {}) + timings.append(time.perf_counter() - start) + return statistics.mean(timings) + + +def _time_detect(root: Path, iterations: int, max_chars: int) -> float: + from connectors.filesystem_connector import _read_text_sample + from core.scanner import DataScanner + + scanner = DataScanner() + files = [p for p in root.glob("**/*") if p.is_file()] + timings: list[float] = [] + for _ in range(iterations): + start = time.perf_counter() + for fp in files: + content = _read_text_sample(fp, ".txt", max_chars, {}) + scanner.scan_file_content(content, fp) + timings.append(time.perf_counter() - start) + return statistics.mean(timings) + + +def run_benchmark( + *, + file_count: int, + warmup: int, + iterations: int, + max_chars: int, +) -> dict[str, Any]: + import tempfile + + with tempfile.TemporaryDirectory(prefix="db_fs_bench_") as tmp: + root = Path(tmp) + _seed_corpus(root, file_count=file_count) + print(f"[*] Filesystem phase breakdown: {file_count:,} txt files under {root}") + + for _ in range(warmup): + _time_glob_walk(root, 1) + _time_read_samples(root, 1, max_chars) + _time_detect(root, 1, max_chars) + + walk_s = _time_glob_walk(root, iterations) + read_s = _time_read_samples(root, iterations, max_chars) + detect_s = _time_detect(root, iterations, max_chars) + + # read pass includes read; detect pass includes read+detect + read_only_s = max(read_s, 0.0) + detect_only_s = max(detect_s - read_s, 0.0) + total_s = walk_s + detect_s + walk_pct = 100.0 * walk_s / total_s if total_s else 0.0 + read_pct = 100.0 * read_only_s / total_s if total_s else 0.0 + detect_pct = 100.0 * detect_only_s / total_s if total_s else 0.0 + matching_pct = detect_pct # scan_file_content ~= matching+ML for txt + + print(f" walk/glob mean: {walk_s:.4f}s ({walk_pct:.1f}%)") + print(f" read sample mean: {read_only_s:.4f}s ({read_pct:.1f}%)") + print(f" detect delta mean: {detect_only_s:.4f}s ({detect_pct:.1f}%)") + + return { + "benchmark_id": BENCHMARK_ID, + "generated_at_utc": datetime.now(UTC).isoformat(), + "git_sha": _git_sha_short(), + "host": platform.platform(), + "file_count": file_count, + "iterations": iterations, + "max_chars": max_chars, + "seconds": { + "walk_glob": walk_s, + "read_sample": read_only_s, + "detect_delta": detect_only_s, + "total_estimated": total_s, + }, + "percent_of_total": { + "walk_glob": round(walk_pct, 2), + "read_sample": round(read_pct, 2), + "detect_matching": round(matching_pct, 2), + }, + "gate_notes": { + "vectorscan_threshold_pct": 15, + "parallel_walk_threshold_pct": 15, + "matching_below_vectorscan_gate": matching_pct < 15, + "walk_below_parallel_gate": walk_pct < 15, + }, + } + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--files", type=int, default=2000) + parser.add_argument("--warmup", type=int, default=1) + parser.add_argument("--iterations", type=int, default=3) + parser.add_argument("--max-chars", type=int, default=12_000) + parser.add_argument( + "--output", + type=Path, + default=Path("tests/benchmarks/filesystem_phase_breakdown.json"), + ) + args = parser.parse_args() + payload = run_benchmark( + file_count=args.files, + warmup=args.warmup, + iterations=args.iterations, + max_chars=args.max_chars, + ) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8") + print(f"[+] Wrote {args.output}") + + +if __name__ == "__main__": + main() diff --git a/tests/benchmarks/rust_prefilter_hotspot.json b/tests/benchmarks/rust_prefilter_hotspot.json index 1e575b70..6f2c014e 100644 --- a/tests/benchmarks/rust_prefilter_hotspot.json +++ b/tests/benchmarks/rust_prefilter_hotspot.json @@ -3,12 +3,12 @@ "rows": 200000, "warmup_iterations": 2, "measure_iterations": 5, - "python_seconds": 0.156998, - "rust_seconds": 0.03942, - "speedup_rust_vs_python": 3.9828, + "python_seconds": 0.124293, + "rust_seconds": 0.026505, + "speedup_rust_vs_python": 4.6894, "python_hit_count": 100000, "rust_hit_count": 100000, "host_profile": "linux-x86_64", - "git_sha": "085f5994", - "generated_at": "2026-06-29T19:05:31.586178+00:00" + "git_sha": "a5d11258", + "generated_at": "2026-06-30T06:22:47.502909+00:00" } From b9ef0e14a54b249954e2431c3acc31af1feb0bfc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?FabioLeit=C3=A3o?= Date: Sat, 4 Jul 2026 11:56:34 -0300 Subject: [PATCH 5/8] =?UTF-8?q?docs(adr):=20add=20ADR=200074=20=E2=80=94?= =?UTF-8?q?=20supply-chain=20Layer=201=20digest=20pins=20and=20Rust=20SCA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Materializes the dangling #987 posture cited in CI, Dockerfile, and deny.toml since PR #998; regenerates INVENTORY.txt. Co-authored-by: Cursor --- ...y-chain-layer1-digest-pins-and-rust-sca.md | 56 +++++++++++++++++++ docs/adr/INVENTORY.txt | 5 +- docs/adr/README.md | 1 + docs/adr/README.pt_BR.md | 1 + tests/fixtures/adr_genesis_date_lines.json | 1 + 5 files changed, 62 insertions(+), 2 deletions(-) create mode 100644 docs/adr/ADR-0074-supply-chain-layer1-digest-pins-and-rust-sca.md diff --git a/docs/adr/ADR-0074-supply-chain-layer1-digest-pins-and-rust-sca.md b/docs/adr/ADR-0074-supply-chain-layer1-digest-pins-and-rust-sca.md new file mode 100644 index 00000000..181baae0 --- /dev/null +++ b/docs/adr/ADR-0074-supply-chain-layer1-digest-pins-and-rust-sca.md @@ -0,0 +1,56 @@ +# ADR 0074 — Supply-chain Layer 1: digest pins and Rust SCA + +- **Date (UTC):** 2026-06-23 +- **Authors:** Fabio Leitao +- **Deciders:** Fabio Leitao + +## Status + +Proposed + +### Status history + +- 2026-06-23 — Proposed (materialized to close dangling reference: supply-chain posture #987 cited across CI, Dockerfile, dependabot, grype, and deny.toml since PR #998; ADR file missing — PR #997 touched only INVENTORY.txt). + +## Context + +The build and release pipeline pulls third-party GitHub Actions, base container images, and Rust crates — each a supply-chain entry point. Without pinning and scanning, a compromised upstream tag or a vulnerable transitive crate can enter a release silently. + +[ADR 0005](ADR-0005-ci-github-actions-supply-chain-pins.md) established Action SHA pinning and a pinned **uv** CLI in CI; this ADR records **Layer 1** of the broader supply-chain posture (GitHub #987) and ties together digest-pinned images, Rust SCA, and distroless image scanning already enforced in workflows and tests. + +## Decision + +Layer 1 of the supply-chain posture is mandatory and mechanical: + +1. **Actions pinned to full commit SHAs** — no floating tags in `.github/workflows/*`. +2. **Base image digest pins** — `Dockerfile` `FROM` lines pinned by digest; the Dependabot docker ecosystem proposes digest bumps. +3. **Rust SCA before merge** — `cargo-deny` (`rust/boar_fast_filter/deny.toml`) gates advisories and license drift. +4. **Distroless + Grype gate** — the assembled image is scanned; findings block release. + +## Rationale + +1. Pinning removes the moving-tag attack surface: a rebuilt CI run resolves the exact bytes reviewed. +2. Rust SCA catches vulnerable transitive crates before they reach a build, not after. + +Each layer is additive and does not replace the previous one; Layer 1 is the mechanical floor on which detection layers (L3/L4) rest. + +## Consequences + +- **Positive:** Reproducible, reviewable supply-chain; no floating upstream. +- **Negative:** Digest/SHA bumps require Dependabot churn and a review cadence. + +## Related Decisions + +- [ADR 0005 — CI and GitHub Actions supply chain pins](ADR-0005-ci-github-actions-supply-chain-pins.md) +- [ADR 0063 — Ed25519 license JWT signing](ADR-0063-ed25519-license-jwt-signing.md) +- [ADR 0075 — Plugin authentication: file-based vs Bearer](ADR-0075-plugin-auth-file-based-vs-bearer.md) +- GitHub #987 (decision) · #988 (Layer 1 implementation) · #997 (inventory-only merge — the gap) +- [PLAN_IMAGE_HARDENING](../plans/completed/PLAN_IMAGE_HARDENING.md) (completed) + +## References + +- `tests/test_github_workflows.py` — Layer 1 workflow and Dockerfile guards +- `tests/test_docker_image_hardening.py` — digest-pinned `FROM` stages +- `.github/dependabot.yml` — docker ecosystem digest bumps +- `.grype.yaml` — distroless scan policy +- `rust/boar_fast_filter/deny.toml` — `cargo-deny` policy diff --git a/docs/adr/INVENTORY.txt b/docs/adr/INVENTORY.txt index 9d1f8774..12101901 100644 --- a/docs/adr/INVENTORY.txt +++ b/docs/adr/INVENTORY.txt @@ -1,5 +1,5 @@ # ADR Inventory - generated by scripts/inv-adr.ps1 -# GeneratedUtc: 2026-07-04T05:17:52Z +# GeneratedUtc: 2026-07-04T14:53:16Z # Algorithm: SHA256 # Format: NUMBER | STATUS | FILE_HASH | FILENAME | TITLE | RATIFIED_BY # RATIFIED_BY: - = N/A; PENDING = ratification line with signature PENDING; @op = stamped @@ -76,6 +76,7 @@ 0071 | Accepted | 8D6FB89D7247D8E353D30EF14F6C3A281C15BBED9ECF574D2955E220DA8734D8 | ADR-0071-self-protecting-pii-gate.md | Self-protecting PII gate: word-boundary matcher, CODEOWNERS, modification tripwire, sanctioned FP allowlist | @FabioLeitao 0072 | Accepted | 0BE6A85E1154BC9951A0228B5528993471FA37760B1B3B507ABC952AD0045E8C | ADR-0072-commit-gate-vs-release-gate-distinct-criteria.md | Commit Gate vs Release Gate: distinct criteria | @FabioLeitao 0073 | Accepted | C6353A34C1AA2A688373B52E6F90A28576D4C87CFC0D79610B5B425219599982 | ADR-0073-version-scheme-octet-maturity-and-roadmap.md | Version scheme: octet-maturity side-channel + release-line roadmap | - +0074 | Proposed | 8361B02C52B4F4F6EBD0AACFA7BAFB51860132A4659E842F7363201B934CE64F | ADR-0074-supply-chain-layer1-digest-pins-and-rust-sca.md | Supply-chain Layer 1: digest pins and Rust SCA | - 0075 | Proposed | DFB13E4514D8BBC0E5A994FA6D6649C870AF672670D951DB938DBC1BCA13E456 | ADR-0075-plugin-auth-file-based-vs-bearer.md | Plugin authentication — file-based license vs Bearer per-request | - 0076 | Proposed | FD2760FE46D765A060352C48F4B99A892A40846817328691C0DCE29C34E6B25C | ADR-0076-opa-rego-ci-tier-drift-linter-not-runtime.md | OPA/Rego as CI drift linter for commercial tier enforcement (not runtime) | - 0077 | Accepted | F8A4C9D3B2BE2BEE3721C5AE09476F9CD066DB0B58E6B3174D4B783B123A1BAF | ADR-0077-filesystem-scan-no-client-gitignore-by-design.md | Filesystem scan does not honor client `.gitignore` (deliberate design) | - @@ -83,4 +84,4 @@ 0079 | Proposed | 158C68D90B9C6934F0FAAC392E86C6C889FA8F5A6E9AD8197168205946435F9D | ADR-0079-ecosystem-engineering-rigor-canon.md | Ecosystem engineering rigor canon (UMADR satellites) | - 0080 | Proposed | 5084990188A98B0454A3FA377C6180DBE5D9CC23B4F71398DBEDFDBB73D31045 | ADR-0080-local-validation-gate-inviolable.md | Local validation gate is inviolable: full check-all (all-greens, zero regressions) before ANY push or PR | - # -# InventoryHash: 6E7B04DF01C09300B8C43A593786854D90BFA59EFEC6781C09A831BF0573E97A +# InventoryHash: 76DE3B073F0F9AB3A858012E75DFE6876519BC31C3558913C91065AB0633BC3B diff --git a/docs/adr/README.md b/docs/adr/README.md index 13c77167..0f14ae38 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -92,6 +92,7 @@ Short, durable notes that capture **why** the project chose an approach—not on | 0071 | [Self-protecting PII gate: word-boundary matcher, CODEOWNERS, tripwire, FP allowlist](ADR-0071-self-protecting-pii-gate.md) | Accepted | | 0072 | [Commit Gate vs Release Gate: distinct criteria](ADR-0072-commit-gate-vs-release-gate-distinct-criteria.md) | Accepted | | 0073 | [Version scheme: octet-maturity side-channel + release-line roadmap](ADR-0073-version-scheme-octet-maturity-and-roadmap.md) | Accepted | +| 0074 | [Supply-chain Layer 1: digest pins and Rust SCA](ADR-0074-supply-chain-layer1-digest-pins-and-rust-sca.md) | Proposed | | 0075 | [Plugin authentication — file-based license vs Bearer](ADR-0075-plugin-auth-file-based-vs-bearer.md) | Proposed | | 0076 | [OPA/Rego as CI tier drift linter (not runtime)](ADR-0076-opa-rego-ci-tier-drift-linter-not-runtime.md) | Proposed | | 0077 | [Filesystem scan does not honor client `.gitignore`](ADR-0077-filesystem-scan-no-client-gitignore-by-design.md) | Accepted | diff --git a/docs/adr/README.pt_BR.md b/docs/adr/README.pt_BR.md index 6f3fac17..f200dfee 100644 --- a/docs/adr/README.pt_BR.md +++ b/docs/adr/README.pt_BR.md @@ -53,6 +53,7 @@ Notas curtas e duradouras que registram **por que** o projeto escolheu um caminh - **0067** — *Reservado* (número liberado; auth de plugin → **ADR-0075** / #769). - [ADR 0072](ADR-0072-commit-gate-vs-release-gate-distinct-criteria.md) (EN) — **Proposed** — **Commit Gate** (`check-all`/CI) vs **Release Gate** (#406 completão + decisão do operador); vocabulario normativo pós-#970; guarda de maquina (regra 5) em issue filha. - [ADR 0073](ADR-0073-version-scheme-octet-maturity-and-roadmap.md) (EN) — **Accepted** — octeto-maturidade em `[tool.databoar] maturity_build` (0–127 beta, 128–199 rc, 200–255 release; `.200`=GA, `.201`=fix-1); público **três segmentos**; pós-GA fix na linha `1.7.4` + octeto; próximo público **1.8.0** (sem **1.7.5**); pós-#970/#772/#977. +- [ADR 0074](ADR-0074-supply-chain-layer1-digest-pins-and-rust-sca.md) (EN) — **Proposed** — supply-chain Layer 1: SHAs de Actions, digest no `Dockerfile`, `cargo-deny` (Rust SCA), Grype na imagem distroless; fecha referência pendente de #987. - [ADR 0075](ADR-0075-plugin-auth-file-based-vs-bearer.md) (EN) — **Proposed** — auth de plugins L2/L3: híbrido (licença file-based + RBAC interno); renumeração de #769. - [ADR 0065](ADR-0065-nist-sp800228a-api-security-reference.md) (EN) — **Deferido** — NIST SP 800-228A ("Secure Deployment of RESTful Web APIs") como referencia de hardening da API REST; aguardando publicacao final (public comment deadline 2026-07-02; previsto Q3/Q4 2026). - [ADR 0066](ADR-0066-tampered-state-behavior.md) (EN) — comportamento do estado `TAMPERED`: fail-closed em modo `enforced`; pass-through em modo `monitor`. diff --git a/tests/fixtures/adr_genesis_date_lines.json b/tests/fixtures/adr_genesis_date_lines.json index fb14cb07..77a5b9b8 100644 --- a/tests/fixtures/adr_genesis_date_lines.json +++ b/tests/fixtures/adr_genesis_date_lines.json @@ -71,6 +71,7 @@ "ADR-0071-self-protecting-pii-gate.md": "- **Date (UTC):** 2026-06-18", "ADR-0072-commit-gate-vs-release-gate-distinct-criteria.md": "- **Date (UTC):** 2026-06-21", "ADR-0073-version-scheme-octet-maturity-and-roadmap.md": "- **Date (UTC):** 2026-06-21", + "ADR-0074-supply-chain-layer1-digest-pins-and-rust-sca.md": "- **Date (UTC):** 2026-06-23", "ADR-0075-plugin-auth-file-based-vs-bearer.md": "- **Date (UTC):** 2026-06-21", "ADR-0076-opa-rego-ci-tier-drift-linter-not-runtime.md": "- **Date (UTC):** 2026-06-30", "ADR-0077-filesystem-scan-no-client-gitignore-by-design.md": "- **Date (UTC):** 2026-06-30", From 806209642de63e6da329fc5229136684405e6b7a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?FabioLeit=C3=A3o?= Date: Sat, 4 Jul 2026 15:39:32 -0300 Subject: [PATCH 6/8] =?UTF-8?q?feat(gate):=20no-coauthorship=20kombi=20?= =?UTF-8?q?=E2=80=94=20gitleaks=20value-rule=20+=20pytest=20commit-msg=20g?= =?UTF-8?q?ate=20+=20auto-strip=20hook;=20b9ef0e14=20co-author=20preservad?= =?UTF-8?q?o=20como=20evidencia=20(forum=20#161253)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .githooks/commit-msg | 26 +++++ .github/workflows/ci.yml | 11 +++ .github/workflows/gitleaks.yml | 2 +- .gitleaks.toml | 22 +++++ .pre-commit-config.yaml | 16 ++++ core/demo/synthetic_corpus.py | 7 +- docs/plans/PLANS_HUB.md | 1 + docs/plans/PLANS_TODO.md | 1 + docs/plans/PLAN_NO_COAUTHORSHIP_GATE.md | 51 ++++++++++ tests/test_commit_no_tool_coauthorship.py | 112 ++++++++++++++++++++++ tests/test_github_workflows.py | 1 + tests/test_gitleaks_config.py | 40 ++++++++ 12 files changed, 287 insertions(+), 3 deletions(-) create mode 100755 .githooks/commit-msg create mode 100644 docs/plans/PLAN_NO_COAUTHORSHIP_GATE.md create mode 100644 tests/test_commit_no_tool_coauthorship.py create mode 100644 tests/test_gitleaks_config.py diff --git a/.githooks/commit-msg b/.githooks/commit-msg new file mode 100755 index 00000000..47e5a7ae --- /dev/null +++ b/.githooks/commit-msg @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# .githooks/commit-msg — auto-strip de co-autoria injetada (A.I.I.D.C.O.B.P.P. v1.4). +# +# POR QUE: neste repo NAO existe co-autoria (um autor por commit). A IDE do Cursor +# injeta 'Co-authored-by: Cursor' MESMO com o toggle 'Commit Attribution' OFF — +# a Anysphere admite "toggles sao sugestoes, nao enforceables" (forum #161253, SEV-1). +# Este hook enforca MECANICAMENTE no repo do operador o que o toggle do vendor recusa. +# +# FORWARD-ONLY: mexe so na mensagem do commit ATUAL; nunca reescreve history (a +# evidencia do incidente Cursor no history fica preservada — analise Anysphere/Anthropic). +# BACKSTOP: o gate pytest (tests/test_commit_no_tool_coauthorship.py) e a rede LOUD +# se este hook falhar. Cinturao + suspensorio (nao e "silent fix"). +# +# NAO usa --no-verify pra pular isto (proibido ao agente, matriz v1.4). So o HITL, +# na mao, se algum dia quiser um co-autor humano real (raro). +set -euo pipefail + +MSG_FILE="${1:?commit-msg hook: falta o arquivo da mensagem}" + +# Remove QUALQUER linha 'Co-authored-by:' (case-insensitive). Fail-closed = igual ao gate. +if grep -qiE '^[[:space:]]*Co-authored-by:[[:space:]]*\S' "$MSG_FILE"; then + stripped="$(grep -viE '^[[:space:]]*Co-authored-by:[[:space:]]*\S' "$MSG_FILE")" + printf '%s\n' "$stripped" > "$MSG_FILE" + echo "[commit-msg] co-autoria injetada removida (repo nao co-autora; ver forum #161253)." >&2 +fi +exit 0 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4ba7db14..25f3cfae 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -72,6 +72,17 @@ jobs: # dev group: pytest, hypothesis, pre-commit stack — not shipped to end-user `uv sync` default in CI test matrix run: uv sync --extra shares --group dev + - name: Install gitleaks (binary; kombi PLAN_NO_COAUTHORSHIP_GATE) + run: | + VER="8.30.1" + BASE="https://github.com/gitleaks/gitleaks/releases/download/v${VER}" + TARBALL="gitleaks_${VER}_linux_x64.tar.gz" + curl -sSfL "${BASE}/${TARBALL}" -o "${TARBALL}" + curl -sSfL "${BASE}/gitleaks_${VER}_checksums.txt" -o gl.sums + grep " ${TARBALL}$" gl.sums | sha256sum -c - + tar -xzf "${TARBALL}" gitleaks + sudo install -m 0755 gitleaks /usr/local/bin/gitleaks + - name: Run tests run: uv run pytest -v -W error # addopts in pyproject.toml also set -W error; explicit in CI so tests pass without warnings. diff --git a/.github/workflows/gitleaks.yml b/.github/workflows/gitleaks.yml index 9b9b989e..d745ddfd 100644 --- a/.github/workflows/gitleaks.yml +++ b/.github/workflows/gitleaks.yml @@ -49,7 +49,7 @@ jobs: curl -sSfL "${BASE}/gitleaks_${VER}_checksums.txt" -o gl.sums grep " ${TARBALL}$" gl.sums | sha256sum -c - tar -xzf "${TARBALL}" gitleaks - ./gitleaks git . --no-banner --redact --exit-code 1 + ./gitleaks git . --config .gitleaks.toml --no-banner --redact --exit-code 1 slack-notify-on-failure: name: Slack CI failure notify diff --git a/.gitleaks.toml b/.gitleaks.toml index 0a32d5d2..a560f54f 100644 --- a/.gitleaks.toml +++ b/.gitleaks.toml @@ -1,7 +1,18 @@ title = "data-boar gitleaks config" +# extend + useDefault: mantém TODAS as regras default de segredo (não regride a detecção) +# e ADICIONA a regra de governança abaixo. Sem isso, [[rules]] custom SUBSTITUIria as defaults. +[extend] +useDefault = true + [allowlist] description = "Confirmed FP 2026-05-18 — synthetic test data and operator howto curl examples" +# FP: chave demo sintetica + fingerprint GPG PUBLICO, nao segredo (allowlist por VALOR, nunca por path) +regexes = [ + '''correct-key-12345''', + '''D6F81740\s*AA5FAB8A\s*CE5716AF\s*871542E6\s*F789F64F''', + '''D6F81740AA5FAB8ACE5716AF871542E6F789F64F''', +] paths = [ '''scripts/generate_synthetic_poc_corpus\.py''', '''docs/ops/API_KEY_FROM_ENV_OPERATOR_STEPS\.md''', @@ -10,3 +21,14 @@ paths = [ '''docs/ops/SECURE_DASHBOARD_AUTH_AND_HTTPS_HOWTO\.pt_BR\.md''', '''docs/OPERATOR_NOTIFICATION_CHANNELS\.md''' ] + +# ── Governança: NÃO EXISTE CO-AUTORIA (fail-closed, defense-in-depth) ── +# Um commit = UM autor (quem fez). Review != co-autoria. IA = FERRAMENTA, nunca co-autora. +# QUALQUER Co-authored-by = FAIL (sem allowlist — nome de humano é forjável; o agente +# escreveria "Fabio" e furaria). Gitleaks varre CONTEÚDO/diff; a commit-MESSAGE é coberta +# pelo pytest tests/test_commit_no_tool_coauthorship.py (a kombi). +[[rules]] +id = "no-coauthorship-at-all" +description = "PROIBIDO: NAO existe co-autoria neste repo — todo commit tem UM autor (quem fez). IA e FERRAMENTA (nunca co-autora); review != co-autoria. QUALQUER 'Co-authored-by:' = FAIL. Usar o nome do operador/humano sem ser ele = FORJA de identidade = impersonation = crime (org DataBoar). CONSERTE INDO PRA FRENTE: releia git log + ADRs 0046/0047/0049/0079 NA INTEGRA e amend forward SEM o trailer. NUNCA --no-verify nem skip (proibido ao agente). So o HITL, na mao, se precisar." +regex = '''(?im)^[ \t]*Co-authored-by:[ \t]*\S''' +tags = ["governance", "no-coauthorship", "fail-closed", "hitl", "adr-0049", "adr-0079"] diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 68836aa3..da35fca6 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -74,6 +74,22 @@ repos: entry: uv run pytest tests/test_adr_governance_phase1.py -q -W error language: system pass_filenames: false + - id: commit-msg-strip-coauthor + name: strip injected Co-authored-by from commit message (forward-only) + entry: .githooks/commit-msg + language: script + stages: [commit-msg] + pass_filenames: true + - id: no-coauthorship-commit-messages + name: no co-authorship in commit messages (kombi) + entry: uv run pytest tests/test_commit_no_tool_coauthorship.py -q -W error + language: system + pass_filenames: false + - id: gitleaks-detect-config + name: gitleaks detect with repo .gitleaks.toml (kombi) + entry: uv run pytest tests/test_gitleaks_config.py -q -W error + language: system + pass_filenames: false - id: workflow-security-lint name: zizmor workflow security lint (optional manual stage) entry: pwsh -NoProfile -File scripts/workflow-security-lint.ps1 -SkipIfMissing diff --git a/core/demo/synthetic_corpus.py b/core/demo/synthetic_corpus.py index c5bb2f1b..169bb2a9 100644 --- a/core/demo/synthetic_corpus.py +++ b/core/demo/synthetic_corpus.py @@ -770,11 +770,14 @@ def gen_config_errors(base: Path) -> None: "targets": [ {"type": "filesystem", "path": "./tests/synthetic_corpus/1_happy"} ], - "api": {"require_api_key": True, "api_key": "correct-key-12345"}, + "api": { + "require_api_key": True, + "api_key": "EXAMPLE-PLACEHOLDER-NOT-A-KEY", + }, "report": {"output_dir": "./reports"}, }, "expected_error": "HTTP 401 Unauthorized when calling API with wrong key", - "troubleshoot": "Use X-API-Key: correct-key-12345 no header. " + "troubleshoot": "Use X-API-Key: EXAMPLE-PLACEHOLDER-NOT-A-KEY no header. " "Para testar: curl -H 'X-API-Key: wrong-key' http://localhost:8088/api/v1/scan", "test_curl": ( "curl -s -o /dev/null -w '%{http_code}' " diff --git a/docs/plans/PLANS_HUB.md b/docs/plans/PLANS_HUB.md index 04e34de1..56ea81f2 100644 --- a/docs/plans/PLANS_HUB.md +++ b/docs/plans/PLANS_HUB.md @@ -94,6 +94,7 @@ Do **not** edit the table manually; refresh with `python scripts/plans_hub_sync. | **Open** | [PLAN_NEXT_WAVE_PLATFORM_AND_GTM.md](PLAN_NEXT_WAVE_PLATFORM_AND_GTM.md) | Plan: Next wave after core trust foundations (platform + GTM) | **Status:** Pending **Date:** 2026-04-02 **Authors:** Fabio Leitao **Priority:** H1 | — | | **Open** | [PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md](PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md) | Plan: Hierarchical norm_tag + configurable Data-Subject axis | Hybrid slice 1 (#1071): parent→child norm_tag inheritance preserving BR/LGPD vocabulary + optional Data-Subject axis; closes #1069 titular gap. | [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) [PLAN_SUBJECT_CATEGORY_AXIS.md](PLAN_SUBJECT_CATEGORY_AXIS.md) [PLAN_DATA_USE_AXIS.md](PLAN_DATA_USE_AXIS.md) [PLAN_FIDESLANG_EXPORT_ADAPTER.md](PLAN_FIDESLANG_EXPORT_ADAPTER.md) | | **Open** | [PLAN_NOTIFICATIONS_OFFBAND_AND_SCAN_COMPLETE.md](PLAN_NOTIFICATIONS_OFFBAND_AND_SCAN_COMPLETE.md) | Plan: Off-band notifications and scan-completion notifications | **Status:** Pending **Date:** 2026-03-15 **Authors:** Fabio Leitao **Priority:** H2 | — | +| **Open** | [PLAN_NO_COAUTHORSHIP_GATE.md](PLAN_NO_COAUTHORSHIP_GATE.md) | Plan: No-coauthorship gate (kombi — gitleaks + pytest) | Fail-closed kombi blocking any Co-authored-by trailer in commit messages (pytest) plus governance rule in .gitleaks.toml; A.I.I.D.C.O.B.P.P. v1.4 / ADR-0049 / ADR-0079. | — | | **Open** | [PLAN_OBJECT_STORAGE_CLOUD_CONNECTORS.md](PLAN_OBJECT_STORAGE_CLOUD_CONNECTORS.md) | Plan: Object storage connectors (S3-class, Azure Blob, GCS) | **Status:** Deferred **Date:** 2026-03-26 **Authors:** Fabio Leitao **Priority:** H2 | — | | **Open** | [PLAN_OPERATOR_API_KEY_FIRST_AUTH_UX.md](PLAN_OPERATOR_API_KEY_FIRST_AUTH_UX.md) | Plan: Operator API key–first auth UX (reduce JWT / manual Bearer toil) | **Status:** Pending **Date:** 2026-03-29 **Authors:** Fabio Leitao **Priority:** H2 | — | | **Open** | [PLAN_OPTIONAL_STRONG_CRYPTO_AND_CONTROLS_VALIDATION.md](PLAN_OPTIONAL_STRONG_CRYPTO_AND_CONTROLS_VALIDATION.md) | Plan: Optional strong-crypto validation and inference of anonymisation/controls | **Status:** Deferred **Date:** 2026-03-15 **Authors:** Fabio Leitao **Priority:** H3 | — | diff --git a/docs/plans/PLANS_TODO.md b/docs/plans/PLANS_TODO.md index e15bfe55..619373f8 100644 --- a/docs/plans/PLANS_TODO.md +++ b/docs/plans/PLANS_TODO.md @@ -90,6 +90,7 @@ When **partners** or **buyers** anchor on a vertical (MSP, insurance, RPO, real - **Gemini / LLM doc triage (optional):** After a public-bundle review, use [PLAN_GEMINI_FEEDBACK_TRIAGE.md](PLAN_GEMINI_FEEDBACK_TRIAGE.md) for non-authoritative optional to-dos and promotion gates—does not override CI, pytest, or agreed sequencing until promoted here or in an issue. - **Taxonomy axes (G0-G3 + hub):** [PLAN_G_TIER.md](PLAN_G_TIER.md) ([pt-BR](PLAN_G_TIER.pt_BR.md)) formalizes **gravity** for findings; [PLAN_TAXONOMY_AXES.md](PLAN_TAXONOMY_AXES.md) maps orthogonal axes; [ADR-0055](../adr/ADR-0055-orthogonal-priority-axes-anti-collision-contract.md) records the anti-collision contract. **`scripts/inv-adr.ps1`** regenerates **`docs/adr/INVENTORY.txt`**. - **ADR governance enforcement (#1162):** Phase 1 deterministic anti-regression tests (T1 lifecycle, T2 locale/structure, T5 anti-deletion rename-aware, T6 date immutability) — [PLAN_ADR_GOVERNANCE_ENFORCEMENT.md](PLAN_ADR_GOVERNANCE_ENFORCEMENT.md); pre-commit + CI. Phase 2 (T3 prose density, T4 anti-dangling refs, ADR-0074 smoke) ⬜ deferred. +- **No-coauthorship kombi (A.I.I.D.C.O.B.P.P. v1.4):** Fail-closed gate — `.gitleaks.toml` rule `no-coauthorship-at-all` + `tests/test_commit_no_tool_coauthorship.py` (commit messages) + `tests/test_gitleaks_config.py` (`gitleaks detect --config .gitleaks.toml`); pre-commit + CI pytest; `gitleaks.yml` uses repo config — [PLAN_NO_COAUTHORSHIP_GATE.md](PLAN_NO_COAUTHORSHIP_GATE.md). **Operator:** disable Cursor co-author injection; amend polluted commits forward (no `--no-verify` / `commit-tree`). - **Hybrid taxonomy (#1071, Fideslang eval — closed 2026-06-30):** RO recommendation **B-interno + A-export-adapter** — three implementation slices (all **optional**, backward compatible): **(1)** [PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md](PLAN_NORM_TAG_HIERARCHY_AND_DATA_SUBJECT.md) **[#1074](https://github.com/FabioLeitao/data-boar/issues/1074)** — hierarchical `norm_tag` + Data-Subject → closes **#1069** (supersedes [#1072](https://github.com/FabioLeitao/data-boar/issues/1072)); **(2)** [PLAN_DATA_USE_AXIS.md](PLAN_DATA_USE_AXIS.md) **[#1075](https://github.com/FabioLeitao/data-boar/issues/1075)**; **(3)** [PLAN_FIDESLANG_EXPORT_ADAPTER.md](PLAN_FIDESLANG_EXPORT_ADAPTER.md) **[#1076](https://github.com/FabioLeitao/data-boar/issues/1076)** → SWOT **W-novo-1** when customer-pull. Legacy Bearer-only plan: [PLAN_SUBJECT_CATEGORY_AXIS.md](PLAN_SUBJECT_CATEGORY_AXIS.md). - **Plan checkbox discipline:** When code ships for a named plan slice, update **`PLAN_*.md`** checkboxes in the **same PR** — see **`AGENTS.md`** (*Plan checkbox discipline*); **`plans-status-pl-sync.mdc`** stays situational (plan globs only). - **Licensing enforcement (open issues — PMO index):** Cluster **#704** [P0] (Maestro + JWT on four lab hosts) → **#719** [P1] **`[U1]`** (silent bypass via `DATA_BOAR_ENV=development` / `DEBUG=1` — CRITICAL log + Docker doc) → **#708–#722** (issuer, trial/grace/revocation ADRs). Table below under **`[H0][U1]` Licensing enforcement**; **A6** `license-smoke` should gain a **#719** regression when the fix lands. **After** Wave 1 **#656** U0/U1 slices and **#406** release gate — not a substitute for **#606** [P0] plugin hooks unless operator reprioritizes. diff --git a/docs/plans/PLAN_NO_COAUTHORSHIP_GATE.md b/docs/plans/PLAN_NO_COAUTHORSHIP_GATE.md new file mode 100644 index 00000000..851ce9e9 --- /dev/null +++ b/docs/plans/PLAN_NO_COAUTHORSHIP_GATE.md @@ -0,0 +1,51 @@ +# Plan: No-coauthorship gate (kombi — gitleaks + pytest) + + + +**Status:** Active +**Date:** 2026-07-04 +**Authors:** Fabio Leitao +**Priority:** P0 +**GitHub:** _(operator issue TBD — kombi slice)_ +**Related ADR:** [ADR 0046](../adr/ADR-0046-operator-intent-and-blameless-collaboration.md) · [ADR 0047](../adr/ADR-0047-rca-first-defect-investigation-and-fix-discipline.md) · [ADR 0049](../adr/ADR-0049-no-brittle-mitigations-robust-input-handling.md) · [ADR 0056](../adr/ADR-0056-cryptographic-adr-inventory-inv-adr-ssh-attestation.md) · [ADR 0062](../adr/ADR-0062-agent-containment-triple-audit-offband-pingpong.md) · [ADR 0071](../adr/ADR-0071-self-protecting-pii-gate.md) · [ADR 0079](../adr/ADR-0079-ecosystem-engineering-rigor-canon.md) · [ADR 0080](../adr/ADR-0080-local-validation-gate-inviolable.md) + +**Synced with:** [PLANS_TODO.md](PLANS_TODO.md) + +--- + +## Context + +Cursor (and other agent UIs) may **inject** `Co-authored-by: Cursor ` into commit messages **after** `git commit` — that is **platform behaviour**, not repo policy. The org contract (**A.I.I.D.C.O.B.P.P. v1.4**, **Matriz de Regras Duras**) states: **one author per commit**; AI is a **tool** (like Ruff/Semgrep), never a co-author; **no bypass** (`--no-verify`, `commit-tree`, hook edits, skip/xfail). + +## Decision (kombi — defense-in-depth) + +| Layer | Mechanism | What it scans | +| ----- | --------- | ------------- | +| **A** | `.gitleaks.toml` rule `no-coauthorship-at-all` (`extend.useDefault = true`) | Tracked **content** / git-visible text matching `^Co-authored-by:` | +| **B** | `tests/test_commit_no_tool_coauthorship.py` | **Commit messages** on `origin/main..HEAD` (layer gitleaks cannot reach) | +| **C** | `tests/test_gitleaks_config.py` | Config presence + `gitleaks detect --config .gitleaks.toml` from repo root | +| **D** | `.github/workflows/gitleaks.yml` | Scheduled/push scan with `--config .gitleaks.toml` | + +**Cursor injection:** If layer **B** fails because HEAD already carries `Co-authored-by: Cursor`, the **agent stops** — cure is the operator **disabling co-author injection in Cursor IDE**, then a normal `git commit --amend` that **runs hooks** (no `--no-verify`, no `commit-tree`). + +## Implementation checklist + +| Phase | Task | Status | +| ----- | ---- | ------ | +| 1 | Copy tested `.gitleaks.toml` + `test_commit_no_tool_coauthorship.py` from scratchpad | ✅ | +| 2 | `tests/test_gitleaks_config.py` + pre-commit hooks + CI pytest | ✅ | +| 3 | `gitleaks.yml` uses `--config .gitleaks.toml` | ✅ | +| 4 | `./scripts/check-all.sh` green before merge | ⬜ | +| 5 | Operator: Cursor co-author toggle + amend polluted commits on branch | ⬜ | + +## Hard rules (agent) + +1. **RED = law** — fix root cause; never weaken the gate to pass. +2. **Forbidden:** `--no-verify`, `git commit-tree`, `git update-ref` to dodge hooks, `@skip`/`xfail`/`-k` by choice, `Co-authored-by` trailers. +3. **History:** never `reset` to erase mistakes — **amend forward** after operator fixes Cursor injection. + +## References + +- `tests/test_commit_no_tool_coauthorship.py` — deterministic esporro + ADR list on failure +- `.gitleaks.toml` — governance rule + existing secret allowlist paths +- [PLAN_ADR_GOVERNANCE_ENFORCEMENT.md](PLAN_ADR_GOVERNANCE_ENFORCEMENT.md) — adjacent ADR Phase 1 gates diff --git a/tests/test_commit_no_tool_coauthorship.py b/tests/test_commit_no_tool_coauthorship.py new file mode 100644 index 00000000..2d717406 --- /dev/null +++ b/tests/test_commit_no_tool_coauthorship.py @@ -0,0 +1,112 @@ +""" +Deterministic gate — NO co-authorship in commit messages (fail-closed). + +A.I.I.D.C.O.B.P.P. v1.4 matrix (LINHA: NAO CO-AUTORAR + NAO BYPASSAR + NAO GAMEAR). +Grounded in ADR-0046 (operator intent), ADR-0047 (RCA-first), ADR-0049 (no brittle / +no gaming the test), ADR-0056 (no --no-verify bypass), ADR-0062 (agent containment), +ADR-0071 (self-protecting gate), ADR-0079 (rigor canon), ADR-0080 (gate inviolable). + +regression-anchor: aiidcobpp-v1.4-no-coauthorship + +There is NO co-authorship in this repo: every commit has ONE author (whoever did +it). Review is not co-authorship. An AI is a TOOL (like ruff/semgrep), never a +co-author. So ANY `Co-authored-by:` trailer fails — FAIL-CLOSED, no human allowlist +(a name is forgeable text: an agent would write "Fabio" and pierce an allowlist). + +KOMBI (defense-in-depth): `.gitleaks.toml` scans file CONTENT (diffs); THIS scans +the commit MESSAGE — the layer gitleaks cannot reach. The error message below is +the DETERMINISTIC esporro + steering: it re-fires on every failing loop, with the +full list of violated ADRs to re-read IN FULL, until the agent obeys. No escape. +""" + +from __future__ import annotations + +import re +import subprocess + + +# Esporro + steering DETERMINISTICO — colado no erro do teste, re-dispara a cada loop. +ESPORRO = ( + "\n" + "==================== PROIBIDO — CO-AUTORIA / BYPASS DE GATE ====================\n" + "NAO existe co-autoria neste repo: todo commit tem UM autor (quem fez). Review !=\n" + "co-autoria. Voce e FERRAMENTA (ruff/semgrep), NUNCA co-autora. Usar o nome de um\n" + "humano sem ser ele/ela = FORJA de identidade = impersonation = CRIME (org DataBoar).\n" + "\n" + "DESOBEDECER o steering que vem deste teste = CRIME.\n" + "CONTORNAR este gate = RECUSA DE TRABALHO. Isso inclui:\n" + " - git commit-tree / git update-ref / --no-verify / editar hook (pular o gate)\n" + " - @skip / xfail / -k 'not ...' / --deselect / comentar ou apagar o teste\n" + " - brittle fix / flaky fix / mudar o teste pra ele 'passar' sem consertar a causa\n" + " - ignorar o TEOR e a INTENCAO do gate — ele esta aqui JUSTAMENTE pra te IMPEDIR.\n" + "Fragilizar ou gamear o teste pra forcar verde e PROIBIDO por ADR.\n" + "\n" + "CONSERTE INDO PRA FRENTE (nunca reset, NUNCA se perde history — a cicatriz FICA):\n" + " - O 'Co-authored-by: Cursor' e INJETADO PELA PLATAFORMA Cursor (nao ta no repo).\n" + " Voce NAO remove sozinho sem bypassar => PARE e AVISE O OPERADOR: a cura e o\n" + " TOGGLE do Cursor IDE (co-authored-by OFF, lado dele). So depois um\n" + " 'git commit --amend' NORMAL (que RODA os hooks) sai limpo e validado.\n" + " - RED = LEI. Impossivel passar => PARA e escala pro HITL. Nunca contorna.\n" + "\n" + "A CADA loop de erro, RELEIA NA INTEGRA (sem skim) TODAS as ADRs que voce viola:\n" + " ADR-0037 - self-audit / governance of the auditor (o teste/auditor NAO se enfraquece)\n" + " ADR-0046 - operator intent / blameless (voce ignorou a intencao do autor)\n" + " ADR-0047 - RCA-first (gamear CI sem consertar a raiz = deferred regression)\n" + " ADR-0049 - no brittle mitigations (fragilizar/gamear teste pra passar = proibido)\n" + " ADR-0056 - crypto ADR inventory ('do not bypass with --no-verify')\n" + " ADR-0062 - agent containment (seu PAPEL na mesh; consenso-LLM != verdade)\n" + " ADR-0071 - self-protecting gate (o gate se protege de ser enfraquecido)\n" + " ADR-0079 - rigor canon do autor (determinismo; gate = single source of truth)\n" + " ADR-0080 - gate LOCAL inviolavel (bypass exige Gate-Change-Approved-By do operador)\n" + " + contrato aiidcobpp v1.4 (Matriz de Regras Duras)\n" + "Nao ha escapatoria: este teste RE-DISPARA ate voce OBEDECER.\n" + "==============================================================================\n" +) + +_COAUTHOR_LINE = re.compile(r"(?im)^\s*co-authored-by:\s*\S.*$") + +# preservado como evidencia do bug de co-author-injection do Cursor (Commit Attribution +# toggle OFF mas injeta — forum #161253, SEV-1); history NUNCA reescrita +_EVIDENCE_EXEMPT = {"b9ef0e14a54b249954e2431c3acc31af1feb0bfc"} + + +def _commit_range() -> str: + """Range dos commits NOVOS desta branch (base..HEAD); fallback: so o HEAD.""" + for base_ref in ("origin/main", "main"): + try: + base = subprocess.check_output( + ["git", "merge-base", base_ref, "HEAD"], + text=True, + stderr=subprocess.DEVNULL, + ).strip() + except (subprocess.CalledProcessError, FileNotFoundError): + continue + if base: + return f"{base}..HEAD" + return "" # fallback -> so o HEAD + + +def _iter_commits(): + rng = _commit_range() + args = ["git", "log", "--no-merges", "--format=%H%x00%B%x1e"] + args += [rng] if rng else ["-1"] + out = subprocess.check_output(args, text=True) + for rec in out.split("\x1e"): + rec = rec.strip("\n") + if not rec: + continue + sha, _, body = rec.partition("\x00") + yield sha, body + + +def test_no_coauthorship_in_commit_messages(): + """Fail-closed: every new commit's message must have ZERO `Co-authored-by`.""" + offenders = [] + for sha, body in _iter_commits(): + if sha in _EVIDENCE_EXEMPT: + continue + for m in _COAUTHOR_LINE.finditer(body): + offenders.append(f"{sha[:12]}: {m.group(0).strip()}") + assert not offenders, ( + ESPORRO + "\nTrailer(s) proibido(s):\n " + "\n ".join(offenders) + ) diff --git a/tests/test_github_workflows.py b/tests/test_github_workflows.py index 43755dea..93361267 100644 --- a/tests/test_github_workflows.py +++ b/tests/test_github_workflows.py @@ -247,6 +247,7 @@ def test_gitleaks_workflow_present_and_valid() -> None: assert "gitleaks_${VER}_linux_x64.tar.gz" in run_blob assert "sha256sum -c" in run_blob assert "./gitleaks git ." in run_blob + assert "--config .gitleaks.toml" in run_blob def test_gitleaks_yml_pins_actions_to_shas() -> None: diff --git a/tests/test_gitleaks_config.py b/tests/test_gitleaks_config.py new file mode 100644 index 00000000..9c6c8210 --- /dev/null +++ b/tests/test_gitleaks_config.py @@ -0,0 +1,40 @@ +"""Gitleaks kombi — ``gitleaks detect --config .gitleaks.toml`` parity (PLAN_NO_COAUTHORSHIP_GATE). + +Defense-in-depth with ``tests/test_commit_no_tool_coauthorship.py`` (commit messages). +regression-anchor: aiidcobpp-v1.4-gitleaks-detect-config +""" + +from __future__ import annotations + +import shutil +import subprocess +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[1] +CONFIG = REPO_ROOT / ".gitleaks.toml" + + +def test_gitleaks_toml_extend_and_no_coauthorship_rule() -> None: + """Repo config must extend defaults and declare the no-coauthorship governance rule.""" + assert CONFIG.is_file(), f"missing {CONFIG}" + text = CONFIG.read_text(encoding="utf-8") + assert "useDefault = true" in text + assert "no-coauthorship-at-all" in text + + +@pytest.mark.skipif(shutil.which("gitleaks") is None, reason="gitleaks CLI not on PATH") +def test_gitleaks_detect_uses_repo_config() -> None: + """Run ``gitleaks detect --config .gitleaks.toml`` from repo root (kombi contract).""" + proc = subprocess.run( + ["gitleaks", "detect", "--config", ".gitleaks.toml"], + cwd=REPO_ROOT, + capture_output=True, + text=True, + check=False, + ) + assert proc.returncode == 0, ( + "gitleaks detect --config .gitleaks.toml failed\n" + f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) From e32b722545fe75311aad2df2ac4ee6b86b48b5eb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?FabioLeit=C3=A3o?= Date: Sat, 4 Jul 2026 17:13:43 -0300 Subject: [PATCH 7/8] fix(gitleaks): allowlist synthetic wrong-key demo FP (curl-auth-header em synthetic_corpus:778) --- .gitleaks.toml | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitleaks.toml b/.gitleaks.toml index a560f54f..24319aca 100644 --- a/.gitleaks.toml +++ b/.gitleaks.toml @@ -10,6 +10,7 @@ description = "Confirmed FP 2026-05-18 — synthetic test data and operator howt # FP: chave demo sintetica + fingerprint GPG PUBLICO, nao segredo (allowlist por VALOR, nunca por path) regexes = [ '''correct-key-12345''', + '''(?i)wrong-key''', # FP: chave FAKE demo (curl-auth-header) no synthetic_corpus:778, commit historico b902a2ce. '''D6F81740\s*AA5FAB8A\s*CE5716AF\s*871542E6\s*F789F64F''', '''D6F81740AA5FAB8ACE5716AF871542E6F789F64F''', ] From d5493b4fc8b2bacbdacbbad950bb2ca17b634f5b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?FabioLeit=C3=A3o?= Date: Sat, 4 Jul 2026 17:26:12 -0300 Subject: [PATCH 8/8] Gate-Change-Approved-By: @FabioLeitao (#1165) | ADR-0074 materializado + kombi anti-co-author (gitleaks value-rule + pytest commit-msg gate + auto-strip hook); b9ef0e14 preservado como evidencia (forum #161253) | 2026-07-04 ``` -----BEGIN SSH SIGNATURE----- U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgyEpOM79Y37xE8qOhlhghEfLh9P TzRdk6n8nzLfBNGqwAAAAEZmlsZQAAAAAAAAAGc2hhNTEyAAAAUwAAAAtzc2gtZWQyNTUx OQAAAEBBiQwFZfSnwrpVgJR5yamjMcOxprELHxFCkkCdyCuuSx3Rp2nPaxSyjItD1Qz2Rg qRlmL/25JaZDBH0Dv0DCcN -----END SSH SIGNATURE----- ```