From 961487f63383a05505472f94e35a44a10b49b76b Mon Sep 17 00:00:00 2001 From: PetrefiedThunder <1@christophersellers.com> Date: Sat, 31 Oct 2026 05:01:07 +0000 Subject: [PATCH 1/2] fix(epcis): keep a truncated export truncated when the fallback adds nothing, and close the #3714 review leftovers Ten CodeRabbit threads were left open when #3714 (EPCIS export cap, FDA package evidence context) merged. Each was verified against main first. services/ingestion/app/epcis/router.py (GET /api/v1/epcis/export) - A truncated database page that the product_id filter emptied entered the in-memory fallback branch and had its regengine:limit / regengine:truncated reset to null / false, so an incomplete export read as complete. The reset now happens only when the fallback actually supplies events; when it supplies nothing the database page's facts stand. - The fallback returned store iteration order while advertising "event_timestamp desc". Fallback records are now sorted by their stored event time, newest first, undated records last. services/shared/rules/versions.py - Each snapshot read runs on its own SAVEPOINT (begin_nested) when the session supports it, so a failed catalog read cannot leave the caller's transaction in InFailedSqlTransaction and poison the evaluated read and every later query on that session. Fake sessions without begin_nested run the statement directly, as before. services/shared/request_workflow/assembly.py - package_contents carries rule_versions_status (the snapshot's catalog_status, "unavailable" when absent) beside rule_versions, so an empty catalog that was read stays distinguishable from one that could not be read; the status is hashed with the rest of the package. Docs - docs/PRODUCT_DESCRIPTION.md names the real cutoff field (regengine:truncated inside regengine:metadata). - docs/runbooks/FDA_RECALL_RESPONSE.md tells reviewers to check catalog_status / evaluated_status before treating rule-version evidence as complete, documents the 250-entry cap behind VALIDATION_ERRORS and the known_gaps fields that say when it bit, and qualifies the package integrity sentence: the verifiers prove internal consistency against manifest.json, not the issuer; compare X-Package-Hash to tie a ZIP to the original export. Tests - epcis router: truncated page emptied by the filter stays truncated with the limit; fallback events come back newest first with undated last. - rule_versions: a failing catalog read is rolled back on its own savepoint and the evaluated read still runs on a second one. - request workflow: rule_versions_status is "ok" for an empty readable catalog and "unavailable" when the catalog read fails. - recall package: the snapshot helper receives (session, tenant, events) and its return value is forwarded as rule_versions. - proof verifier and admin verifier: a same-size edit inside evidence_context is rejected by the hash check, not the size check. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GyhDYG3ddQvVKyu2qtYfCj --- docs/PRODUCT_DESCRIPTION.md | 2 +- docs/runbooks/FDA_RECALL_RESPONSE.md | 13 ++++-- .../admin/tests/test_verify_audit_pack.py | 16 +++++-- services/ingestion/app/epcis/router.py | 30 +++++++++++-- .../ingestion/tests/test_epcis_router_1342.py | 45 +++++++++++++++++++ .../tests/test_fda_export_recall_1342.py | 16 ++++++- .../tests/test_proof_verify_router.py | 18 ++++++-- services/shared/request_workflow/assembly.py | 9 +++- services/shared/rules/versions.py | 22 ++++++++- .../tests/test_rule_versions_snapshot.py | 43 ++++++++++++++++++ tests/test_request_workflow_unit.py | 26 +++++++++-- 11 files changed, 217 insertions(+), 23 deletions(-) diff --git a/docs/PRODUCT_DESCRIPTION.md b/docs/PRODUCT_DESCRIPTION.md index 2e08c4933..be8c28182 100644 --- a/docs/PRODUCT_DESCRIPTION.md +++ b/docs/PRODUCT_DESCRIPTION.md @@ -31,7 +31,7 @@ A browser verifier and a downloadable Python bundle verifier let reviewers inspe | Readiness assessments and recall reporting | Release-enabled: [readiness](../services/ingestion/app/readiness_router.py), [recall reports](../services/ingestion/app/recall_report.py). | | Bundle verifier and recall-drill endpoint | [Browser verifier](https://regengine.co/verify), [verifier source](../frontend/public/sdk/verify_chain.py). The recall-drill endpoint currently returns no drill data. | | Supplier manifest remediation and correction links | Implemented in [remediation](../services/ingestion/app/remediation/) and the [buyer dashboard](../frontend/src/app/remediation/page.tsx); excluded from the production router allowlist. Notification delivery also requires recipient/provider configuration. | -| EPCIS JSON-LD export and broader recall simulations | Implemented in [EPCIS export](../services/ingestion/app/epcis_export.py) and [simulations](../services/ingestion/app/recall_simulations.py); these routers are experimental and excluded from the production allowlist. A single EPCIS export returns at most 2,000 events, newest first, and its `metadata.truncated` flag says when it was cut off. Public sample EPCIS downloads are a separate proof surface. | +| EPCIS JSON-LD export and broader recall simulations | Implemented in [EPCIS export](../services/ingestion/app/epcis_export.py) and [simulations](../services/ingestion/app/recall_simulations.py); these routers are experimental and excluded from the production allowlist. A single EPCIS export returns at most 2,000 events, newest first, and the `regengine:truncated` flag inside its `regengine:metadata` block says when it was cut off. Public sample EPCIS downloads are a separate proof surface. | | Inbound email CSV ingestion | [Postmark integration](../services/ingestion/app/email_ingest/); excluded from the production allowlist, with no self-service provisioning UI. See [rollout notes](engineering/EMAIL_INGESTION.md). | | Passwordless login and administration | [Email OTP/passkey auth](../services/admin/app/auth/challenges_router.py), tenant/user/API-key management, and session controls. Email delivery requires provider and identity provisioning. | | Subscription access and usage metering | [Subscription gate](../services/ingestion/app/subscription_gate.py), [metering design and limitations](../DESIGN_metering-billing.md). Usage reporting to Stripe requires an operator call; there is no periodic reporting job. | diff --git a/docs/runbooks/FDA_RECALL_RESPONSE.md b/docs/runbooks/FDA_RECALL_RESPONSE.md index 1280b2f33..d1361bf6c 100644 --- a/docs/runbooks/FDA_RECALL_RESPONSE.md +++ b/docs/runbooks/FDA_RECALL_RESPONSE.md @@ -165,14 +165,19 @@ The ZIP bundle (`_build_fda_package()` in `services/ingestion/app/fda_export_ser |------|----------| | `fda_spreadsheet__.csv` | FDA-oriented spreadsheet, one row per event (`FDA_COLUMNS`) | | `chain_verification__.json` | Hash chain verification result, row hash coverage, completeness summary, redaction flag | -| `evidence_context__.json` | Rule catalog and the rule versions that evaluated these events, enforcement mode at export time, and a known-gaps section (KDE coverage, unchecked event types, missing hashes, whether the coverage gate was bypassed with `allow_incomplete`) | -| `VALIDATION_ERRORS__.log` | Only when events are missing required KDEs: one line per affected event | +| `evidence_context__.json` | Rule catalog and the rule versions that evaluated these events, enforcement mode at export time, and a known-gaps section (KDE coverage, unchecked event types, missing hashes, whether the coverage gate was bypassed with `allow_incomplete`). Before treating the rule-version evidence as complete, check `rule_versions.catalog_status` and `rule_versions.evaluated_status` (also mirrored as `known_gaps.rules_catalog_status` / `known_gaps.rules_evaluated_status`): `ok` means the read succeeded, `unavailable` means the export could not read that table and the section is empty for that reason, `not_requested` means no evaluations were looked up. | +| `VALIDATION_ERRORS__.log` | Only when events are missing required KDEs: one line per affected event, built from the completeness summary's `missing_required_events` list, which is capped at 250 entries. When more events are affected than the log lists, `known_gaps.missing_required_events_truncated` in the evidence context is `true` and `known_gaps.missing_required_events_listed` gives the count written; `events_with_missing_required_fields` is the full count | | `manifest.json` | Package metadata, per-file sizes and SHA-256 digests, Merkle root over the hashed members, pointer to the evidence context file. The manifest is not one of its own Merkle leaves. | | `README.txt` | Human-readable description of the package and how to verify it | The PDF format is a separate response (`format=pdf`), not a member of the ZIP. Every member except -`manifest.json` is hashed and listed in `manifest.files`, so a change to any of them, the known-gaps -section included, fails both the browser verifier and `services/admin/scripts/verify_audit_pack.py`. +`manifest.json` is hashed and listed in `manifest.files`. The browser verifier and +`services/admin/scripts/verify_audit_pack.py` compare each member against the per-file hashes and the +Merkle root in `manifest.json`, so a change to any member, the known-gaps section included, without a +matching manifest update fails both. These checks establish that the package is internally consistent; +they do not authenticate the issuer, so a package rebuilt with a regenerated manifest passes them. To +tie a ZIP to the original export, compare it against the `X-Package-Hash` value captured when it was +downloaded (section 3.4). ### 3.3 FDA CSV Column Specification diff --git a/services/admin/tests/test_verify_audit_pack.py b/services/admin/tests/test_verify_audit_pack.py index 7815d1ef3..2720ad6a0 100644 --- a/services/admin/tests/test_verify_audit_pack.py +++ b/services/admin/tests/test_verify_audit_pack.py @@ -521,12 +521,22 @@ def test_pack_declares_evidence_context_member(tmp_path): assert verifier.verify_pack(str(pack_path)) is True -def test_tampered_evidence_context_fails(tmp_path): +def test_tampered_evidence_context_fails(tmp_path, capsys): + # Flip one character inside a string value so the member keeps its + # serialized size: the rejection must come from the hash comparison, + # not from the cheaper size check that runs first. pack_path, _ = _make_pack(tmp_path) members = _read_members(pack_path) name = _evidence_name(members) context = json.loads(members[name]) - context["known_gaps"]["coverage_gate_bypassed"] = not context["known_gaps"]["coverage_gate_bypassed"] - members[name] = json.dumps(context, indent=2, sort_keys=True).encode("utf-8") + status = context["known_gaps"]["verification_status"] + assert isinstance(status, str) and status + context["known_gaps"]["verification_status"] = status[:-1] + ("X" if status[-1] != "X" else "Y") + mutated = json.dumps(context, indent=2, sort_keys=True).encode("utf-8") + assert len(mutated) == len(members[name]), "the mutation must keep the serialized size" + members[name] = mutated tampered = _write_members(tmp_path, "tampered-evidence.zip", members) assert verifier.verify_pack(str(tampered)) is False + out = capsys.readouterr().out + assert f"hash mismatch for {name}" in out + assert "size mismatch" not in out diff --git a/services/ingestion/app/epcis/router.py b/services/ingestion/app/epcis/router.py index b21306a36..014662122 100644 --- a/services/ingestion/app/epcis/router.py +++ b/services/ingestion/app/epcis/router.py @@ -159,6 +159,20 @@ def _parse_stored_event_time(value: object) -> datetime | None: return parsed +_EPOCH_UTC = datetime(1970, 1, 1, tzinfo=timezone.utc) + + +def _fallback_export_sort_key(entry: dict) -> tuple[bool, datetime]: + """Sort key for fallback-store records: event time, unparseable last. + + Mirrors the DB export's ``ORDER BY event_timestamp DESC`` when used + with ``reverse=True``: dated records come first, newest to oldest, + then the records whose event time is missing or unparseable. + """ + event_dt = _parse_stored_event_time((entry.get("normalized_cte") or {}).get("event_time")) + return (event_dt is not None, event_dt or _EPOCH_UTC) + + @router.post("/events", status_code=201, summary="Ingest EPCIS 2.0 event") async def ingest_epcis_event( event: dict, @@ -367,9 +381,19 @@ async def export_epcis_events( if product_id and normalized.get("product_id") != product_id: continue filtered_records.append(entry) - events = [entry["epcis_document"] for entry in filtered_records] - limit = None - truncated = False + if filtered_records: + # The DB path orders by event_timestamp DESC; the fallback + # store iterates in insertion order, so sort it the same way + # to keep ``regengine:ordering`` truthful. Records without a + # parseable event time go last. + filtered_records.sort(key=_fallback_export_sort_key, reverse=True) + events = [entry["epcis_document"] for entry in filtered_records] + # Only the fallback's own events are uncapped. When it supplies + # nothing, the DB page's limit/truncated still describe what + # the caller got: a truncated page that the product filter + # emptied must not read as a complete export. + limit = None + truncated = False filters = { f"regengine:{key}": value diff --git a/services/ingestion/tests/test_epcis_router_1342.py b/services/ingestion/tests/test_epcis_router_1342.py index a4bdd31eb..639318b9e 100644 --- a/services/ingestion/tests/test_epcis_router_1342.py +++ b/services/ingestion/tests/test_epcis_router_1342.py @@ -339,6 +339,51 @@ def test_fallback_path_reports_no_cap(self, client: TestClient, monkeypatch: pyt assert meta["regengine:limit"] is None assert meta["regengine:truncated"] is False + def test_truncated_db_page_emptied_by_filter_stays_truncated( + self, client: TestClient, monkeypatch: pytest.MonkeyPatch + ) -> None: + """A truncated DB page that the product filter emptied is still an incomplete export. + + The fallback branch runs because ``events`` is empty, but when the + fallback store has nothing to add it must not reset the DB page's + limit/truncated facts to "uncapped, complete". + """ + monkeypatch.setattr( + router_mod, + "_list_events_from_db_detailed", + lambda *a, **kw: _page([], truncated=True), + ) + monkeypatch.setattr(router_mod, "_allow_in_memory_fallback", lambda: True) + monkeypatch.setattr(router_mod, "_fallback_store_for", lambda t: {}) + + resp = client.get("/api/v1/epcis/export?product_id=prod-rare") + assert resp.status_code == 200 + meta = resp.json()["regengine:metadata"] + assert meta["regengine:event_count"] == 0 + assert meta["regengine:truncated"] is True + assert meta["regengine:limit"] == persistence_mod.EXPORT_EVENT_LIMIT + + def test_fallback_events_are_newest_first(self, client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None: + """The fallback honours the advertised ``event_timestamp desc`` ordering. + + Records are inserted oldest-first (store iteration order); the + response must still be newest-first, with undated records last. + """ + monkeypatch.setattr(router_mod, "_list_events_from_db_detailed", _page) + monkeypatch.setattr(router_mod, "_allow_in_memory_fallback", lambda: True) + + early, _, late = _ordered_export_times() + fake_store = { + "undated": {"normalized_cte": {}, "epcis_document": {"eventID": "undated"}}, + "older": {"normalized_cte": {"event_time": early}, "epcis_document": {"eventID": "older"}}, + "newer": {"normalized_cte": {"event_time": late}, "epcis_document": {"eventID": "newer"}}, + } + monkeypatch.setattr(router_mod, "_fallback_store_for", lambda t: fake_store) + + body = client.get("/api/v1/epcis/export").json() + assert [e["eventID"] for e in body["epcisBody"]["eventList"]] == ["newer", "older", "undated"] + assert body["regengine:metadata"]["regengine:ordering"] == "event_timestamp desc" + def test_db_error_without_fallback_returns_503(self, client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None: """Line 225: DB error + fallback disabled → 503.""" diff --git a/services/ingestion/tests/test_fda_export_recall_1342.py b/services/ingestion/tests/test_fda_export_recall_1342.py index 39feff4be..090621f62 100644 --- a/services/ingestion/tests/test_fda_export_recall_1342.py +++ b/services/ingestion/tests/test_fda_export_recall_1342.py @@ -417,11 +417,19 @@ def test_non_package_format_still_goes_csv_path(self, monkeypatch): class TestPackageExport: def test_package_happy_path(self, monkeypatch): events = [{"id": "e1"}, {"id": "e2"}, {"id": "e3"}] - _, _, cap = _install_stubs( + sess, _, cap = _install_stubs( monkeypatch, rows=[{"id": 1}], events=events, ) + snapshot = {"catalog_status": "ok", "catalog": {"rule-a": {"version": 2}}, "evaluated": [], "evaluated_status": "ok"} + seen: dict = {} + + def _snapshot(db_session, tenant_id, evts): + seen["args"] = (db_session, tenant_id, evts) + return snapshot + + monkeypatch.setattr(r, "_rule_versions_for_events", _snapshot) resp = _call(format="package", include_pii=False) assert resp == "PKG-RESP" pkg_k = cap["package_response_kwargs"] @@ -433,8 +441,12 @@ def test_package_happy_path(self, monkeypatch): assert pkg_k["extra_headers"]["X-KDE-Coverage"] == "0.95" assert pkg_k["include_pii"] is False # S7: gate outcome and rule-version snapshot reach the package builder. + # The snapshot is taken on the export's own session, for this + # tenant, over exactly the exported events, and is forwarded as-is. assert pkg_k["coverage_gate_bypassed"] is False - assert "catalog_status" in pkg_k["rule_versions"] + assert seen["args"] == (sess, "t-1", events) + assert seen["args"][0] is sess + assert pkg_k["rule_versions"] is snapshot # Did NOT go through the CSV response path assert cap["csv_response_kwargs"] is None diff --git a/services/ingestion/tests/test_proof_verify_router.py b/services/ingestion/tests/test_proof_verify_router.py index 6cf19681d..3b8ee8460 100644 --- a/services/ingestion/tests/test_proof_verify_router.py +++ b/services/ingestion/tests/test_proof_verify_router.py @@ -971,11 +971,17 @@ def test_member_present_and_package_verifies(self, client: TestClient, tmp_path: path.write_bytes(package_bytes) assert sdk_verify_bundle(str(path)) == 0 - def test_tampered_known_gaps_fail_both_verifiers(self, client: TestClient, tmp_path: Path): + def test_tampered_known_gaps_fail_both_verifiers(self, client: TestClient, tmp_path: Path, capsys): + """A same-size edit inside the member is rejected by the hash check, not the size check.""" + def _flip(data: bytes) -> bytes: context = json.loads(data) - context["known_gaps"]["coverage_gate_bypassed"] = True - return json.dumps(context, indent=2, sort_keys=True).encode("utf-8") + status = context["known_gaps"]["verification_status"] + assert isinstance(status, str) and status + context["known_gaps"]["verification_status"] = status[:-1] + ("X" if status[-1] != "X" else "Y") + mutated = json.dumps(context, indent=2, sort_keys=True).encode("utf-8") + assert len(mutated) == len(data), "the mutation must keep the serialized size" + return mutated tampered = self._rewrite(self._package(), "evidence_context_", _flip) response = client.post("/api/v1/proof/verify", files={"bundle": ("t.zip", tampered, "application/zip")}) @@ -983,8 +989,14 @@ def _flip(data: bytes) -> bytes: assert response.json()["verified"] is False checks = {check["name"]: check for check in response.json()["checks"]} assert checks["file_hashes_match_manifest"]["passed"] is False + detail = checks["file_hashes_match_manifest"]["detail"] + assert "hash_mismatched=['evidence_context_" in detail + assert "size_mismatched=[]" in detail sdk_verify_bundle = runpy.run_path(str(REPO_ROOT / "frontend/public/sdk/verify_chain.py"))["verify_bundle"] path = tmp_path / "t.zip" path.write_bytes(tampered) assert sdk_verify_bundle(str(path)) == 1 + output = capsys.readouterr().out + assert "sha256 mismatch" in output + assert "!= declared" not in output, "the SDK verifier must reject on the hash, not the size" diff --git a/services/shared/request_workflow/assembly.py b/services/shared/request_workflow/assembly.py index 1cc033aac..2b5aecbd0 100644 --- a/services/shared/request_workflow/assembly.py +++ b/services/shared/request_workflow/assembly.py @@ -157,10 +157,14 @@ def assemble_response_package( # package so both evidence artefacts describe rules the same way; # the stored shape stays the catalog map this package always had. # The per-event evaluations are already gathered above, so only the - # catalog is requested here (one query, as before). + # catalog is requested here (one query, as before). The catalog + # status is stored beside it so an empty catalog ("ok", no rules) + # stays distinguishable from one that could not be read. from shared.rules.versions import snapshot_rule_versions - rule_versions = snapshot_rule_versions(self.db).get("catalog", {}) + rule_versions_snapshot = snapshot_rule_versions(self.db) + rule_versions = rule_versions_snapshot.get("catalog", {}) + rule_versions_status = rule_versions_snapshot.get("catalog_status", "unavailable") # Snapshot identity state at assembly time (single query with LEFT JOIN # to avoid N+1 — previously fired one alias query per entity). @@ -230,6 +234,7 @@ def assemble_response_package( "scoped_event_ids": sorted_event_ids, "rule_evaluation_ids": eval_ids, "rule_versions": rule_versions, + "rule_versions_status": rule_versions_status, "identity_state": identity_state, "exception_state": { str(e.get("exception_id", "")): { diff --git a/services/shared/rules/versions.py b/services/shared/rules/versions.py index dbc82929b..e5039bd19 100644 --- a/services/shared/rules/versions.py +++ b/services/shared/rules/versions.py @@ -42,6 +42,22 @@ ) +def _execute_isolated(db: Any, statement: Any, params: Optional[Dict[str, Any]] = None) -> Any: + """Run one read on its own SAVEPOINT when the session supports it. + + On PostgreSQL a failed statement aborts the enclosing transaction + (``InFailedSqlTransaction``), so without a savepoint a missing + ``fsma.rule_definitions`` would poison the evaluated read below and + every later query the caller issues on the same session. Sessions + without ``begin_nested`` (unit-test fakes) run the statement directly. + """ + begin_nested = getattr(db, "begin_nested", None) + if callable(begin_nested): + with begin_nested(): + return db.execute(statement, params) if params is not None else db.execute(statement) + return db.execute(statement, params) if params is not None else db.execute(statement) + + def snapshot_rule_versions( db: Any, tenant_id: Optional[str] = None, @@ -73,7 +89,7 @@ def snapshot_rule_versions( } try: - rows = db.execute(_CATALOG_SQL).mappings().fetchall() + rows = _execute_isolated(db, _CATALOG_SQL).mappings().fetchall() for row in rows: snapshot["catalog"][str(row["rule_id"])] = { "version": row.get("rule_version"), @@ -88,7 +104,9 @@ def snapshot_rule_versions( ids = [str(e) for e in (event_ids or []) if e] if tenant_id and ids: try: - rows = db.execute(_EVALUATED_SQL, {"tid": str(tenant_id), "event_ids": ids}).mappings().fetchall() + rows = ( + _execute_isolated(db, _EVALUATED_SQL, {"tid": str(tenant_id), "event_ids": ids}).mappings().fetchall() + ) snapshot["evaluated"] = [ { "rule_id": str(row["rule_id"]), diff --git a/services/shared/tests/test_rule_versions_snapshot.py b/services/shared/tests/test_rule_versions_snapshot.py index 2f47b7495..59c9708e6 100644 --- a/services/shared/tests/test_rule_versions_snapshot.py +++ b/services/shared/tests/test_rule_versions_snapshot.py @@ -40,6 +40,28 @@ def execute(self, stmt, params=None): return _Result(nxt) +class _SavepointSession(_FakeSession): + """Fake session with ``begin_nested``; records every savepoint enter/exit.""" + + def __init__(self, *results: Any): + super().__init__(*results) + self.savepoints: list[str] = [] + + def begin_nested(self): + session = self + + class _Savepoint: + def __enter__(self_inner): + session.savepoints.append("enter") + return self_inner + + def __exit__(self_inner, exc_type, exc, tb): + session.savepoints.append("exit:rollback" if exc_type else "exit:commit") + return False + + return _Savepoint() + + _CATALOG_ROWS = [ {"rule_id": "rule-a", "rule_version": 3, "title": "A", "severity": "critical", "category": "kde"}, {"rule_id": "rule-b", "rule_version": 1, "title": "B", "severity": "warning", "category": "cte"}, @@ -101,6 +123,27 @@ def test_catalog_failure_degrades_not_raises() -> None: assert snap["evaluated_status"] == "ok" +def test_each_read_runs_in_its_own_savepoint() -> None: + """A failed catalog read is rolled back on its own savepoint. + + On PostgreSQL a failed statement aborts the enclosing transaction, so + without the savepoint the evaluated read (and every later query on the + caller's session) would fail too. The evaluated read must still run, + in a second savepoint, and succeed independently. + """ + sess = _SavepointSession( + RuntimeError("relation does not exist"), + [{"rule_id": "r", "rule_version": 1, "evaluation_count": 1}], + ) + snap = mod.snapshot_rule_versions(sess, tenant_id="t", event_ids=["e"]) + assert snap["catalog_status"] == "unavailable" + assert snap["catalog"] == {} + assert snap["evaluated_status"] == "ok" + assert snap["evaluated"] == [{"rule_id": "r", "rule_version": 1, "evaluation_count": 1}] + assert len(sess.calls) == 2 + assert sess.savepoints == ["enter", "exit:rollback", "enter", "exit:commit"] + + def test_evaluated_failure_degrades_not_raises() -> None: sess = _FakeSession(_CATALOG_ROWS, RuntimeError("timeout")) snap = mod.snapshot_rule_versions(sess, tenant_id="t", event_ids=["e"]) diff --git a/tests/test_request_workflow_unit.py b/tests/test_request_workflow_unit.py index cb3bd563d..3cde5f515 100644 --- a/tests/test_request_workflow_unit.py +++ b/tests/test_request_workflow_unit.py @@ -584,7 +584,7 @@ class TestAssembledPackageHashLinkage: stored hash equals the helper's digest of the stored contents. """ - def _fake_db(self, captured): + def _fake_db(self, captured, *, catalog_raises=None): import json as _json from sqlalchemy.sql.elements import TextClause @@ -617,6 +617,8 @@ def _execute(statement, params=None): sql = statement.text if isinstance(statement, TextClause) else str(statement) if "MAX(version_number)" in sql: return _Result(scalar_value=0) + if catalog_raises is not None and "FROM fsma.rule_definitions" in sql: + raise catalog_raises if "INSERT INTO fsma.response_packages" in sql: captured["contents_json"] = params["contents"] captured["hash"] = params["hash"] @@ -628,11 +630,11 @@ def _execute(statement, params=None): db.execute.side_effect = _execute return db - def _assemble(self): + def _assemble(self, *, catalog_raises=None): from services.shared.request_workflow import RequestWorkflow captured = {} - db = self._fake_db(captured) + db = self._fake_db(captured, catalog_raises=catalog_raises) wf = RequestWorkflow(db) wf._get_case = MagicMock(return_value=_mock_case_row(status="assembling")) wf._get_scope_event_ids = MagicMock(return_value=[]) @@ -683,6 +685,24 @@ def test_insert_receives_helper_digest(self): check = {k: v for k, v in contents.items() if k != "package_hash"} assert captured["hash"] == compute_manifest_hash(check) + def test_rule_versions_status_stored_beside_catalog(self): + """An empty catalog that was read is 'ok'; the status is hashed with the rest.""" + import json as _json + + _, captured = self._assemble() + contents = _json.loads(captured["contents_json"]) + assert contents["rule_versions"] == {} + assert contents["rule_versions_status"] == "ok" + + def test_unreadable_rule_catalog_is_marked_unavailable(self): + """A catalog read failure leaves rule_versions empty but says why.""" + import json as _json + + _, captured = self._assemble(catalog_raises=RuntimeError("relation does not exist")) + contents = _json.loads(captured["contents_json"]) + assert contents["rule_versions"] == {} + assert contents["rule_versions_status"] == "unavailable" + # --------------------------------------------------------------------------- # 9. Submit Package — validation From ea0ce2f21f67d9bb6507a46ab07eabeb417d7e15 Mon Sep 17 00:00:00 2001 From: PetrefiedThunder <1@christophersellers.com> Date: Sat, 31 Oct 2026 13:03:11 +0000 Subject: [PATCH 2/2] fix(deps): port #3773 pypdf, pyasn1 and follow-redirects bumps The Security Audit, Production Readiness Gate and per-service pip-audit jobs went red on this PR after advisories for pypdf 6.21.2, pyasn1 0.6.4 and follow-redirects 1.16.0-1.16.1 were published. Main is red on the same jobs. This ports #3773 byte for byte (pypdf 6.21.4, pyasn1 0.6.5, npm overrides follow-redirects >=1.16.2 and minimatch >=11.0.1). It no-ops once main carries #3773. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01GyhDYG3ddQvVKyu2qtYfCj --- frontend/package-lock.json | 21 ++++++++++++++------- frontend/package.json | 3 ++- packages/factory-local/requirements.lock | 6 +++--- packages/factory-local/requirements.txt | 2 +- requirements.in | 2 +- requirements.lock | 12 ++++++------ 6 files changed, 27 insertions(+), 19 deletions(-) diff --git a/frontend/package-lock.json b/frontend/package-lock.json index c23efbbc5..a24cc87ab 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1405,6 +1405,12 @@ "url": "https://opencollective.com/libvips" } }, + "node_modules/@isaacs/nonrecursive-glob-match": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/@isaacs/nonrecursive-glob-match/-/nonrecursive-glob-match-1.0.3.tgz", + "integrity": "sha512-SI7kyKHq4XXDRozw+r9GlRZkia1ZkFrpLW98r2IC2Bpr8YkFGE3fvjzmxCG0x78A++69HmDNrDvSMYBZmHGpKA==", + "license": "BlueOak-1.0.0" + }, "node_modules/@jest/diff-sequences": { "version": "30.0.1", "resolved": "https://registry.npmjs.org/@jest/diff-sequences/-/diff-sequences-30.0.1.tgz", @@ -7866,9 +7872,9 @@ "license": "ISC" }, "node_modules/follow-redirects": { - "version": "1.16.1", - "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.1.tgz", - "integrity": "sha512-FNvFGzoMLWmE6Yj9spb/zjd7yiNCHiAW9/Tg9CXrQ8wuu32HtlJOwWO11OJafl5FfY3DxTdQ0vj42zU1kvv5jg==", + "version": "1.16.2", + "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.2.tgz", + "integrity": "sha512-/kVP2U8z596Ia0gwFrwyIeo74ta5aVmSeU1FfAUMAmcEP5EP7zydmag3SLiN64NaRKef3lxsq7i/g7ltJQDGow==", "funding": [ { "type": "individual", @@ -9823,12 +9829,13 @@ } }, "node_modules/minimatch": { - "version": "10.2.4", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.4.tgz", - "integrity": "sha512-oRjTw/97aTBN0RHbYCdtF1MQfvusSIBQM0IZEgzl6426+8jSC0nF1a/GmnVLpfB9yyr6g6FTqWqiZVbxrtaCIg==", + "version": "11.0.3", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-11.0.3.tgz", + "integrity": "sha512-iny/40PEGkNJQ5/Pv6z3nm6FbRAby32GeZRjCUJk2WnNDbgPSTcdI+DDvLqsgBdD4cD5sMtMheqPPnehCCa8Nw==", "license": "BlueOak-1.0.0", "dependencies": { - "brace-expansion": "^5.0.2" + "@isaacs/nonrecursive-glob-match": "^1.0.3", + "brace-expansion": "^5.0.12" }, "engines": { "node": "18 || 20 || >=22" diff --git a/frontend/package.json b/frontend/package.json index 7fbbfb069..f754526bd 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -105,7 +105,8 @@ "rollup": "4.59.0", "esbuild": "0.27.3", "fast-uri": "^3.1.5", - "minimatch": ">=3.1.3", + "follow-redirects": ">=1.16.2", + "minimatch": ">=11.0.1", "postcss": "^8.5.18", "serialize-javascript": "7.0.3", "workbox-build": { diff --git a/packages/factory-local/requirements.lock b/packages/factory-local/requirements.lock index 925eeb74c..700b730d0 100644 --- a/packages/factory-local/requirements.lock +++ b/packages/factory-local/requirements.lock @@ -1235,9 +1235,9 @@ pygments==2.21.0 \ --hash=sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9 \ --hash=sha256:610ca751c9bc2492b38eb9a38a7fbc93edbbb2d7182edaf34e66ae493dee5c8c # via pytest -pypdf==6.21.2 \ - --hash=sha256:0a3e1c5c886a1be03646c16a04827cc166c2a0575c971b26a4bf89686551c2d4 \ - --hash=sha256:1d44d1a3a200f25062adff0085b02040e4646431ac2ccac22e42679e91466084 +pypdf==6.21.4 \ + --hash=sha256:33e4fb41ca352810eb4e9838aa0d82d6888a5f8bf682f43e99cce5faa666255c \ + --hash=sha256:c8817e3f888d47a087872ded69812c1ef30cacb871d6e053e0c00bbaf0bc2163 # via -r requirements.txt pytest==9.1.1 \ --hash=sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313 \ diff --git a/packages/factory-local/requirements.txt b/packages/factory-local/requirements.txt index 890ce9304..778f4f886 100644 --- a/packages/factory-local/requirements.txt +++ b/packages/factory-local/requirements.txt @@ -8,6 +8,6 @@ neo4j>=5.23,<6 streamlit>=1.38,<2 pytest>=9.0.3,<10 beautifulsoup4>=4.12,<5 -pypdf>=6.21.2,<7 +pypdf>=6.21.4,<7 openai>=1.40,<3 python-dotenv>=1.0,<2 diff --git a/requirements.in b/requirements.in index 7d228d4e5..0d6a98310 100644 --- a/requirements.in +++ b/requirements.in @@ -163,7 +163,7 @@ beautifulsoup4>=4.15.0 lxml>=6.1.3 pdfminer.six>=20260107 pdfplumber>=0.11.10 -pypdf>=6.21.2,<7 # CVE-2026-115572/115580/115585; CVE-2026-71852, CVE-2026-71870; earlier: CVE-2026-59935/59936 (inline-image DoS), CVE-2026-48155/48156, GHSA-jm82-fx9c-mx94 +pypdf>=6.21.4,<7 # CVE-2026-119203/119204/119205 (6.21.4); CVE-2026-119206/119207/119208/119209 (6.21.3); earlier: CVE-2026-115572/115580/115585, CVE-2026-71852/71870, CVE-2026-59935/59936, CVE-2026-48155/48156, GHSA-jm82-fx9c-mx94 fpdf2>=2.8.9,<3 python-docx>=1.2.0 docx2txt>=0.9 diff --git a/requirements.lock b/requirements.lock index 98bb01f8b..133aa6865 100644 --- a/requirements.lock +++ b/requirements.lock @@ -2999,9 +2999,9 @@ psycopg-binary==3.3.6 \ --hash=sha256:fa34eb47969297471db7b7f193622c7e3ee839ec05abd05f1fe104d5b1b1dcf4 \ --hash=sha256:fdccb3a0e184b03e9baa673b15a809cf36c339c85dbda0ebc25a698846dfbee8 # via psycopg -pyasn1==0.6.4 \ - --hash=sha256:9c447d8431c947fe4c8febc4ed9e760bc29011a5b01e5c74b67025bd9fb8ce81 \ - --hash=sha256:deda9277cfd454080ec40b207fb6df82206a3a2688735233cdcd8d3d565f088b +pyasn1==0.6.5 \ + --hash=sha256:4017b0fce125096b8573b39c989fdb1a4b19cf761e683398184550fea0e66587 \ + --hash=sha256:6b11616a6bcf3a40e635f082bb232fa57b5e0dd7f517bd7ddd69ec3794508a47 # via # pyasn1-modules # webauthn @@ -3202,9 +3202,9 @@ pyotp==2.10.0 \ --hash=sha256:1df2f6a1bcc3bb0716172a5215ddc2f8c7c7fd26a13df9927d52e1746934836c \ --hash=sha256:d01e9703443616b03c57c700b5cbffd56a1f929c1b0f8f03131bc78c1fca9d3f # via -r requirements.in -pypdf==6.21.2 \ - --hash=sha256:0a3e1c5c886a1be03646c16a04827cc166c2a0575c971b26a4bf89686551c2d4 \ - --hash=sha256:1d44d1a3a200f25062adff0085b02040e4646431ac2ccac22e42679e91466084 +pypdf==6.21.4 \ + --hash=sha256:33e4fb41ca352810eb4e9838aa0d82d6888a5f8bf682f43e99cce5faa666255c \ + --hash=sha256:c8817e3f888d47a087872ded69812c1ef30cacb871d6e053e0c00bbaf0bc2163 # via -r requirements.in pypdfium2==5.13.0 \ --hash=sha256:07f58e91b8c45ca144a1ff3008faf3c73ef8a5e9fb32988831788363288228cd \