diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ec8ae1e..62fc759 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -45,6 +45,22 @@ jobs: PGDATABASE: policyweave_test PGPASSWORD: policyweave_ci_password run: sh db/tests/policy_revision_runtime.sh + - name: Verify PostgreSQL concurrent policy writers + env: + PGHOST: 127.0.0.1 + PGPORT: 5432 + PGUSER: policyweave_ci + PGDATABASE: policyweave_test + PGPASSWORD: policyweave_ci_password + run: sh db/tests/policy_revision_concurrency.sh + - name: Verify PostgreSQL policy revision restore + env: + PGHOST: 127.0.0.1 + PGPORT: 5432 + PGUSER: policyweave_ci + PGDATABASE: policyweave_test + PGPASSWORD: policyweave_ci_password + run: sh db/tests/policy_revision_restore.sh - run: npx playwright install --with-deps chromium - run: npm run test:e2e - name: Upload exact-head browser evidence diff --git a/.gitignore b/.gitignore index 90e0f30..b224117 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,4 @@ coverage playwright-report test-results/ *.tsbuildinfo +.codegraph/ diff --git a/AGENTS.md b/AGENTS.md index b384d91..e8e9074 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,7 +12,7 @@ PolicyWeave is a local-first privacy-policy fact-authoring workspace. It structu - Do not encode legal conclusions from memory. Every legal/rule/template decision needs an authoritative source, effective date, source revision, and implementation/test trace. - Do not commit identifying customer, individual, or real operational-organization data in tests, examples, fixtures, or product documentation. Publicly documented legal authorities, official document titles, standards bodies, source publishers, and the repository owner may be named when required for accurate provenance and citation. Production must not consume synthetic demo data. - Keep persistence objects semantically named with at least two words and `snake_case` unless a framework contract requires another convention. Avoid generic named persistence objects such as a standalone `id` table/collection. -- Hosted persistence/publication must be introduced only behind explicit tenant, authorization, audit, encryption, immutable revision, and supersession contracts. +- Hosted persistence/publication must be introduced only behind explicit tenant, authorization, audit, encryption, immutable revision, and supersession contracts. CI PostgreSQL restart and dump/restore evidence lives in `db/tests/policy_revision_restore.sh` and does not enable a hosted adapter. Seed `retention_status = applies` only in the same transaction as its `retention_rule`; autocommit fails the deferred fact contract. - GitHub Actions dependencies stay SHA pinned and checkout credentials must not persist. ## Verification diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index dd323bb..a28f5d9 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -35,10 +35,10 @@ The active MVP is a React/Vite browser workspace. State is in memory and there i Authoring completeness is deliberately separate from legal sufficiency. Current readiness rules prove that product-defined fact responsibilities were explicitly addressed; they do not assert that a policy complies with law. Source/effective-date-bound legal validation belongs to the Legal Source Registry -> Review & Publication boundary. -## Persistence boundary (Proposed schema; no runtime) +## Persistence boundary (Proposed schema; CI-only runtime) ADR-0003 and `db/migrations/0001_policy_revision.sql` propose the first PostgreSQL contract. The 3NF write model uses `policy_revision` as aggregate root; `service_profile`, `collection_item`, `processing_purpose`, and `retention_rule` are revision-owned facts. `(tenant_account_id, revision_number)` identifies a version, while `(policy_revision_id, collection_item_key)` is the item-level UPSERT/idempotency key. Deferred database constraints lock the owning revision row, reject collection items under explicit no-collection, and reject retention-rule/status contradictions at transaction commit. -This source contract is not an active datastore, released API, or migration-execution result. Hosted tenant authorization, immutable audit events, encryption, concurrent-lock evidence, migration rollback, and backup/restore remain open. Publication remains append-only/immutable with explicit supersession; writes across unrelated aggregates must not share a transaction merely for convenience. Named database/schema/persistence objects use at least two semantic words and `snake_case`, for example `policy_revision`, `collection_item`, `processing_purpose`, `review_finding`, `publication_revision`, and `legal_source_revision`. +Exact-head CI executes that migration against digest-pinned PostgreSQL 18, including rollback, two-session locks with NULL-safe complete-value assertions, process restart, and custom-format dump/restore of complete collection-item plus independent collection and retention facts. This is not an active datastore, released API, or hosted adapter. Hosted tenant authorization, immutable audit events, encryption, deletion, operational backup/restore, and production-scale contention remain open. Publication remains append-only/immutable with explicit supersession; writes across unrelated aggregates must not share a transaction merely for convenience. Named database/schema/persistence objects use at least two semantic words and `snake_case`, for example `policy_revision`, `collection_item`, `processing_purpose`, `review_finding`, `publication_revision`, and `legal_source_revision`. Separate write-side draft commands from read-side rendered/review projections once hosted traffic justifies it. Account for revision hot spots and optimistic/constrained writes before adding collaborative editing. Keep source/customer integrations behind ACLs; do not form a shared kernel with unrelated ContextualWisdomLab products without demonstrated reuse. diff --git a/CHANGELOG.md b/CHANGELOG.md index 470a4e9..d22276c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,8 @@ All notable product changes are recorded here. PolicyWeave is pre-release; entri ## Unreleased ### Added +- PostgreSQL restart and custom-format dump/restore evidence that preserves NULL-safe complete service/collection-item values, a collecting-without-retention cross-state fixture, and independent no-collection and applies-retention facts, then re-executes no-collection plus both retention-status/rule contradictions against the restored database. +- PostgreSQL two-session concurrency evidence that observes real lock waits, rejects a collection-item writer racing with a no-collection update, and proves competing same-item UPSERTs converge to one row carrying the second writer's label, mode, and path with NULL-safe complete-value assertions and without timing-based transaction sleeps. - PostgreSQL 18 runtime contract coverage for migration apply/down/apply cycles, item-key UPSERT idempotency, and deferred rejection of no-collection, missing-retention-rule, and revision-owner contradictions. The database remains CI-only and is not a hosted product backend. - Proposed PostgreSQL `policy_revision` 3NF migration contract with tenant-scoped version identity, normalized revision-owned facts, owner-key immutability, parent-row-serialized deferred no-collection/retention consistency checks, and item-level natural-key UPSERT. It is source-validated only and does not claim a deployed database or hosted persistence. - Seven-step authoring workflow with distinct editing surfaces for service information, collection items, processing purposes, retention, third-party transfer, international transfer, and privacy contact. @@ -53,6 +55,6 @@ All notable product changes are recorded here. PolicyWeave is pre-release; entri ### Not yet shipped - Authoritative legal-rule snapshots that can determine legal sufficiency beyond product-defined fact completeness. - Manual zoom and screen-reader evidence beyond the automated desktop/tablet/mobile Chromium accessibility and focus checks. -- A product persistence adapter, durable hosted storage, tenant authorization, immutable audit history, concurrent-writer scheduling, restart, encryption, and backup/restore evidence. CI migration and rollback execution do not constitute a hosted runtime. +- A product persistence adapter, durable hosted storage, tenant authorization, immutable audit history, encryption, operational backup/restore, and production-scale contention evidence. Bounded CI database execution, including process restart and dump/restore, does not constitute a hosted runtime. - Authenticated immutable publication revisions and public URL lifecycle. - Hosted tenant/security/operability evidence and endpoint load testing. diff --git a/CLAUDE.md b/CLAUDE.md index bff26e2..60e4e51 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,3 +5,5 @@ Follow `AGENTS.md` as the repository-wide operating contract. Before changing code, read `docs/PRD.md`, `ARCHITECTURE.md`, the applicable ADRs, `docs/TRD.md`, `docs/research-traceability.md`, and `docs/product-technical-gap-baseline.md`. Preserve PolicyWeave's boundary: verified operator facts in, deterministic reviewable policy projection out; no legal-advice or compliance-guarantee claims. For behavior changes, add or strengthen tests before production code. Reconcile documentation and the product-gap ledger on the same branch. Use the exact current PR head for reviews, checks, and merge decisions, and never bypass governance to compensate for a failing or unassigned check. + +Deferred `policy_revision` fact triggers evaluate the final commit state. CI restore seeds `retention_status = applies` and `retention_rule` in one transaction; the owner runbook is `db/tests/policy_revision_restore.sh`. diff --git a/db/tests/policy_revision_concurrency.sh b/db/tests/policy_revision_concurrency.sh new file mode 100644 index 0000000..43573ae --- /dev/null +++ b/db/tests/policy_revision_concurrency.sh @@ -0,0 +1,195 @@ +#!/bin/sh +set -eu + +: "${PGHOST:=127.0.0.1}" +: "${PGPORT:=5432}" +: "${PGUSER:=policyweave_ci}" +: "${PGDATABASE:=policyweave_test}" +: "${PGPASSWORD:=policyweave_ci_password}" +export PGHOST PGPORT PGUSER PGDATABASE PGPASSWORD + +test_run_id=$$ +parent_release_fifo="/tmp/policyweave_parent_release_${test_run_id}" +parent_ready_file="/tmp/policyweave_parent_ready_${test_run_id}" +parent_writer_log="/tmp/policyweave_parent_writer_${test_run_id}.log" +item_writer_log="/tmp/policyweave_item_writer_${test_run_id}.log" +upsert_release_fifo="/tmp/policyweave_upsert_release_${test_run_id}" +upsert_ready_file="/tmp/policyweave_upsert_ready_${test_run_id}" +first_upsert_log="/tmp/policyweave_first_upsert_${test_run_id}.log" +second_upsert_log="/tmp/policyweave_second_upsert_${test_run_id}.log" +parent_writer_process_id= +item_writer_process_id= +first_upsert_process_id= +second_upsert_process_id= + +psql_command() { + psql --no-psqlrc --set ON_ERROR_STOP=1 "$@" +} + +cleanup_concurrency_test() { + for writer_process_id in \ + ${parent_writer_process_id:-} \ + ${item_writer_process_id:-} \ + ${first_upsert_process_id:-} \ + ${second_upsert_process_id:-}; do + kill "$writer_process_id" 2>/dev/null || true + wait "$writer_process_id" 2>/dev/null || true + done + rm -f \ + "$parent_release_fifo" "$parent_ready_file" "$parent_writer_log" "$item_writer_log" \ + "$upsert_release_fifo" "$upsert_ready_file" "$first_upsert_log" "$second_upsert_log" +} + +wait_for_file() { + expected_file=$1 + wait_name=$2 + attempt_count=0 + while [ ! -f "$expected_file" ]; do + attempt_count=$((attempt_count + 1)) + if [ "$attempt_count" -ge 200 ]; then + echo "timed out waiting for ${wait_name}" >&2 + return 1 + fi + sleep 0.05 + done +} + +wait_for_lock() { + writer_application_name=$1 + writer_log=$2 + attempt_count=0 + while :; do + waiting_count=$(psql_command --tuples-only --no-align --command \ + "select count(*) from pg_stat_activity where application_name = '${writer_application_name}' and wait_event_type = 'Lock'") + if [ "$waiting_count" -ge 1 ]; then + return 0 + fi + attempt_count=$((attempt_count + 1)) + if [ "$attempt_count" -ge 200 ]; then + cat "$writer_log" >&2 || true + echo "timed out waiting for PostgreSQL lock: ${writer_application_name}" >&2 + return 1 + fi + sleep 0.05 + done +} + +trap cleanup_concurrency_test 0 1 2 15 + +mkfifo "$parent_release_fifo" "$upsert_release_fifo" +psql_command --file db/migrations/0001_policy_revision.sql + +psql_command <<'SQL' +insert into policy_revision (policy_revision_id, tenant_account_id, revision_number) +values ('30000000-0000-4000-8000-000000000001', '40000000-0000-4000-8000-000000000001', 1); +SQL + +PGAPPNAME=policyweave_parent_writer PGOPTIONS='-c statement_timeout=15000' \ + psql --no-psqlrc --set ON_ERROR_STOP=1 >"$parent_writer_log" 2>&1 <"$item_writer_log" 2>&1 <<'SQL' & +begin; +insert into collection_item (policy_revision_id, collection_item_key, collection_item_label) +values ('30000000-0000-4000-8000-000000000001', 'contact_email', 'Contact email'); +commit; +SQL +item_writer_process_id=$! +wait_for_lock policyweave_item_writer "$item_writer_log" +printf 'release\n' >"$parent_release_fifo" +wait "$parent_writer_process_id" +parent_writer_process_id= + +set +e +wait "$item_writer_process_id" +item_writer_status=$? +set -e +item_writer_process_id= +if [ "$item_writer_status" -eq 0 ]; then + echo 'competing collection item unexpectedly committed' >&2 + exit 1 +fi +grep -F -- 'no-collection confirmation conflicts with collection items' "$item_writer_log" + +psql_command <<'SQL' +insert into policy_revision (policy_revision_id, tenant_account_id, revision_number) +values ('30000000-0000-4000-8000-000000000002', '40000000-0000-4000-8000-000000000001', 2); +SQL + +PGAPPNAME=policyweave_first_upsert_writer PGOPTIONS='-c statement_timeout=15000' \ + psql --no-psqlrc --set ON_ERROR_STOP=1 >"$first_upsert_log" 2>&1 <"$second_upsert_log" 2>&1 <<'SQL' & +begin; +select upsert_collection_item( + '30000000-0000-4000-8000-000000000002', + 'contact_email', + 'Concurrent contact email', + 'optional', + 'Account profile form' +); +commit; +SQL +second_upsert_process_id=$! +wait_for_lock policyweave_second_upsert_writer "$second_upsert_log" +printf 'release\n' >"$upsert_release_fifo" +wait "$first_upsert_process_id" +first_upsert_process_id= +wait "$second_upsert_process_id" +second_upsert_process_id= + +psql_command <<'SQL' +do $concurrency_assertion$ +declare + stored_count integer; + stored_label text; + stored_mode text; + stored_path text; +begin + select count(*), + max(collection_item_label), + max(collection_mode::text), + max(collection_path) + into stored_count, stored_label, stored_mode, stored_path + from collection_item + where policy_revision_id = '30000000-0000-4000-8000-000000000002' + and collection_item_key = 'contact_email'; + + if stored_count <> 1 + or stored_label <> 'Concurrent contact email' + or stored_mode is distinct from 'optional' + or stored_path is distinct from 'Account profile form' then + raise exception 'concurrent collection-item UPSERTs did not converge'; + end if; +end; +$concurrency_assertion$; +SQL + +psql_command --file db/migrations/0001_policy_revision.down.sql +cleanup_concurrency_test +trap - 0 1 2 15 diff --git a/db/tests/policy_revision_restore.sh b/db/tests/policy_revision_restore.sh new file mode 100644 index 0000000..ea8fadb --- /dev/null +++ b/db/tests/policy_revision_restore.sh @@ -0,0 +1,311 @@ +#!/bin/sh +set -eu + +: "${PGHOST:=127.0.0.1}" +: "${PGPORT:=5432}" +: "${PGUSER:=policyweave_ci}" +: "${PGDATABASE:=policyweave_test}" +: "${PGPASSWORD:=policyweave_ci_password}" +export PGHOST PGPORT PGUSER PGDATABASE PGPASSWORD + +test_run_id=$$ +dump_file="/tmp/policyweave_restore_${test_run_id}.dump" +list_file="/tmp/policyweave_restore_${test_run_id}.list" +container_dump_file="/tmp/policyweave_restore.dump" +postgres_container_id= + +psql_command() { + psql --no-psqlrc --set ON_ERROR_STOP=1 "$@" +} + +cleanup_restore_test() { + rm -f "$dump_file" "$list_file" + if [ -n "$postgres_container_id" ]; then + docker exec "$postgres_container_id" rm -f "$container_dump_file" >/dev/null 2>&1 || true + fi + psql --no-psqlrc --command "drop database if exists policyweave_restore" >/dev/null 2>&1 || true +} + +wait_for_postgres() { + attempt_count=0 + while :; do + if docker exec "$postgres_container_id" pg_isready -U "$PGUSER" -d "$PGDATABASE" >/dev/null 2>&1 \ + && psql_command --command 'select 1' >/dev/null 2>&1; then + return 0 + fi + attempt_count=$((attempt_count + 1)) + if [ "$attempt_count" -ge 60 ]; then + echo 'PostgreSQL did not accept connections after restart' >&2 + return 1 + fi + sleep 1 + done +} + +find_postgres_container() { + container_id=$(docker ps --filter name=postgres --format '{{.ID}}' | head -n 1) + if [ -n "$container_id" ]; then + printf '%s\n' "$container_id" + return 0 + fi + docker ps --format '{{.ID}} {{.Ports}}' | awk '/5432/ { print $1; exit }' +} + +expect_failure() { + failure_name=$1 + expected_message=$2 + failure_log="/tmp/${failure_name}.log" + if psql_command >"$failure_log" 2>&1; then + echo "expected PostgreSQL failure: ${failure_name}" >&2 + return 1 + fi + if ! grep -F -- "$expected_message" "$failure_log"; then + cat "$failure_log" >&2 + echo "unexpected PostgreSQL failure: ${failure_name}" >&2 + return 1 + fi +} + +assert_restored_facts() { + scene_name=$1 + psql_command < 1 + or service_profile_count <> 1 + or stored_service_name is distinct from 'Restore Probe Service' + or stored_service_url is distinct from 'https://restore.example.test' + or collection_item_count <> 1 + or stored_item_label <> 'Restore contact email' + or stored_item_mode is distinct from 'required' + or stored_item_path is distinct from 'Account registration form' + or collection_purpose_count <> 1 + or stored_purpose <> 'Account notices' + or collection_rule_count <> 1 + or stored_period <> '1 year after account closure' + or no_collection_revision_count <> 1 + or no_collection_item_count <> 0 + or no_collection_rule_count <> 0 + or collection_without_retention_count <> 1 then + raise exception '${scene_name} did not preserve independent collection and retention facts'; + end if; +end; +\$restore_assertion\$; +SQL +} + +if ! command -v docker >/dev/null 2>&1; then + echo 'docker is required for PostgreSQL restart evidence' >&2 + exit 1 +fi + +postgres_container_id=$(find_postgres_container) +if [ -z "$postgres_container_id" ]; then + echo 'postgres container not found for restart evidence' >&2 + docker ps >&2 || true + exit 1 +fi + +trap cleanup_restore_test 0 1 2 15 + +psql_command --file db/migrations/0001_policy_revision.sql + +psql_command <<'SQL' +begin; +insert into policy_revision ( + policy_revision_id, tenant_account_id, revision_number, retention_status +) values ( + '60000000-0000-4000-8000-000000000001', '50000000-0000-4000-8000-000000000001', 1, 'applies' +); +insert into service_profile (policy_revision_id, service_name, service_url) +values ( + '60000000-0000-4000-8000-000000000001', + 'Restore Probe Service', + 'https://restore.example.test' +); +select upsert_collection_item( + '60000000-0000-4000-8000-000000000001', + 'contact_email', + 'Restore contact email', + 'required', + 'Account registration form' +); +insert into processing_purpose (policy_revision_id, collection_item_key, purpose_text) +values ( + '60000000-0000-4000-8000-000000000001', + 'contact_email', + 'Account notices' +); +insert into retention_rule (policy_revision_id, retention_period) +values ( + '60000000-0000-4000-8000-000000000001', + '1 year after account closure' +); + +insert into policy_revision ( + policy_revision_id, tenant_account_id, revision_number, no_collection_confirmed, retention_status +) values ( + '60000000-0000-4000-8000-000000000002', '50000000-0000-4000-8000-000000000001', 2, true, 'none' +); + +insert into policy_revision ( + policy_revision_id, tenant_account_id, revision_number, retention_status +) values ( + '60000000-0000-4000-8000-000000000003', '50000000-0000-4000-8000-000000000001', 3, 'none' +); +select upsert_collection_item( + '60000000-0000-4000-8000-000000000003', + 'support_email', + 'Support email', + 'optional', + 'Support request form' +); +commit; +SQL + +psql_command --command 'checkpoint' + +docker restart "$postgres_container_id" >/dev/null +wait_for_postgres +assert_restored_facts restart + +docker exec "$postgres_container_id" pg_dump \ + -U "$PGUSER" \ + -d "$PGDATABASE" \ + --no-owner \ + --no-acl \ + --format=custom \ + --schema=public \ + -f "$container_dump_file" + +docker cp "$postgres_container_id:$container_dump_file" "$dump_file" +if [ ! -s "$dump_file" ]; then + echo 'pg_dump produced an empty custom archive' >&2 + exit 1 +fi + +psql_command --file db/migrations/0001_policy_revision.down.sql +psql_command <<'SQL' +do $rollback_assertion$ +begin + if to_regclass('public.policy_revision') is not null then + raise exception 'rollback left policy_revision behind before restore'; + end if; +end; +$rollback_assertion$; +SQL + +psql_command --command "drop database if exists policyweave_restore" +psql_command --command "create database policyweave_restore" + +docker exec "$postgres_container_id" pg_restore -l "$container_dump_file" \ + | grep -v 'SCHEMA - public' >"$list_file" +docker cp "$list_file" "$postgres_container_id:$container_dump_file.list" +docker exec "$postgres_container_id" pg_restore \ + -U "$PGUSER" \ + -d policyweave_restore \ + --no-owner \ + --no-acl \ + --exit-on-error \ + -L "$container_dump_file.list" \ + "$container_dump_file" + +PGDATABASE=policyweave_restore assert_restored_facts restore +PGDATABASE=policyweave_restore expect_failure restored_no_collection_conflict 'no-collection confirmation conflicts with collection items' <<'SQL' +begin; +insert into collection_item (policy_revision_id, collection_item_key, collection_item_label) +values ('60000000-0000-4000-8000-000000000002', 'contact_email', 'Contact email'); +commit; +SQL +PGDATABASE=policyweave_restore expect_failure restored_status_without_rule 'retention status applies requires a retention rule' <<'SQL' +begin; +update policy_revision + set retention_status = 'applies' + where policy_revision_id = '60000000-0000-4000-8000-000000000002'; +commit; +SQL +PGDATABASE=policyweave_restore expect_failure restored_rule_without_status 'retention rule requires retention status applies' <<'SQL' +begin; +insert into retention_rule (policy_revision_id, retention_period) +values ('60000000-0000-4000-8000-000000000002', '1 year'); +commit; +SQL + +psql_command --command "drop database policyweave_restore" +cleanup_restore_test +trap - 0 1 2 15 diff --git a/docs/ADR-0003-policy-revision-persistence.md b/docs/ADR-0003-policy-revision-persistence.md index d3d6a7a..bb5b3fc 100644 --- a/docs/ADR-0003-policy-revision-persistence.md +++ b/docs/ADR-0003-policy-revision-persistence.md @@ -16,7 +16,7 @@ The browser workspace holds an operator's draft only in memory. A hosted product - `retention_status = applies` requires exactly one current `retention_rule`; other statuses cannot retain one. - Organization-owned persistence identifiers use two or more semantic words and `snake_case`. - Retries update one collection item through its declared natural key; they do not replace an entire revision. -- Hosted authorization, audit, encryption, publication, backup/restore, and runtime migration evidence remain mandatory before production use. +- Hosted authorization, audit, encryption, publication, and operational backup/restore remain mandatory before production use. CI restart and dump/restore evidence does not replace those hosted controls. ## Decision @@ -42,12 +42,12 @@ Rejected. No released owner contract currently supplies PolicyWeave's product-do ## Evidence -`src/persistence-schema.test.ts` fixes the stable schema markers for revision identity, normalized ownership, deferred fact consistency, and natural-key UPSERT. `db/tests/policy_revision_runtime.sh` applies the migration to a digest-pinned PostgreSQL 18 service, exercises natural-key retry, requires each negative scene to emit its expected domain error, executes the down migration, verifies object removal, and repeats the apply/down cycle. This remains CI evidence rather than a deployed storage claim. +`src/persistence-schema.test.ts` fixes the stable schema markers for revision identity, normalized ownership, deferred fact consistency, and natural-key UPSERT. `db/tests/policy_revision_runtime.sh` applies the migration to a digest-pinned PostgreSQL 18 service, exercises natural-key retry, requires each negative scene to emit its expected domain error, executes the down migration, verifies object removal, and repeats the apply/down cycle. `db/tests/policy_revision_concurrency.sh` coordinates two real sessions with FIFOs, observes PostgreSQL lock waits, and verifies conflicting fact writers fail closed while same-item UPSERT writers converge to one row carrying the second writer's label, mode, and path; nullable mode/path comparisons use `IS DISTINCT FROM` so missing values cannot satisfy the evidence assertion. `db/tests/policy_revision_restore.sh` checkpoints, restarts the service container, dumps a custom-format archive, restores it into a fresh database, and proves a collecting revision may independently retain `retention_status = none`, NULL-safe complete service/collection-item values survive, and no-collection plus both retention status/rule deferred contradictions still fail against the restored schema; nullable restored values are checked with `IS DISTINCT FROM`. Applies retention and its required rule are seeded in one transaction because the deferred fact trigger evaluates the final commit state. This remains CI evidence rather than a deployed storage claim. ## Risks and effects - The migration is not a production backend and grants no network access. -- CI proves PostgreSQL 18 execution for the covered single-session cases but does not prove concurrent lock scheduling, restart safety, tenant authorization, or backup/restore. +- Exact-head CI must prove PostgreSQL 18 execution for the covered single-session, two-session, process-restart, and dump/restore cases; it does not prove tenant authorization, encryption, deletion, or production-scale contention. - The `tenant_account_id` is deliberately not linked to an identity table until a released Keyverse contract and PolicyWeave authorization design exist. - Draft facts may remain nullable while unresolved; database constraints protect contradictions, while completeness remains the deterministic review responsibility. - The collection mode enum uses locale-neutral values. UI labels are translated at the application boundary rather than stored as database truth. @@ -58,8 +58,10 @@ Rejected. No released owner contract currently supplies PolicyWeave's product-do - A client tries to add an item to a revision confirmed as no-collection: commit fails with a constraint violation. - A client changes retention from `applies` to `none` but forgets to remove the old rule: commit fails, so stale retention evidence cannot survive. - A client attempts to move a retention rule between revisions: commit fails instead of leaving the original `applies` revision without its required rule. +- A no-collection writer holds the revision lock while a second client adds an item: the second client waits, observes the committed parent fact, and fails closed. +- Two clients UPSERT the same item key: the second waits for the first and updates the same row rather than creating a duplicate. - Two clients claim the same tenant revision number: the unique constraint rejects one rather than creating ambiguous versions. ## Follow-up -Measure concurrent UPSERT/lock behavior, verify restart and backup/restore, add tenant-purpose authorization and immutable audit events, and only then connect a hosted asynchronous API. Immutable publication and supersession remain a separate Review & Publication decision. +Add tenant-purpose authorization and immutable audit events, measure production-scale contention, and only then connect a hosted asynchronous API. Immutable publication and supersession remain a separate Review & Publication decision. diff --git a/docs/TRD.md b/docs/TRD.md index 70121d2..425d035 100644 --- a/docs/TRD.md +++ b/docs/TRD.md @@ -40,7 +40,7 @@ The separation between collection and retention follows the PIPC Standard Person ## Hosted persistence/publication entry criteria Before network persistence lands, define a versioned policy-data schema, migration policy, 3NF relational model, per-item UPSERT/idempotency rules, immutable publication receipt, supersession/rollback semantics, tenant/purpose authorization, audit evidence, encryption/key management, retention/deletion behavior, and backup/restore testing. Use two-or-more-word semantic persistence object names in `snake_case` by default. The revision model must preserve explicit no-collection and explicit retention status independently; `none` must not be materialized from collection absence, and an inapplicable/non-retained state must not carry a live `retention_rule` value. -ADR-0003 and migration `0001_policy_revision.sql` satisfy only the source-contract portion of that entry criterion: tenant-scoped revision identity, normalized service/collection/purpose/retention facts, parent-row-serialized deferred contradiction checks, and collection-item natural-key UPSERT. Current CI validates stable DDL markers but does not execute PostgreSQL. Runtime migration/rollback, concurrency, authorization, audit, encryption, deletion, and backup/restore evidence remain required before any adapter may enable network persistence. +ADR-0003 and migration `0001_policy_revision.sql` satisfy only the source-contract portion of that entry criterion: tenant-scoped revision identity, normalized service/collection/purpose/retention facts, parent-row-serialized deferred contradiction checks, and collection-item natural-key UPSERT. Current CI executes PostgreSQL 18 for apply/down/apply, exact domain-error, two-session lock with NULL-safe complete-value assertions, process-restart, and custom-format dump/restore scenes. Authorization, audit, encryption, deletion, production-scale contention, and a hosted adapter remain required before network persistence may be enabled. A publication command must operate on a specific reviewed `policy_revision` and create a new immutable `publication_revision`. It must fail closed when blocking findings, incompatible rule/template versions, missing source receipts, or missing authorization exist. Publication never mutates foreign customer databases. diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 9c96a36..6ad587b 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -48,6 +48,8 @@ Action-runtime cleanup used hosted CI `34088150412` as a real control-plane RED: Queue RCA then observed distinct stale heads `cbfbdae4a4bb72433d7bdcc51afdbd8a29c102a1`, `3d576aa9e4079af0749cb03f6cc20e4cb30e6639`, and `7dd5c24eed855ef7acdfc0dcb217c86dd00d2df3` retaining separate queued CI runs because the repository workflow had no concurrency contract. Test-only head `bdaf5e8560289b7c4774d46fff2afa7489ad51c9` specifies PR-or-ref grouping and stale-head cancellation. Repository CI now groups by workflow plus pull-request number (or branch ref for push events) and sets `cancel-in-progress: true`; this prevents future superseded heads from consuming runner capacity while keeping unrelated PRs and refs independent. +The restart/restore slice remains bounded CI evidence and does not enable a hosted adapter. After the initial atomic-seed repair, review found that paired collection/applies and no-collection/none fixtures could not prove collection and retention are independent, the restored database re-exercised only the no-collection trigger, and a nullable restored `service_name` could evade `<>` through SQL three-valued logic. Test-only head `5e54834873e125b3e3ce4f599e4037e017330638` added the missing cross-state and NULL-safe assertions; exact-head CI `34204279846` was RED only in the restore step with `restart did not preserve independent collection and retention facts`. The next commit seeds a valid collecting revision with `retention_status = none`, keeps authored service and collection-item assertions NULL-safe, and executes status-side missing-rule plus rule-side unexpected-rule transactions against the restored database. Pre-documentation head `202e69d95c94e4432365d6599016a371c0f2cbc3` CI `34204464388` then passed the complete suite. A later exact-head review found that the nullable authored service URL was not selected or asserted even though the evidence claim covered complete service values. Mutation-probe head `aaef3b5489493669cdb53c08a72b6a109fc0b687` deliberately nulled that URL after restart; CI `34205653966` passed every preceding step and failed only the new NULL-safe restore assertion. Commit `57732c6dbec872ad29e97a7f22096dbba9613e9a` removes the probe while retaining literal name/URL checks. These immutable runs establish the TDD transitions but are not substitutes for the final current-head verdict. This is CI durability evidence, not operational backup, tenant authorization, or a released datastore. + ## Current baseline | Area | Evidence | Status | Commercialization gap | Owner/action | Next verification | @@ -57,10 +59,10 @@ Queue RCA then observed distinct stale heads `cbfbdae4a4bb72433d7bdcc51afdbd8a29 | Review workspace | Live preview, total blocker count, warning-to-owner navigation, buyer-facing readiness guidance, deterministic step-heading focus transfer, Playwright viewport/screenshot harness | Implemented foundation; offscreen focus repaired and exact-head browser verified | Automated desktop/tablet/mobile focus-scroll evidence is GREEN; broader interaction coverage remains bounded | UX: retain exact-head artifacts, then extend interaction coverage | Exact-head screenshots, keyboard/focus and accessibility checks | | Accessibility | Semantic controls, visible focus behavior, focus-token >=3:1 regression, muted-text >=4.5:1 authored-surface regression, jsdom focus transition, axe/browser, focused-heading viewport checks, responsive retention transitions, and effective 200% browser-zoom reflow | Partial; bounded exact-head browser GREEN | Native browser UI zoom automation, screen-reader, and manual WCAG evidence remain absent | UX/Test Engineering: add a manual interaction record and remaining cases without claiming conformance from automation alone | Exact-head WCAG/browser matrix plus screen-reader and manual evidence | | Legal/rule traceability | `docs/research-traceability.md`; PIPA/PIPC source/effective-date model; PIPC Notice No. 2025-4 terminology bound to collection/retention separation | Source baseline established | Current readiness rules are product completeness only; no versioned article/section-level legal-rule engine | Legal Source Registry: map future legal validations to source/effective date/revision | Snapshot fixtures across effective dates | -| Policy model | ADRs, ARCHITECTURE, TRD, ADR-0003, Proposed ERD, up/down migration, schema and PostgreSQL runtime contract tests | Proposed 3NF foundation; PostgreSQL 18 CI execution required; browser runtime remains memory-only | Apply/down/apply, single-session item UPSERT, and three expected domain-specific deferred failures are executable in CI; concurrent scheduling, restart, authorization, audit, encryption, deletion, and backup/restore remain unproved | Platform: retain exact-head PostgreSQL evidence, then add the hosted authorization/audit boundary while keeping the adapter disabled | Concurrent idempotency, restart and backup/restore tests | +| Policy model | ADRs, ARCHITECTURE, TRD, ADR-0003, Proposed ERD, up/down migration, schema, runtime, two-session concurrency, restart, and dump/restore contract tests | Proposed 3NF foundation; PostgreSQL 18 exact-head CI execution required; browser runtime remains memory-only | Apply/down/apply, exact negative errors, observed lock waits, conflicting-fact rejection, same-item UPSERT convergence with NULL-safe complete label/mode/path assertions, process restart, and custom-format restore with a collection/no-retention cross-state, NULL-safe complete service name/URL and item assertions, and restored no-collection plus both retention contradiction checks are implemented; authorization, audit, encryption, deletion, and production-scale contention remain unproved | Platform: retain exact-head PostgreSQL evidence, then add the hosted authorization/audit boundary while keeping the adapter disabled | Tenant authorization, immutable audit, and encryption tests | | Publication | Readiness CTA truthfully does not pretend to publish; immutable `publication_revision` is designed | Planned | No authenticated approval, immutable publication, supersession, rollback, or public URL lifecycle | Review & Publication: implement after persistence/security entry criteria | Authorization, replay/digest, supersession tests | | Security/privacy | `docs/SECURITY.md`; local-first runtime; SHA-pinned checkout | Baseline documented | Hosted tenant model, encryption/key handling, audit/incident/retention evidence absent | Platform/Security: threat-model hosted boundary before backend | Exact-head security tests and org scans | -| Tests | 58 unit/UI/schema/workflow regressions plus thirteen executed Playwright/axe cases, two project-scoped skips, and a screenshot artifact contract | Improved; exact-head unit/build/browser/PostgreSQL GREEN required | Repository-wide 100% execution/docstring coverage is not yet evidenced | Test Engineering: measure coverage and extend realistic edge cases | Exact-head coverage + browser/database evidence | +| Tests | 63 unit/UI/schema/workflow regressions plus PostgreSQL runtime/concurrency/restore scripts, thirteen Playwright/axe cases, two project-scoped skips, and a screenshot artifact contract | Improved; exact-head unit/build/browser/PostgreSQL GREEN required | Repository-wide 100% execution/docstring coverage is not yet evidenced | Test Engineering: measure coverage and extend realistic edge cases | Exact-head coverage + browser/database evidence | | Performance | Static Vite client | Unevidenced | No realistic buyer-flow browser performance baseline; no network backend exists for meaningful k6 endpoint evidence | Operability: record browser baseline now; add k6 only when hosted network surfaces exist | Real measurements before latency claims | | CI/security merge gate | Repo CI plus active organization ruleset-required workflows; immutable Node 24-based checkout, setup/cache, and artifact-upload action releases | Live external gate; warning-free evidence is re-fetched for the exact merge candidate | Every branch movement invalidates predecessor evidence and stale approval; current hosted jobs may remain queued before runner assignment and dependency/reviewer control-plane failures can fail closed independently | Re-fetch exact-head workflows/reviews; use the central owner path for runner/dependency-review incidents rather than leaf-side churn or bypass | Terminal exact-head checks with no action-runtime deprecation warnings + independent approval + resolved threads | @@ -77,6 +79,6 @@ A publishable release requires complete fact-to-warning navigation, deterministi ## Active commercialization order 1. Reacquire exact-current-head CI/security/SAST/central workflow execution and independent approval; merge PR #1 only through ordinary protection. 2. Extend the exact-head responsive/accessibility evidence with screen-reader checks, native browser-zoom automation when the harness supports it, and manual records; CSS focus contrast, logical step-focus transfer, focused-heading viewport visibility, retention transitions, effective 200% browser-zoom reflow, and current desktop/tablet/mobile screenshots are already verified. -3. Preserve exact-head PostgreSQL 18 apply/down and single-session UPSERT evidence, then prove concurrent item-level UPSERT scheduling, restart, tenant authorization, immutable audit, and backup/restore before enabling any hosted adapter. +3. Preserve exact-head PostgreSQL 18 migration, domain-error, two-session lock, restart, and dump/restore evidence, then prove tenant authorization, immutable audit, and encryption before enabling any hosted adapter. 4. Implement secure review/publication with immutable releases and explicit supersession. 5. Add hosted tenant isolation, audit/incident evidence, compose deployment, and realistic performance/load tests when network services exist. diff --git a/src/postgres-concurrency-contract.test.ts b/src/postgres-concurrency-contract.test.ts new file mode 100644 index 0000000..21af59d --- /dev/null +++ b/src/postgres-concurrency-contract.test.ts @@ -0,0 +1,29 @@ +import { existsSync, readFileSync } from 'node:fs' +import { fileURLToPath } from 'node:url' +import { describe, expect, it } from 'vitest' + +const workflowPath = fileURLToPath(new URL('../.github/workflows/ci.yml', import.meta.url)) +const concurrencyPath = fileURLToPath(new URL('../db/tests/policy_revision_concurrency.sh', import.meta.url)) + +const workflowSource = readFileSync(workflowPath, 'utf8') +const concurrencyTest = existsSync(concurrencyPath) ? readFileSync(concurrencyPath, 'utf8') : '' + +describe('PostgreSQL concurrent writer evidence contract', () => { + it('observes real PostgreSQL lock waits without timing-based transaction sleeps', () => { + expect(concurrencyTest).toContain('mkfifo') + expect(concurrencyTest).toContain("wait_event_type = 'Lock'") + expect(concurrencyTest).toContain('policyweave_parent_writer') + expect(concurrencyTest).toContain('policyweave_item_writer') + expect(concurrencyTest).toContain('statement_timeout=15000') + expect(concurrencyTest).not.toContain('pg_sleep') + }) + + it('proves competing facts fail closed and same-item UPSERTs converge', () => { + expect(concurrencyTest).toContain('no-collection confirmation conflicts with collection items') + expect(concurrencyTest).toMatch(/count\(\*\)[\s\S]*Concurrent contact email/) + expect(concurrencyTest).toMatch( + /stored_mode[\s\S]*stored_path[\s\S]*if[\s\S]*stored_mode is distinct from 'optional'[\s\S]*stored_path is distinct from 'Account profile form'/i, + ) + expect(workflowSource).toContain('run: sh db/tests/policy_revision_concurrency.sh') + }) +}) diff --git a/src/postgres-restore-contract.test.ts b/src/postgres-restore-contract.test.ts new file mode 100644 index 0000000..792c019 --- /dev/null +++ b/src/postgres-restore-contract.test.ts @@ -0,0 +1,45 @@ +import { existsSync, readFileSync } from 'node:fs' +import { fileURLToPath } from 'node:url' +import { describe, expect, it } from 'vitest' + +const workflowPath = fileURLToPath(new URL('../.github/workflows/ci.yml', import.meta.url)) +const restorePath = fileURLToPath(new URL('../db/tests/policy_revision_restore.sh', import.meta.url)) + +const workflowSource = readFileSync(workflowPath, 'utf8') +const restoreTest = existsSync(restorePath) ? readFileSync(restorePath, 'utf8') : '' + +describe('PostgreSQL restart and restore evidence contract', () => { + it('restarts the digest-pinned PostgreSQL service and reconnects without fabricated delays', () => { + expect(restoreTest).toContain('docker restart') + expect(restoreTest).toContain('pg_isready') + expect(restoreTest).toContain('checkpoint') + expect(restoreTest).not.toContain('pg_sleep') + expect(restoreTest).not.toContain('sleep 5') + }) + + it('restores a custom dump after rollback and keeps collection independent from retention', () => { + expect(restoreTest).toContain('--format=custom') + expect(restoreTest).toContain('0001_policy_revision.down.sql') + expect(restoreTest).toContain('pg_restore') + expect(restoreTest).toContain("no_collection_confirmed = true") + expect(restoreTest).toContain("retention_status = 'none'") + expect(restoreTest).toContain("retention_status = 'applies'") + expect(restoreTest).toMatch( + /stored_item_mode[\s\S]*stored_item_path[\s\S]*stored_item_mode is distinct from 'required'[\s\S]*stored_item_path is distinct from 'Account registration form'/i, + ) + expect(restoreTest).toContain('no-collection confirmation conflicts with collection items') + expect(workflowSource).toContain('run: sh db/tests/policy_revision_restore.sh') + }) + + it('commits applies retention only after the required rule exists in the same transaction', () => { + const appliesIndex = restoreTest.indexOf("1, 'applies'") + const beginIndex = restoreTest.lastIndexOf('begin;', appliesIndex) + const ruleIndex = restoreTest.indexOf('insert into retention_rule', appliesIndex) + const commitIndex = restoreTest.indexOf('commit;', Math.max(appliesIndex, ruleIndex)) + expect(appliesIndex).toBeGreaterThan(-1) + expect(beginIndex).toBeGreaterThan(-1) + expect(beginIndex).toBeLessThan(appliesIndex) + expect(ruleIndex).toBeGreaterThan(appliesIndex) + expect(commitIndex).toBeGreaterThan(ruleIndex) + }) +})