feat(data): add bounded tenant provenance portability

[seonghobae]

Current migrated integration receipt (2026-09-05)

Head 69f50ae684f50c501ff2f49be2969f1d211d7f3c, tree 839d62316b5ff5f1e84e346e48982f0d4494c8c7, normally merges checkpoint 7ee6e68c31b2e716210fc8b62e287a78b765062b with #1427 cb08b1c3ea2aba8844fc29ef703c34368cc55e47. The preserved stack is #1565 → #1503 → #1572 → #1468 → #1427 → #1497. No force push, PR close, or valid-delta deletion was used.

345 tests pass, zero failed/skipped, with -W error, after exact uv sync --locked and fresh/repeated actual migration to the single 0021_merge_provenance_workspace head. The terminal run took 176.52 seconds and exit zero; its exact isolated Compose resources were cleaned up. Ruff, diff checks and 13 touched-document file links pass. The unchanged 8 MiB-class test_export_counts_cited_segment_bytes_once now passes: the preceding 288-pass/1-failure checkpoint had exposed GiST overflow, repaired at #1572 instead of shrinking the fixture or dropping the indexes.

Real migration regressions preserve existing portable-identity mappings through rollback/re-upgrade. Forced simultaneous identical and incompatible imports preserve the winning records; incompatible input yields one rejected writer with no losing rows/mappings. ADR-0022 corrects the prior lock description: the first lock keys the complete sorted portable-identity sets, followed by sorted per-email locks scoped to target user and organization. It does not claim to serialize every arbitrary partial overlap.

The local not-yet-pushed merge revision was renamed from 0020 to 0021 before publication, joining 0018_provenance_identity and 0020_search_trigram_storage; inherited migration IDs remain intact. ADR-0007 collided with #1361, so its proposal/content/discussion lineage is preserved as ADR-0022. The PDF parent similarly carries ADR-0021, formerly 0005. The detailed source/history/artifact receipt is docs/doctoring/tenant_provenance_integration.md; no competing canonical Gap ledger is introduced here.

Keep Draft. The GIN storage candidate does not accelerate distance-only top-k ordering; representative p95 and migration costs, NewsDOM's immutable 64MiB release/exact pin, current-head hosted Checks and qualifying review, and signed HTTP/browser restore remain open acceptance gates. Local API success tests use service overrides, so no signed end-to-end restore or full customer-exit portability is claimed. Prior numerical/ADR receipts below retain their historical scope only.

---

Current evidence — 2026-09-05

Exact head: 705d8ece2c97edc8575ea59766fd8f68bf4cdb82; direct base #1427 codex/pdf-dom-upload-64m@3a5f2b3751987fa8baaf85f96641e1659679aa80. Remains Draft/Proposed.

• Disposable PostgreSQL 16.15 + pgvector: uv run --frozen python -m pytest -q -W error -ra --tb=short tests/test_tenant_provenance_bundle.py tests/test_data_api.py returned 224 passed, 0 failed, 0 skipped in 12.53 seconds. This supersedes the earlier environment-skipped observation. The existing worktree environment retained packages outside the lock, so this is not clean-lock proof.
• Actual fresh uv run --frozen python scripts/migrate_db.py on the same source head failed, exit 1, at 0011_email_read_state: relation "emails" does not exist. The suite creates tables through ORM metadata, not that migration chain. A schema-fixture pass cannot be promoted to installability.
• The existing migration owner fix: create workspace_entities/workspace_documents registry and provision Workspace rows #1503 is now 19d5860bc27e860acba940390f5792721cd99e5e, non-force stacked on fix(test): install Starlette TestClient dependency #1565. Exact synchronization, fresh/repeated migration to 0019_email_read_state_repair, 75 strict PostgreSQL/dependency tests, Ruff, and diff checks pass there. This PR has not yet inherited that repair.
• Next: propagate owner through test(db): align PostgreSQL smoke fixtures with current schema #1468 → fix(data): align PDF DOM upload budget with sidecar #1427 → feat(data): add bounded tenant provenance portability #1497, reconcile structured bootstrap behavior while preserving provenance tests, and join the resulting Alembic heads with a forward merge revision. Then verify fresh/historical deployment, rollback, and conflicting concurrent imports with one rejected writer and no losing partial rows.
• Current-head hosted checks are queued, not passing. API success tests mock the import/export service; full signed HTTP-to-database restore, deployed runtime, release, and protected merge are unverified. No bypass or count-only PR closure.

Historical PR description and receipts — superseded where inconsistent above

The following is retained for lineage. Its older heads, counts, and readiness statements are historical, not current gate evidence.

Summary

Adds a deterministic bounded BagIt + RO-Crate 1.3 + PROV archive for the exact project-evidence closure, with transactional tenant-scoped import, portable identity remapping, conflict detection, fail-closed metadata/security validation, and authoritative signed Data API export/import endpoints.

Exact current identity

• protected base: develop@042b0c70531b229af3acbd0421a2f23098d848b3
• branch: feat/tenant-provenance-roundtrip
• exact head: 152d1998c4e8024be9dc7026c8789d343c884fd0
• mechanical mergeability: mergeable; this is not merge authorization.
• lifecycle: Draft while current-head required review controls are failing and no qualifying independent approval exists.

The latest product-source change ensures PostgreSQL pool disposal still runs if connection cleanup itself fails. All current inline Devin/CodeRabbit review threads are resolved. Historical CHANGES_REQUESTED/dismissed submissions are predecessor governance evidence only and are not treated as approval.

Security and scope

• exact user, organization, and workspace selection;
• OIDC-authoritative provenance scope only for the exposed transfer endpoints;
• portable remapping for globally unique project/content graph identities while preserving owner-scoped mail/attachment evidence;
• deterministic content-bound bundle/activity identity;
• cross-email/cross-tenant reference validation before persistence;
• bounded archive profile: 64 MiB archive, 64 entries, 32 MiB per entry, 100:1 compression ratio;
• database-side size preflight and serialized maximum-size export materialization;
• rejects non-finite JSON, sensitive nested metadata, invalid provenance references, malformed timestamps and non-authoritative sessions;
• excludes binary payloads, embeddings, credentials, provider/connector state and audit history;
• tests/real_datasets was not accessed or included.

Exact-head repository evidence

For unchanged head 152d1998c4e8024be9dc7026c8789d343c884fd0 at the latest fresh refetch:

• Application CI: completed / success;
• Security Scan: completed / success;
• Dependency Review: completed / success;
• SAST Semgrep: completed / success;
• Bandit Security Scan: completed / success;
• Build and Publish Docker Images: completed / success;
• coverage-evidence: completed / success;
• PR governance metadata controller: completed / success;
• all current inline review threads: resolved;
• qualifying formal APPROVED review: absent;
• required opencode-review check-run 99830468507: completed / failure because the authenticated dispatch never produced an opencode-agent APPROVED or CHANGES_REQUESTED verdict on this exact head before the fail-closed poll ended;
• required noema-review check-run 99928013216: completed / failure after exact-head validation, sidecar startup and orchestrator/free preflight; terminal model output was malformed JSON (Expecting ',' delimiter, response length 3540, SHA-256 2ad0e386c189243f). Raw model output was correctly not logged.

The two review-control failures are not Naruon product-source findings. Open central owner issues already cover the reproduced classes: ContextualWisdomLab/.github#1531 tracks OpenCode review queue/dispatch starvation and missing current-head verdicts; ContextualWisdomLab/.github#1637 tracks malformed/truncated Noema verdict envelopes and typed fail-closed review-unavailable handling. Protected .github/main has advanced beyond the immutable workflow sources used by these historical runs, so rerunning the old Actions jobs would re-materialize predecessor workflow code and would not validate current owner repairs.

Earlier local/backend pass counts, predecessor reviews, cancelled/skipped checks and predecessor central failures remain useful development history only and are not transferred as current merge gates.

Owner-path recovery boundary

Do not mutate Naruon product code to compensate for a canonical review-control defect. The central owner must reach protected GREEN for the failing class, after which this unchanged Naruon exact head needs a fresh trigger that materializes the then-current central workflow source. A normal merge requires fresh schema-valid OpenCode and Noema evidence on the exact current head, plus all other then-live required checks and a qualifying post-last-push independent approval.

Do not force-push, destructively rebase, self-approve, bypass branch protection, fabricate review evidence, transfer predecessor checks, broaden reviewer credentials or weaken a required gate.

Remaining product scope

Full mailbox-only mail portability, binary lifecycle, credentials, provider/connector state, embeddings, audit-history portability, and operator-safe remediation diagnostics remain explicitly outside this bounded slice unless a later accepted contract adds them.

Merge boundary

Do not merge until the unchanged exact head satisfies every then-live protected-branch/ruleset required check, fresh Noema/OpenCode current-head evidence is terminal passing, all valid review threads remain resolved, and the qualifying independent non-author approval required by live governance exists after the latest push. Failed, queued, pending, in-progress, skipped-required, absent, stale, predecessor, dismissed, model-only, status-only and author-only evidence is non-passing.

[coderabbitai]

📝 Walkthrough

Walkthrough

This change adds deterministic BagIt/RO-Crate provenance bundle export and import for scoped tenant project evidence. It adds OIDC-authoritative API routes, bounded archive validation, transactional restoration, identity mapping, conditional schema backfill handling, tests, ADR-0005, and planning updates.

Changes

Tenant provenance portability

Layer / File(s)	Summary
Archive contract and validation backend/services/tenant_provenance_bundle.py, backend/tests/test_tenant_provenance_bundle.py	Defines deterministic BagIt/RO-Crate ZIP archives, canonical JSON records, stable UIDs, metadata restrictions, ZIP safety checks, size limits, manifests, and parser validation.
Scoped export and transactional import backend/services/tenant_provenance_bundle.py, backend/db/models.py, backend/alembic/versions/0018_provenance_identity_mappings.py, backend/tests/test_tenant_provenance_bundle.py	Exports scoped project-evidence closures and imports validated records with deterministic identity mapping, scope checks, duplicate handling, advisory locking, and transactional insertion.
Signed-session API integration backend/api/data.py, backend/tests/test_data_api.py	Adds provenance bundle download and import routes with OIDC checks, bounded streaming reads, ZIP responses, import receipts, and fixed error responses.
Portability design and project records docs/adr/0005-tenant-provenance-portability.md, docs/superpowers/specs/..., docs/superpowers/plans/..., docs/planning/naruon-platform-plan.md, docs/product-technical-gap-baseline.md, CHANGELOG.md	Documents the archive contract, identity remapping, verification results, feature-gated project graph ingestion, and remaining portability scope.

Schema bootstrap compatibility

Layer / File(s)	Summary
Conditional schema backfill backend/scripts/bootstrap_db.py, backend/alembic/versions/0001_initial_control_plane.py, backend/alembic/versions/0018_provenance_identity_mappings.py, backend/tests/test_bootstrap_db.py, backend/tests/test_alembic_migrations.py	Moves backfill execution into execute_schema_backfill, conditionally creates the legacy emails index, and makes mapping-table migration operations idempotent. Tests cover migration references and index creation when the table exists.

Estimated code review effort: 5 (Critical) | ~120 minutes

Merge Risk: ⚪ Minimal · up to 00342

The change is merge-ready after normal checks and review; no actionable merge-blocking risk remains. A localized test helper should narrow its error handling so genuine database failures are not reported as skipped tests.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant DataAPI
  participant AuthContext
  participant ProvenanceBundleService
  participant PostgreSQL

  Client->>DataAPI: Request export or import
  DataAPI->>AuthContext: Resolve OIDC-authoritative scope
  AuthContext-->>DataAPI: Return user, organization, and workspace
  DataAPI->>ProvenanceBundleService: Process bounded provenance archive
  ProvenanceBundleService->>PostgreSQL: Query or restore scoped records
  PostgreSQL-->>ProvenanceBundleService: Return archive data or import receipt
  ProvenanceBundleService-->>DataAPI: Return ZIP archive or receipt
  DataAPI-->>Client: Return response

Loading

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding bounded tenant provenance portability. It matches the new export/import APIs and related implementation.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/tenant-provenance-roundtrip

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

PR governance metadata gate is not ready for 152d1998c4e8024be9dc7026c8789d343c884fd0:

• Required check opencode-review is FAILURE on the current head.

[seonghobae]

Current-head remediation evidence for 092e39383ec2e8578750456bc480f92f64e8ad0f:

• fresh-schema Alembic 0018 now guards the current-schema baseline and creates the index idempotently;
• provenance bundle APIs now require session_verifier == "oidc";
• attachment references are translated only under typed metadata keys, preserving arbitrary strings;
• mixed native/multiple-origin workspaces re-export as one target-scoped archive, while a single complete mapped origin retains portable UIDs;
• native parent-node closure and optional structural-edge endpoints are covered.

Verification: full backend with warnings as errors: 1993 passed, 3 skipped; focused mixed-origin PostgreSQL coverage and broader provenance selection passed; Ruff and git diff --check passed; independent re-review found no remaining P1/P2/P3.

[seonghobae]

@coderabbitai review

Please re-review exact current head e2968cdd96bf9faf4d263785f1a4d0bdee78dc83; the current-head database smoke-test finding is fixed.

[seonghobae]

@opencode-agent

Please review exact current head e2968cdd96bf9faf4d263785f1a4d0bdee78dc83. Merge/update actions remain disabled.

[coderabbitai]

@seonghobae I will re-review pull request #1497 at e2968cdd96bf9faf4d263785f1a4d0bdee78dc83, including the database smoke-test fix.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[seonghobae]

@coderabbitai review

Please re-review exact current head 152d1998c4e8024be9dc7026c8789d343c884fd0; both current-head PostgreSQL smoke findings are fixed.

[seonghobae]

@opencode-agent

Please review exact current head 152d1998c4e8024be9dc7026c8789d343c884fd0. Merge/update actions remain disabled.

[coderabbitai]

@seonghobae I will re-review pull request #1497 at 152d1998c4e8024be9dc7026c8789d343c884fd0, including the two PostgreSQL smoke-test fixes.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Dismissed as obsolete predecessor-head CHANGES_REQUESTED after the cited findings were addressed and their current review threads are resolved. This does not constitute approval and does not transfer any predecessor review evidence to current head 152d199.

[seonghobae]

Fresh exact-head gate correction (2026-09-03): the PR body still describes noema-review check 99928013216 as queued, but that check is now terminal failure on unchanged head 152d1998c4e8024be9dc7026c8789d343c884fd0.

Run 33462446295, attempt 3 acquired a hosted runner and successfully completed stale-trigger rejection, superseded-run cancellation, repository-scoped reviewer credential selection/minting, exact live-head validation, target visibility resolution, and contextual-orchestrator sidecar provisioning. The first failing step is Prepare Noema model verdict, which ran from 2026-09-02 03:38:53Z to 05:06:53Z and failed; publication/token-refresh steps were correctly skipped afterward. This establishes a review-control/model-verdict failure boundary, not a Naruon source-code finding by itself.

Repository-local Application CI, Security Scan, Dependency Review, SAST Semgrep, Bandit, and Docker checks on this exact head remain terminal-success. Do not mutate product code or transfer predecessor review evidence to satisfy the failed Noema control. Merge remains blocked until a fresh exact-head Noema result is terminal-valid, all then-live required checks are passing, and the required independent post-last-push approval exists.

[seonghobae]

Exact-head review evidence

Reviewed unchanged head 152d1998c4e8024be9dc7026c8789d343c884fd0 against current develop with CodeGraph call-path inspection.

• Archive framing, canonical JSON, duplicate-key rejection, member and compression bounds, tenant/workspace scope, portable identity locks, conflict preflight, and transactional import paths were inspected at the shared owner service.
• Exact hash locks from backend/requirements-hashes.txt and backend/requirements-agent.txt were installed under Python 3.14.
• Focused provenance, Data API, Alembic, and bootstrap suite: 154 passed, 117 skipped; Ruff passed.
• The skipped database paths were then rerun against a localhost-only disposable PostgreSQL 16 + pgvector container: 115 passed, 108 deselected, with warnings treated as errors.
• The test container was removed after the run.
• No unresolved review threads exist. The PR remains Draft because historical central review failures are not passing evidence and fresh protected exact-head review/check results are still required.

[seonghobae]

현재 exact head 705d8ece2c97edc8575ea59766fd8f68bf4cdb82를 #1427(3a5f2b3751987fa8baaf85f96641e1659679aa80) 위에 non-force stack하고 base를 codex/pdf-dom-upload-64m으로 retarget했습니다. 전체 순서는 #1565 → #1468 → #1427 → #1497이며, 각 선행 delta를 보존합니다.

충돌은 기존 64 MiB PDF 경계, PostgreSQL 조건부 legacy-index bootstrap, provenance portability를 함께 유지하도록 통합했습니다. 중복 ADR-0005는 ADR-0007로 바꾸고 foundation 병합 전 상태를 Proposed로 낮췄습니다. 유효 child delta는 17개 파일이며 ADR-0007만 포함합니다.

검증: 비-PostgreSQL focused suite 157 passed / 117 skipped, Ruff 통과. 격리된 실제 pgvector/pgvector:pg16 smoke는 117 passed / 135 deselected였습니다. 이 head의 Checks와 독립 리뷰가 새로 완료되기 전에는 기존 증거를 승계하지 않습니다.