Repository navigation
Refactor/db mv - #290
Refactor/db mv#290AlexAxthelm wants to merge 12 commits into
Conversation
Serve the `list` and `filter-options` endpoints from a precomputed, permission-scoped `og_field_resource_state` table instead of re-running the coalescing window (rank + pivot) on every request. `filter-options` in particular drops from a full-table ranked scan to a plain indexed DISTINCT. The read model stores one coalesced variant per resource per visibility profile, keyed by a two-bit `permission_mask` (wm | ccr); all other sources are a shared public tier, so a resource has at most four rows. It is built by reusing the existing live coalescer per mask (`utils.coalesce_resources`), so the cache is defined by the same ranking the live path uses and cannot drift. Read path: callers with a canonical visibility profile (all public sources plus any combination of wm/ccr) are served from the read model; every other caller (unscoped `None`, or a partial public grant) falls back to live coalescing, which is always correct. Wire models and REST surface are unchanged, so there is no frontend impact. The table is kept current in-band, inside the writing transaction, via app-controlled hooks on the create/attach, merge, and reprioritize paths. It is fully rebuildable from scratch (`rebuild_all_resource_state`). The Alembic migration is schema-only: running the async ORM coalescer inside a sync migration is unsafe, so backfill is an explicit post-upgrade step. Run `python -m stitch.api.db.read_model.rebuild` once after `alembic upgrade head` (and after any data restore); until then, canonical profiles see an empty list. Tests: read-model vs live parity across all four profiles and the filter/sort/ q/paging scenarios, permission masking, and sync-equals-rebuild after each mutating action. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ute store Replace og_field_source_priority (global defaults) and og_field_resource_source_priority (per-field overrides) with one table, og_field_resource_attribute_priority, at the (resource, colname, source_pk) value grain. Coalescing ranking collapses from a four-key, two-tier ROW_NUMBER() window to a single ORDER BY priority: curated rows take the low positions and outrank defaults, which follow in global SOURCE_PRIORITY order. This linearizes the previous override-then-default tiering exactly. Default rows are seeded from SOURCE_PRIORITY on create/attach/merge and curated rows written on reprioritize, all in-band; global source rank now derives from the SOURCE_PRIORITY constant (db.source_priority) rather than a lookup table. The membership FK to og_field_source_priority is dropped; source-key validity is already guaranteed by the OGSISrcKey enum. The migration backfills the new table (linearizing the old two tiers via a window function) and drops both old tables. Behavior is unchanged: the existing override/default coalescing tests now exercise the single-table ranking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-0290-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-0290-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-290.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-0290-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0290 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0290 |
Jobs (2)
| job | image | postgres_db | api_url | auth_mode |
|---|---|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0290@sha256:826e22124d886642b3c602128caa6ffaf591c8474f3a7654b4ddf850dd327a1a |
pr_0290 |
||
| seed | ghcr.io/rmi/stitch-seed:pr-0290@sha256:ff2982a48de38006ba7b40471d6d53a9c5af9442cf906be95d47b5596d939e59 |
https://pr-0290-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io/api/v1 |
stitch-client-bearer-token |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-09-25T17:02:31Z | 2026-09-25T17:02:08Z | 953f480 | ghcr.io/rmi/stitch-api:pr-0290 |
ghcr.io/rmi/stitch-api:pr-0290@sha256:826e22124d886642b3c602128caa6ffaf591c8474f3a7654b4ddf850dd327a1a |
| 2026-09-25T17:02:28Z | 2026-09-25T17:02:08Z | 953f480 | ghcr.io/rmi/stitch-entity-linkage:pr-0290 |
ghcr.io/rmi/stitch-entity-linkage:pr-0290@sha256:4452a57f3a85fea8c45c4ee804ea87613a5e73dce2a96dd38d7b43cc2aebc83d |
| 2026-09-25T17:02:28Z | 2026-09-25T17:02:08Z | 953f480 | ghcr.io/rmi/stitch-seed:pr-0290 |
ghcr.io/rmi/stitch-seed:pr-0290@sha256:ff2982a48de38006ba7b40471d6d53a9c5af9442cf906be95d47b5596d939e59 |
| 2026-09-25T17:02:28Z | 2026-09-25T17:02:08Z | 953f480 | ghcr.io/rmi/stitch-stitch-llm:pr-0290 |
ghcr.io/rmi/stitch-stitch-llm:pr-0290@sha256:0fe23424b6344482b0a65e1a0035f5f1a1372051d61bea85fa9c430ecb969079 |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The migrations can fail or expose empty production reads, and concurrent writes can corrupt derived-state maintenance.
Review effort: Balanced
Findings: 2
Open (4)
What changed in this PR
Introduces a permission-scoped resource-state read model and consolidates source-ranking data into one per-attribute priority store.
Changes:
- Adds precomputed list/filter state for four permission profiles.
- Replaces global and override priority tables with one priority model.
- Updates migrations, write hooks, and parity/synchronization tests.
| File | Description |
|---|---|
PLAN.md |
Documents the refactor design and rollout. |
deployments/api/tests/routers/test_merge_candidates_integration.py |
Updates merge priority assertions. |
deployments/api/tests/routers/test_licensed_sources_routes.py |
Seeds priorities in direct fixtures. |
deployments/api/tests/observability/test_query_name_actions.py |
Updates priority setup and query labels. |
deployments/api/tests/db/test_source_priority_seed.py |
Tests constant-derived source ranking. |
deployments/api/tests/db/test_resource_state_read_model.py |
Tests parity, permissions, and synchronization. |
deployments/api/tests/db/test_resource_actions.py |
Updates priority behavior tests. |
deployments/api/tests/conftest.py |
Removes obsolete priority-table seeding. |
deployments/api/src/stitch/api/db/source_priority.py |
Implements global rank helpers. |
deployments/api/src/stitch/api/db/read_model/state.py |
Builds and refreshes resource state. |
deployments/api/src/stitch/api/db/read_model/rebuild.py |
Adds offline rebuild command. |
deployments/api/src/stitch/api/db/read_model/permissions.py |
Encodes permission masks. |
deployments/api/src/stitch/api/db/read_model/__init__.py |
Defines the read-model package. |
deployments/api/src/stitch/api/db/queries.py |
Uses consolidated priority rows. |
deployments/api/src/stitch/api/db/priorities.py |
Maintains default and curated priorities. |
deployments/api/src/stitch/api/db/og_field_source_actions.py |
Refreshes derived data after attachment. |
deployments/api/src/stitch/api/db/og_field_resource_actions.py |
Serves cached reads and maintains state. |
deployments/api/src/stitch/api/db/model/resource.py |
Uses constant-derived source ordering. |
deployments/api/src/stitch/api/db/model/og_field_source_priority.py |
Removes the global priority model. |
deployments/api/src/stitch/api/db/model/og_field_resource_state.py |
Defines stored resource state. |
deployments/api/src/stitch/api/db/model/og_field_resource_source_priority.py |
Removes the override model. |
deployments/api/src/stitch/api/db/model/og_field_resource_attribute_priority.py |
Defines consolidated priorities. |
deployments/api/src/stitch/api/db/model/membership.py |
Removes the obsolete source FK. |
deployments/api/src/stitch/api/db/model/__init__.py |
Exports replacement models. |
deployments/api/src/stitch/api/db/merge_candidate_actions.py |
Uses canonical source ordering directly. |
deployments/api/alembic/versions/c2d3e4f5a6b7_single_attribute_priority_table.py |
Migrates priority storage. |
deployments/api/alembic/versions/b1f2c3d4e5a6_add_resource_state_read_model.py |
Creates the read-model schema. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Address code-review findings on the read-model refactor: - Read path falls back to live coalescing when og_field_resource_state is unpopulated (list: only on an empty result; filter-options: up front), so a canonical caller is never served an empty page before the post-migration rebuild has run. An un-rebuilt table now degrades to correct-but-slower. - rebuild_all_resource_state wipes with TRUNCATE on PostgreSQL (no dead-tuple churn) and reserves the full rebuild for bootstrap / coalescing-logic changes; steady-state drift stays incremental via the in-band refresh hooks. - Add targeted rebuilds so recovery need not touch the whole table: refresh_resource_states(ids) and refresh_changed_since(ts), exposed on the rebuild CLI as --ids and --since. - Priority-table migration: discover the membership FK by its referenced table (confrelid) instead of a fragile constraint-def LIKE, and drop all matching FKs, so the og_field_source_priority drop can't fail on a DB-generated name. Tests: fallback-when-unpopulated (query + filter-options), targeted refresh_resource_states scope, and refresh_changed_since bounds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve code-review and Copilot findings on the read-model refactor: - Empty-state fallback: reads fall back to live coalescing when og_field_resource_state is unpopulated (list only on an empty result; filter-options up front), so a canonical caller is never served an empty page before the post-migration rebuild has run. Un-rebuilt = correct-but-slower. - rebuild_all_resource_state wipes with TRUNCATE on PostgreSQL (no dead-tuple churn); full rebuild is reserved for bootstrap / coalescing-logic changes. - Targeted rebuilds so recovery need not touch the whole table: refresh_resource_states(ids) and refresh_changed_since(ts), exposed on the rebuild CLI as --ids / --since. - Concurrency: take a per-resource row lock (SELECT ... FOR UPDATE, no-op on SQLite) at the start of every derived-state rebuild path (attach, reprioritize, merge — originals locked in sorted order) so concurrent mutations of one resource can't interleave their delete+reinsert and collide. - Migration: use CURRENT_TIMESTAMP (not now(), unsupported on SQLite) in the priority backfill; discover the membership FK by its referenced table and drop all matches, so dropping og_field_source_priority can't fail on a generated name. Tests: read-model fallback (query + filter-options), targeted refresh scope, refresh_changed_since bounds, and JSON (owners/operators) + numeric round-trip through the typed-column store. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CD summary
|
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0290 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0290 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-09-28T15:07:13Z | 2026-09-28T15:06:57Z | 8927530 | ghcr.io/rmi/stitch-api:pr-0290 |
ghcr.io/rmi/stitch-api:pr-0290@sha256:5520251dd215ce66485cb4ec2e960dc17388293858d1af0b00a32c6b32edd045 |
| 2026-09-28T15:07:16Z | 2026-09-28T15:06:57Z | 8927530 | ghcr.io/rmi/stitch-entity-linkage:pr-0290 |
ghcr.io/rmi/stitch-entity-linkage:pr-0290@sha256:4f13ade190093f69a789c79dd0b444c993ca3dac5c03e550b890fdac08ea4a69 |
| 2026-09-28T15:07:17Z | 2026-09-28T15:06:57Z | 8927530 | ghcr.io/rmi/stitch-seed:pr-0290 |
ghcr.io/rmi/stitch-seed:pr-0290@sha256:56471f2442d99a535aec18d8683db330a93ac3d09b0c0ed8e2f386a1651415ca |
| 2026-09-28T15:07:13Z | 2026-09-28T15:06:57Z | 8927530 | ghcr.io/rmi/stitch-stitch-llm:pr-0290 |
ghcr.io/rmi/stitch-stitch-llm:pr-0290@sha256:385daba8ce29891257ea0d52caabbfb7a76e9ebeecac2d7d438ff063caf9aaba |
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-0290-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-0290-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-290.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-0290-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0290 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0290 |
Jobs (1)
| job | image | postgres_db |
|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0290@sha256:2aac5b7932ee32a16bf6faef596751d7cc6033c570dfe694c76ea31d3cf95f60 |
pr_0290 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-09-30T10:12:43Z | 2026-09-30T10:12:25Z | dc7b588 | ghcr.io/rmi/stitch-api:pr-0290 |
ghcr.io/rmi/stitch-api:pr-0290@sha256:2aac5b7932ee32a16bf6faef596751d7cc6033c570dfe694c76ea31d3cf95f60 |
| 2026-09-30T10:12:43Z | 2026-09-30T10:12:25Z | dc7b588 | ghcr.io/rmi/stitch-entity-linkage:pr-0290 |
ghcr.io/rmi/stitch-entity-linkage:pr-0290@sha256:1537f8d81d9392e6b18d46d697fe153ef87028999bfc3d9cccaf74911efefda8 |
| 2026-09-30T10:12:45Z | 2026-09-30T10:12:25Z | dc7b588 | ghcr.io/rmi/stitch-seed:pr-0290 |
ghcr.io/rmi/stitch-seed:pr-0290@sha256:9082fcc8014f9f6b8dec5bb5ca386c2965bb47c0066bf36b2aeaa8028a4c5ee2 |
| 2026-09-30T10:12:43Z | 2026-09-30T10:12:25Z | dc7b588 | ghcr.io/rmi/stitch-stitch-llm:pr-0290 |
ghcr.io/rmi/stitch-stitch-llm:pr-0290@sha256:7f9c2a398f625ff9b42f96b9f8e14f56ae82c499362446ec9a633ec8bf673dbe |
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-0290-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-0290-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-290.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-0290-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0290 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0290 |
Jobs (1)
| job | image | postgres_db |
|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0290@sha256:2d751f130ba50beee7be5c086b99bd513fabd43491c5cae4e6af85b44e52fca6 |
pr_0290 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-10-07T11:51:00Z | 2026-10-07T11:50:06Z | 3879033 | ghcr.io/rmi/stitch-api:pr-0290 |
ghcr.io/rmi/stitch-api:pr-0290@sha256:2d751f130ba50beee7be5c086b99bd513fabd43491c5cae4e6af85b44e52fca6 |
| 2026-10-07T11:51:02Z | 2026-10-07T11:50:06Z | 3879033 | ghcr.io/rmi/stitch-entity-linkage:pr-0290 |
ghcr.io/rmi/stitch-entity-linkage:pr-0290@sha256:621dd378e502374ebc0f3b76ba7b86e08c743fa6349a88cfe18776bd85eb1064 |
| 2026-10-07T11:50:57Z | 2026-10-07T11:50:06Z | 3879033 | ghcr.io/rmi/stitch-seed:pr-0290 |
ghcr.io/rmi/stitch-seed:pr-0290@sha256:93626ec70391ff3c852887f49635bec9c5f8a7fdff4d77e7cd87836639d0f626 |
| 2026-10-07T11:50:59Z | 2026-10-07T11:50:06Z | 3879033 | ghcr.io/rmi/stitch-stitch-llm:pr-0290 |
ghcr.io/rmi/stitch-stitch-llm:pr-0290@sha256:797e2495d6e774125f7c7ab48031b211cee2befe6d1102f59c9d56d2848f8e52 |
Memberships had only their PK indexed, so every read that scans a resource's active memberships (single-resource source_data join, coalescing universe filter on resource_id + status=ACTIVE) did a full-table seq scan. Add a composite index; load testing showed ~17ms -> 1.4ms on that query and fixed a detail-endpoint regression (35ms -> 10ms). General win independent of the read-model work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-0290-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-0290-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-290.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-0290-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0290 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0290 |
Jobs (1)
| job | image | postgres_db |
|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0290@sha256:7747fd80eb49e0580609ea5d43d7fc3b8af9a731ddb598eeed3ec88939b525e6 |
pr_0290 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-10-08T14:40:40Z | 2026-10-08T14:40:23Z | c164a23 | ghcr.io/rmi/stitch-api:pr-0290 |
ghcr.io/rmi/stitch-api:pr-0290@sha256:7747fd80eb49e0580609ea5d43d7fc3b8af9a731ddb598eeed3ec88939b525e6 |
| 2026-10-08T14:40:45Z | 2026-10-08T14:40:23Z | c164a23 | ghcr.io/rmi/stitch-entity-linkage:pr-0290 |
ghcr.io/rmi/stitch-entity-linkage:pr-0290@sha256:c93af4f186620806281099d212eea27e900c6849dacffab1d8f93aeaaa22ce76 |
| 2026-10-08T14:40:40Z | 2026-10-08T14:40:23Z | c164a23 | ghcr.io/rmi/stitch-seed:pr-0290 |
ghcr.io/rmi/stitch-seed:pr-0290@sha256:cb4f32f9447a5b57ba02c6b71030755517253d3e9a9c7fc0538786fe2998f6d6 |
| 2026-10-08T14:40:43Z | 2026-10-08T14:40:23Z | c164a23 | ghcr.io/rmi/stitch-stitch-llm:pr-0290 |
ghcr.io/rmi/stitch-stitch-llm:pr-0290@sha256:d25b40b33619fe4fbc323de3c8bb2b0050603c935d91fcbaf6b490901d5965ba |


Implements designs from PR #279, resulting in big performance increase.
refactor(db): precomputed current-state read model + single priority store (STIT-766)
Relates to STIT-766 (epic STIT-689).
Summary
The
list,filter-options, anddetailendpoints rebuild the same coalesced "current state" on every request — a 5-table join →ROW_NUMBER()winner-pick → pivot that scales ~linearly with data and dominates latency. This PR:og_field_resource_state, a precomputed, permission-scoped table (one pre-pivoted winner row per resource per permission profile —wm/ccrmask, ≤4 today) that serveslistandfilter-options. Non-canonical callers (or an unpopulated table) fall back to live coalescing, so results are always correct.og_field_source_priority(global defaults) +og_field_resource_source_priority(per-field overrides) into oneog_field_resource_attribute_prioritytable, turning the two-tierNULLS LASTranking window into a singleORDER BY priority.Public REST surface and response models are unchanged — only how the responses are produced. No frontend changes.
Performance (local, 50k resources ≈ prod scale,
listmedian vsmain)filter-options: ~10–21× faster (indexedDISTINCTper mask vs a full-table ranked scan).mainat 1k/10k across all profiles. (A 2-row seed drift observed at 50k was a dataset artifact, not a coalescing difference.)Measured with the loadtest harness (deterministic multi-source dataset, real tokens for the 4 permission profiles, single-stream, 1k/10k/50k volume ladder; baseline = merge-base).
Behavior preservation & testing
list/filter-options/GET {id}equal live coalescing across all four permission profiles and filter/sort/q/paging scenarios.owners/operators) and numeric fields; seeding-completeness test; membership-uniqueness + merge-dedup tests; Phase B ranking-parity via the existing override/default/dup-source coalescing tests.Deploy / ops
pr_0290).alembic upgrade headfor the perf win:--ids,--since); steady-state stays current in-band via write hooks.wm,ccr).Review findings addressed
All Copilot findings (C1 empty-state, C2
now()/SQLite, C3 membership-grain FK, C4 concurrency lock) and internal review findings resolved: live fallback +TRUNCATErebuild + targeted refresh; cross-dialect timestamps;UNIQUE(resource_id, source_pk)+ dedup + composite priority→membership FK; per-resourceFOR UPDATEon mutating paths;set_curatedinput guard; scoped attach seeding.AI assistance
Implemented with Claude Code (read model, priority-store collapse, migrations, tests). Behavior is validated by the parity/sync/round-trip/completeness test suites above; performance by the loadtest harness. Reviewers should still verify the migration and coalescing logic.
Reviewer notes / open questions
resource_id-scoped membership index is a general win worth folding into a migration here.