From 91b1223d2b19bac720634372e67920ac4999664d Mon Sep 17 00:00:00 2001 From: Michael Barlow Date: Thu, 17 Sep 2026 20:25:49 -0600 Subject: [PATCH 1/8] initial draft --- docs/STIT-766-design.md | 307 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 307 insertions(+) create mode 100644 docs/STIT-766-design.md diff --git a/docs/STIT-766-design.md b/docs/STIT-766-design.md new file mode 100644 index 00000000..b146a0a7 --- /dev/null +++ b/docs/STIT-766-design.md @@ -0,0 +1,307 @@ +# STIT-766 — Current-state read model: design proposal + +**Status:** draft +**Author:** Michael Barlow +**Reviewer:** John McGrath +**Jira:** [STIT-766](https://rmi1.atlassian.net/browse/STIT-766) (epic [STIT-689](https://rmi1.atlassian.net/browse/STIT-689)) +**Created**: 2026-09-17 + +## 1. Objective + +Design a database architecture and application logic to serve the key Stitch endpoints with lower latency. + +## 2. Background + +### Problem + +The `list`, `filter-options`, and `detail` endpoints all rebuild the same thing on +every request: one coalesced, flat row per resource that represents the current top-level "state". +`queries.py` does this by joining membership → resource → default priority → source → value rows, +left-joining per-field overrides, ranking candidates with a four-key `ROW_NUMBER()` +window, and then pivoting the winners into columns with `max(case(...))` per field. + +Combined with serializing to/from Pydantic models, this has led to poor response times that +negatively impact user experience. Furthermore, given that our entire db is less than 100MB +in size, this raises questions about our overall db design and access patterns. + +### Observations + +With the newly added attribute-level priority feature ([STIT-494](https://rmi1.atlassian.net/browse/STIT-494)), we now +have a more deterministic and bounded way to establish the different variations we +display for a flat resource. Our permissions model only has 2 restricted options: `wm` and `cc`. +All the remaining sources are public, and ALL users have read permissions for them. + +> [!NOTE] +> It was more straightforward to treat these individually when implemented, but reviewing our permissions model +> may help simplify in the future. However, it's not strictly necessary for this work. + +If we think of the set of `llm`, `rmi`, `bc`, `alb`, and `gem` as a single `public` visibility. +Then there are some fairly significant implications in how our coalescing logic actually +plays out. For any given attribute on a coalesced resource, users will see at most 3 +possible options: `wm`, `cc`, and the highest priority `public` source. To illustrate, +see the following scenarios for a resource's `latitude` data: + +- priorities: `wm`, `cc`, `gem`, `bc` + - user with `wm + cc` sees `wm` + - user with `wm` sees `wm` + - user with `cc` sees `cc` + - remaining users see `gem` +- priorities: `wm`, `cc`, `gem`, `rmi`, `llm`, `alb` + - user with `wm + cc` sees `wm` + - user with `wm` sees `wm` + - user with `cc` sees `cc` + - remaining users see `gem` +- priorities: `cc`, `gem`, `wm`, `bc` + - user with `wm + cc` sees `cc` + - user with `cc` sees `cc` + - all remaining users (including `wm` viewers) see `gem` +- priorities: `bc`, `cc`, `wm`, `gem` + - all users see `bc` +- priorities: `bc`, `gem`, `cc`, `wm` + - all users see `bc` + +For any view or operation that interacts only with the flattened representation of a resource: + +- we only need to store/cache up to the first public source in the priority list +- a resource attribute has at most 3 variations: `wm`, `cc`, and `public` +- an entire coalesced resource has **at most 4 possible variations** depending on user permissions + and attribute priorities: + - `public`: all attributes have a public source as the highest priority + - `public + wm`: 1 or more attributes has `wm` as highest priority and all next-highest priorities are public + - said another way, all top 2 priorities are either `wm` or `public` and at least 1 is `wm` + - `public + cc`: 1 or more attributes has `cc` as highest priority and all next-highest priorities are public + - all top 2 priorities are either `cc` or `public` and at least 1 is `cc` + - `public + wm + cc`: at least 1 attribute where `wm` is highest **AND** 1 attribute where `cc` is highest + - Note: `wm + cc` must see different data from both `wm` and `cc` individually + +Thus, whatever structure we use here is bounded by at most 4 x the number of top-level (unmerged) resources: 1 variant for each +possible licensed representation. + +> [!NOTE] +> One caveat here is that the number of possible variants grows exponentially with the number of distinct licenses: +> +> - 2 licensed sources = 4 variants +> - 3 licensed sources = 8 variants +> - 4 licensed sources = 16 variants +> +> so even if we add 2 proprietary sources, this structure grows to a max of 16 x the number of top-level (unmerged) resources. +> I don't see this as a real issue even in the long term. How many licensed sources are even realistic? 6? 10? Even at 10 +> proprietary sources, the view/table would be 2^10 = 1024 x resources. The upper bound of **all** named fields is probably +> somewhere in the ~65k range, so ~65M rows is still manageable...if we'd ever even get close to that. + +## 2. Goals and non-goals + +**Goals** + +- Serve `list` and `filter-options` with <100ms latency +- Give the database one unambiguous stored answer to "what is the value for each attribute of a resource for a user with X permissions" +- Express the invariants declaratively in schema, so the sync/refresh mechanism has lower/zero risk regarding correctness +- Preserve the public REST surface and today's behavior + +**Non-goals** + +- A general permission-based solution that accounts for all future/unknown licensed and public permissions +- The details of the sync mechanism. See §6. +- Redesigning the permission model. + +## 3. Proposal + +Part of the goal is to trade application complexity for schema complexity. Making schemas more complex +but in a way that enables tighter constraints and simpler queries (i.e. through mostly joins) frees +out application code from being the enforcer of invariants. + +### Attributes table + +Add `og_field_attributes` table: + +- id: primary key +- name: attribute name +- (optional) description +- (optional) type: text, int, float, json, etc... + +Priority rows and EAV rows reference `og_field_attribute.id` + +### Single priority table + +Replace the two-table priority split with a single table at the +`(resource, attribute, source record)` grain, and use defaults when +writing new data rather than as a SQL fallback rule. + +- Add `og_field_resource_attribute_priority` table. + - resource_id: fk to resources + - attribute_id: fk to `og_field_attributes` + - source_id: fk to `oil_gas_field_sources` + - priority: int + - is_curated: bool (false if default, true if set by user) +- Drop `og_field_source_priority` and `og_field_resource_source_priority`. +- On resource or source creation, write explicit priority rows for every attribute + the source has a value for, seeded from the existing `SOURCE_PRIORITY` tuple in + `stitch.ogsi.model`. +- Add constraints: + - unique priority across resource and attribute: no 2 sources for a resource + attribute can have the same priority + `UNIQUE (resource_id, attribute_id, priority)` + - priority > 0 + - fk to memberships on resource_id, source_id: ensure source is attached to resource + - fk to `` + +### Benefits + +#### Simpler, faster SQL + +Provenance for a single resource is a series of joins. + +```sql +SELECT + p.resource_id, + a.name AS attribute, + p.source_id, + sk.key_name AS source_key, + p.priority, + v.value_text, + v.value_num, + v.value_json +FROM og_field_resource_attribute_priority AS p +JOIN oil_gas_field_source_values AS v + ON v.source_id = p.source_id + AND v.attribute_id = p.attribute_id +JOIN og_field_attributes AS a + ON a.attribute_id = p.attribute_id +JOIN oil_gas_field_sources AS s + ON s.source_id = p.source_id +JOIN source_key AS sk + ON sk.source_key_id = s.source_key_id +JOIN user_source_key_permission AS permission + ON permission.source_key_id = s.source_key_id + AND permission.user_id = $1 + AND permission.action = 'view' +WHERE p.resource_id = $2 +ORDER BY p.priority, p.source_id; +``` + +| Today | After | +| ----------------------------------------- | ---------------------------------------------------- | +| Tiered `ROW_NUMBER()` over four sort keys | `ORDER BY priority` on one column | +| Curated and default rows form two tiers | Every candidate has an explicit row; no tiers | +| NULL values skipped by the query | A priority row exists only where a value exists (FK) | +| Complex CTE chain | Joins | + +Two consequences to handle in the migration: + +- `MembershipModel.source` and the override table's `source` column currently FK to + `og_field_source_priority.source`. Dropping that table needs a new target for + that closed set (a `og_field_source_keys` table, or a CHECK). +- `og_field_memberships` has a surrogate `id` primary key and no unique constraint + on `(resource_id, source_pk)`. Adding one is a prerequisite for the priority + table to enforce "the source is attached to this resource" by foreign key. +- `is_override` is part of the public per-field source-values response and is + currently derived from "an override row exists." With every row explicit, it + needs an explicit `is_curated` flag to survive. + +## 4. Proposal, part 2 — a flattened per-profile state table + +`og_field_resource_state`, one row per `(resource_id, profile)`: + +- one typed column per `OilGasFieldBase` attribute, following `ATTRIBUTE_KINDS` +- `provenance jsonb` — `{attribute_name: source_key}` for the winning source +- primary key `(resource_id, profile)` + +**Why four rows per resource is enough.** Every registered user is granted read on +the five public sources (`gem`, `rmi`, `llm`, `alb`, `bc`) by our Auth0 tenant +policy. The only variables are `wm` and `ccr`, so a caller's visibility is two +bits and the profile domain is exactly: + +| Profile | `wm` | `ccr` | +| --------------- | ---- | ----- | +| `public` | no | no | +| `public_wm` | yes | no | +| `public_ccr` | no | yes | +| `public_wm_ccr` | yes | yes | + +Within one attribute, any candidate ranked below the highest-priority _public_ +candidate can never win for anyone, so only candidates ranked above it matter, and +those are `wm` or `ccr` rows. Most attributes therefore produce the same value in +all four rows; only an attribute where a licensed source outranks the best public +source differs. A resource whose `name_local` is ordered `wm, ccr, gem` and whose +other attributes are ordered `gem, wm, ccr` differs across profiles in +`name_local` alone. + +Rows exist even when every attribute is null for that profile, which is what +preserves the null-shell behavior of `_resource_universe()`. + +## 5. Invariants + +The point of §3 and §4 is to push these into the schema. Being explicit about which +ones the database can enforce and which remain obligations on the sync is what +makes the sync mechanism a free choice. + +**Enforced declaratively** + +| # | Invariant | Mechanism | +| --- | ---------------------------------------------------------------------------- | ----------------------------------------------------- | +| 1 | A priority row references a source attached to that resource | composite FK to membership `(resource_id, source_pk)` | +| 2 | A priority row references a populated `(source, attribute)` value | composite FK to the value table | +| 3 | Two sources cannot share a priority position for one `(resource, attribute)` | UNIQUE `(resource_id, attribute_id, priority)` | +| 4 | A source appears at most once per `(resource, attribute)` | primary key | +| 5 | `priority >= 0` | CHECK | +| 6 | Attribute references come from the closed attribute set | FK to `og_field_attributes` | +| 7 | One state row per `(resource, profile)` | primary key | +| 8 | `profile` is one of the four values | CHECK or enum | +| 9 | A state row references an existing resource | FK | + +**Sync obligations** (not expressible as row constraints; a trigger or a +consistency check job, decided with §6) + +| # | Obligation | +| --- | ---------------------------------------------------------------------------- | +| A | Only resources with `repointed_id IS NULL` appear in the state table | +| B | All four profile rows exist for every resource in the state table | +| C | A provenance entry exists exactly when the matching value column is non-null | +| D | A provenance source key is visible in that row's profile | + +Obligation C is expressible as a jsonb CHECK in Postgres but not in SQLite, so +whether to enforce it in the schema depends on the STIT-603 outcome. + +## 6. Refresh + +Deferred by design. Any implementation must uphold §5 and must be triggered by: +a source value write, a priority change, a membership status change, a resource +repoint (merge), and a bulk ETL load. + +The candidate mechanisms are application-transaction maintenance plus a full +rebuild command, database triggers, or a scheduled rebuild. The choice turns +partly on an open question: the ETL apps live in a separate repository and may +write source data without passing through the API's write paths. + +## 7. Tradeoffs and risks + +- **The four-profile bound rests on the Auth0 policy, not the schema.** + `permissions.licensed_sources()` derives the set from individual + `source:read:` claims, so a partial public grant is representable. If one + were ever issued, that user would read values from a source they are not + licensed for. Mitigation: map a caller's claims to a profile at request time by + checking only `wm` and `ccr`, and fail loudly on an unmappable set. +- **`detail` is only half solved.** Its `source_data` payload needs every + candidate row, not just winners, so the candidate query stays on that path. +- **Write amplification.** One source value write can rewrite up to four state + rows. +- **Two-table schema changes.** Adding a field to `OilGasFieldBase` becomes a + migration against both `og_field_attributes` and the state table's columns, + where today it is a change to `ATTRIBUTE_KINDS` and a CHECK. +- **Priority row volume.** Rows scale as resources × sources × populated + attributes rather than the current handful. Needs an estimate before approval. +- **`og_field_attributes` is a second source of truth** for the attribute set, + alongside `OilGasFieldBase`. It needs a startup drift guard like the existing + `OGFieldName` check. + +## 8. Open questions + +1. Priority row count at current and projected resource volume. +2. Should `filter-options` read `SELECT DISTINCT` from the state table, or does it + warrant its own small per-profile distinct-values table? +3. Do the ETL apps write source data directly to the database, bypassing the API? +4. Migration and backfill sequencing: can the state table be built and validated + against the live query path before any endpoint reads from it? +5. Does anything depend on priority numbers being globally unique per source, as + `og_field_source_priority.priority` is today? + +## Related Documents and Links From 3bbbe6a4dae5aa19e7504f69038a5cdefdeb8148 Mon Sep 17 00:00:00 2001 From: Michael Barlow Date: Thu, 17 Sep 2026 20:29:50 -0600 Subject: [PATCH 2/8] structural revisions, and wording --- docs/STIT-766-design.md | 199 ++++++++++++++++++++++++---------------- 1 file changed, 119 insertions(+), 80 deletions(-) diff --git a/docs/STIT-766-design.md b/docs/STIT-766-design.md index b146a0a7..6121f2f4 100644 --- a/docs/STIT-766-design.md +++ b/docs/STIT-766-design.md @@ -8,17 +8,12 @@ ## 1. Objective -Design a database architecture and application logic to serve the key Stitch endpoints with lower latency. - +Design a database architecture and application logic to serve the key Stitch endpoints with minimal latency. ## 2. Background ### Problem -The `list`, `filter-options`, and `detail` endpoints all rebuild the same thing on -every request: one coalesced, flat row per resource that represents the current top-level "state". -`queries.py` does this by joining membership → resource → default priority → source → value rows, -left-joining per-field overrides, ranking candidates with a four-key `ROW_NUMBER()` -window, and then pivoting the winners into columns with `max(case(...))` per field. +The `list`, `filter-options`, and `detail` endpoints all rebuild the same thing on every request: one coalesced, flat row per resource that represents the current top-level "state". `queries.py` does this by joining membership → resource → default priority → source → value rows, left-joining per-field overrides, ranking candidates with a four-key `ROW_NUMBER()` window, and then pivoting the winners into columns with `max(case(...))` per field. Combined with serializing to/from Pydantic models, this has led to poor response times that negatively impact user experience. Furthermore, given that our entire db is less than 100MB @@ -26,14 +21,10 @@ in size, this raises questions about our overall db design and access patterns. ### Observations -With the newly added attribute-level priority feature ([STIT-494](https://rmi1.atlassian.net/browse/STIT-494)), we now -have a more deterministic and bounded way to establish the different variations we -display for a flat resource. Our permissions model only has 2 restricted options: `wm` and `cc`. -All the remaining sources are public, and ALL users have read permissions for them. +With the newly added attribute-level priority feature ([STIT-494](https://rmi1.atlassian.net/browse/STIT-494)), we now have a more deterministic and bounded way to establish the different variations we display for a flat resource. Our permissions model only has 2 restricted options: `wm` and `cc`. All the remaining sources are public, and ALL users have read permissions for them. > [!NOTE] -> It was more straightforward to treat these individually when implemented, but reviewing our permissions model -> may help simplify in the future. However, it's not strictly necessary for this work. +> It was more straightforward to treat these individually when implemented, but reviewing our permissions model may help simplify in the future. However, it's not strictly necessary for this work. If we think of the set of `llm`, `rmi`, `bc`, `alb`, and `gem` as a single `public` visibility. Then there are some fairly significant implications in how our coalescing logic actually @@ -64,8 +55,7 @@ For any view or operation that interacts only with the flattened representation - we only need to store/cache up to the first public source in the priority list - a resource attribute has at most 3 variations: `wm`, `cc`, and `public` -- an entire coalesced resource has **at most 4 possible variations** depending on user permissions - and attribute priorities: +- an entire coalesced resource has **at most 4 possible variations** depending on user permissions and attribute priorities: - `public`: all attributes have a public source as the highest priority - `public + wm`: 1 or more attributes has `wm` as highest priority and all next-highest priorities are public - said another way, all top 2 priorities are either `wm` or `public` and at least 1 is `wm` @@ -74,8 +64,7 @@ For any view or operation that interacts only with the flattened representation - `public + wm + cc`: at least 1 attribute where `wm` is highest **AND** 1 attribute where `cc` is highest - Note: `wm + cc` must see different data from both `wm` and `cc` individually -Thus, whatever structure we use here is bounded by at most 4 x the number of top-level (unmerged) resources: 1 variant for each -possible licensed representation. +Thus, whatever structure we use here is bounded by at most 4 x the number of top-level (unmerged) resources: 1 variant for each possible licensed representation. > [!NOTE] > One caveat here is that the number of possible variants grows exponentially with the number of distinct licenses: @@ -84,19 +73,17 @@ possible licensed representation. > - 3 licensed sources = 8 variants > - 4 licensed sources = 16 variants > -> so even if we add 2 proprietary sources, this structure grows to a max of 16 x the number of top-level (unmerged) resources. -> I don't see this as a real issue even in the long term. How many licensed sources are even realistic? 6? 10? Even at 10 -> proprietary sources, the view/table would be 2^10 = 1024 x resources. The upper bound of **all** named fields is probably -> somewhere in the ~65k range, so ~65M rows is still manageable...if we'd ever even get close to that. +> so even if we add 2 proprietary sources, this structure grows to a max of 16 x the number of top-level (unmerged) resources. I don't see this as a real issue even in the long term. How many licensed sources are even realistic? 6? 10? Even at 10 proprietary sources, the view/table would be 2^10 = 1024 x resources. The upper bound of **all** named fields is probably somewhere in the ~65k range, so ~65M rows is still manageable...if we'd ever even get close to that. ## 2. Goals and non-goals **Goals** -- Serve `list` and `filter-options` with <100ms latency -- Give the database one unambiguous stored answer to "what is the value for each attribute of a resource for a user with X permissions" -- Express the invariants declaratively in schema, so the sync/refresh mechanism has lower/zero risk regarding correctness -- Preserve the public REST surface and today's behavior +1. Serve `list` and `filter-options` with <100ms latency +2. Give the database one unambiguous stored answer to "what is the value for each attribute of a resource for a user with X permissions" +3. As a follow-up to 2, provide one unambiguous stored representation for each resource for each effective user permission set. +4. Express the invariants declaratively in schema, so the sync/refresh mechanism has lower/zero risk regarding correctness +5. Preserve the public REST surface and today's behavior **Non-goals** @@ -104,45 +91,128 @@ possible licensed representation. - The details of the sync mechanism. See §6. - Redesigning the permission model. -## 3. Proposal +## 3. Schema Changes + +Part of the goal is to trade application complexity for schema complexity. Making schemas more complex but in a way that enables tighter constraints and simpler queries (i.e. through mostly joins) frees our application code from being the enforcer of invariants. + +### Resource State View/Table +The purpose is to provide a durable store that houses the flattened/coalesced resources. We're effectively precomputing the coalescing logic and saving it to a single table. The actual schema is less important than its function and the constraints we place on it. -Part of the goal is to trade application complexity for schema complexity. Making schemas more complex -but in a way that enables tighter constraints and simpler queries (i.e. through mostly joins) frees -out application code from being the enforcer of invariants. +It must: +- only store unmerged resources (i.e. where `repointed_id == NULL`), merging triggers deletion from the table +- provide highly performant, permission-scoped querying of resource data +- not expose proprietary information to unlicensed users +- provide resource representations that are consistent with existing coalescing logic +- be able to be rebuilt from scratch at any time, we should be able to derive the data for the table easily & quickly -### Attributes table +The top contender for a schema is: +```sql +CREATE TABLE og_field_resource_view ( + resource_id bigint not null, + permission_mask smallint not null, + record jsonb not null, + + -- cannot have the same resource with the same permission + primary key (resource_id, permission_mask), + + constraint og_field_resource_view + foreign key (resource_id) + references og_field_resources(id) +) +``` +The one slight drawback is that it's slightly tricky to maintain that we only retain `resource_id` values that correspond to `og_field_resources.id` where `og_field_resources.repointed_id is NULL`. -Add `og_field_attributes` table: +**permission mask** +The `permission_mask` is a bitmask where `public` = 0, `cc` = 1, `wm` = 2, and `wm + cc` = 3. -- id: primary key -- name: attribute name -- (optional) description -- (optional) type: text, int, float, json, etc... +| wm | cc |perm | +| --- | --- | --- | +| 0 | 0 | 0 | +| 0 | 1 | 1 | +| 1 | 0 | 2 | +| 1 | 1 | 3 | -Priority rows and EAV rows reference `og_field_attribute.id` +This allows for 2 filtering options: +- strict `permission_mask = ` (incurs minor cost of possibly duplicating data across rows) +- bit comparison to filter where `( | permission_mask) = ` + - if we sort by `permission_mask` (desc), this would allow us to only store the minimum number of resource variants + - for example, if a resource had ALL public sources as the highest priorities, we'd only need 1 row in the table with `permission_mask = 0` because ALL users would see the same version + - or if a resource had only `wm` and `public` variants, a `wm + cc` permission would get the `wm` version: 3 (`cc + wm`) | 2 (`wm`) = 3 + - but a `cc` permission would skip the `wm` row and see the `public` variant: + 1 (`cc`) | 2 (`wm`) = 3 => exclude + 1 (`cc`) | 0 (`pub`) = 1 => include + + +> [!NOTE] Permission Alternative +> We can also use permission columns for the minor cost of duplicating data across columns. Benefits from being a simpler more understandable approach. + +**record column as jsonb** +We'd effectively house the entire flat Pydantic model in json. The main reasoning is that we can likely expand to near-instant full-text search without much difficulty. It's also partially as an experiment to assess the difficulty of working with Postgres JSON syntax and investigate whether there are performance trade-offs. Should it prove easy to use while still being performant, it opens the door to using it in other places across our application where we might want greater flexibility in our data handling. + +**sync overview** +Very roughly speaking, when a user or process updates relevant data (merge resources, reprioritize, new sources from ETL), we compute the updated view state(s), and write them to the table, deleting where necessary. + +### Attribute metadata: `og_field_attributes` +```sql +CREATE TABLE og_field_attributes ( + id serial primary key, + name text not null unique, + + -- optional columns + description text, + type text, + created_at timestamptz not null default now(), + unit text, +) +``` -### Single priority table +Priority rows and EAV rows reference `og_field_attribute.id` +### Single priority store: `og_field_resource_attribute_priority` Replace the two-table priority split with a single table at the `(resource, attribute, source record)` grain, and use defaults when writing new data rather than as a SQL fallback rule. +```sql +create table og_field_resource_attribute_priority ( + resource_id bigint not null, + attribute_id bigint not null, + source_id bigint not null, + priority integer not null, + is_curated boolean not null default false, + created_at timestamptz not null default now(), + + primary key (resource_id, attribute_id, source_id), + + constraint resource_attribute_priority_position_uq + unique (resource_id, attribute_id, priority), + + constraint resource_attribute_priority_nonnegative_ck + check (priority >= 0), + + -- the source must be attached to the resource. + constraint resource_attribute_priority_memberships_fk + foreign key (resource_id, source_id) + references memberships(resource_id, source_id) + on delete cascade, + + -- the source must have a populated value for the selected attribute. + constraint resource_attribute_priority_source_value_fk + foreign key (source_id, attribute_id) + references oil_gas_field_source_values(source_id, attribute_id) +); + +``` -- Add `og_field_resource_attribute_priority` table. - - resource_id: fk to resources - - attribute_id: fk to `og_field_attributes` - - source_id: fk to `oil_gas_field_sources` - - priority: int - - is_curated: bool (false if default, true if set by user) - Drop `og_field_source_priority` and `og_field_resource_source_priority`. - On resource or source creation, write explicit priority rows for every attribute the source has a value for, seeded from the existing `SOURCE_PRIORITY` tuple in `stitch.ogsi.model`. - Add constraints: - - unique priority across resource and attribute: no 2 sources for a resource + attribute can have the same priority + - unique priority across resource and attribute: no 2 rows for a resource & attribute can have the same priority `UNIQUE (resource_id, attribute_id, priority)` - priority > 0 - - fk to memberships on resource_id, source_id: ensure source is attached to resource - - fk to `` + - fk constraint to memberships on resource_id, source_id: ensure source is attached to resource + - fk constraint to `og_field_source_values` on `source_id, attribute_id` ### Benefits @@ -197,38 +267,7 @@ Two consequences to handle in the migration: currently derived from "an override row exists." With every row explicit, it needs an explicit `is_curated` flag to survive. -## 4. Proposal, part 2 — a flattened per-profile state table - -`og_field_resource_state`, one row per `(resource_id, profile)`: - -- one typed column per `OilGasFieldBase` attribute, following `ATTRIBUTE_KINDS` -- `provenance jsonb` — `{attribute_name: source_key}` for the winning source -- primary key `(resource_id, profile)` - -**Why four rows per resource is enough.** Every registered user is granted read on -the five public sources (`gem`, `rmi`, `llm`, `alb`, `bc`) by our Auth0 tenant -policy. The only variables are `wm` and `ccr`, so a caller's visibility is two -bits and the profile domain is exactly: - -| Profile | `wm` | `ccr` | -| --------------- | ---- | ----- | -| `public` | no | no | -| `public_wm` | yes | no | -| `public_ccr` | no | yes | -| `public_wm_ccr` | yes | yes | - -Within one attribute, any candidate ranked below the highest-priority _public_ -candidate can never win for anyone, so only candidates ranked above it matter, and -those are `wm` or `ccr` rows. Most attributes therefore produce the same value in -all four rows; only an attribute where a licensed source outranks the best public -source differs. A resource whose `name_local` is ordered `wm, ccr, gem` and whose -other attributes are ordered `gem, wm, ccr` differs across profiles in -`name_local` alone. - -Rows exist even when every attribute is null for that profile, which is what -preserves the null-shell behavior of `_resource_universe()`. - -## 5. Invariants +## 4. Invariants The point of §3 and §4 is to push these into the schema. Being explicit about which ones the database can enforce and which remain obligations on the sync is what @@ -261,7 +300,7 @@ consistency check job, decided with §6) Obligation C is expressible as a jsonb CHECK in Postgres but not in SQLite, so whether to enforce it in the schema depends on the STIT-603 outcome. -## 6. Refresh +## 5. Refresh Deferred by design. Any implementation must uphold §5 and must be triggered by: a source value write, a priority change, a membership status change, a resource @@ -272,7 +311,7 @@ rebuild command, database triggers, or a scheduled rebuild. The choice turns partly on an open question: the ETL apps live in a separate repository and may write source data without passing through the API's write paths. -## 7. Tradeoffs and risks +## 6. Tradeoffs and risks - **The four-profile bound rests on the Auth0 policy, not the schema.** `permissions.licensed_sources()` derives the set from individual From ed07a6f73ccf4a28c9f29f822e8a062939a47e84 Mon Sep 17 00:00:00 2001 From: Michael Barlow Date: Fri, 18 Sep 2026 11:23:33 -0600 Subject: [PATCH 3/8] shorten sections, add optional tables and query/API explanation --- .gitignore | 3 +- docs/01_schema.md | 63 ++ docs/01_schema.sql | 223 ++++++ docs/02_schema_variants.sql | 342 +++++++++ docs/03_schema_operations_api.md | 675 ++++++++++++++++++ ...-state view design drafted and proposed.md | 23 + docs/STIT-766-design.md | 266 +++---- 7 files changed, 1441 insertions(+), 154 deletions(-) create mode 100644 docs/01_schema.md create mode 100644 docs/01_schema.sql create mode 100644 docs/02_schema_variants.sql create mode 100644 docs/03_schema_operations_api.md create mode 100644 docs/STIT-766 Materialized current-state view design drafted and proposed.md diff --git a/.gitignore b/.gitignore index 7e201db2..09659c10 100644 --- a/.gitignore +++ b/.gitignore @@ -70,7 +70,6 @@ instance/ # Sphinx documentation docs/_build/ -docs/ # PyBuilder .pybuilder/ @@ -223,7 +222,7 @@ __marimo__/ **/.DS_Store # Local design notes and plans -/docs/ +# /docs/ # impeccable design skill (local design context and live-mode config) /PRODUCT.md diff --git a/docs/01_schema.md b/docs/01_schema.md new file mode 100644 index 00000000..4199de79 --- /dev/null +++ b/docs/01_schema.md @@ -0,0 +1,63 @@ +```mermaid +erDiagram + app_user { + uuid user_id + } + resource { + bigint resource_id + timestamptz created_at + } + source_key { + bigint source_key_id + text key_name + } + source { + bigint source_id + bigint source_key_id + text external_id + timestamptz created_at + } + source_key ||--o{ source : "" + attribute_definition { + bigint attribute_id + text name + text description + text unit + timestamptz created_at + } + resource_source { + bigint resource_id + bigint source_id + timestamptz attached_at + } + resource ||--o{ resource_source : "" + source ||--o{ resource_source : "" + source_attribute_value { + bigint source_id + bigint attribute_id + text value_text + numeric value_num + jsonb value_json + timestamptz created_at + timestamptz updated_at + } + source ||--o{ source_attribute_value : "" + attribute_definition ||--o{ source_attribute_value : "" + resource_attribute_priority { + bigint resource_id + bigint attribute_id + bigint source_id + integer priority + timestamptz created_at + } + resource_source ||--o{ resource_attribute_priority : "" + source_attribute_value ||--o{ resource_attribute_priority : "" + user_source_key_permission { + uuid user_id + bigint source_key_id + source_key_action action + timestamptz granted_at + } + app_user ||--o{ user_source_key_permission : "" + source_key ||--o{ user_source_key_permission : "" +``` diff --git a/docs/01_schema.sql b/docs/01_schema.sql new file mode 100644 index 00000000..4df990ca --- /dev/null +++ b/docs/01_schema.sql @@ -0,0 +1,223 @@ +-- Source-prioritized, permission-aware EAV schema +-- Target database: PostgreSQL 15+ +-- +-- Core assumptions: +-- * Lower integer values represent higher priority. +-- * A source attribute row exists only when the source has a value. +-- * Exactly one typed value column is populated per source attribute. +-- * Source visibility is granted by source key and action. +-- * Permission filtering occurs before priority resolution. + +BEGIN; + +CREATE TYPE source_key_action AS ENUM ( + 'view', + 'edit' +); + +CREATE TABLE users ( + user_id uuid PRIMARY KEY +); + +CREATE TABLE og_field_resources ( + resource_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + created_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE source_keys ( + source_key_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + key_name text NOT NULL UNIQUE +); + +CREATE TABLE oil_gas_field_sources ( + source_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + source_key_id bigint NOT NULL, + external_id text, + created_at timestamptz NOT NULL DEFAULT now(), + + CONSTRAINT source_source_key_fk + FOREIGN KEY (source_key_id) + REFERENCES source_keys(source_key_id) +); + +CREATE TABLE attribute_definition ( + attribute_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + name text NOT NULL UNIQUE, + description text, + unit text, + created_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE memberships ( + resource_id bigint NOT NULL, + source_id bigint NOT NULL, + attached_at timestamptz NOT NULL DEFAULT now(), + + PRIMARY KEY (resource_id, source_id), + + CONSTRAINT resource_source_resource_fk + FOREIGN KEY (resource_id) + REFERENCES og_field_resources(resource_id) + ON DELETE CASCADE, + + CONSTRAINT resource_source_source_fk + FOREIGN KEY (source_id) + REFERENCES oil_gas_field_sources(source_id) + ON DELETE CASCADE +); + +CREATE TABLE oil_gas_field_source_values ( + source_id bigint NOT NULL, + attribute_id bigint NOT NULL, + + value_text text, + value_num numeric, + value_json jsonb, + + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now(), + + PRIMARY KEY (source_id, attribute_id), + + CONSTRAINT oil_gas_field_source_values_source_fk + FOREIGN KEY (source_id) + REFERENCES oil_gas_field_sources(source_id) + ON DELETE CASCADE, + + CONSTRAINT oil_gas_field_source_values_attribute_fk + FOREIGN KEY (attribute_id) + REFERENCES attribute_definition(attribute_id), + + CONSTRAINT oil_gas_field_source_values_exactly_one_value_ck + CHECK (num_nonnulls(value_text, value_num, value_json) = 1), + + CONSTRAINT oil_gas_field_source_values_nonnull_json_ck + CHECK (value_json IS NULL OR value_json <> 'null'::jsonb) +); + +CREATE TABLE og_field_resource_attribute_priority ( + resource_id bigint NOT NULL, + attribute_id bigint NOT NULL, + source_id bigint NOT NULL, + priority integer NOT NULL, + created_at timestamptz NOT NULL DEFAULT now(), + + PRIMARY KEY (resource_id, attribute_id, source_id), + + CONSTRAINT resource_attribute_priority_position_uq + UNIQUE (resource_id, attribute_id, priority), + + CONSTRAINT resource_attribute_priority_nonnegative_ck + CHECK (priority >= 0), + + -- The source must be attached to the resource. + CONSTRAINT resource_attribute_priority_resource_source_fk + FOREIGN KEY (resource_id, source_id) + REFERENCES memberships(resource_id, source_id) + ON DELETE CASCADE, + + -- The source must have a populated value for the selected attribute. + CONSTRAINT resource_attribute_priority_source_value_fk + FOREIGN KEY (source_id, attribute_id) + REFERENCES oil_gas_field_source_values(source_id, attribute_id) +); + +-- handles the permissions for a user associated with a given source key +-- user_id | src_key_id | action +-- 1 | 1 (wm) | view -> only wm + public +-- 1 | 2 (rmi) | view +-- 1 | 3 (gem) | view +-- 1 | 4 (alb) | view +-- 1 | 5 (bc) | view +-- 1 | 6 (llm) | view + +-- 2 | 7 (cc) | view -> only cc + public +-- 2 | 2 (rmi) | view +-- 2 | 3 (gem) | view +-- 2 | 4 (alb) | view +-- 2 | 5 (bc) | view +-- 2 | 6 (llm) | view + +-- 3 | 2 (rmi) | view -> only public +-- 3 | 3 (gem) | view +-- 3 | 4 (alb) | view +-- 3 | 5 (bc) | view +-- 3 | 6 (llm) | view + +-- 4 | 1 (wm) | view -> all: wm + cc + public +-- 4 | 7 (cc) | view +-- 4 | 2 (rmi) | view +-- 4 | 3 (gem) | view +-- 4 | 4 (alb) | view +-- 4 | 5 (bc) | view +-- 4 | 6 (llm) | view +CREATE TABLE user_source_key_permission ( + user_id uuid NOT NULL, + source_key_id bigint NOT NULL, + action source_key_action NOT NULL, + granted_at timestamptz NOT NULL DEFAULT now(), + + PRIMARY KEY (user_id, source_key_id, action), + + CONSTRAINT user_source_key_permission_user_fk + FOREIGN KEY (user_id) + REFERENCES users(user_id) + ON DELETE CASCADE, + + CONSTRAINT user_source_key_permission_key_fk + FOREIGN KEY (source_key_id) + REFERENCES source_keys(source_key_id) + ON DELETE CASCADE +); + +-- Resolution path: resource/attribute -> priority -> source -> source key. +CREATE INDEX resource_attribute_priority_resolution_idx + ON og_field_resource_attribute_priority ( + resource_id, + attribute_id, + priority, + source_id + ); + +CREATE INDEX resource_attribute_priority_source_idx + ON og_field_resource_attribute_priority ( + source_id, + attribute_id + ); + +CREATE INDEX source_key_lookup_idx + ON oil_gas_field_sources ( + source_key_id, + source_id + ); + +CREATE INDEX user_source_key_view_idx + ON user_source_key_permission ( + user_id, + source_key_id + ) + WHERE action = 'view'; + +-- Optional lookup indexes for filtering before or after resolution. +CREATE INDEX oil_gas_field_source_values_text_lookup_idx + ON oil_gas_field_source_values ( + attribute_id, + value_text, + source_id + ) + WHERE value_text IS NOT NULL; + +CREATE INDEX oil_gas_field_source_values_num_lookup_idx + ON oil_gas_field_source_values ( + attribute_id, + value_num, + source_id + ) + WHERE value_num IS NOT NULL; + +CREATE INDEX oil_gas_field_source_values_json_gin_idx + ON oil_gas_field_source_values + USING gin (value_json) + WHERE value_json IS NOT NULL; + +COMMIT; diff --git a/docs/02_schema_variants.sql b/docs/02_schema_variants.sql new file mode 100644 index 00000000..021fc76b --- /dev/null +++ b/docs/02_schema_variants.sql @@ -0,0 +1,342 @@ +-- Schema variants and optional extensions +-- Target database: PostgreSQL 15+ +-- +-- This file is not intended to be executed as a single migration. Each section +-- is an alternative to, or extension of, the canonical schema in 01_schema.sql. + +------------------------------------------------------------------------------- +-- Variant A: direct view grants only +------------------------------------------------------------------------------- +-- Use when the only key-scoped permission is "view". The presence of a row +-- means the user can view sources with that key. + +CREATE TABLE user_source_key_view_permission ( + user_id uuid NOT NULL, + source_key_id bigint NOT NULL, + granted_at timestamptz NOT NULL DEFAULT now(), + + PRIMARY KEY (user_id, source_key_id), + + FOREIGN KEY (user_id) + REFERENCES app_user(user_id) + ON DELETE CASCADE, + + FOREIGN KEY (source_key_id) + REFERENCES source_key(source_key_id) + ON DELETE CASCADE +); + +------------------------------------------------------------------------------- +-- Variant B: role-based grants +------------------------------------------------------------------------------- +-- Use when many users share the same effective visibility profile. This can +-- substantially reduce duplicate permission rows and enables caching by role +-- or visibility profile instead of by individual user. + +CREATE TABLE app_role ( + role_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + role_name text NOT NULL UNIQUE +); + +CREATE TABLE user_role ( + user_id uuid NOT NULL, + role_id bigint NOT NULL, + + PRIMARY KEY (user_id, role_id), + + FOREIGN KEY (user_id) + REFERENCES app_user(user_id) + ON DELETE CASCADE, + + FOREIGN KEY (role_id) + REFERENCES app_role(role_id) + ON DELETE CASCADE +); + +CREATE TABLE role_source_key_permission ( + role_id bigint NOT NULL, + source_key_id bigint NOT NULL, + action source_key_action NOT NULL, + + PRIMARY KEY (role_id, source_key_id, action), + + FOREIGN KEY (role_id) + REFERENCES app_role(role_id) + ON DELETE CASCADE, + + FOREIGN KEY (source_key_id) + REFERENCES source_key(source_key_id) + ON DELETE CASCADE +); + +CREATE VIEW effective_user_source_key_permission AS +SELECT DISTINCT + ur.user_id, + rp.source_key_id, + rp.action +FROM user_role AS ur +JOIN role_source_key_permission AS rp + ON rp.role_id = ur.role_id; + +------------------------------------------------------------------------------- +-- Variant C: groups plus direct grants +------------------------------------------------------------------------------- +-- Effective grants are the union of direct grants and group-derived grants. + +CREATE TABLE app_group ( + group_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + group_name text NOT NULL UNIQUE +); + +CREATE TABLE user_group ( + user_id uuid NOT NULL, + group_id bigint NOT NULL, + + PRIMARY KEY (user_id, group_id), + + FOREIGN KEY (user_id) + REFERENCES app_user(user_id) + ON DELETE CASCADE, + + FOREIGN KEY (group_id) + REFERENCES app_group(group_id) + ON DELETE CASCADE +); + +CREATE TABLE group_source_key_permission ( + group_id bigint NOT NULL, + source_key_id bigint NOT NULL, + action source_key_action NOT NULL, + + PRIMARY KEY (group_id, source_key_id, action), + + FOREIGN KEY (group_id) + REFERENCES app_group(group_id) + ON DELETE CASCADE, + + FOREIGN KEY (source_key_id) + REFERENCES source_key(source_key_id) + ON DELETE CASCADE +); + +CREATE VIEW effective_user_source_key_permission AS +SELECT + p.user_id, + p.source_key_id, + p.action +FROM user_source_key_permission AS p + +UNION + +SELECT + ug.user_id, + gp.source_key_id, + gp.action +FROM user_group AS ug +JOIN group_source_key_permission AS gp + ON gp.group_id = ug.group_id; + +------------------------------------------------------------------------------- +-- Variant D: retain a wide source-data table +------------------------------------------------------------------------------- +-- If source data must remain in physical columns, use a companion presence +-- table to expose only populated attributes to the priority system. A trigger +-- or ingestion process must keep this table synchronized. + +CREATE TABLE source_attribute_presence ( + source_id bigint NOT NULL, + attribute_id bigint NOT NULL, + + PRIMARY KEY (source_id, attribute_id), + + FOREIGN KEY (source_id) + REFERENCES source(source_id) + ON DELETE CASCADE, + + FOREIGN KEY (attribute_id) + REFERENCES attribute_definition(attribute_id) +); + +-- In this variant, replace the priority FK to source_attribute_value with: +-- +-- ALTER TABLE resource_attribute_priority +-- DROP CONSTRAINT resource_attribute_priority_source_value_fk, +-- ADD CONSTRAINT resource_attribute_priority_source_presence_fk +-- FOREIGN KEY (source_id, attribute_id) +-- REFERENCES source_attribute_presence(source_id, attribute_id); + +------------------------------------------------------------------------------- +-- Variant E: allow multiple source records per key for the same resource +------------------------------------------------------------------------------- +-- The canonical schema already permits this. Add a uniqueness constraint only +-- if the domain requires at most one source per source key per resource. +-- PostgreSQL cannot enforce this using only resource_source because source_key +-- resides on source. A denormalized source_key_id column or trigger is needed. + +ALTER TABLE resource_source + ADD COLUMN source_key_id bigint; + +ALTER TABLE resource_source + ADD CONSTRAINT resource_source_key_fk + FOREIGN KEY (source_key_id) + REFERENCES source_key(source_key_id); + +ALTER TABLE resource_source + ADD CONSTRAINT resource_source_one_source_per_key_uq + UNIQUE (resource_id, source_key_id); + +-- Synchronize resource_source.source_key_id with source.source_key_id in the +-- application or with a trigger. + +------------------------------------------------------------------------------- +-- Variant F: nullable priority values with automatic fallback +------------------------------------------------------------------------------- +-- Not recommended with the canonical model. The canonical source_attribute_value +-- table stores only populated values, so priority rows always reference a value. +-- If null-valued rows must be retained for audit purposes, separate "observed" +-- from "candidate" values instead of weakening the FK invariant. + +CREATE TABLE source_attribute_observation ( + observation_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + source_id bigint NOT NULL REFERENCES source(source_id) ON DELETE CASCADE, + attribute_id bigint NOT NULL REFERENCES attribute_definition(attribute_id), + value_text text, + value_num numeric, + value_json jsonb, + observed_at timestamptz NOT NULL DEFAULT now(), + + CHECK (num_nonnulls(value_text, value_num, value_json) <= 1), + CHECK (value_json IS NULL OR value_json <> 'null'::jsonb) +); + +-- Only non-null observations should be promoted into source_attribute_value. + +------------------------------------------------------------------------------- +-- Variant G: row-level security for source values +------------------------------------------------------------------------------- +-- This variant assumes every transaction sets: +-- +-- SET LOCAL app.user_id = ''; +-- +-- The application role must not have BYPASSRLS. Table owners normally bypass +-- RLS unless FORCE ROW LEVEL SECURITY is enabled. + +ALTER TABLE source_attribute_value ENABLE ROW LEVEL SECURITY; +ALTER TABLE source_attribute_value FORCE ROW LEVEL SECURITY; + +CREATE POLICY source_attribute_value_select_policy +ON source_attribute_value +FOR SELECT +USING ( + EXISTS ( + SELECT 1 + FROM source AS s + JOIN user_source_key_permission AS p + ON p.source_key_id = s.source_key_id + AND p.action = 'view' + WHERE s.source_id = source_attribute_value.source_id + AND p.user_id = current_setting('app.user_id', true)::uuid + ) +); + +-- Consider corresponding policies on source and resource_attribute_priority if +-- the existence or provenance of restricted sources is itself sensitive. + +------------------------------------------------------------------------------- +-- Variant H: permission-aware resolution view +------------------------------------------------------------------------------- +-- A view cannot accept a user_id parameter. This form expects app.user_id to be +-- set transaction-locally. SECURITY INVOKER is the default and is preferred. + +CREATE VIEW resolved_resource_attribute AS +SELECT DISTINCT ON ( + p.resource_id, + p.attribute_id +) + p.resource_id, + p.attribute_id, + p.source_id, + s.source_key_id, + p.priority, + v.value_text, + v.value_num, + v.value_json +FROM resource_attribute_priority AS p +JOIN source_attribute_value AS v + ON v.source_id = p.source_id + AND v.attribute_id = p.attribute_id +JOIN source AS s + ON s.source_id = p.source_id +JOIN user_source_key_permission AS permission + ON permission.source_key_id = s.source_key_id + AND permission.user_id = current_setting('app.user_id', true)::uuid + AND permission.action = 'view' +ORDER BY + p.resource_id, + p.attribute_id, + p.priority, + p.source_id; + +------------------------------------------------------------------------------- +-- Variant I: permission-aware resolution function +------------------------------------------------------------------------------- +-- Convenient when the database receives an explicit, trusted user identifier. +-- Do not expose this function to clients that may substitute arbitrary user IDs +-- unless authorization is enforced elsewhere. + +CREATE FUNCTION resolved_resource_values(p_user_id uuid) +RETURNS TABLE ( + resource_id bigint, + attribute_id bigint, + source_id bigint, + source_key_id bigint, + priority integer, + value_text text, + value_num numeric, + value_json jsonb +) +LANGUAGE sql +STABLE +SECURITY INVOKER +AS $$ + SELECT DISTINCT ON ( + p.resource_id, + p.attribute_id + ) + p.resource_id, + p.attribute_id, + p.source_id, + s.source_key_id, + p.priority, + v.value_text, + v.value_num, + v.value_json + FROM resource_attribute_priority AS p + JOIN source_attribute_value AS v + ON v.source_id = p.source_id + AND v.attribute_id = p.attribute_id + JOIN source AS s + ON s.source_id = p.source_id + JOIN user_source_key_permission AS permission + ON permission.source_key_id = s.source_key_id + AND permission.user_id = p_user_id + AND permission.action = 'view' + ORDER BY + p.resource_id, + p.attribute_id, + p.priority, + p.source_id +$$; + +------------------------------------------------------------------------------- +-- Variant J: dense, contiguous priority positions +------------------------------------------------------------------------------- +-- UNIQUE(resource_id, attribute_id, priority) prevents duplicates but does not +-- prevent gaps. Enforce contiguous positions in application logic, a deferred +-- constraint trigger, or avoid positional integers entirely by storing a stable +-- rank value that permits insertion between neighbors. +-- +-- Examples of alternatives: +-- * numeric rank: 1000, 2000, 3000, insert 1500 between first and second +-- * explicit linked list: predecessor_source_id +-- * full replacement API: delete and reinsert the complete ordered list diff --git a/docs/03_schema_operations_api.md b/docs/03_schema_operations_api.md new file mode 100644 index 00000000..cc0efc79 --- /dev/null +++ b/docs/03_schema_operations_api.md @@ -0,0 +1,675 @@ +# Source-Prioritized Resource Schema: Operational API + +This document describes how application code should operate on the schema in +`01_schema.sql`. It treats the database model as an API: the supported commands, +read semantics, invariants, transaction boundaries, and expected failure modes. + +## 1. Core semantics + +A **resource** is a minimal logical record assembled from one or more **sources**. +Each source belongs to a **source key**, such as `A`, `B`, or `licensed_vendor`. +A user may view only the source keys for which they have `view` permission. + +Source data is represented as populated attribute rows: + +```text +(source_id, attribute_id) -> exactly one of value_text, value_num, value_json +``` + +For each resource and attribute, `resource_attribute_priority` defines an ordered +list of source candidates. Lower numeric priority wins. + +The resolved value is therefore: + +> the highest-priority populated value whose source key is visible to the user + +Permission filtering must happen **before** selecting the winning priority. If a +restricted source at priority 0 contains a value and the user cannot view its +key, the user receives the next visible candidate. + +## 2. Important invariants + +The schema enforces these relationships: + +1. A source can participate in a resource only through `resource_source`. +2. A priority entry can reference only a source attached to the resource. +3. A priority entry can reference only a populated source attribute. +4. A source attribute contains exactly one typed value. +5. Two sources cannot occupy the same priority position for the same resource + and attribute. +6. A source cannot appear twice in the priority list for the same resource and + attribute. +7. Visibility is determined by the source's key, not by the resource or + attribute independently. + +The schema does **not** require priorities to be contiguous. Positions `0, 10, +20` are valid and often easier to maintain than `0, 1, 2`. + +## 3. Suggested application service surface + +A service layer can expose operations similar to the following: + +```text +create_resource() -> resource_id +create_source(source_key, external_id?) -> source_id +attach_source(resource_id, source_id) +define_attribute(name, description?, unit?) -> attribute_id +put_source_value(source_id, attribute, typed_value) +delete_source_value(source_id, attribute) +set_attribute_priorities(resource_id, attribute, ordered_source_ids) +grant_source_key_permission(user_id, source_key, action = "view") +revoke_source_key_permission(user_id, source_key, action = "view") +resolve_resource(user_id, resource_id) -> flat object +search_resources(user_id, predicates) -> resource ids or flat objects +explain_resolved_value(user_id, resource_id, attribute) -> provenance +``` + +Database functions are optional. These can be implemented in application code +using parameterized SQL while preserving the same semantics. + +## 4. Creating resources and sources + +### Create a resource + +```sql +INSERT INTO resource DEFAULT VALUES +RETURNING resource_id; +``` + +### Resolve or create a source key + +```sql +INSERT INTO source_key (key_name) +VALUES ($1) +ON CONFLICT (key_name) +DO UPDATE SET key_name = EXCLUDED.key_name +RETURNING source_key_id; +``` + +### Create a source + +```sql +INSERT INTO source (source_key_id, external_id) +VALUES ($1, $2) +RETURNING source_id; +``` + +### Attach a source to a resource + +```sql +INSERT INTO resource_source (resource_id, source_id) +VALUES ($1, $2) +ON CONFLICT DO NOTHING; +``` + +Attaching a source does not automatically prioritize every value from that +source. Priority remains explicit per attribute. + +## 5. Defining attributes + +Use a stable attribute name at the API boundary, but use `attribute_id` +internally. + +```sql +INSERT INTO attribute_definition (name, description, unit) +VALUES ($1, $2, $3) +ON CONFLICT (name) +DO UPDATE SET + description = COALESCE(EXCLUDED.description, + attribute_definition.description), + unit = COALESCE(EXCLUDED.unit, attribute_definition.unit) +RETURNING attribute_id; +``` + +Renaming an attribute should be treated as a controlled schema operation. The +priority and value tables remain stable because they reference `attribute_id`, +not the textual name. + +## 6. Writing source values + +A source value exists only when it is populated. SQL `NULL` means "no candidate +value" and should normally be represented by deleting the row. + +### Text value + +```sql +INSERT INTO source_attribute_value ( + source_id, + attribute_id, + value_text +) +VALUES ($1, $2, $3) +ON CONFLICT (source_id, attribute_id) +DO UPDATE SET + value_text = EXCLUDED.value_text, + value_num = NULL, + value_json = NULL, + updated_at = now(); +``` + +### Numeric value + +```sql +INSERT INTO source_attribute_value ( + source_id, + attribute_id, + value_num +) +VALUES ($1, $2, $3) +ON CONFLICT (source_id, attribute_id) +DO UPDATE SET + value_text = NULL, + value_num = EXCLUDED.value_num, + value_json = NULL, + updated_at = now(); +``` + +### JSON value + +```sql +INSERT INTO source_attribute_value ( + source_id, + attribute_id, + value_json +) +VALUES ($1, $2, $3::jsonb) +ON CONFLICT (source_id, attribute_id) +DO UPDATE SET + value_text = NULL, + value_num = NULL, + value_json = EXCLUDED.value_json, + updated_at = now(); +``` + +The JSON literal `null` is rejected. Use row deletion to represent absence. + +### Delete a value + +```sql +DELETE FROM source_attribute_value +WHERE source_id = $1 + AND attribute_id = $2; +``` + +This deletion fails while a priority row references the value. The application +must first remove or replace that source in affected priority lists. This is an +intentional consistency guard. + +A safe transaction is: + +```sql +BEGIN; + +DELETE FROM resource_attribute_priority +WHERE source_id = $1 + AND attribute_id = $2; + +DELETE FROM source_attribute_value +WHERE source_id = $1 + AND attribute_id = $2; + +COMMIT; +``` + +## 7. Managing priority lists + +Treat a priority list as one ordered aggregate rather than many unrelated rows. +Replacing the complete list in one transaction is usually safer than issuing +individual moves. + +Given an ordered input such as: + +```json +[17, 42, 91] +``` + +assign priorities using ordinality: + +```sql +BEGIN; + +DELETE FROM resource_attribute_priority +WHERE resource_id = $1 + AND attribute_id = $2; + +INSERT INTO resource_attribute_priority ( + resource_id, + attribute_id, + source_id, + priority +) +SELECT + $1, + $2, + input.source_id, + input.ordinality - 1 +FROM unnest($3::bigint[]) WITH ORDINALITY AS input(source_id, ordinality); + +COMMIT; +``` + +The insert fails when: + +- a source is not attached to the resource; +- a source lacks a value for the attribute; +- the source appears more than once; +- the input creates duplicate priority positions. + +Those failures are useful API validation errors and should normally map to a +client error rather than an internal server error. + +### Incremental insertion with sparse ranks + +To avoid renumbering, use ranks such as `1000, 2000, 3000`. Insert a new source +between the first and second at `1500`. Renormalize only when no gap remains. + +## 8. Granting and revoking visibility + +### Grant permission + +```sql +INSERT INTO user_source_key_permission ( + user_id, + source_key_id, + action +) +VALUES ($1, $2, 'view') +ON CONFLICT DO NOTHING; +``` + +### Revoke permission + +```sql +DELETE FROM user_source_key_permission +WHERE user_id = $1 + AND source_key_id = $2 + AND action = 'view'; +``` + +Permission changes take effect immediately for dynamically resolved reads. +Caches must include the user's effective visibility profile in the cache key or +be invalidated when permissions change. + +## 9. Resolving one resource for one user + +The permission join removes inaccessible candidates before `DISTINCT ON` +chooses the winner. + +```sql +WITH resolved AS ( + SELECT DISTINCT ON ( + p.resource_id, + p.attribute_id + ) + p.resource_id, + p.attribute_id, + p.source_id, + s.source_key_id, + p.priority, + v.value_text, + v.value_num, + v.value_json + FROM resource_attribute_priority AS p + JOIN source_attribute_value AS v + ON v.source_id = p.source_id + AND v.attribute_id = p.attribute_id + JOIN source AS s + ON s.source_id = p.source_id + JOIN user_source_key_permission AS permission + ON permission.source_key_id = s.source_key_id + AND permission.user_id = $1 + AND permission.action = 'view' + WHERE p.resource_id = $2 + ORDER BY + p.resource_id, + p.attribute_id, + p.priority, + p.source_id +) +SELECT + r.resource_id, + jsonb_object_agg( + a.name, + COALESCE( + to_jsonb(r.value_text), + to_jsonb(r.value_num), + r.value_json + ) + ) AS attributes +FROM resolved AS r +JOIN attribute_definition AS a + ON a.attribute_id = r.attribute_id +GROUP BY r.resource_id; +``` + +The result is a user-specific flat JSON object. + +### Empty resources + +The query above returns no row when the user can see no resolved attributes. +To return the resource with an empty JSON object, start from `resource` and use +a lateral or left join. + +## 10. Resolving many resources + +The same resolution CTE can omit the `resource_id` predicate. For large result +sets, page by `resource_id` and resolve only the requested page. + +Avoid flattening every attribute for every resource before applying selective +filters. Resolve only the attributes needed for filtering or output. + +## 11. Provenance and explanation + +A useful API should expose why a user received a particular value. + +```sql +SELECT + p.resource_id, + a.name AS attribute, + p.source_id, + sk.key_name AS source_key, + p.priority, + v.value_text, + v.value_num, + v.value_json +FROM resource_attribute_priority AS p +JOIN source_attribute_value AS v + ON v.source_id = p.source_id + AND v.attribute_id = p.attribute_id +JOIN attribute_definition AS a + ON a.attribute_id = p.attribute_id +JOIN source AS s + ON s.source_id = p.source_id +JOIN source_key AS sk + ON sk.source_key_id = s.source_key_id +JOIN user_source_key_permission AS permission + ON permission.source_key_id = s.source_key_id + AND permission.user_id = $1 + AND permission.action = 'view' +WHERE p.resource_id = $2 + AND a.name = $3 +ORDER BY p.priority, p.source_id; +``` + +The first row is the visible winner. Remaining rows are visible fallbacks. +Do not reveal hidden candidates or their keys unless the user has separate +permission to inspect restricted provenance. + +## 12. Dynamic search by unknown attributes + +The search API may accept an arbitrary list of predicates. An extensible input +shape is: + +```json +[ + { + "attribute": "country", + "operator": "eq", + "value_text": "US" + }, + { + "attribute": "production", + "operator": "gte", + "value_num": 1000 + } +] +``` + +The correct order of operations is: + +```text +parse predicates + -> resolve attribute names to IDs + -> restrict candidates to visible source keys + -> choose the highest-priority visible value + -> apply predicates + -> require all predicates to match +``` + +Filtering raw source values before priority resolution answers a different +question: "does any permitted source contain this value?" The usual API should +instead filter the flattened representation the user actually sees. + +### Equality-only search + +This query accepts typed equality predicates as JSON and requires every +predicate to match. + +```sql +WITH requested_filters AS ( + SELECT + definition.attribute_id, + filter.attribute, + filter.value_text, + filter.value_num, + filter.value_json + FROM jsonb_to_recordset($2::jsonb) AS filter( + attribute text, + value_text text, + value_num numeric, + value_json jsonb + ) + JOIN attribute_definition AS definition + ON definition.name = filter.attribute +), +resolved AS ( + SELECT DISTINCT ON ( + p.resource_id, + p.attribute_id + ) + p.resource_id, + p.attribute_id, + v.value_text, + v.value_num, + v.value_json + FROM resource_attribute_priority AS p + JOIN requested_filters AS filter + ON filter.attribute_id = p.attribute_id + JOIN source_attribute_value AS v + ON v.source_id = p.source_id + AND v.attribute_id = p.attribute_id + JOIN source AS s + ON s.source_id = p.source_id + JOIN user_source_key_permission AS permission + ON permission.source_key_id = s.source_key_id + AND permission.user_id = $1 + AND permission.action = 'view' + ORDER BY + p.resource_id, + p.attribute_id, + p.priority, + p.source_id +), +matched AS ( + SELECT + resolved.resource_id, + resolved.attribute_id + FROM resolved + JOIN requested_filters AS filter + ON filter.attribute_id = resolved.attribute_id + AND ( + ( + filter.value_text IS NOT NULL + AND resolved.value_text = filter.value_text + ) + OR ( + filter.value_num IS NOT NULL + AND resolved.value_num = filter.value_num + ) + OR ( + filter.value_json IS NOT NULL + AND resolved.value_json = filter.value_json + ) + ) +) +SELECT matched.resource_id +FROM matched +GROUP BY matched.resource_id +HAVING count(*) = (SELECT count(*) FROM requested_filters) +ORDER BY matched.resource_id; +``` + +Input validation should require exactly one filter value column per predicate, +just as the value table requires exactly one storage column. + +### Supporting comparison operators + +Add an `operator` field and dispatch by value type: + +```sql +AND CASE filter.operator + WHEN 'eq' THEN resolved.value_num = filter.value_num + WHEN 'neq' THEN resolved.value_num <> filter.value_num + WHEN 'lt' THEN resolved.value_num < filter.value_num + WHEN 'lte' THEN resolved.value_num <= filter.value_num + WHEN 'gt' THEN resolved.value_num > filter.value_num + WHEN 'gte' THEN resolved.value_num >= filter.value_num + ELSE false +END +``` + +Do not accept arbitrary SQL operators or column names from clients. Parse a +small allowlisted operator vocabulary and bind all values as parameters. + +### Duplicate attributes + +Decide explicitly whether repeated predicates for one attribute mean AND, OR, +or invalid input. A simple first version should reject duplicates. + +### Unknown attributes + +The join to `attribute_definition` silently drops unknown names. For an API, +validate before executing the search and return a clear client error listing +unknown attributes. + +## 13. Pagination and result shape + +A search endpoint may return only resource IDs first: + +```text +search_resources(user, predicates, cursor, limit) -> [resource_id] +``` + +Then fetch flattened representations for those IDs in a second query. This is +often easier to optimize than resolving and aggregating every matching resource +inside one large statement. + +Prefer keyset pagination: + +```sql +WHERE resource_id > $cursor +ORDER BY resource_id +LIMIT $limit +``` + +## 14. Concurrency + +Priority-list replacement should occur in one transaction. Concurrent writers +for the same `(resource_id, attribute_id)` can serialize using a row lock or an +advisory transaction lock. + +One option is to lock the resource row: + +```sql +SELECT 1 +FROM resource +WHERE resource_id = $1 +FOR UPDATE; +``` + +A narrower advisory lock can be derived from the resource and attribute IDs: + +```sql +SELECT pg_advisory_xact_lock($1, $2); +``` + +Choose one strategy and use it consistently in all priority mutation paths. + +## 15. Row-level security + +Application-level permission joins make semantics explicit. PostgreSQL row-level +security can add defense in depth so restricted source values cannot be read by +accident. + +A common request transaction is: + +```sql +BEGIN; +SET LOCAL app.user_id = ''; +-- Execute permission-aware queries. +COMMIT; +``` + +The authenticated identity must come from trusted server-side authentication, +not a client-controlled query parameter. + +RLS does not eliminate the need to reason about provenance leakage. Protect +`source`, `resource_attribute_priority`, and diagnostic views as needed if even +the existence of restricted sources is confidential. + +## 16. Caching and materialization + +The flattened representation is user-dependent. A cache keyed only by +`resource_id` is incorrect when users have different source-key permissions. + +Valid cache keys include: + +```text +(resource_id, user_id, permission_version) +(resource_id, role_id, permission_version) +(resource_id, visibility_profile_hash) +``` + +Role- or profile-based caching is preferable when many users share identical +grants. + +A globally materialized resolved table works only for a canonical visibility +profile. For multiple profiles, materialize per profile or resolve dynamically. + +## 17. Deletion behavior + +Deleting a resource cascades through its attachments and priority rows. + +Deleting a source cascades through its values, attachments, and priority rows. +This is convenient but may be too destructive for audited systems. Alternatives +include soft deletion or `ON DELETE RESTRICT` plus explicit archival workflows. + +Deleting an attribute definition is restricted while values or priorities refer +to it. Treat attribute deletion as a migration rather than a routine API call. + +## 18. Recommended validation errors + +Map database constraint failures to stable API errors: + +| Condition | Suggested API error | +|---|---| +| Source not attached to resource | `SOURCE_NOT_ATTACHED` | +| Source has no value for attribute | `SOURCE_ATTRIBUTE_MISSING` | +| Duplicate priority position | `PRIORITY_CONFLICT` | +| Duplicate source in list | `DUPLICATE_PRIORITY_SOURCE` | +| No typed value or multiple typed values | `INVALID_TYPED_VALUE` | +| JSON literal null supplied | `INVALID_JSON_NULL` | +| Unknown attribute name | `UNKNOWN_ATTRIBUTE` | +| Unsupported operator | `UNSUPPORTED_OPERATOR` | +| User lacks permission | `FORBIDDEN_SOURCE_KEY` or omit existence details | + +For security-sensitive operations, avoid distinguishing "does not exist" from +"exists but forbidden" unless the caller is allowed to know that distinction. + +## 19. Recommended initial implementation + +A practical first version is: + +1. Use the canonical typed EAV tables. +2. Store direct `view` grants by source key. +3. Replace full priority lists transactionally. +4. Resolve with a permission join followed by `DISTINCT ON`. +5. Accept equality-only dynamic filters initially. +6. Resolve only requested attributes during search. +7. Add RLS after application queries and transaction identity handling are + stable. +8. Add role-based permission profiles if caching or permission-row volume + becomes significant. + +This preserves strong relational constraints while keeping the application API +predictable and extensible. diff --git a/docs/STIT-766 Materialized current-state view design drafted and proposed.md b/docs/STIT-766 Materialized current-state view design drafted and proposed.md new file mode 100644 index 00000000..aaa097ca --- /dev/null +++ b/docs/STIT-766 Materialized current-state view design drafted and proposed.md @@ -0,0 +1,23 @@ +- one initial issue: sqlite does not support materialized views, we may need to drop support for sqlite in general + +Objective +Alter the database structure to eliminate all slow responses, focusing on `list`, `filter-options`, and `detail` endpoints. + +Proposal +- drop the base priority table, all new resources + sources get rows in the `resource_source_priority` table +- add a `og_field_attributes` table: `id`, `attr_name`, `type`(optional) + +## Important Invariants + +The schema should enforces these relationships: + +1. A source can participate in a resource only through `resource_source`. +2. A priority entry can reference only a source attached to the resource. +3. A priority entry can reference only a populated source attribute. +4. A source attribute contains exactly one typed value. +5. Two sources cannot occupy the same priority position for the same resource + and attribute. +6. A source cannot appear twice in the priority list for the same resource and + attribute. +7. Visibility is determined by the source's key, not by the resource or + attribute independently. diff --git a/docs/STIT-766-design.md b/docs/STIT-766-design.md index 6121f2f4..857dc82c 100644 --- a/docs/STIT-766-design.md +++ b/docs/STIT-766-design.md @@ -9,6 +9,7 @@ ## 1. Objective Design a database architecture and application logic to serve the key Stitch endpoints with minimal latency. + ## 2. Background ### Problem @@ -106,21 +107,14 @@ It must: - be able to be rebuilt from scratch at any time, we should be able to derive the data for the table easily & quickly The top contender for a schema is: -```sql -CREATE TABLE og_field_resource_view ( - resource_id bigint not null, - permission_mask smallint not null, - record jsonb not null, - - -- cannot have the same resource with the same permission - primary key (resource_id, permission_mask), - - constraint og_field_resource_view - foreign key (resource_id) - references og_field_resources(id) -) +```mermaid +erDiagram + og_field_resource_view { + bignt resource_id + smallint permission_mask + jsonb record + } ``` -The one slight drawback is that it's slightly tricky to maintain that we only retain `resource_id` values that correspond to `og_field_resources.id` where `og_field_resources.repointed_id is NULL`. **permission mask** The `permission_mask` is a bitmask where `public` = 0, `cc` = 1, `wm` = 2, and `wm + cc` = 3. @@ -151,19 +145,25 @@ We'd effectively house the entire flat Pydantic model in json. The main reasonin **sync overview** Very roughly speaking, when a user or process updates relevant data (merge resources, reprioritize, new sources from ETL), we compute the updated view state(s), and write them to the table, deleting where necessary. +```mermaid +flowchart LR + A[db updates] -->|Trigger| B[compute coalesced state] + B --> C[Write coalesced state to table] + D(merge) --> A + E(reprioritize) --> A + F(llm enrich) --> A + G(ETL) --> A +``` ### Attribute metadata: `og_field_attributes` -```sql -CREATE TABLE og_field_attributes ( - id serial primary key, - name text not null unique, - - -- optional columns - description text, - type text, - created_at timestamptz not null default now(), - unit text, -) + +```mermaid +erDiagram + og_field_attributes { + serial id + text name + } + ``` Priority rows and EAV rows reference `og_field_attribute.id` @@ -172,35 +172,16 @@ Priority rows and EAV rows reference `og_field_attribute.id` Replace the two-table priority split with a single table at the `(resource, attribute, source record)` grain, and use defaults when writing new data rather than as a SQL fallback rule. -```sql -create table og_field_resource_attribute_priority ( - resource_id bigint not null, - attribute_id bigint not null, - source_id bigint not null, - priority integer not null, - is_curated boolean not null default false, - created_at timestamptz not null default now(), - - primary key (resource_id, attribute_id, source_id), - - constraint resource_attribute_priority_position_uq - unique (resource_id, attribute_id, priority), - - constraint resource_attribute_priority_nonnegative_ck - check (priority >= 0), - - -- the source must be attached to the resource. - constraint resource_attribute_priority_memberships_fk - foreign key (resource_id, source_id) - references memberships(resource_id, source_id) - on delete cascade, - - -- the source must have a populated value for the selected attribute. - constraint resource_attribute_priority_source_value_fk - foreign key (source_id, attribute_id) - references oil_gas_field_source_values(source_id, attribute_id) -); +```mermaid +erDiagram + og_field_resource_attribute_priority { + bigint resource_id + bigint attribute_id + bigint source_id + int priority + boolean is_curated + } ``` - Drop `og_field_source_priority` and `og_field_resource_source_priority`. @@ -213,13 +194,86 @@ create table og_field_resource_attribute_priority ( - priority > 0 - fk constraint to memberships on resource_id, source_id: ensure source is attached to resource - fk constraint to `og_field_source_values` on `source_id, attribute_id` +- `oil_gas_field_source_values` currently has an `id` primary key, so we could replace `(attribute_id, source_id)` with `source_value_id` +- we set `is_curated` to `True` when a user makes an update + +### Optional Additional Tables +These are primarily for some convenience in constructing simpler SQL statements and permissions handling. + +```mermaid +erDiagram + og_source_keys { + bigint id + text key_name + } +``` -### Benefits +References to `source` or `src_key` become foreign keys with `source_key_id`. -#### Simpler, faster SQL +```mermaid +erDiagram + user_source_key_permissions { + uuid user_id + bigint source_key_id + source_key_permission perm + } +``` + +Where the `source_key_permission` is a new ENUM type with `read`, `edit`. + +What this would allow is comparatively smaller and more straightforward SQL statements. -Provenance for a single resource is a series of joins. +**single resource for a user** +```sql +WITH resolved AS ( + SELECT DISTINCT ON ( + p.resource_id, + p.attribute_id + ) + p.resource_id, + p.attribute_id, + p.source_id, + s.source_key_id, + p.priority, + v.value_text, + v.value_num, + v.value_json + FROM og_field_resource_attribute_priority AS p + JOIN oil_gas_field_source_values AS v + ON v.source_id = p.source_id + AND v.attribute_id = p.attribute_id + JOIN oil_gas_field_sources AS s + ON s.source_id = p.source_id + JOIN user_source_key_permission AS permission + ON permission.source_key_id = s.source_key_id + AND permission.user_id = $1 -- pass in specific user id + AND permission.action = 'read' + WHERE p.resource_id = $2 -- drop this predicate to fetch many resources + ORDER BY + p.resource_id, + p.attribute_id, + p.priority + -- p.source_id <-- unnecessary with the priority uniqueness constraint +) +SELECT + r.resource_id, + jsonb_object_agg( + a.name, + COALESCE( + to_jsonb(r.value_text), + to_jsonb(r.value_num), + r.value_json + ) + ) AS attributes +FROM resolved AS r +JOIN og_field_attributes AS a + ON a.attribute_id = r.attribute_id +GROUP BY r.resource_id; + +``` +Note: The above can be paged and totaled as well with minimal alteration. +**single resource detailed provenance** ```sql SELECT p.resource_id, @@ -236,9 +290,9 @@ JOIN oil_gas_field_source_values AS v AND v.attribute_id = p.attribute_id JOIN og_field_attributes AS a ON a.attribute_id = p.attribute_id -JOIN oil_gas_field_sources AS s +JOIN og_field_sources AS s ON s.source_id = p.source_id -JOIN source_key AS sk +JOIN og_source_keys AS sk ON sk.source_key_id = s.source_key_id JOIN user_source_key_permission AS permission ON permission.source_key_id = s.source_key_id @@ -248,99 +302,7 @@ WHERE p.resource_id = $2 ORDER BY p.priority, p.source_id; ``` -| Today | After | -| ----------------------------------------- | ---------------------------------------------------- | -| Tiered `ROW_NUMBER()` over four sort keys | `ORDER BY priority` on one column | -| Curated and default rows form two tiers | Every candidate has an explicit row; no tiers | -| NULL values skipped by the query | A priority row exists only where a value exists (FK) | -| Complex CTE chain | Joins | - -Two consequences to handle in the migration: - -- `MembershipModel.source` and the override table's `source` column currently FK to - `og_field_source_priority.source`. Dropping that table needs a new target for - that closed set (a `og_field_source_keys` table, or a CHECK). -- `og_field_memberships` has a surrogate `id` primary key and no unique constraint - on `(resource_id, source_pk)`. Adding one is a prerequisite for the priority - table to enforce "the source is attached to this resource" by foreign key. -- `is_override` is part of the public per-field source-values response and is - currently derived from "an override row exists." With every row explicit, it - needs an explicit `is_curated` flag to survive. - -## 4. Invariants - -The point of §3 and §4 is to push these into the schema. Being explicit about which -ones the database can enforce and which remain obligations on the sync is what -makes the sync mechanism a free choice. - -**Enforced declaratively** - -| # | Invariant | Mechanism | -| --- | ---------------------------------------------------------------------------- | ----------------------------------------------------- | -| 1 | A priority row references a source attached to that resource | composite FK to membership `(resource_id, source_pk)` | -| 2 | A priority row references a populated `(source, attribute)` value | composite FK to the value table | -| 3 | Two sources cannot share a priority position for one `(resource, attribute)` | UNIQUE `(resource_id, attribute_id, priority)` | -| 4 | A source appears at most once per `(resource, attribute)` | primary key | -| 5 | `priority >= 0` | CHECK | -| 6 | Attribute references come from the closed attribute set | FK to `og_field_attributes` | -| 7 | One state row per `(resource, profile)` | primary key | -| 8 | `profile` is one of the four values | CHECK or enum | -| 9 | A state row references an existing resource | FK | - -**Sync obligations** (not expressible as row constraints; a trigger or a -consistency check job, decided with §6) - -| # | Obligation | -| --- | ---------------------------------------------------------------------------- | -| A | Only resources with `repointed_id IS NULL` appear in the state table | -| B | All four profile rows exist for every resource in the state table | -| C | A provenance entry exists exactly when the matching value column is non-null | -| D | A provenance source key is visible in that row's profile | - -Obligation C is expressible as a jsonb CHECK in Postgres but not in SQLite, so -whether to enforce it in the schema depends on the STIT-603 outcome. - -## 5. Refresh - -Deferred by design. Any implementation must uphold §5 and must be triggered by: -a source value write, a priority change, a membership status change, a resource -repoint (merge), and a bulk ETL load. - -The candidate mechanisms are application-transaction maintenance plus a full -rebuild command, database triggers, or a scheduled rebuild. The choice turns -partly on an open question: the ETL apps live in a separate repository and may -write source data without passing through the API's write paths. - -## 6. Tradeoffs and risks - -- **The four-profile bound rests on the Auth0 policy, not the schema.** - `permissions.licensed_sources()` derives the set from individual - `source:read:` claims, so a partial public grant is representable. If one - were ever issued, that user would read values from a source they are not - licensed for. Mitigation: map a caller's claims to a profile at request time by - checking only `wm` and `ccr`, and fail loudly on an unmappable set. -- **`detail` is only half solved.** Its `source_data` payload needs every - candidate row, not just winners, so the candidate query stays on that path. -- **Write amplification.** One source value write can rewrite up to four state - rows. -- **Two-table schema changes.** Adding a field to `OilGasFieldBase` becomes a - migration against both `og_field_attributes` and the state table's columns, - where today it is a change to `ATTRIBUTE_KINDS` and a CHECK. -- **Priority row volume.** Rows scale as resources × sources × populated - attributes rather than the current handful. Needs an estimate before approval. -- **`og_field_attributes` is a second source of truth** for the attribute set, - alongside `OilGasFieldBase`. It needs a startup drift guard like the existing - `OGFieldName` check. - -## 8. Open questions +## 8. Open Issues 1. Priority row count at current and projected resource volume. -2. Should `filter-options` read `SELECT DISTINCT` from the state table, or does it - warrant its own small per-profile distinct-values table? -3. Do the ETL apps write source data directly to the database, bypassing the API? -4. Migration and backfill sequencing: can the state table be built and validated - against the live query path before any endpoint reads from it? -5. Does anything depend on priority numbers being globally unique per source, as - `og_field_source_priority.priority` is today? - -## Related Documents and Links +2. How to handle priorities upon merge? From bf8e329186f15e7a04a3aae713ef88fd90db2c0f Mon Sep 17 00:00:00 2001 From: Michael Barlow Date: Fri, 18 Sep 2026 11:49:34 -0600 Subject: [PATCH 4/8] delete outdated/unused files --- docs/02_schema_variants.sql | 342 ------------------ ...-state view design drafted and proposed.md | 23 -- 2 files changed, 365 deletions(-) delete mode 100644 docs/02_schema_variants.sql delete mode 100644 docs/STIT-766 Materialized current-state view design drafted and proposed.md diff --git a/docs/02_schema_variants.sql b/docs/02_schema_variants.sql deleted file mode 100644 index 021fc76b..00000000 --- a/docs/02_schema_variants.sql +++ /dev/null @@ -1,342 +0,0 @@ --- Schema variants and optional extensions --- Target database: PostgreSQL 15+ --- --- This file is not intended to be executed as a single migration. Each section --- is an alternative to, or extension of, the canonical schema in 01_schema.sql. - -------------------------------------------------------------------------------- --- Variant A: direct view grants only -------------------------------------------------------------------------------- --- Use when the only key-scoped permission is "view". The presence of a row --- means the user can view sources with that key. - -CREATE TABLE user_source_key_view_permission ( - user_id uuid NOT NULL, - source_key_id bigint NOT NULL, - granted_at timestamptz NOT NULL DEFAULT now(), - - PRIMARY KEY (user_id, source_key_id), - - FOREIGN KEY (user_id) - REFERENCES app_user(user_id) - ON DELETE CASCADE, - - FOREIGN KEY (source_key_id) - REFERENCES source_key(source_key_id) - ON DELETE CASCADE -); - -------------------------------------------------------------------------------- --- Variant B: role-based grants -------------------------------------------------------------------------------- --- Use when many users share the same effective visibility profile. This can --- substantially reduce duplicate permission rows and enables caching by role --- or visibility profile instead of by individual user. - -CREATE TABLE app_role ( - role_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - role_name text NOT NULL UNIQUE -); - -CREATE TABLE user_role ( - user_id uuid NOT NULL, - role_id bigint NOT NULL, - - PRIMARY KEY (user_id, role_id), - - FOREIGN KEY (user_id) - REFERENCES app_user(user_id) - ON DELETE CASCADE, - - FOREIGN KEY (role_id) - REFERENCES app_role(role_id) - ON DELETE CASCADE -); - -CREATE TABLE role_source_key_permission ( - role_id bigint NOT NULL, - source_key_id bigint NOT NULL, - action source_key_action NOT NULL, - - PRIMARY KEY (role_id, source_key_id, action), - - FOREIGN KEY (role_id) - REFERENCES app_role(role_id) - ON DELETE CASCADE, - - FOREIGN KEY (source_key_id) - REFERENCES source_key(source_key_id) - ON DELETE CASCADE -); - -CREATE VIEW effective_user_source_key_permission AS -SELECT DISTINCT - ur.user_id, - rp.source_key_id, - rp.action -FROM user_role AS ur -JOIN role_source_key_permission AS rp - ON rp.role_id = ur.role_id; - -------------------------------------------------------------------------------- --- Variant C: groups plus direct grants -------------------------------------------------------------------------------- --- Effective grants are the union of direct grants and group-derived grants. - -CREATE TABLE app_group ( - group_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - group_name text NOT NULL UNIQUE -); - -CREATE TABLE user_group ( - user_id uuid NOT NULL, - group_id bigint NOT NULL, - - PRIMARY KEY (user_id, group_id), - - FOREIGN KEY (user_id) - REFERENCES app_user(user_id) - ON DELETE CASCADE, - - FOREIGN KEY (group_id) - REFERENCES app_group(group_id) - ON DELETE CASCADE -); - -CREATE TABLE group_source_key_permission ( - group_id bigint NOT NULL, - source_key_id bigint NOT NULL, - action source_key_action NOT NULL, - - PRIMARY KEY (group_id, source_key_id, action), - - FOREIGN KEY (group_id) - REFERENCES app_group(group_id) - ON DELETE CASCADE, - - FOREIGN KEY (source_key_id) - REFERENCES source_key(source_key_id) - ON DELETE CASCADE -); - -CREATE VIEW effective_user_source_key_permission AS -SELECT - p.user_id, - p.source_key_id, - p.action -FROM user_source_key_permission AS p - -UNION - -SELECT - ug.user_id, - gp.source_key_id, - gp.action -FROM user_group AS ug -JOIN group_source_key_permission AS gp - ON gp.group_id = ug.group_id; - -------------------------------------------------------------------------------- --- Variant D: retain a wide source-data table -------------------------------------------------------------------------------- --- If source data must remain in physical columns, use a companion presence --- table to expose only populated attributes to the priority system. A trigger --- or ingestion process must keep this table synchronized. - -CREATE TABLE source_attribute_presence ( - source_id bigint NOT NULL, - attribute_id bigint NOT NULL, - - PRIMARY KEY (source_id, attribute_id), - - FOREIGN KEY (source_id) - REFERENCES source(source_id) - ON DELETE CASCADE, - - FOREIGN KEY (attribute_id) - REFERENCES attribute_definition(attribute_id) -); - --- In this variant, replace the priority FK to source_attribute_value with: --- --- ALTER TABLE resource_attribute_priority --- DROP CONSTRAINT resource_attribute_priority_source_value_fk, --- ADD CONSTRAINT resource_attribute_priority_source_presence_fk --- FOREIGN KEY (source_id, attribute_id) --- REFERENCES source_attribute_presence(source_id, attribute_id); - -------------------------------------------------------------------------------- --- Variant E: allow multiple source records per key for the same resource -------------------------------------------------------------------------------- --- The canonical schema already permits this. Add a uniqueness constraint only --- if the domain requires at most one source per source key per resource. --- PostgreSQL cannot enforce this using only resource_source because source_key --- resides on source. A denormalized source_key_id column or trigger is needed. - -ALTER TABLE resource_source - ADD COLUMN source_key_id bigint; - -ALTER TABLE resource_source - ADD CONSTRAINT resource_source_key_fk - FOREIGN KEY (source_key_id) - REFERENCES source_key(source_key_id); - -ALTER TABLE resource_source - ADD CONSTRAINT resource_source_one_source_per_key_uq - UNIQUE (resource_id, source_key_id); - --- Synchronize resource_source.source_key_id with source.source_key_id in the --- application or with a trigger. - -------------------------------------------------------------------------------- --- Variant F: nullable priority values with automatic fallback -------------------------------------------------------------------------------- --- Not recommended with the canonical model. The canonical source_attribute_value --- table stores only populated values, so priority rows always reference a value. --- If null-valued rows must be retained for audit purposes, separate "observed" --- from "candidate" values instead of weakening the FK invariant. - -CREATE TABLE source_attribute_observation ( - observation_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - source_id bigint NOT NULL REFERENCES source(source_id) ON DELETE CASCADE, - attribute_id bigint NOT NULL REFERENCES attribute_definition(attribute_id), - value_text text, - value_num numeric, - value_json jsonb, - observed_at timestamptz NOT NULL DEFAULT now(), - - CHECK (num_nonnulls(value_text, value_num, value_json) <= 1), - CHECK (value_json IS NULL OR value_json <> 'null'::jsonb) -); - --- Only non-null observations should be promoted into source_attribute_value. - -------------------------------------------------------------------------------- --- Variant G: row-level security for source values -------------------------------------------------------------------------------- --- This variant assumes every transaction sets: --- --- SET LOCAL app.user_id = ''; --- --- The application role must not have BYPASSRLS. Table owners normally bypass --- RLS unless FORCE ROW LEVEL SECURITY is enabled. - -ALTER TABLE source_attribute_value ENABLE ROW LEVEL SECURITY; -ALTER TABLE source_attribute_value FORCE ROW LEVEL SECURITY; - -CREATE POLICY source_attribute_value_select_policy -ON source_attribute_value -FOR SELECT -USING ( - EXISTS ( - SELECT 1 - FROM source AS s - JOIN user_source_key_permission AS p - ON p.source_key_id = s.source_key_id - AND p.action = 'view' - WHERE s.source_id = source_attribute_value.source_id - AND p.user_id = current_setting('app.user_id', true)::uuid - ) -); - --- Consider corresponding policies on source and resource_attribute_priority if --- the existence or provenance of restricted sources is itself sensitive. - -------------------------------------------------------------------------------- --- Variant H: permission-aware resolution view -------------------------------------------------------------------------------- --- A view cannot accept a user_id parameter. This form expects app.user_id to be --- set transaction-locally. SECURITY INVOKER is the default and is preferred. - -CREATE VIEW resolved_resource_attribute AS -SELECT DISTINCT ON ( - p.resource_id, - p.attribute_id -) - p.resource_id, - p.attribute_id, - p.source_id, - s.source_key_id, - p.priority, - v.value_text, - v.value_num, - v.value_json -FROM resource_attribute_priority AS p -JOIN source_attribute_value AS v - ON v.source_id = p.source_id - AND v.attribute_id = p.attribute_id -JOIN source AS s - ON s.source_id = p.source_id -JOIN user_source_key_permission AS permission - ON permission.source_key_id = s.source_key_id - AND permission.user_id = current_setting('app.user_id', true)::uuid - AND permission.action = 'view' -ORDER BY - p.resource_id, - p.attribute_id, - p.priority, - p.source_id; - -------------------------------------------------------------------------------- --- Variant I: permission-aware resolution function -------------------------------------------------------------------------------- --- Convenient when the database receives an explicit, trusted user identifier. --- Do not expose this function to clients that may substitute arbitrary user IDs --- unless authorization is enforced elsewhere. - -CREATE FUNCTION resolved_resource_values(p_user_id uuid) -RETURNS TABLE ( - resource_id bigint, - attribute_id bigint, - source_id bigint, - source_key_id bigint, - priority integer, - value_text text, - value_num numeric, - value_json jsonb -) -LANGUAGE sql -STABLE -SECURITY INVOKER -AS $$ - SELECT DISTINCT ON ( - p.resource_id, - p.attribute_id - ) - p.resource_id, - p.attribute_id, - p.source_id, - s.source_key_id, - p.priority, - v.value_text, - v.value_num, - v.value_json - FROM resource_attribute_priority AS p - JOIN source_attribute_value AS v - ON v.source_id = p.source_id - AND v.attribute_id = p.attribute_id - JOIN source AS s - ON s.source_id = p.source_id - JOIN user_source_key_permission AS permission - ON permission.source_key_id = s.source_key_id - AND permission.user_id = p_user_id - AND permission.action = 'view' - ORDER BY - p.resource_id, - p.attribute_id, - p.priority, - p.source_id -$$; - -------------------------------------------------------------------------------- --- Variant J: dense, contiguous priority positions -------------------------------------------------------------------------------- --- UNIQUE(resource_id, attribute_id, priority) prevents duplicates but does not --- prevent gaps. Enforce contiguous positions in application logic, a deferred --- constraint trigger, or avoid positional integers entirely by storing a stable --- rank value that permits insertion between neighbors. --- --- Examples of alternatives: --- * numeric rank: 1000, 2000, 3000, insert 1500 between first and second --- * explicit linked list: predecessor_source_id --- * full replacement API: delete and reinsert the complete ordered list diff --git a/docs/STIT-766 Materialized current-state view design drafted and proposed.md b/docs/STIT-766 Materialized current-state view design drafted and proposed.md deleted file mode 100644 index aaa097ca..00000000 --- a/docs/STIT-766 Materialized current-state view design drafted and proposed.md +++ /dev/null @@ -1,23 +0,0 @@ -- one initial issue: sqlite does not support materialized views, we may need to drop support for sqlite in general - -Objective -Alter the database structure to eliminate all slow responses, focusing on `list`, `filter-options`, and `detail` endpoints. - -Proposal -- drop the base priority table, all new resources + sources get rows in the `resource_source_priority` table -- add a `og_field_attributes` table: `id`, `attr_name`, `type`(optional) - -## Important Invariants - -The schema should enforces these relationships: - -1. A source can participate in a resource only through `resource_source`. -2. A priority entry can reference only a source attached to the resource. -3. A priority entry can reference only a populated source attribute. -4. A source attribute contains exactly one typed value. -5. Two sources cannot occupy the same priority position for the same resource - and attribute. -6. A source cannot appear twice in the priority list for the same resource and - attribute. -7. Visibility is determined by the source's key, not by the resource or - attribute independently. From 83996660b11c3149291400013319cb89765a8de4 Mon Sep 17 00:00:00 2001 From: Michael Barlow Date: Fri, 18 Sep 2026 11:50:02 -0600 Subject: [PATCH 5/8] delete mermaid test doc --- docs/01_schema.md | 63 ----------------------------------------------- 1 file changed, 63 deletions(-) delete mode 100644 docs/01_schema.md diff --git a/docs/01_schema.md b/docs/01_schema.md deleted file mode 100644 index 4199de79..00000000 --- a/docs/01_schema.md +++ /dev/null @@ -1,63 +0,0 @@ -```mermaid -erDiagram - app_user { - uuid user_id - } - resource { - bigint resource_id - timestamptz created_at - } - source_key { - bigint source_key_id - text key_name - } - source { - bigint source_id - bigint source_key_id - text external_id - timestamptz created_at - } - source_key ||--o{ source : "" - attribute_definition { - bigint attribute_id - text name - text description - text unit - timestamptz created_at - } - resource_source { - bigint resource_id - bigint source_id - timestamptz attached_at - } - resource ||--o{ resource_source : "" - source ||--o{ resource_source : "" - source_attribute_value { - bigint source_id - bigint attribute_id - text value_text - numeric value_num - jsonb value_json - timestamptz created_at - timestamptz updated_at - } - source ||--o{ source_attribute_value : "" - attribute_definition ||--o{ source_attribute_value : "" - resource_attribute_priority { - bigint resource_id - bigint attribute_id - bigint source_id - integer priority - timestamptz created_at - } - resource_source ||--o{ resource_attribute_priority : "" - source_attribute_value ||--o{ resource_attribute_priority : "" - user_source_key_permission { - uuid user_id - bigint source_key_id - source_key_action action - timestamptz granted_at - } - app_user ||--o{ user_source_key_permission : "" - source_key ||--o{ user_source_key_permission : "" -``` From 37991400c5826a76e7cb027495765a6b245564d2 Mon Sep 17 00:00:00 2001 From: Michael Barlow Date: Fri, 18 Sep 2026 11:54:24 -0600 Subject: [PATCH 6/8] fix section numbering --- docs/STIT-766-design.md | 51 ++++++++++++++++++++++++----------------- 1 file changed, 30 insertions(+), 21 deletions(-) diff --git a/docs/STIT-766-design.md b/docs/STIT-766-design.md index 857dc82c..1722b6b8 100644 --- a/docs/STIT-766-design.md +++ b/docs/STIT-766-design.md @@ -76,7 +76,7 @@ Thus, whatever structure we use here is bounded by at most 4 x the number of top > > so even if we add 2 proprietary sources, this structure grows to a max of 16 x the number of top-level (unmerged) resources. I don't see this as a real issue even in the long term. How many licensed sources are even realistic? 6? 10? Even at 10 proprietary sources, the view/table would be 2^10 = 1024 x resources. The upper bound of **all** named fields is probably somewhere in the ~65k range, so ~65M rows is still manageable...if we'd ever even get close to that. -## 2. Goals and non-goals +## 3. Goals and non-goals **Goals** @@ -92,14 +92,16 @@ Thus, whatever structure we use here is bounded by at most 4 x the number of top - The details of the sync mechanism. See §6. - Redesigning the permission model. -## 3. Schema Changes +## 4. Schema Changes Part of the goal is to trade application complexity for schema complexity. Making schemas more complex but in a way that enables tighter constraints and simpler queries (i.e. through mostly joins) frees our application code from being the enforcer of invariants. ### Resource State View/Table + The purpose is to provide a durable store that houses the flattened/coalesced resources. We're effectively precomputing the coalescing logic and saving it to a single table. The actual schema is less important than its function and the constraints we place on it. It must: + - only store unmerged resources (i.e. where `repointed_id == NULL`), merging triggers deletion from the table - provide highly performant, permission-scoped querying of resource data - not expose proprietary information to unlicensed users @@ -107,8 +109,9 @@ It must: - be able to be rebuilt from scratch at any time, we should be able to derive the data for the table easily & quickly The top contender for a schema is: + ```mermaid -erDiagram +erDiagram og_field_resource_view { bignt resource_id smallint permission_mask @@ -117,25 +120,25 @@ erDiagram ``` **permission mask** -The `permission_mask` is a bitmask where `public` = 0, `cc` = 1, `wm` = 2, and `wm + cc` = 3. +The `permission_mask` is a bitmask where `public` = 0, `cc` = 1, `wm` = 2, and `wm + cc` = 3. -| wm | cc |perm | -| --- | --- | --- | -| 0 | 0 | 0 | -| 0 | 1 | 1 | -| 1 | 0 | 2 | -| 1 | 1 | 3 | +| wm | cc | perm | +| --- | --- | ---- | +| 0 | 0 | 0 | +| 0 | 1 | 1 | +| 1 | 0 | 2 | +| 1 | 1 | 3 | This allows for 2 filtering options: + - strict `permission_mask = ` (incurs minor cost of possibly duplicating data across rows) - bit comparison to filter where `( | permission_mask) = ` - - if we sort by `permission_mask` (desc), this would allow us to only store the minimum number of resource variants - - for example, if a resource had ALL public sources as the highest priorities, we'd only need 1 row in the table with `permission_mask = 0` because ALL users would see the same version - - or if a resource had only `wm` and `public` variants, a `wm + cc` permission would get the `wm` version: 3 (`cc + wm`) | 2 (`wm`) = 3 - - but a `cc` permission would skip the `wm` row and see the `public` variant: - 1 (`cc`) | 2 (`wm`) = 3 => exclude - 1 (`cc`) | 0 (`pub`) = 1 => include - + - if we sort by `permission_mask` (desc), this would allow us to only store the minimum number of resource variants + - for example, if a resource had ALL public sources as the highest priorities, we'd only need 1 row in the table with `permission_mask = 0` because ALL users would see the same version + - or if a resource had only `wm` and `public` variants, a `wm + cc` permission would get the `wm` version: 3 (`cc + wm`) | 2 (`wm`) = 3 + - but a `cc` permission would skip the `wm` row and see the `public` variant: + 1 (`cc`) | 2 (`wm`) = 3 => exclude + 1 (`cc`) | 0 (`pub`) = 1 => include > [!NOTE] Permission Alternative > We can also use permission columns for the minor cost of duplicating data across columns. Benefits from being a simpler more understandable approach. @@ -145,6 +148,7 @@ We'd effectively house the entire flat Pydantic model in json. The main reasonin **sync overview** Very roughly speaking, when a user or process updates relevant data (merge resources, reprioritize, new sources from ETL), we compute the updated view state(s), and write them to the table, deleting where necessary. + ```mermaid flowchart LR A[db updates] -->|Trigger| B[compute coalesced state] @@ -158,7 +162,7 @@ flowchart LR ### Attribute metadata: `og_field_attributes` ```mermaid -erDiagram +erDiagram og_field_attributes { serial id text name @@ -169,6 +173,7 @@ erDiagram Priority rows and EAV rows reference `og_field_attribute.id` ### Single priority store: `og_field_resource_attribute_priority` + Replace the two-table priority split with a single table at the `(resource, attribute, source record)` grain, and use defaults when writing new data rather than as a SQL fallback rule. @@ -193,11 +198,12 @@ erDiagram `UNIQUE (resource_id, attribute_id, priority)` - priority > 0 - fk constraint to memberships on resource_id, source_id: ensure source is attached to resource - - fk constraint to `og_field_source_values` on `source_id, attribute_id` + - fk constraint to `og_field_source_values` on `source_id, attribute_id` - `oil_gas_field_source_values` currently has an `id` primary key, so we could replace `(attribute_id, source_id)` with `source_value_id` - we set `is_curated` to `True` when a user makes an update ### Optional Additional Tables + These are primarily for some convenience in constructing simpler SQL statements and permissions handling. ```mermaid @@ -219,11 +225,12 @@ erDiagram } ``` -Where the `source_key_permission` is a new ENUM type with `read`, `edit`. +Where the `source_key_permission` is a new ENUM type with `read`, `edit`. What this would allow is comparatively smaller and more straightforward SQL statements. **single resource for a user** + ```sql WITH resolved AS ( SELECT DISTINCT ON ( @@ -271,9 +278,11 @@ JOIN og_field_attributes AS a GROUP BY r.resource_id; ``` + Note: The above can be paged and totaled as well with minimal alteration. **single resource detailed provenance** + ```sql SELECT p.resource_id, @@ -302,7 +311,7 @@ WHERE p.resource_id = $2 ORDER BY p.priority, p.source_id; ``` -## 8. Open Issues +## 5. Open Issues 1. Priority row count at current and projected resource volume. 2. How to handle priorities upon merge? From 761c823391ffddc059aed4cfcc529f4c58f9a106 Mon Sep 17 00:00:00 2001 From: Michael Barlow Date: Fri, 18 Sep 2026 11:56:49 -0600 Subject: [PATCH 7/8] rename supporting docs --- ...03_schema_operations_api.md => draft-schema-operations-api.md} | 0 docs/{01_schema.sql => draft-schema.sql} | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename docs/{03_schema_operations_api.md => draft-schema-operations-api.md} (100%) rename docs/{01_schema.sql => draft-schema.sql} (100%) diff --git a/docs/03_schema_operations_api.md b/docs/draft-schema-operations-api.md similarity index 100% rename from docs/03_schema_operations_api.md rename to docs/draft-schema-operations-api.md diff --git a/docs/01_schema.sql b/docs/draft-schema.sql similarity index 100% rename from docs/01_schema.sql rename to docs/draft-schema.sql From c2b2e63928ab313fc2e86ed23501c4a345f50a0f Mon Sep 17 00:00:00 2001 From: MB <5711147+mbarlow12@users.noreply.github.com> Date: Fri, 2 Oct 2026 09:41:18 -0600 Subject: [PATCH 8/8] Update docs/STIT-766-design.md Co-authored-by: Alex Axthelm --- docs/STIT-766-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/STIT-766-design.md b/docs/STIT-766-design.md index 1722b6b8..902dec16 100644 --- a/docs/STIT-766-design.md +++ b/docs/STIT-766-design.md @@ -2,7 +2,7 @@ **Status:** draft **Author:** Michael Barlow -**Reviewer:** John McGrath +**Reviewer:** John McGrath, Alex Axthelm **Jira:** [STIT-766](https://rmi1.atlassian.net/browse/STIT-766) (epic [STIT-689](https://rmi1.atlassian.net/browse/STIT-689)) **Created**: 2026-09-17