Skip to content

docs(db): Draft design for DB refactor to improve performance and simplify (STIT-766) - #279

Open
mbarlow12 wants to merge 9 commits into
mainfrom
design/db-state-refactor
Open

mbarlow12 wants to merge 9 commits into
mainfrom
design/db-state-refactor

Conversation

@mbarlow12

@mbarlow12 mbarlow12 commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Attempt 1 at a design doc for a DB architecture proposal.

Design Doc (rendered)

Would welcome feedback along 2 separate axes:

  1. The design itself.
    I've tried to hit a relatively high level of abstraction to provide freedom in implementation for more minute details. Any glaring oversights? Potential bugs? Missing components?
  2. Presentation.
    Unclear wording or missing sections? Too detailed or not detailed enough?

AI helped with an initial draft, but I essentially overwrote the entire thing. I also used AI to draft draft-schema.sql and draft-schema-operations-api.md files. They're very early iterations and are largely outdated, but they represent a rough initial solution to the db structural issues.

@github-actions

Copy link
Copy Markdown

CD summary 8399666

Database (1)
db_name postgres_host postgres_port postgres_db
pr_0279 stitch-dev.postgres.database.azure.com 5432 pr_0279
Images (4)
build_time commit_time git_sha image image_digest
2026-09-18T17:52:26Z 2026-09-18T17:52:07Z 3a41c37 ghcr.io/rmi/stitch-api:pr-0279 ghcr.io/rmi/stitch-api:pr-0279@sha256:ea4eebf824117a19d566cb9c8f9d5edce8eeb44e15275ad177503c273c75382d
2026-09-18T17:52:25Z 2026-09-18T17:52:07Z 3a41c37 ghcr.io/rmi/stitch-entity-linkage:pr-0279 ghcr.io/rmi/stitch-entity-linkage:pr-0279@sha256:c45c7eae3d959263de3c5a2d716b703691655b765bab7cd56f677a55a916c5b0
2026-09-18T17:52:26Z 2026-09-18T17:52:07Z 3a41c37 ghcr.io/rmi/stitch-seed:pr-0279 ghcr.io/rmi/stitch-seed:pr-0279@sha256:2e36f7b1441fd214a2a2caf5db59404fccd9fe5d380a15905ab199ccf85a6501
2026-09-18T17:52:25Z 2026-09-18T17:52:07Z 3a41c37 ghcr.io/rmi/stitch-stitch-llm:pr-0279 ghcr.io/rmi/stitch-stitch-llm:pr-0279@sha256:970d5c451bf8493d2ea1c7dc28620ff0fd5707f4cb5e4eb80225ed9347aa3b8c

@github-actions

Copy link
Copy Markdown

CD summary 3799140

Database (1)
db_name postgres_host postgres_port postgres_db
pr_0279 stitch-dev.postgres.database.azure.com 5432 pr_0279
Images (4)
build_time commit_time git_sha image image_digest
2026-09-18T17:55:18Z 2026-09-18T17:54:35Z f1cb57c ghcr.io/rmi/stitch-api:pr-0279 ghcr.io/rmi/stitch-api:pr-0279@sha256:e2cea949199016e474d2f15cb5b42e281d31c479a8c940a750eb9912a26a6643
2026-09-18T17:55:19Z 2026-09-18T17:54:35Z f1cb57c ghcr.io/rmi/stitch-entity-linkage:pr-0279 ghcr.io/rmi/stitch-entity-linkage:pr-0279@sha256:450bc7e9c8bb0c30d3ff6d0f5591140106eee11951d93e9c3dec5d2d2e64b8d9
2026-09-18T17:55:18Z 2026-09-18T17:54:35Z f1cb57c ghcr.io/rmi/stitch-seed:pr-0279 ghcr.io/rmi/stitch-seed:pr-0279@sha256:fbeed77c90f8d1a8ae10ee06d2dde35b18187a78061b628561b88ea362cfdebd
2026-09-18T17:55:20Z 2026-09-18T17:54:35Z f1cb57c ghcr.io/rmi/stitch-stitch-llm:pr-0279 ghcr.io/rmi/stitch-stitch-llm:pr-0279@sha256:5d3a8479df3800c2c21f23817b77c30349a4c54e848fc752812abd85f3157baa

@github-actions

Copy link
Copy Markdown

CD summary ce2e06f

Database (1)
db_name postgres_host postgres_port postgres_db
pr_0279 stitch-dev.postgres.database.azure.com 5432 pr_0279
Jobs (1)
job image postgres_db
db-migrations ghcr.io/rmi/stitch-api:pr-0279@sha256:f2707c67a42f86b6da89ddbb65864b8bb041b9403f25416824310292c8c4183d pr_0279
Images (4)
build_time commit_time git_sha image image_digest
2026-09-18T17:56:27Z 2026-09-18T17:55:42Z 5b43d27 ghcr.io/rmi/stitch-api:pr-0279 ghcr.io/rmi/stitch-api:pr-0279@sha256:f2707c67a42f86b6da89ddbb65864b8bb041b9403f25416824310292c8c4183d
2026-09-18T17:56:27Z 2026-09-18T17:55:42Z 5b43d27 ghcr.io/rmi/stitch-entity-linkage:pr-0279 ghcr.io/rmi/stitch-entity-linkage:pr-0279@sha256:de6c95e3d392dbd6353c9396882d07641a86d30d35eb2106ef2047c44a7e05e1
2026-09-18T17:56:28Z 2026-09-18T17:55:42Z 5b43d27 ghcr.io/rmi/stitch-seed:pr-0279 ghcr.io/rmi/stitch-seed:pr-0279@sha256:2b9b6bacc12ce6f47c9a6942b9a512e4ee15b8216fcd215b136dd2f1600f04b6
2026-09-18T17:56:29Z 2026-09-18T17:55:42Z 5b43d27 ghcr.io/rmi/stitch-stitch-llm:pr-0279 ghcr.io/rmi/stitch-stitch-llm:pr-0279@sha256:6bdb510aeb48f43e39615fb06020f2438b3c0fffc77cc16250cf46f22b0aa469

@github-actions

Copy link
Copy Markdown

CD summary 761c823

Frontend: https://witty-mushroom-017a3dc1e-279.westus2.1.azurestaticapps.net

Deployments (4)
service url fqdn
api open pr-0279-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io
entity-linkage open pr-0279-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io
frontend https://witty-mushroom-017a3dc1e-279.westus2.1.azurestaticapps.net
stitch-llm open pr-0279-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io
Database (1)
db_name postgres_host postgres_port postgres_db
pr_0279 stitch-dev.postgres.database.azure.com 5432 pr_0279
Jobs (1)
job image postgres_db
db-migrations ghcr.io/rmi/stitch-api:pr-0279@sha256:539a0c72039d84f6e4b8a12bb346c209e6ea787341350008706cbda4d0257bcf pr_0279
Images (4)
build_time commit_time git_sha image image_digest
2026-09-18T17:57:38Z 2026-09-18T17:57:00Z 18df0f8 ghcr.io/rmi/stitch-api:pr-0279 ghcr.io/rmi/stitch-api:pr-0279@sha256:539a0c72039d84f6e4b8a12bb346c209e6ea787341350008706cbda4d0257bcf
2026-09-18T17:57:34Z 2026-09-18T17:57:00Z 18df0f8 ghcr.io/rmi/stitch-entity-linkage:pr-0279 ghcr.io/rmi/stitch-entity-linkage:pr-0279@sha256:02f78ae66c7fd0e6252d784a739434a82c37850640ea5071149beeb7be5d5de2
2026-09-18T17:57:37Z 2026-09-18T17:57:00Z 18df0f8 ghcr.io/rmi/stitch-seed:pr-0279 ghcr.io/rmi/stitch-seed:pr-0279@sha256:c308e54799ac800f27c50540a562768e170cb7cd99f743066d58706a343d9dbf
2026-09-18T17:57:35Z 2026-09-18T17:57:00Z 18df0f8 ghcr.io/rmi/stitch-stitch-llm:pr-0279 ghcr.io/rmi/stitch-stitch-llm:pr-0279@sha256:ea0a6018690b07af41bb6c47bcf13c759f72967636725050ccbee35fc66e1680

Comment thread docs/STIT-766-design.md
- user with `wm + cc` sees `wm`
- user with `wm` sees `wm`
- user with `cc` sees `cc`
- remaining users see `gem`

@john john Sep 23, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be gem, or "all combined public sources?" Like why couldnt' they see GEM+BC? Same question repeats below. You say it above, just confirming in the bullets too.

Re-read, think I get it--the presumption is if both BC and GEM have coords, we'll defaul to GEM?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. This was one spot where I suspected a diagram/table would be more illustrative.

Comment thread docs/STIT-766-design.md

The purpose is to provide a durable store that houses the flattened/coalesced resources. We're effectively precomputing the coalescing logic and saving it to a single table. The actual schema is less important than its function and the constraints we place on it.

It must:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We talked about this and it's inherent, but just to be clear/pedantic, it must also be re-creatable. Like if this table is inadvertently dropped, it would straightforward to recreate it by marching through the resources and re-precomputing. We should be careful to maintain that, ie not add to the table in the future in a way that it would hold data that could be recontructed--so we will be adding complexity, but it's low-risk complexity.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@john Is that sufficiently captured in:

  • be able to be rebuilt from scratch at any time, we should be able to derive the data for the table easily & quickly

Or is there some rewording you think it needs?

Comment thread docs/STIT-766-design.md
> We can also use permission columns for the minor cost of duplicating data across columns. Benefits from being a simpler more understandable approach.

**record column as jsonb**
We'd effectively house the entire flat Pydantic model in json. The main reasoning is that we can likely expand to near-instant full-text search without much difficulty. It's also partially as an experiment to assess the difficulty of working with Postgres JSON syntax and investigate whether there are performance trade-offs. Should it prove easy to use while still being performant, it opens the door to using it in other places across our application where we might want greater flexibility in our data handling.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having done this before, JSON search semantics are weird but not hard. But also we can encapsulate them so they can be used or not, and it's hidden, and can be changed without effects on calling code. Also I suspect it's avoidable in most other cases, which we may want to aim for just to reduce the set of things most devs need to understand.

Comment thread docs/STIT-766-design.md
@AlexAxthelm AlexAxthelm mentioned this pull request Sep 28, 2026

This branch was successfully deployed

1 active (outdated) deployment
development — 761c8233 Deployed Sep 23, 2026 by AlexAxthelm via drop-db #289
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants