feat(schema): retrofit storage_parameters onto existing partition children - #160
Merged
tthayer merged 1 commit intoJul 31, 2026
Conversation
…ldren Children created before a model declared storage_parameters previously kept the default forever. Tenant-list partitions never roll over, so nothing would ever fix them; KSUID and date partitions only self-heal via retention. Migrations now diffs each child's reloptions and emits an ALTER per drifted child, last in the statement list and capped at 50 per run. Last because the consumer abandons a table's remaining statements on the first lock_timeout, so a contended partition must not cost the column and index statements. Capped to bound lock churn on a many-tenant table; converged children render nothing, so successive runs advance through the set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tthayer
marked this pull request as ready for review
July 31, 2026 21:29
tthayer
deleted the
worktree-iga-3716-protoc-gen-pgdb-retrofit-storage-parameters
branch
July 31, 2026 21:33
arreyder
pushed a commit
that referenced
this pull request
Aug 21, 2026
Adds autovacuum_vacuum_insert_threshold and autovacuum_vacuum_insert_scale_factor (PG13+) to StorageParameters, fields 13 and 14. Why these are not covered by the existing pair. vacuum_threshold and vacuum_scale_factor trigger on DEAD tuples, so on an append-mostly table they may rarely fire however tightly they are set -- dead tuples accumulate slowly relative to inserts, while freshly-inserted pages pile up all-visible-but-unmarked. That stale visibility map is what makes an Index Only Scan fall back to heap fetches, which shows up as a poor buffer-cache hit ratio on a scan that should be cheap. The insert-driven pair triggers on inserts since the last vacuum instead, which is what refreshes the visibility map. PostgreSQL added them for exactly this gap, and until now there was no way to declare them here: a dead-tuple trigger is the wrong knob for that symptom no matter how aggressively it is set. Both render paths are covered: storageParams2with for CREATE, and storageParams2alterTable for the drift-detection ALTER, so the retrofit machinery from #160 applies these to existing tables -- which matters, since a table large enough to need this is not one anybody recreates. The float parameter goes through needsUpdate's epsilon comparison like the other floats. Tests: one case per path. The retrofit case has the threshold already correct and the scale factor drifted, asserting only the drifted parameter is emitted. Note: five tests in example/models/food/v1 (TestSchemaFoodPasta, TestPastaIngredientBitVector, TestPastaIngredientBitVectorRetrieval, TestKSUIDCollationV17, TestPartitionChildStorageParamRetrofit) fail identically on clean origin/main in my environment and are unrelated to this change -- they appear to need a live PostgreSQL. Verified by stashing and re-running. Regenerated with protoc-gen-go v1.36.11 to match the version main was built with, so the diff carries no toolchain bump.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was broken
PR #159 (v0.12.2) made
storage_parametersreach partition children at creationtime. Nothing retrofitted children that already existed.
Which strategy that actually mattered for:
partitioned = true) — the real problem. They neverroll over. A child created before the model declared the option keeps the default
forever, silently.
are created with the declared
WITH (...).The change
At the end of
Migrations():Three deliberate properties:
pkg/postgres/xpgdb/schema.go) breaks out of atable's statement loop on the first
55P03lock_timeout. Putting child ALTERs lastmeans a contended tenant partition can only cost other child ALTERs — never the
column-adds and index statements ahead of them.
maxPartitionChildStorageParamAlters), ordered by childname. Bounds lock churn on a many-tenant table — each
ALTER TABLE ... SET (...)takes a brief ACCESS EXCLUSIVE lock. Self-advancing: converged children render
nothing, so the next run's first 50 are the next 50 that actually drift.
storageParams2alterTable(desc, tableName, existingParams);storageParams2alteris now a one-line wrapper, so
TestStorageParams2alteris unchanged and stillpasses. No blanket re-SET, no churn on already-correct children.
Also folds the duplicated
reloptionskey=valueparsing into a sharedparseReloptionsused by both the parent and the new child reader. The newpg_inheritsreader mirrors the existingreadStorageParametersshape (same goquparameterization, same
nspname = 'public'scoping).Only costs an extra catalog query when the model is a partitioned parent and
declares storage parameters.
Migrationsstill early-returnsCreateSchemawhen thetable does not exist, so the retrofit block never runs on a fresh table.
Tests
pgdb/v1/schema_sql_test.go—TestPartitionChildAlters: 120 drifted childrentruncate to exactly 50 in name order (first is
parent_000, lastparent_049); aconverged set emits nothing; a descriptor with no declaration emits nothing.
example/models/food/v1/food_test.go—TestPartitionChildStorageParamRetrofitsimulates the real case end to end against a live Postgres: create the tenant-list
partitioned
Pastatable (declaresfillfactor: 90) plus three tenant children,assert
Migrationsemits nothing while they are converged, thenALTER TABLE <child> RESET (fillfactor)on one of the three to look like apre-declaration child. Asserts exactly that child is retrofitted, applies the ALTER,
verifies
fillfactor = 90landed, and asserts a thirdMigrationscall emits nothing.Negative test confirmed: with the retrofit block disabled,
TestPartitionChildStorageParamRetrofitfails with"[]" should have 1 item(s), but has 0 / only the stale child should be retrofitted. The test genuinely guards thebehaviour rather than passing vacuously.
Residual limitation
execReconcilein the consumer notes that a55P03skip is "not self-healingin-process" — it retries only on the next
EnsureSchema, i.e. a redeploy. So a largeretrofit converges across deploys rather than within one: at 50/run, a 5,000-tenant
table needs ~100 deploys. Related: because the consumer abandons the remainder of a
table's statements on the first lock_timeout, a perpetually contended child head-of-line
blocks the children after it in name order for that run. Both are inherent to the
consumer's break-on-lock_timeout semantics, and both are strictly better than the status
quo, where nothing converged at all.
maxPartitionChildStorageParamAltersis the knobif the deploy count ever matters.
Linear: IGA-3716
Note: the ticket's "route lock behaviour through reconcileLockTimeout / 55P03"
criterion required no code here — that machinery lives in the consumer, not this
library.
🤖 Generated with Claude Code