Skip to content

feat(schema): retrofit storage_parameters onto existing partition children - #160

Merged
tthayer merged 1 commit into
mainfrom
worktree-iga-3716-protoc-gen-pgdb-retrofit-storage-parameters
Jul 31, 2026
Merged

feat(schema): retrofit storage_parameters onto existing partition children#160
tthayer merged 1 commit into
mainfrom
worktree-iga-3716-protoc-gen-pgdb-retrofit-storage-parameters

Conversation

@tthayer

@tthayer tthayer commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

What was broken

PR #159 (v0.12.2) made storage_parameters reach partition children at creation
time
. Nothing retrofitted children that already existed.

Which strategy that actually mattered for:

  • Tenant-list partitions (partitioned = true) — the real problem. They never
    roll over. A child created before the model declared the option keeps the default
    forever, silently.
  • KSUID and date partitions self-heal: retention rolls old children out and new ones
    are created with the declared WITH (...).

The change

At the end of Migrations():

if isPartitionedParent(desc) && desc.GetStorageParameters() != nil {
    children, err := readPartitionChildStorageParameters(ctx, db, desc)
    ...
    rv = append(rv, partitionChildAlters(desc, children, maxPartitionChildStorageParamAlters)...)
}

Three deliberate properties:

  1. Emitted last. The consumer (c1 pkg/postgres/xpgdb/schema.go) breaks out of a
    table's statement loop on the first 55P03 lock_timeout. Putting child ALTERs last
    means a contended tenant partition can only cost other child ALTERs — never the
    column-adds and index statements ahead of them.
  2. Capped at 50 per run (maxPartitionChildStorageParamAlters), ordered by child
    name.
    Bounds lock churn on a many-tenant table — each ALTER TABLE ... SET (...)
    takes a brief ACCESS EXCLUSIVE lock. Self-advancing: converged children render
    nothing, so the next run's first 50 are the next 50 that actually drift.
  3. Diff-driven. Reuses the existing per-key diff via a new
    storageParams2alterTable(desc, tableName, existingParams); storageParams2alter
    is now a one-line wrapper, so TestStorageParams2alter is unchanged and still
    passes. No blanket re-SET, no churn on already-correct children.

Also folds the duplicated reloptions key=value parsing into a shared
parseReloptions used by both the parent and the new child reader. The new
pg_inherits reader mirrors the existing readStorageParameters shape (same goqu
parameterization, same nspname = 'public' scoping).

Only costs an extra catalog query when the model is a partitioned parent and
declares storage parameters. Migrations still early-returns CreateSchema when the
table does not exist, so the retrofit block never runs on a fresh table.

Tests

pgdb/v1/schema_sql_test.goTestPartitionChildAlters: 120 drifted children
truncate to exactly 50 in name order (first is parent_000, last parent_049); a
converged set emits nothing; a descriptor with no declaration emits nothing.

example/models/food/v1/food_test.goTestPartitionChildStorageParamRetrofit
simulates the real case end to end against a live Postgres: create the tenant-list
partitioned Pasta table (declares fillfactor: 90) plus three tenant children,
assert Migrations emits nothing while they are converged, then
ALTER TABLE <child> RESET (fillfactor) on one of the three to look like a
pre-declaration child. Asserts exactly that child is retrofitted, applies the ALTER,
verifies fillfactor = 90 landed, and asserts a third Migrations call emits nothing.

Negative test confirmed: with the retrofit block disabled,
TestPartitionChildStorageParamRetrofit fails with "[]" should have 1 item(s), but has 0 / only the stale child should be retrofitted. The test genuinely guards the
behaviour rather than passing vacuously.

Residual limitation

execReconcile in the consumer notes that a 55P03 skip is "not self-healing
in-process" — it retries only on the next EnsureSchema, i.e. a redeploy. So a large
retrofit converges across deploys rather than within one: at 50/run, a 5,000-tenant
table needs ~100 deploys. Related: because the consumer abandons the remainder of a
table's statements on the first lock_timeout, a perpetually contended child head-of-line
blocks the children after it in name order for that run. Both are inherent to the
consumer's break-on-lock_timeout semantics, and both are strictly better than the status
quo, where nothing converged at all. maxPartitionChildStorageParamAlters is the knob
if the deploy count ever matters.

Linear: IGA-3716

Note: the ticket's "route lock behaviour through reconcileLockTimeout / 55P03"
criterion required no code here — that machinery lives in the consumer, not this
library.

🤖 Generated with Claude Code

…ldren

Children created before a model declared storage_parameters previously kept
the default forever. Tenant-list partitions never roll over, so nothing would
ever fix them; KSUID and date partitions only self-heal via retention.

Migrations now diffs each child's reloptions and emits an ALTER per drifted
child, last in the statement list and capped at 50 per run. Last because the
consumer abandons a table's remaining statements on the first lock_timeout, so
a contended partition must not cost the column and index statements. Capped to
bound lock churn on a many-tenant table; converged children render nothing, so
successive runs advance through the set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@linear-code

linear-code Bot commented Jul 31, 2026

Copy link
Copy Markdown

IGA-3716

@tthayer
tthayer marked this pull request as ready for review July 31, 2026 21:29
@tthayer
tthayer merged commit dbdb1f7 into main Jul 31, 2026
4 checks passed
@tthayer
tthayer deleted the worktree-iga-3716-protoc-gen-pgdb-retrofit-storage-parameters branch July 31, 2026 21:33
arreyder pushed a commit that referenced this pull request Aug 21, 2026
Adds autovacuum_vacuum_insert_threshold and
autovacuum_vacuum_insert_scale_factor (PG13+) to StorageParameters, fields
13 and 14.

Why these are not covered by the existing pair. vacuum_threshold and
vacuum_scale_factor trigger on DEAD tuples, so on an append-mostly table
they may rarely fire however tightly they are set -- dead tuples accumulate
slowly relative to inserts, while freshly-inserted pages pile up
all-visible-but-unmarked. That stale visibility map is what makes an Index
Only Scan fall back to heap fetches, which shows up as a poor buffer-cache
hit ratio on a scan that should be cheap.

The insert-driven pair triggers on inserts since the last vacuum instead,
which is what refreshes the visibility map. PostgreSQL added them for
exactly this gap, and until now there was no way to declare them here: a
dead-tuple trigger is the wrong knob for that symptom no matter how
aggressively it is set.

Both render paths are covered: storageParams2with for CREATE, and
storageParams2alterTable for the drift-detection ALTER, so the retrofit
machinery from #160 applies these to existing tables -- which matters,
since a table large enough to need this is not one anybody recreates.
The float parameter goes through needsUpdate's epsilon comparison like the
other floats.

Tests: one case per path. The retrofit case has the threshold already
correct and the scale factor drifted, asserting only the drifted parameter
is emitted.

Note: five tests in example/models/food/v1 (TestSchemaFoodPasta,
TestPastaIngredientBitVector, TestPastaIngredientBitVectorRetrieval,
TestKSUIDCollationV17, TestPartitionChildStorageParamRetrofit) fail
identically on clean origin/main in my environment and are unrelated to
this change -- they appear to need a live PostgreSQL. Verified by stashing
and re-running.

Regenerated with protoc-gen-go v1.36.11 to match the version main was built
with, so the diff carries no toolchain bump.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant