Skip to content

feat(schema): insert-driven autovacuum storage parameters - #165

Open
arreyder wants to merge 1 commit into
mainfrom
crr/autovacuum-insert-threshold
Open

feat(schema): insert-driven autovacuum storage parameters#165
arreyder wants to merge 1 commit into
mainfrom
crr/autovacuum-insert-threshold

Conversation

@arreyder

@arreyder arreyder commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Adds autovacuum_vacuum_insert_threshold and autovacuum_vacuum_insert_scale_factor (PG13+) to StorageParameters, fields 13 and 14.

Why the existing pair doesn't cover this

autovacuum_vacuum_threshold / autovacuum_vacuum_scale_factor trigger on dead tuples. On an append-mostly table that means they may rarely fire however tightly they are set: dead tuples accumulate slowly relative to inserts, while freshly-inserted pages pile up all-visible-but-unmarked. That stale visibility map is what makes an Index Only Scan fall back to heap fetches, which surfaces as a poor buffer-cache hit ratio on a scan that ought to be cheap.

The insert-driven pair triggers on inserts since the last vacuum, which is what refreshes the visibility map. PostgreSQL added them for exactly this gap. On a large append-mostly table the dead-tuple trigger is the wrong knob for that symptom, and tightening it further does not become the right one.

The generator plumbing is a hand-maintained mirror — worth knowing about

The storage-parameter path is enumerated parameter-by-parameter in three places: storageParamsTemplateContext, its extraction block in internal/pgdb/pgdb_descriptor.go, and internal/pgdb/templates/descriptor.tmpl. A parameter added only to the .proto and to the SQL renderers is silently dropped before it reaches GetStorageParameters() — and if a message declared only such parameters, hasStorageParams stays false and the whole block emits return nil.

All three sites are updated here. To stop that being re-discovered the hard way, Pet in the example models now declares both parameters and TestPetStorageParameters asserts they survive codegen — the only test in the repo covering declaration → generated descriptor. That also gets the reloption names validated against a real server for free by the existing PostgreSQL-backed animals suite, which is the one thing no unit test can do.

Drift detection: relative epsilon

needsUpdate compared floats with a fixed absolute epsilon of 1e-4. The scale factors worth setting on a large append-mostly table sit at or below that, so e.g. declared 0.001 against an existing 0.0011 (10% off) compared equal — drift was reported as agreement and never converged. It's now relative (1e-9 + 1e-4 * |desired|), which still absorbs float32 round-trip noise; the existing precision-difference case is unchanged, and there are now cases on both sides of that boundary.

Coverage

Both render paths — storageParams2with (CREATE) and storageParams2alterTable (drift-detection ALTER), so the retrofit machinery applies these to existing tables, which is the case that matters since a table big enough to need this is not one anybody recreates. Cases cover: each of the pair set alone (a guard checking the sibling's presence would pass the both-set case), the pair alongside other parameters, -1 (PostgreSQL's documented sentinel for disabling the trigger), both-drifted and one-drifted retrofit, and small-scale-factor drift vs. round-trip noise.

Notes for the reviewer

  • int32 for the threshold, matching the sibling autovacuum_vacuum_threshold and PostgreSQL's 32-bit reloption.
  • Regenerated with protoc-gen-go v1.36.11, the version main was generated with, so the diff carries no spurious toolchain bump. Verified codegen reproduces main byte-for-byte before applying the change.
  • ./pgdb/... and ./example/models/{animals,zoo}/... pass. Five tests in example/models/food/v1 fail in my environment on ERROR: extension "vector" is not available — no pgvector locally; they fail identically on clean main, verified by stashing.
  • Fields are placed at the end in field-number order, matching the file's existing convention.
  • Heads-up: the unmerged branch crr/add-storage-parameters reserves field 13 for default_statistics_target. Nothing on main occupies 13/14, but whichever lands second needs renumbering.

@arreyder
arreyder force-pushed the crr/autovacuum-insert-threshold branch 2 times, most recently from c09152c to 7ef6c41 Compare August 21, 2026 02:47
Adds autovacuum_vacuum_insert_threshold and
autovacuum_vacuum_insert_scale_factor (PG13+) to StorageParameters.

The existing autovacuum_vacuum_threshold / autovacuum_vacuum_scale_factor
pair triggers on DEAD tuples. On an append-mostly table that pair may
rarely fire however tightly it is set: dead tuples accumulate slowly
relative to inserts while freshly-inserted pages pile up all-visible but
unmarked. The stale visibility map is what makes an Index Only Scan fall
back to heap fetches. The insert-driven pair triggers on inserts since
the last vacuum, which is what refreshes the visibility map -- the gap
PostgreSQL added these for.

Both render paths are covered: storageParams2with (CREATE) and
storageParams2alterTable (drift-detection ALTER), so the retrofit
machinery applies these to existing tables -- which is the case that
matters, since a table large enough to need this is not one anybody
recreates.

The generator's storage-parameter plumbing is a hand-maintained mirror of
the proto message (storageParamsTemplateContext, its extraction block,
and descriptor.tmpl all enumerate parameters one by one). A parameter
added only to the .proto and the SQL renderers is silently dropped before
it ever reaches GetStorageParameters(), and if a message declared only
such parameters, hasStorageParams would stay false and the whole
storage_parameters block would emit `return nil`. All three sites are
updated here, and the new example declaration on Pet plus its assertions
in TestPetStorageParameters cover that declaration-to-descriptor path so
the next parameter added cannot regress it silently. That also gets the
reloption names validated against a real server by the existing
PostgreSQL-backed animals suite.

needsUpdate's float comparison used a fixed absolute epsilon of 1e-4.
The scale factors worth setting on a large append-mostly table sit at or
below that, so drift was reported as agreement and never converged. It
is now relative (1e-9 + 1e-4 * |desired|), which still absorbs float32
round-trip noise -- the existing precision-difference case is unchanged.

int32 for the threshold, matching the sibling autovacuum_vacuum_threshold
and PostgreSQL's 32-bit reloption.

Generated with protoc-gen-go v1.36.11, the version main was generated
with, so the diff carries no spurious toolchain bump.
@arreyder
arreyder force-pushed the crr/autovacuum-insert-threshold branch from 7ef6c41 to ad18c7f Compare August 21, 2026 02:48
@arreyder
arreyder requested a review from pquerna August 21, 2026 03:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant