From 344ac5f011ffdcd6ac20c4dbeddbadd004173c74 Mon Sep 17 00:00:00 2001
From: choiyounggi <74581798+choiyounggi@users.noreply.github.com>
Date: Thu, 23 Jul 2026 13:06:54 +0900
Subject: [PATCH] ingest(databases): online schema changes +
 autovacuum/wraparound (from Postgres survival guide)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Two gaps the databases domain didn't cover, both production-severe for a
write-heavy startup, both cross-checked against the official PostgreSQL docs:

- schema-design/online-schema-changes — avoiding the ACCESS EXCLUSIVE outage:
  non-volatile DEFAULT fast path, ADD CONSTRAINT NOT VALID + VALIDATE (SHARE
  UPDATE EXCLUSIVE), the CHECK-proves-NOT-NULL trick, CREATE INDEX CONCURRENTLY,
  ADD FOREIGN KEY's lighter lock, lock_timeout for lock-queue pileups, and
  expand-and-contract to decouple DB migration from app deploy.
- operations/autovacuum-and-wraparound (new category) — per-table scale_factor/
  cost_limit tuning for hot tables, age(datfrozenxid)/relfrozenxid + n_dead_tup
  monitoring, the ~3M-XID read-only cliff and superuser VACUUM recovery (not
  FULL/FREEZE), VACUUM FULL vs pg_repack.

Registers both in databases/index.md (new operations category), appends log.md,
and adds two-way related links from column-data-types and index-write-cost.

Source: Hatchet "Postgres survival guide" + postgresql.org docs (sql-altertable,
sql-createindex, routine-vacuuming). Independently fact-checked against the
official docs; one default corrected (autovacuum_vacuum_scale_factor 0.2, not 0.1).
---
 log.md                                        |  1 +
 wiki/databases/index.md                       |  7 +++
 wiki/databases/indexing/index-write-cost.md   |  2 +-
 .../operations/autovacuum-and-wraparound.md   | 57 ++++++++++++++++++
 .../schema-design/column-data-types.md        |  2 +-
 .../schema-design/online-schema-changes.md    | 58 +++++++++++++++++++
 6 files changed, 125 insertions(+), 2 deletions(-)
 create mode 100644 wiki/databases/operations/autovacuum-and-wraparound.md
 create mode 100644 wiki/databases/schema-design/online-schema-changes.md

diff --git a/log.md b/log.md
index bcc206d..c9fd5a8 100644
--- a/log.md
+++ b/log.md
@@ -32,3 +32,4 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
 ## [2026-07-11] ingest | security +1 (new category api-exposure): exposing-an-origin-http-api — code-vs-edge control split for putting an origin HTTP API behind a public tunnel/proxy. Captures the non-obvious 5xx-bypasses-request-middleware gotcha (set security headers in the exception handler too — verify with a forced 500), header-only fail-closed auth, docs/OpenAPI off at the origin by default, sanitized 5xx bodies, and the code(validate/headers/errors)-vs-edge(TLS/HSTS/rate-limit/WAF) boundary. Derived from hardening korea-data-suite for external exposure + independent pentest; sources OWASP Secure Headers, MDN, Starlette middleware, FastAPI docs.
 ## [2026-07-12] ingest | databases +1 (new category sqlite): concurrent-access-for-a-read-api — WAL + busy_timeout for a read-heavy API with a background writer; set journal_mode once (per-connection pragma cost ~0.4ms measured vs 0.04ms busy_timeout), single writer, run the batch writer as a separate process (interpreter-lock isolation), scale reads with worker processes not threads. Derived from load-testing korea-data-suite (measured connect/query/pragma costs, GIL-bound concurrency, separate-sync-process deployment). Sources: SQLite WAL/pragma/FAQ docs.
 ## [2026-07-12] revise | security/secrets-in-code +1 edge case: third-party HTTP client (httpx/requests) logs the full request URL — including a query-param API key — at INFO, so root/DEBUG logging leaks it; keep the client logger above INFO. Found when a standalone sync process set logging.basicConfig(INFO) and httpx wrote the data.go.kr serviceKey to the log file. last_verified bumped to 2026-07-12.
+## [2026-07-23] ingest | databases +2: schema-design/online-schema-changes (ACCESS EXCLUSIVE lock avoidance — non-volatile default fast path, ADD CONSTRAINT NOT VALID + VALIDATE at SHARE UPDATE EXCLUSIVE, CHECK-NOT-NULL trick, CREATE INDEX CONCURRENTLY, expand-and-contract to decouple DB migration from app deploy, lock_timeout for lock-queue pile-up) + operations/autovacuum-and-wraparound (NEW category operations: per-table scale_factor/cost_limit tuning for hot tables, age(datfrozenxid)/relfrozenxid + n_dead_tup monitoring, wraparound read-only cliff and superuser VACUUM recovery, VACUUM FULL vs pg_repack). Derived from the Hatchet "Postgres survival guide"; both cross-checked against PostgreSQL official docs (sql-altertable, routine-vacuuming).
diff --git a/wiki/databases/index.md b/wiki/databases/index.md
index 2865ff2..aaafc93 100644
--- a/wiki/databases/index.md
+++ b/wiki/databases/index.md
@@ -36,6 +36,13 @@ Match your situation to a "load when" line; load only matching pages.
 | [column-data-types](schema-design/column-data-types.md) | Picking column types: money, time, text, enums, JSON, binary; changing a type on a live table |
 | [nullability-and-defaults](schema-design/nullability-and-defaults.md) | Declaring column nullability/defaults; queries dropping rows around NULLs |
 | [soft-delete](schema-design/soft-delete.md) | Deleted records themselves must be restorable or kept (deleted_at schemas); deciding what a parent's deletion does to children that must survive (for who-changed-what history → requirements-to-tables) |
+| [online-schema-changes](schema-design/online-schema-changes.md) | Running ALTER TABLE / CREATE INDEX on a large table under live traffic; a migration blocks reads/writes (ACCESS EXCLUSIVE); adding a column/constraint/NOT NULL/index/type change safely; expand-and-contract to decouple DB migration from app deploy |
+
+## operations
+
+| Page | Load when |
+|------|-----------|
+| [autovacuum-and-wraparound](operations/autovacuum-and-wraparound.md) | A write-heavy table bloats or slows over time; tuning autovacuum for a hot table; monitoring/preventing transaction-ID wraparound (age(datfrozenxid)); the database starts refusing writes to avoid wraparound; deciding VACUUM vs VACUUM FULL vs pg_repack |
 
 ## sqlite
 
diff --git a/wiki/databases/indexing/index-write-cost.md b/wiki/databases/indexing/index-write-cost.md
index 7119fcf..96a751d 100644
--- a/wiki/databases/indexing/index-write-cost.md
+++ b/wiki/databases/indexing/index-write-cost.md
@@ -8,7 +8,7 @@ sources:
   - https://www.postgresql.org/docs/current/indexes.html
   - https://use-the-index-luke.com/
 last_verified: 2026-07-10
-related: [databases-indexing-index-selection, databases-indexing-covering-indexes]
+related: [databases-indexing-index-selection, databases-indexing-covering-indexes, databases-schema-design-online-schema-changes, databases-operations-autovacuum-and-wraparound]
 ---
 
 # Budgeting Index Maintenance Cost
diff --git a/wiki/databases/operations/autovacuum-and-wraparound.md b/wiki/databases/operations/autovacuum-and-wraparound.md
new file mode 100644
index 0000000..f5a0fd3
--- /dev/null
+++ b/wiki/databases/operations/autovacuum-and-wraparound.md
@@ -0,0 +1,57 @@
+---
+id: databases-operations-autovacuum-and-wraparound
+domain: databases
+category: operations
+applies_to: [postgresql]
+confidence: verified
+sources:
+  - https://www.postgresql.org/docs/current/routine-vacuuming.html
+last_verified: 2026-07-23
+related: [databases-indexing-index-write-cost, databases-query-optimization-reading-execution-plans, databases-schema-design-online-schema-changes]
+---
+
+# Keeping Autovacuum Ahead of a Write-Heavy Table (and Off the Wraparound Cliff)
+
+## When this applies
+
+A table takes sustained `UPDATE`/`DELETE`/`INSERT` traffic in production. Two
+failure modes grow silently: **bloat** (dead tuples autovacuum hasn't reclaimed,
+so scans and indexes slow down) and **transaction-ID wraparound** (unfrozen old
+rows approaching the 32-bit XID limit). Wraparound is the severe one: at the
+cliff, Postgres refuses all writes cluster-wide until you vacuum. Default
+autovacuum is tuned for average tables and can fall behind a hot one.
+
+## Do this
+
+1. **Monitor both clocks on a schedule** (weekly is enough until a table is hot):
+
+   | Signal | Query | Act when |
+   |--------|-------|----------|
+   | Wraparound age | `SELECT datname, age(datfrozenxid) FROM pg_database ORDER BY 2 DESC;` | Investigate above ~150M; `autovacuum_freeze_max_age` default is 200M, and aggressive autovacuum should hold it well below that |
+   | Per-table freeze age | `age(relfrozenxid)` from `pg_stat_user_tables` | A single table climbing while others are flat means autovacuum can't finish it (long txn holding it back, or cost limits too low) |
+   | Bloat | `n_dead_tup`, `last_autovacuum` from `pg_stat_user_tables` | Dead ratio stays high and `last_autovacuum` is stale → autovacuum isn't triggering often enough |
+
+2. **Tune the hot table specifically, not the cluster.** Autovacuum triggers at
+   `threshold + scale_factor × rows`; the default `autovacuum_vacuum_scale_factor`
+   of 0.2 means a 1M-row table must reach ~200k dead tuples first. Lower it
+   per-table so it fires earlier:
+   `ALTER TABLE t SET (autovacuum_vacuum_scale_factor = 0.02, autovacuum_vacuum_cost_limit = 2000);`
+   The default cost limit (200) with a 2ms delay throttles I/O and is often why
+   autovacuum can't keep up on a busy table.
+
+3. **Set `log_autovacuum_min_duration = 0`** so every autovacuum is logged — you
+   need the record to see whether it's running and how long it takes.
+
+## Edge cases
+
+| Case | Then |
+|------|------|
+| An autovacuum on one table runs for over an hour or never seems to finish | Its cost limits are too low for the table's churn, or a long-running transaction / idle-in-transaction session is holding `xmin` back so nothing can be frozen — find it in `pg_stat_activity` and end it |
+| Insert-only table (append log) never gets vacuumed, then triggers a huge aggressive freeze | Insert-triggered autovacuum (PG13+) helps, but also lower the table's `autovacuum_freeze_min_age` so ordinary vacuums freeze rows early and spread the work |
+| Wraparound warning already in the log ("must be vacuumed within N transactions") | Run a plain database-wide `VACUUM` (not `VACUUM FULL` — it needs an XID and fails; not `VACUUM FREEZE` — it does more than needed). First clear what pins `xmin`: long transactions, orphaned prepared transactions (`pg_prepared_xacts`), and stale replication slots (`pg_replication_slots`) |
+| Database already refusing writes (read-only, ~3M XIDs left) | Only `VACUUM` and reads work. Vacuum in single-user mode if needed; this is an incident, so the real fix is the monitoring above so you never reach it |
+| `VACUUM FULL` proposed to reclaim space on a live table | It takes `ACCESS EXCLUSIVE` and rewrites the whole table — an outage. Prefer `pg_repack` for online bloat reclamation; reserve `VACUUM FULL` for a maintenance window |
+
+## Sources
+
+- https://www.postgresql.org/docs/current/routine-vacuuming.html — vacuum's four jobs, wraparound mechanics and thresholds, autovacuum trigger formula and cost settings, recovery steps and the VACUUM FULL/FREEZE caveats
diff --git a/wiki/databases/schema-design/column-data-types.md b/wiki/databases/schema-design/column-data-types.md
index d03cdea..50de921 100644
--- a/wiki/databases/schema-design/column-data-types.md
+++ b/wiki/databases/schema-design/column-data-types.md
@@ -9,7 +9,7 @@ sources:
   - https://wiki.postgresql.org/wiki/Don%27t_Do_This
   - https://dev.mysql.com/doc/refman/8.0/en/data-types.html
 last_verified: 2026-07-10
-related: [databases-schema-design-primary-key-choice, databases-schema-design-nullability-and-defaults, databases-schema-design-requirements-to-tables]
+related: [databases-schema-design-primary-key-choice, databases-schema-design-nullability-and-defaults, databases-schema-design-requirements-to-tables, databases-schema-design-online-schema-changes]
 ---
 
 # Choosing Column Data Types
diff --git a/wiki/databases/schema-design/online-schema-changes.md b/wiki/databases/schema-design/online-schema-changes.md
new file mode 100644
index 0000000..2c564c9
--- /dev/null
+++ b/wiki/databases/schema-design/online-schema-changes.md
@@ -0,0 +1,58 @@
+---
+id: databases-schema-design-online-schema-changes
+domain: databases
+category: schema-design
+applies_to: [postgresql]
+confidence: verified
+sources:
+  - https://www.postgresql.org/docs/current/sql-altertable.html
+last_verified: 2026-07-23
+related: [databases-schema-design-column-data-types, databases-schema-design-foreign-keys-and-referential-actions, databases-indexing-index-write-cost, databases-schema-design-nullability-and-defaults]
+---
+
+# Applying Schema Changes to a Live Table Without Blocking Writes
+
+## When this applies
+
+Running `ALTER TABLE` / `CREATE INDEX` against a table that takes concurrent
+production traffic, on a table large enough that a full scan or rewrite is not
+instant. Most `ALTER TABLE` forms take an `ACCESS EXCLUSIVE` lock — it blocks
+reads *and* writes for the whole operation, so a multi-second rewrite is a
+multi-second outage. The goal: make each change either instant-under-lock or
+lock-light.
+
+## Do this
+
+Split every risky change into an instant metadata step plus a concurrent/validated
+step. Pick the row for your change:
+
+| Change | Do |
+|--------|----|
+| Add a column with a default | Postgres 11+ stores a **non-volatile** default as metadata — instant, no rewrite. Keep the default constant (`DEFAULT now()` is fine; `DEFAULT clock_timestamp()`, a stored generated column, or an identity column force a full rewrite) |
+| Add a `CHECK` or `FOREIGN KEY` constraint | Add it `NOT VALID` (skips the scan, takes the lock only briefly), then `ALTER TABLE ... VALIDATE CONSTRAINT` in a separate statement — validation takes only `SHARE UPDATE EXCLUSIVE`, so writes continue |
+| Make a column `NOT NULL` | First add `CHECK (col IS NOT NULL) NOT VALID`, `VALIDATE` it, then `SET NOT NULL` — a valid CHECK proving no NULLs lets `SET NOT NULL` skip its table scan |
+| Add an index | `CREATE INDEX CONCURRENTLY` — builds without an `ACCESS EXCLUSIVE` lock ([databases-indexing-index-write-cost]). Never inside a transaction block |
+| Add a foreign key | `ADD FOREIGN KEY` needs only `SHARE ROW EXCLUSIVE` (not ACCESS EXCLUSIVE), but still add it `NOT VALID` + `VALIDATE` to avoid the reference scan under lock |
+| Rename/drop a column, or change a type | These force `ACCESS EXCLUSIVE` (and a type change rewrites the whole table by default — see the binary-coercible exception in Edge cases). Use expand-and-contract instead of an in-place change (see below) |
+
+**Expand and contract** — for renames, type changes, and column splits, never
+mutate in place under load. Expand: add the new column/table (instant steps
+above) and dual-write from the app. Migrate: backfill old rows in batches.
+Contract: switch reads to the new shape, then drop the old column in a final
+quick `ACCESS EXCLUSIVE` step. This also decouples the deploy: the app tolerates
+both shapes across the window, so DB migration and app release need not be atomic.
+
+## Edge cases
+
+| Case | Then |
+|------|------|
+| Type change where old type is binary-coercible to new (e.g. `text`→`varchar`, no collation change) | No rewrite, and indexes are not rebuilt — the fast path. Verify against your exact types before assuming instant |
+| `CREATE INDEX CONCURRENTLY` fails midway | It leaves an `INVALID` index that still costs writes; `DROP INDEX` it and retry, don't leave it |
+| The brief `ACCESS EXCLUSIVE` step queues behind a long-running query | The `ALTER` waits for the lock *and every query that arrives behind it also waits* — a lock queue pile-up. Set a short `lock_timeout` on the DDL and retry, so it backs off instead of freezing traffic |
+| Batched backfill during expand | Keep each batch a short transaction and throttle — a single `UPDATE` over the whole table holds row locks and generates dead tuples faster than autovacuum clears them ([databases-operations-autovacuum-and-wraparound]) |
+| Adding a volatile default / generated / identity column is unavoidable | It rewrites the table under `ACCESS EXCLUSIVE`; schedule it as a maintenance-window operation, not a live deploy |
+
+## Sources
+
+- https://www.postgresql.org/docs/current/sql-altertable.html — lock levels per ALTER form, NOT VALID / VALIDATE, non-volatile default fast path, type-change rewrite rules
+- https://www.postgresql.org/docs/current/sql-createindex.html — CREATE INDEX CONCURRENTLY lock behavior