Repository navigation
Detect and report schema (DDL) changes - #14
Merged
Merged
Conversation
PostgreSQL's logical decoding does not stream DDL, but pgoutput re-describes
a published table whenever its shape changed, right before that table's next
row change. Diffing consecutive relation descriptions turns that into a
first-class feature with no DDL triggers, no schema-history topic and no
superuser:
- Detects columns added, dropped and retyped, primary key / replica identity
changes, and renames (the table OID survives a rename, which is how a
rename is told apart from a new table). Type OIDs resolve to names.
- Classifies each change: adding a column is additive, everything else can
break consumers holding documents for the table.
- Reports through every channel the product already uses — WARN logs for
breaking changes, cdc_schema_changes_total{schema,table,kind}, a rolling
history at GET /api/schema, a new dashboard page, and (with REDIS_URL) a
"<prefix>:schema" notification stream gated by CDC_SCHEMA_EVENTS.
Schema notifications are deliberately best-effort, unlike row events: a
reconnect re-describes tables with nothing to diff against, so a change
cannot be re-derived on redelivery. ports.SchemaObserver therefore returns
no error and a failed publish is counted in cdc_schema_publish_errors_total
instead of stalling replication. Row delivery keeps at-least-once unchanged.
Relation messages carry no WAL position of their own, so a change is stamped
with the commit LSN of the transaction it arrived in, and documented as the
position where the change was detected rather than where the DDL committed.
The integration test now performs real DDL against PostgreSQL and asserts
both the API and the notification stream see it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dEj24Ta2o9QxzW9ZrSPtb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PR 2 of 3 — base is #13 (
claude/gtc-doctor), so review only the top commit; GitHub retargets this tomainonce #13 merges. The stacking is for a clean diff, not a code dependency.What does this PR do?
PostgreSQL's logical decoding has no
ALTER TABLEevent — which is exactly why Debezium needs a schema-history topic. But pgoutput re-describes a published table whenever its shape changed, immediately before that table's next row change. Diffing consecutive descriptions turns that into a first-class feature with no DDL triggers, no history topic, no superuser — it works on RDS.I verified this behavior against a real PostgreSQL before building on it.
Detected, with the effect of the change rather than the statement text:
column_addedcolumn_droppedcolumn_type_changedkey_changedREPLICA IDENTITYalteredreplica_identity_changedtable_renamedRenames are told apart from new tables by the relation OID, which survives them. Type OIDs resolve to names (
int4→int8), falling back tooid:<n>for user-defined types.Reported through every channel the product already uses: WARN logs for breaking changes (so existing log alerting catches them),
cdc_schema_changes_total{schema,table,kind}, a rolling history atGET /api/schema, a new dashboard Schema page, and — withREDIS_URL— a<prefix>:schemanotification stream gated byCDC_SCHEMA_EVENTS.Type of change
Checklist
go test ./...passes (including-race)gofmt/go vetare cleanALTER TABLEs and asserts both the API and the notification stream)README.md/CLAUDE.mdupdated (new "Schema changes (DDL)" section,CDC_SCHEMA_EVENTS, endpoint table, delivery-semantics note)Notes for the reviewer
Schema notifications are deliberately best-effort, unlike row events. A reconnect re-describes tables with nothing to diff against, so a missed change genuinely cannot be re-derived on redelivery — at-least-once is not achievable here. Rather than imply otherwise,
ports.SchemaObserverreturns no error by design, a failed publish incrementscdc_schema_publish_errors_totaland logs, and row delivery's at-least-once guarantee is untouched.Two further limitations follow from the mechanism and are documented in the README:
Also: relation messages carry no WAL position of their own (the LSN came through as
0/0), so a change is stamped with the commit LSN of the transaction it arrived in, and documented as where it was detected rather than where the DDL committed.🤖 Generated with Claude Code
https://claude.ai/code/session_019dEj24Ta2o9QxzW9ZrSPtb
Generated by Claude Code