Repository navigation
Add webhook and NATS sinks - #15
Merged
Merged
Conversation
Two sinks that widen who can consume GTC without adding infrastructure: - **Webhook**: one signed POST per event to any HTTP endpoint, so a serverless function or third-party service can consume changes with no Redis or search dependency. Deliveries carry X-GTC-Event-Id (the LSN) for deduplication, plus table/operation headers for routing, and an optional HMAC-SHA256 signature over "<timestamp>.<body>" so receivers can verify the payload came from GTC and reject replays. Non-2xx and timeouts fail the event, so it retries and dead-letters rather than being lost. - **NATS**: publishes to <prefix>.<schema>.<table> through JetStream by default, which acknowledges each publish and — because the message ID is the event's LSN — collapses the duplicates at-least-once redelivery produces. The capturing stream is auto-created, matching the plug-and-play behavior of slot/publication creation. Core NATS remains available but is fire-and-forget, and doctor warns about it. Subjects are sanitized: wildcards and whitespace legal in PostgreSQL identifiers but not in NATS subjects become underscores. Both reuse the existing per-table filtering, transforms, resilience and DLQ layers, and appear in the dashboard's sink table automatically. Neither package depends on the Redis sink: each declares the small interface it needs and modules.go supplies the shared key resolver. Supporting changes: - Rename RedisKeyedSinkConfig/RedisTableConfig to KeyedSinkConfig/TableConfig now that four sinks share them, and give every keyed sink the mirror-everything default when no sink YAML is present. Without this the new sinks silently delivered nothing, which the integration test caught. - doctor: validate the webhook URL (warning on unsigned or plain-HTTP delivery) without probing the endpoint, verify NATS connectivity and that JetStream is actually enabled, and warn when an enabled sink selects no tables — a silent-delivery footgun of the config-file convention. - Tests: webhook against a real httptest server (signing, auth, filtering, non-2xx, timeout, cancellation), NATS against an in-process nats-server with JetStream (publish, LSN dedupe, auto-created stream, core-NATS mode, health, sanitization). The integration test now runs a real HTTP receiver the gateway posts to, and asserts JetStream delivery when GTC_TEST_NATS_URL is set; CI provides it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019dEj24Ta2o9QxzW9ZrSPtb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PR 3 of 3 — base is #14 (
claude/gtc-ddl), so review only the top commit; GitHub retargets as the stack merges. This one has a real dependency on #13: it adds checks todoctor.go.What does this PR do?
Two sinks that widen who can consume GTC without adding infrastructure.
Webhook
One signed POST per event to any HTTP endpoint — a serverless function, an internal service, a third-party integration — with no Redis or search dependency.
X-GTC-Event-IdX-GTC-Table/X-GTC-OperationX-GTC-Timestamp+X-GTC-Signature"<timestamp>.<body>"— the timestamp is bound into the signature so replays can be rejectedAny 2xx is success; anything else, or a timeout, fails the event so it retries and dead-letters rather than being silently lost. The receiver sits in the replication path, so
WEBHOOK_TIMEOUTdefaults to 5s andgtc doctorchecks the sink-count × timeout budget againstwal_sender_timeout.NATS
Publishes to
<prefix>.<schema>.<table>through JetStream by default, for two reasons: JetStream acknowledges each publish (success means persisted, not just sent), and using the event's LSN as the message ID makes the server collapse the duplicates that at-least-once redelivery produces. The capturing stream auto-creates, matching how slots and publications already work. Core NATS remains available viaNATS_JETSTREAM=falsebut is fire-and-forget, so doctor warns that events vanish when nobody is subscribed. Subjects are sanitized — wildcards and whitespace that PostgreSQL allows in identifiers but NATS rejects in subjects become underscores.Both reuse the existing per-table filtering, transforms, resilience and DLQ layers, and appear in the dashboard's sink table automatically. Neither package imports the Redis sink: each declares the small interface it needs, and
modules.gosupplies the shared key resolver.Supporting changes
RedisKeyedSinkConfig/RedisTableConfig→KeyedSinkConfig/TableConfignow that four sinks share them, and gave every keyed sink the mirror-everything default when no sink YAML is present.doctor: validate the webhook URL (warning on unsigned or plain-HTTP delivery) without probing it, verify NATS connectivity and that JetStream is actually enabled, and warn when an enabled sink selects no tables.Type of change
Checklist
go test ./...passes (including-race)gofmt/go vetare cleanhttptestserver (signing, auth, filtering, non-2xx, timeout, cancellation), NATS against an in-process nats-server with JetStream (publish, LSN dedupe, auto-created stream, core-NATS mode, health, sanitization). The integration test runs a real HTTP receiver the gateway posts to and asserts JetStream delivery; CI now starts anats:2 -jscontainer.README.md/CLAUDE.mdupdated (new sink section with signature-verification example, all new env vars,config/sinks.example.yaml)Notes for the reviewer
A bug the integration test caught: with the new YAML sections added, both sinks initialized fine and then silently skipped every event — the "mirror everything when there's no sink YAML" default only covered the two Redis sinks. Fixed, and I added the general
doctorwarning above, because a sink that is fully configured yet selects no tables is a silent-delivery footgun of the config-file convention.Not included: ClickHouse. It is a materially bigger lift than these two — target-table DDL and type mapping, batching for acceptable throughput, and a real interaction with #14 (an upstream
ALTERimplies a target migration). It deserves its own PR rather than a shallow version bolted onto this one.🤖 Generated with Claude Code
https://claude.ai/code/session_019dEj24Ta2o9QxzW9ZrSPtb
Generated by Claude Code