Skip to content

Add webhook and NATS sinks - #15

Merged
emoss08 merged 1 commit into
claude/gtc-ddlfrom
claude/gtc-sinks
Aug 12, 2026
Merged

emoss08 merged 1 commit into
claude/gtc-ddlfrom
claude/gtc-sinks

Conversation

@emoss08

@emoss08 emoss08 commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Stacked PR 3 of 3 — base is #14 (claude/gtc-ddl), so review only the top commit; GitHub retargets as the stack merges. This one has a real dependency on #13: it adds checks to doctor.go.

What does this PR do?

Two sinks that widen who can consume GTC without adding infrastructure.

Webhook

One signed POST per event to any HTTP endpoint — a serverless function, an internal service, a third-party integration — with no Redis or search dependency.

Header Purpose
X-GTC-Event-Id The LSN — dedupe on this, delivery is at-least-once
X-GTC-Table / X-GTC-Operation Routing without parsing the body
X-GTC-Timestamp + X-GTC-Signature HMAC-SHA256 over "<timestamp>.<body>" — the timestamp is bound into the signature so replays can be rejected

Any 2xx is success; anything else, or a timeout, fails the event so it retries and dead-letters rather than being silently lost. The receiver sits in the replication path, so WEBHOOK_TIMEOUT defaults to 5s and gtc doctor checks the sink-count × timeout budget against wal_sender_timeout.

NATS

Publishes to <prefix>.<schema>.<table> through JetStream by default, for two reasons: JetStream acknowledges each publish (success means persisted, not just sent), and using the event's LSN as the message ID makes the server collapse the duplicates that at-least-once redelivery produces. The capturing stream auto-creates, matching how slots and publications already work. Core NATS remains available via NATS_JETSTREAM=false but is fire-and-forget, so doctor warns that events vanish when nobody is subscribed. Subjects are sanitized — wildcards and whitespace that PostgreSQL allows in identifiers but NATS rejects in subjects become underscores.

Both reuse the existing per-table filtering, transforms, resilience and DLQ layers, and appear in the dashboard's sink table automatically. Neither package imports the Redis sink: each declares the small interface it needs, and modules.go supplies the shared key resolver.

Supporting changes

  • Renamed RedisKeyedSinkConfig/RedisTableConfig → KeyedSinkConfig/TableConfig now that four sinks share them, and gave every keyed sink the mirror-everything default when no sink YAML is present.
  • doctor: validate the webhook URL (warning on unsigned or plain-HTTP delivery) without probing it, verify NATS connectivity and that JetStream is actually enabled, and warn when an enabled sink selects no tables.

Type of change

  • New feature / sink

Checklist

  • go test ./... passes (including -race)
  • gofmt / go vet are clean
  • Tests added or updated for changed behavior — webhook against a real httptest server (signing, auth, filtering, non-2xx, timeout, cancellation), NATS against an in-process nats-server with JetStream (publish, LSN dedupe, auto-created stream, core-NATS mode, health, sanitization). The integration test runs a real HTTP receiver the gateway posts to and asserts JetStream delivery; CI now starts a nats:2 -js container.
  • README.md / CLAUDE.md updated (new sink section with signature-verification example, all new env vars, config/sinks.example.yaml)
  • If this touches delivery semantics: both sinks are at-least-once like every other sink — failures propagate so the WAL position does not advance. NATS additionally deduplicates redelivered events server-side by LSN.

Notes for the reviewer

A bug the integration test caught: with the new YAML sections added, both sinks initialized fine and then silently skipped every event — the "mirror everything when there's no sink YAML" default only covered the two Redis sinks. Fixed, and I added the general doctor warning above, because a sink that is fully configured yet selects no tables is a silent-delivery footgun of the config-file convention.

Not included: ClickHouse. It is a materially bigger lift than these two — target-table DDL and type mapping, batching for acceptable throughput, and a real interaction with #14 (an upstream ALTER implies a target migration). It deserves its own PR rather than a shallow version bolted onto this one.

🤖 Generated with Claude Code

https://claude.ai/code/session_019dEj24Ta2o9QxzW9ZrSPtb


Generated by Claude Code

Two sinks that widen who can consume GTC without adding infrastructure:

- **Webhook**: one signed POST per event to any HTTP endpoint, so a
  serverless function or third-party service can consume changes with no
  Redis or search dependency. Deliveries carry X-GTC-Event-Id (the LSN) for
  deduplication, plus table/operation headers for routing, and an optional
  HMAC-SHA256 signature over "<timestamp>.<body>" so receivers can verify
  the payload came from GTC and reject replays. Non-2xx and timeouts fail
  the event, so it retries and dead-letters rather than being lost.
- **NATS**: publishes to <prefix>.<schema>.<table> through JetStream by
  default, which acknowledges each publish and — because the message ID is
  the event's LSN — collapses the duplicates at-least-once redelivery
  produces. The capturing stream is auto-created, matching the plug-and-play
  behavior of slot/publication creation. Core NATS remains available but is
  fire-and-forget, and doctor warns about it. Subjects are sanitized:
  wildcards and whitespace legal in PostgreSQL identifiers but not in NATS
  subjects become underscores.

Both reuse the existing per-table filtering, transforms, resilience and DLQ
layers, and appear in the dashboard's sink table automatically. Neither
package depends on the Redis sink: each declares the small interface it
needs and modules.go supplies the shared key resolver.

Supporting changes:
- Rename RedisKeyedSinkConfig/RedisTableConfig to KeyedSinkConfig/TableConfig
  now that four sinks share them, and give every keyed sink the
  mirror-everything default when no sink YAML is present. Without this the
  new sinks silently delivered nothing, which the integration test caught.
- doctor: validate the webhook URL (warning on unsigned or plain-HTTP
  delivery) without probing the endpoint, verify NATS connectivity and that
  JetStream is actually enabled, and warn when an enabled sink selects no
  tables — a silent-delivery footgun of the config-file convention.
- Tests: webhook against a real httptest server (signing, auth, filtering,
  non-2xx, timeout, cancellation), NATS against an in-process nats-server
  with JetStream (publish, LSN dedupe, auto-created stream, core-NATS mode,
  health, sanitization). The integration test now runs a real HTTP receiver
  the gateway posts to, and asserts JetStream delivery when
  GTC_TEST_NATS_URL is set; CI provides it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dEj24Ta2o9QxzW9ZrSPtb
@emoss08
emoss08 merged commit 119a652 into main Aug 12, 2026
7 checks passed
@emoss08
emoss08 deleted the claude/gtc-sinks branch August 12, 2026 01:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants