Repository navigation
End-to-end integration test in CI — and the startup bug it caught - #12
Merged
Merged
Conversation
…connection PostgreSQL walsender connections reject the extended (parameterized) query protocol, so the pg_publication and pg_replication_slots existence checks failed with SQLSTATE 08P01 on every fresh start — meaning auto-creation of the slot and publication (the plug-and-play first-run path) never succeeded and the reader retried until SLOT_RETRY_TIMEOUT. Run both checks as simple-protocol queries with the name embedded as a quoted literal (names come from config, quoting is still applied). Found by the new end-to-end integration test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019dEj24Ta2o9QxzW9ZrSPtb
Runs the compiled gateway as a subprocess against a live PostgreSQL (wal_level=logical) and Redis, exercising the full pipeline the unit tests mock at the WAL boundary: - auto-create of the replication slot and publication on first start - automatic backfill: READ events for rows that existed before start - live streaming: INSERT/UPDATE/DELETE arrive on the Redis stream in per-key order with the right payloads (duplicates tolerated, as at-least-once semantics allow) - /api/stats agrees with the stream, and SIGTERM shuts down cleanly Excluded from `go test ./...` by an `integration` build tag; a new CI job starts postgres:16 and redis:7 with docker run (job services can't override the container command, and Postgres needs -c wal_level=logical) and runs it on every PR. CONTRIBUTING.md and CLAUDE.md document how to run it locally. The test's first run immediately caught the replication-connection protocol bug fixed in the previous commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019dEj24Ta2o9QxzW9ZrSPtb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a real end-to-end test — the compiled gateway running against a live PostgreSQL (
wal_level=logical) and Redis — and fixes the genuine startup bug it caught on its very first run.The bug: the WAL reader's existence checks for the publication and replication slot used parameterized queries (extended query protocol), which PostgreSQL walsender connections reject with
SQLSTATE 08P01. Auto-creation of the slot and publication — the plug-and-play first-run path — never succeeded: the reader retried untilCDC_SLOT_RETRY_TIMEOUTand the gateway never became ready on a fresh database. Both checks now run as simple-protocol queries with the names embedded as quoted literals (names come from config; quoting is still applied).The test (
test/integration, behind anintegrationbuild tag sogo test ./...is unaffected) builds the gateway binary, starts it as a subprocess, and asserts:/readinessflips to 200READevents on the Redis streamINSERT/UPDATE/DELETEarrive in per-key order with correct payloads (duplicates collapsed — at-least-once semantics allow them;DELETEcarries nonew_data)/api/statsagrees with the stream (streaming true, events counted)On failure it dumps the gateway's full log into the test output. It cleans up all state it creates (table, publication, slot, stream, backfill resume-state table) so back-to-back runs pass.
CI: a new
integrationjob startspostgres:16 -c wal_level=logicalandredis:7via plaindocker run(job services can't override the container command, which the wal_level flag requires) and runs the test on every PR, with service-log dumps on failure.Type of change
Checklist
go test ./...passes (including-race; the integration test ran twice back-to-back locally against real services, ~12s each)gofmt/go vetare cleanREADME.md/CLAUDE.mdupdated if env vars or behavior changed (CONTRIBUTING.md + CLAUDE.md document the local invocation)Notes for the reviewer
🤖 Generated with Claude Code
https://claude.ai/code/session_019dEj24Ta2o9QxzW9ZrSPtb
Generated by Claude Code