Repository navigation
Backfill all replays completed tables; backfill status survives restarts - #19
Merged
Merged
Conversation
POST /backfill {"all":true} called EnqueueAll, which skips tables that
already completed, so on a running deployment it queued nothing and
reported success. It now calls ReplayAll: every published table is
replayed from the start, as the README and dashboard describe, and the
response says how many were queued. The startup auto-backfill keeps
EnqueueAll's resume semantics.
Backfill status lived only in memory, so after a restart GET /backfill
and the dashboard showed nothing although gtc_backfill_state recorded
every table as done. The state table gains rows_copied (added in place on
existing installs); the coordinator loads it at start and reports
completed backfills with their row counts and completion time. The
stats table list includes every backfilled table, so it is not empty
until the first live change.
Replaying a table cleared its persisted state in a goroutine after
queueing it; for a small table that delete could land after the replay
saved its completion. The state is now cleared before the table is
queued.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
POST /backfill {"all":true}(and the dashboard's "backfill all") now replays every published table from the start, completed or not, and returns{"status":"enqueued","tables":N}. Before, it calledEnqueueAll, which skips completed tables, so on a running deployment it queued 0 tables and still reported success. Startup auto-backfill keeps the resume-onlyEnqueueAll.gtc_backfill_stategainsrows_copied(ADD COLUMN IF NOT EXISTSon existing installs), and the coordinator loads completed tables into status at start, with row counts and completion time. The dashboard's table list includes every backfilled table, so it is not empty after a restart.Found running GTC in production: after a redeploy the dashboard showed nothing, and "backfill all" silently did nothing.
Testing
ReplayAllre-queues completed tables (andEnqueueAllstill skips them) and does not double-queue; progress saves carryrows_copied.go vet ./...,go test ./...,go test -raceon the backfill package.🤖 Generated with Claude Code