Repository navigation
feat(ingest): reconcile orphaned published parse_jobs (F5) - #23
Merged
Merged
Conversation
… jobs - Add 0012 migration introducing the 'reconciled' parse_job_history action - Add 'reconciled' to the ParseJobHistoryAction union - Add reclaimStalePublishedJobs repository method (FOR UPDATE SKIP LOCKED re-queue of stale published jobs + reconciled history row per reclaim) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…r shape - ParseJobReconciler.reconcileStale calls reclaimStalePublishedJobs and, per reclaimed job, notifies the observer and logs job bindings - Narrow ParseJobReconcilerRepository contract + noop observer/logger defaults - Factory-injection unit test covers populated/empty/noop-default paths Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… task - Surface PARSE_JOB_STALE_AFTER_MS (default 1h) and PARSE_JOB_RECONCILE_BATCH_SIZE (default 25) on config.ingest - createIngestRuntime constructs ParseJobReconciler and runs a third parse-job-reconciler IntervalTask, started/closed with the others - server.ts passes staleAfterMs/reconcileBatchSize through - Update runtime/app/queue/storage constructors for the new fields - Real-pg integration: stale published -> queued re-queue + reconciled history, fresh/queued untouched, idempotent second pass Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s-reconciler # Conflicts: # .planning/STATE.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
F5 — reconcile orphaned
publishedparse_jobsparse_jobsstuck inpublishedwere never recovered:listPublishableJobsonly selectsqueued/retryable, so a published job whose parser result never returned was a permanent dead-end (75 orphaned on staging, oldest 33 days).Adds a
parse-job-reconcilerIntervalTaskthat re-queuespublishedjobs whosepublished_atis older than a configurable staleness threshold, so the existing publisher re-publishes them. Re-queue is idempotent (a late parser result still supersedes viaparser_results);FOR UPDATE SKIP LOCKEDmakes concurrent replicas claim disjoint batches; every reclaim writes areconciledparse_job_historyaudit row (new enum value, migration 0012).Config:
PARSE_JOB_STALE_AFTER_MS(default 1h) andPARSE_JOB_RECONCILE_BATCH_SIZE(default 25).Scope: the parser-side root cause (ack-after-persist in
replay-parser-2) is a separate change, out of scope here — this is the server-2 recovery loop only.Tests
Factory-injection reconciler unit test + real-pg integration (stale→queued,
reconciledhistory, fresh/queued untouched, idempotent second pass).pnpm verifygreen, 100% coverage.🤖 Generated with Claude Code