feat(ops): bind PostgreSQL backups to exact recovery provenance

[seonghobae]

Purpose

Advance #188 from exact protected develop@622e5e6c3d534f230c390f10e3832efadfc01825 with a bounded PostgreSQL recovery-evidence path. Persistent Docker volumes are not backups, and a backup archive is not recoverability. This Draft provides an integrity-bound logical backup bundle plus a fail-closed disposable empty-target restore rehearsal. It still does not claim destructive production recovery, PITR, Kafka/Debezium/DLT rewind, external-target compensation, application-startup/readiness recovery, or measured RPO/RTO.

Exact current identity

• protected/live base: develop@622e5e6c3d534f230c390f10e3832efadfc01825;
• branch: ops/postgres-recovery-manifest-622e5e6;
• representative durable-data RED head: 5ea1d328a40544240504d271ebf8a5f01292d723;
• durable-row GREEN predecessor: 41f840d49f170496bf60cf2fd0d2f210712b88a4;
• exact failure-code-grammar RED head: 510d033982f935aa9d933ea90dcb52824f2c6b7a;
• exact current GREEN source head: f33de0a8a6709947c0b19a94a931f1870afa386c;
• current synthetic merge preview: fb4743f2d2908944e455dd717e3d57750da582e5;
• Draft / GitHub mergeability: true / true;
• formal reviews: none; unresolved review threads: zero at the latest refresh.

Every predecessor-head/base check or review is historical after movement. Current protected PR workflows execute GitHub synthetic merge revisions, so aggregate-green jobs remain merged-tree evidence rather than accepted literal-source proof.

RCA chain

1. Durability != recoverability: protected local deployment had persistent PostgreSQL storage but no independently verifiable backup artifact.
2. Backup != restore: a valid archive does not prove it can be restored into a clean target while preserving application migration identity.
3. Publication identity race: second-resolution timestamp + source SHA allowed concurrent backups to select the same final identity before publication.
4. Archive integrity != manifest integrity: archive SHA did not authenticate mutable provenance fields in manifest.txt.
5. Restore failure != atomic failure: pg_restore --exit-on-error alone can leave partial objects after an earlier command committed.
6. Schema presence != application recovery: Flyway version and required relation existence do not prove representative restored durable rows still satisfy mightyETL's persisted integrity/lifecycle invariants.
7. Recovery checks must match authoritative schema constraints: a looser recovery regex is not equivalent to the protected migration's persisted failure-code grammar.
8. Recovery-domain separation: PostgreSQL restore cannot implicitly rewind Kafka, Debezium state, DLT records, or external-target side effects.

TDD / recovery evidence

The branch preserves fail-first history for backup creation, truthful recovery documentation, changelog discoverability, concurrent backup identity reservation, disposable-target restore rehearsal, independently authenticated manifest provenance, atomic pg_restore --single-transaction, required relation checks, and representative durable-row invariants.

Representative durable-row RED -> GREEN

Test-only 5ea1d328a40544240504d271ebf8a5f01292d723 added PostgresLogicalRestoreDurableDataInvariantTest before production changed. Hosted CI 31379582437, macOS job 93426522475, compiled normally and ran ETL with 287 tests, exactly one failure, zero errors/skips. The new recovery assertion failed because the restore command did not evaluate restored idempotency rows. Existing recovery contracts were green. This is valid RED at the post-restore application-integrity boundary.

41f840d49f170496bf60cf2fd0d2f210712b88a4 then added post-restore count-only, fail-closed checks for:

• lowercase 64-hex idempotency and durable-job identity/digest fields;
• durable-job lifecycle state membership;
• nonnegative attempt counts;
• active-versus-terminal request-payload lifecycle;
• bounded persisted failure-code syntax.

Hosted CI 31379887799 completed successfully on macOS/Ubuntu/Windows; ETL ran 287/287 green, CDC 106/106 green, gateway 3/3 green, and the full reactor succeeded. The same log still proved the inherited protected JaCoCo false green (Analyzed bundle 'etl-service' with 0 classes).

Follow-on RED -> GREEN — exact protected failure-code grammar

Fresh source comparison against protected migration V2__create_etl_job_records.sql found the first durable-row implementation used a looser recovery-only failure-code regex than the database constraint. The protected authority is:

^[a-z][a-z0-9_]{2,127}$

Test-only 510d033982f935aa9d933ea90dcb52824f2c6b7a changed only the recovery contract to require that exact grammar before production changed. CI 31380086329, macOS job 93428106291, checked out synthetic merge ac755919ebb0845dffc31b2549378750d6f5d6a1. Production/test compilation succeeded and ETL ran 287 tests with exactly one failure, zero errors/skips: restore must enforce the exact protected durable-job failure-code grammar. All earlier recovery assertions were green. This is valid RED at the intended schema/recovery consistency boundary.

Exact current source f33de0a8a6709947c0b19a94a931f1870afa386c is one fast-forward commit over that RED head and changes exactly one line in the restore script: the failure-code validation now matches the protected V2 constraint exactly. Immediate compare is ahead 1 / behind 0 with one file and one addition/one deletion.

Exact-current-head hosted verification

For exact source head f33de0a8a6709947c0b19a94a931f1870afa386c every exposed aggregate is terminal-success:

• CI 31380283009: success on macOS, Ubuntu and Windows;
• Dependency Review 31380283063: success;
• CycloneDX SBOM 31380283122: success;
• SAST Semgrep 31380283074: success;
• Security Scan 31380283100: aggregate success.

CI macOS job 93428715109 checked out synthetic merge fb4743f2d2908944e455dd717e3d57750da582e5, not literal source. On that tree:

• PostgresLogicalRestoreDurableDataInvariantTest: green;
• ETL: 287 tests, 0 failures/errors/skips;
• CDC: 106 tests, 0 failures/errors/skips;
• gateway: 3 tests, 0 failures/errors/skips;
• full reactor: BUILD SUCCESS.

The same CI log still says Analyzed bundle 'etl-service' with 0 classes before JaCoCo reports success. #162/#164 owns selected-class non-vacuity and #205 owns repository-wide coverage completeness. The log also repeats already-owned protected-base diagnostics such as resource-encoding (#217), Mockito agent setup (#194), ETL resource-ID logging (#218/#219), and CDC diagnostic logging (#170–#176/#210–#212); none is duplicated into this recovery branch.

Shared evidence boundaries

Aggregate success is not protected merge authorization:

• source-executing workflows remain synthetic PR merge checkout rather than accepted literal source (ci: schedule NVIDIA OpenCode maintenance agent #121 / read-only central dependencies);
• [Quality gate] Make JaCoCo fail closed when production class selection is empty #162/fix(coverage): make JaCoCo production gate non-vacuous #164 and [Quality evidence] Extend non-vacuous 100% coverage to every owned production module #205 own non-vacuous coverage;
• [Security evidence] Fail closed when Trivy cannot resolve Maven BOM-managed dependency versions #196 owns Maven dependency-resolution completeness for vulnerability evidence;
• qualifying independent non-author formal approval is absent;
• all exact-head/base evidence must be regenerated after source/base movement.

Do not copy shared workflow/security/coverage fixes into this branch merely to manufacture acceptance evidence.

Remaining #188 boundary

This remains bounded PostgreSQL artifact/rehearsal evidence, not complete disaster recovery. Later acceptance still needs realistic proof for application startup/readiness on a restored database, destructive-loss replacement/retry procedure, replay/cancellation/tenant invariants as those schemas integrate, Kafka/Debezium/DLT/external-target reconciliation, protected release linkage, and measured elapsed recovery before any RPO/RTO/SLA claim.

Merge boundary

Keep Draft. Merge only when the unchanged exact source head has accepted literal-source deterministic/security evidence, complete same-revision dependency/vulnerability/SBOM evidence, non-vacuous applicable owned-production coverage, zero valid unresolved review findings, qualifying independent non-author formal approval where governance requires it, and branch protection/rulesets permit expected-head merge. No predecessor-head, other-PR, incomplete-scanner, skipped-required, status-only, or synthetic-merge-only evidence transfers.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f2c3d930-2bda-465d-94de-07b8ec0f5084

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}