[Data governance/Reliability] Govern CDC dead-letter payload retention and replay authority

[seonghobae]

Buyer / data-governance gap

Protected develop@622e5e6c3d534f230c390f10e3832efadfc01825 configures replica-consumer failure recovery with Spring Kafka DeadLetterPublishingRecoverer and sends exhausted/non-retryable records to <source-topic>.DLT. The repository operations guide advertises this as a real reliability feature, but the repository has no corresponding DLT retention, access-control, encryption, redrive/replay, deletion, or data-classification contract.

This matters because the DLT is not merely a failure counter. Protected KafkaConfig constructs the standard recoverer without a custom producer-record sanitizer/transformer. Spring Kafka's maintained API documents that the default producer-record construction copies the failed consumer record key/value and adds original-record and DLT/exception headers; the framework exposes customization specifically when the contents sent to dead-letter/retry topics must change. Therefore business CDC payload retained for operational recovery can also become a second durable data copy with additional diagnostic metadata.

Primary Spring evidence to reverify at implementation time:

• Spring for Apache Kafka DeadLetterPublishingRecoverer API (createProducerRecord default copies key/value and adds headers);
• Spring Kafka reference, Managing Dead Letter Record Headers (original/exception header behavior and growth semantics).

RCA

• Immediate cause: exhausted/non-retryable replica records are republished to .DLT through the default recoverer behavior.
• Technical root cause: retry/recovery semantics exist, but durable dead-letter data ownership is not modeled as a governed product boundary.
• Control failure: KafkaConfigTest proves that a dead-letter recoverer/backoff is configured, but does not prove payload/header classification, retention, ACLs, encryption, redrive authority, deletion, or bounded diagnostic headers; docs/cdc/ops-and-reliability.md names the DLT but contains no lifecycle/control contract.
• Risk: DLT records can contain the same business/PII-bearing Debezium value/key data required for replay, plus failure/origin metadata. Blanket payload masking is not automatically acceptable because it may destroy recovery utility; the product needs purpose-bound governance instead.

Distinct remediation options / feasibility

Preferred — purpose-bound raw-record DLT with explicit governance

Keep replay-capable record key/value only if operational recovery genuinely requires it, but make the boundary explicit and testable:

• versioned DLT naming/provisioning policy rather than undocumented accidental lifecycle;
• bounded topic retention/deletion policy appropriate to supported deployments;
• least-privilege producer/consumer ACL ownership and tenant/environment separation;
• encryption in transit and broker/storage controls owned by the deployment profile;
• explicit classification of key/value and retained original/exception headers;
• exclude or bound exception stack/message headers when they can contain payload/provider/credential-adjacent diagnostics while retaining stable finite failure classification;
• auditable redrive/replay authority and idempotency/fencing so replay cannot silently duplicate effects;
• deletion/incident/runbook semantics and metrics without payload/high-cardinality identifiers.

Alternative — sanitized/quarantine envelope

Publish a bounded quarantine envelope with source identity, digest, stable failure code and an authorized pointer to separately protected payload evidence only if a real governed evidence store exists. Do not invent a new database/object store merely to avoid Kafka retention; this option needs a separately accepted persistence/authority design.

Reject

• silently discarding payload by masking/truncating it when that prevents valid recovery;
• retaining raw payload indefinitely with no ownership/retention decision;
• logging DLT payloads as the diagnostic mechanism;
• automatic replay without owner authorization, exact source identity, idempotency/fencing and duplicate-effect analysis;
• assuming the broker's deployment-specific defaults constitute mightyETL's retention/security contract.

TDD / acceptance

Before production mutation, add a fail-first contract at the actual DeadLetterPublishingRecoverer/Kafka record boundary. The accepted design must prove, as applicable:

1. the exact DLT producer record contents (key/value + allowed headers) are deliberate and bounded;
2. credentials, JDBC URLs, SQL, raw exception stack/message material, and unrelated headers are not added merely for diagnostics;
3. business payload remains available only when the selected recovery model explicitly requires it;
4. DLT topic naming cannot collide across unsupported tenant/environment boundaries;
5. supported deployment profiles provide an explicit finite retention/deletion policy rather than relying on unspecified broker defaults;
6. producer/consumer/redrive authority is least-privilege and documented; ordinary app callers cannot invoke replay by knowing a topic name;
7. redrive/replay uses a reviewed idempotent/fenced contract and does not claim exactly-once where the target cannot prove it;
8. poison messages cannot loop indefinitely between source and DLT;
9. topic/record/header size bounds and exception-header growth are handled fail-closed;
10. observability contains finite outcome/failure classifications without payload, key, row ID, principal, SQL or exception text;
11. canonical Data Governance/Security/Threat Model/Operability/Architecture/Traceability explains DLT ownership, retention, encryption, access, replay and deletion;
12. exact-source CI, dependency, SBOM, SAST/security, non-vacuous coverage and review gates pass before protected integration.

Current implementation classification

planned / known_gap. Do not open a source PR until the exact payload-retention/redrive policy is selected from the evidence above and current CDC writer overlap is rechecked. This issue can advance in parallel through documentation/data-governance analysis without racing #139 acknowledgement, #170–#174 diagnostics, or connector-retirement work.

Relationship: #139 owns Kafka acknowledgement-before-offset progress; #141 owns graceful CDC stop; #170–#176 own diagnostic confidentiality; #159 owns canonical data-governance/operability traceability. This issue owns the durable dead-letter data lifecycle and replay authority, not ordinary logging.

[seonghobae]

Fresh implementation evidence: Draft PR #192 is now the active bounded header-confidentiality slice on exact source 901a98e3bef05f63002602da8f228963ff698e63 over protected develop@622e5e6c3d534f230c390f10e3832efadfc01825. Its hosted CI RED is valid: macOS job 93332556332 checked out synthetic merge a3818e868509d60bd7522d5ddf9a35480fd7731b, reached KafkaConfigTest.deadLetterRecordKeepsReplayPayloadButDropsRawExceptionMessageAndStackTrace, and failed only because Spring Kafka's DLT record still carried kafka_dlt-exception-message containing a credential-bearing JDBC URL; CDC ran 107 tests with exactly one failure and zero errors/skips. The test deliberately preserves replay key/value while requiring raw exception message/stack headers to disappear, so this addresses diagnostic minimization without blanket payload masking.

Current #192 Dependency Review 31347680671, SBOM 31347680672, and SAST 31347680688 are success. Security Scan 31347680680 fails only on the six inherited protected-base Jackson findings (CVE-2026-54515, CVE-2026-59889, GHSA-mhm7-754m-9p8w at root/etl POM), not on the DLT test diff; #160 remains the shared remediation owner. #192 is under another active writer and is source-frozen for this invocation; its smallest next production remedy should be implemented by that writer after the observed RED. This slice does not close #181: DLT retention/ACL/encryption/redrive/deletion governance remains separate even after unsafe diagnostic headers are removed.

[seonghobae]

2026-08-10 source-backed DLT lifecycle progress — two bounded controls are now active PRs

This issue remains open and continues to own the broader dead-letter data lifecycle/replay authority. Two independently reviewable source defects inside that boundary have now been implemented test-first without pretending the overall governance decision is complete.

#192 — raw exception diagnostics are removed from DLT headers

Draft PR #192, current source head 58e21a9bda5aa1fe3fa2654f1f70f6c45bd3bbfd, proves that protected DeadLetterPublishingRecoverer defaults add raw exception-message/stack-trace headers. Its fail-first test injected a credential-bearing PostgreSQL URL and observed kafka_dlt-exception-message at the actual producer-record boundary. The GREEN keeps the failed key/value plus bounded original/classification metadata but calls excludeHeader(EX_MSG, EX_STACKTRACE) so raw exception text/stack traces are not transported as recovery metadata.

This is a confidentiality control only. It does not decide payload retention, broker ACLs/encryption, tenant/environment separation, deletion or redrive authority.

#197 — DLT records are explicitly terminal before replica appliers

Draft PR #197, current source head c88bf401f819a9cc7c9735746c4b55ae517b764f, proves that the default live replica topic pattern matches .DLT records and the protected consumer routes such a record into ProcessedDataReplicaApplier. The fail-first test produced exactly that applier interaction. The GREEN adds an explicit .DLT guard before either replica applier, making terminal quarantine independent of downstream table allow-lists or future applier changes.

The investigation also falsified a stronger initial hypothesis: with today's default replica.tables=processed_data, the downstream data applier sees DLT as the table suffix and returns, so protected default behavior does not itself prove recursive .DLT.DLT publication. #197 is therefore correctly classified as an explicit defense-in-depth terminal-routing contract, not as remediation of a proven current recursive loop.

Evidence limitations remain independent and non-passing

Both active PRs currently have aggregate-green hosted workflow sets on GitHub synthetic PR merge revisions, not accepted literal source heads. Protected JaCoCo still reports Analyzed bundle 'etl-service' with 0 classes on these integration trees (#162/#164), so that is not valid 100% owned-production coverage evidence. Their hard Trivy jobs also warn Dependency version cannot be determined. Child dependencies will not be found. before reporting zero findings; #196 now owns that scanner-completeness false-green condition. Those evidence gaps do not invalidate the local DLT fixes, but they block protected merge/release acceptance.

What still belongs to #181

Keep this issue open until the product selects and proves the remaining durable DLT authority:

• explicit supported DLT topic naming/provisioning and tenant/environment collision policy;
• finite retention/deletion by supported deployment profile;
• producer/consumer/redrive least privilege;
• encryption/TLS and deployment-owned key/credential provenance;
• classification and bounds for retained business key/value and all allowed headers;
• authorized redrive/replay with idempotency/fencing and external-target duplicate-effect analysis;
• poison-message size/growth controls, schema/version compatibility and unsupported-message handling;
• incident, deletion and recovery runbooks;
• finite-cardinality observability without payload/key/row/principal/SQL/exception text;
• canonical PRD/TRD/Architecture/UML/Data Governance/Security/Threat Model/Operability/Traceability once docs: establish canonical commercial architecture baseline #149's documentation writer lane is safe.

Accordingly the issue maturity should now be understood as known_gap with bounded active_pr sub-controls (#192, #197), not as completed DLT governance and not as implemented_on_develop.