[Data integrity] Reject invalid AMOUNT values instead of silently coercing them to zero

[seonghobae]

Buyer / data-integrity defect

Protected develop@622e5e6c3d534f230c390f10e3832efadfc01825 silently transforms invalid, empty, excessive-precision, and extreme-scale AMOUNT values to the valid-looking business value 0.00 in EtlService.formatAmount(...). Existing tests and docs/etl/bounded-atomic-batches.md explicitly preserve that legacy fallback.

For an ETL product, this is lossy data corruption rather than safe validation: an upstream malformed amount such as "not-a-number", an empty amount, or a value outside the supported numeric contract becomes indistinguishable downstream from a genuine zero monetary amount. Because the whole batch is otherwise prevalidated before JDBC, the service can fail closed before any write instead of manufacturing data.

Status: known_gap; direct protected-source remediation is independently valuable and should be test-first.

RCA

• Immediate cause: formatAmount catches NumberFormatException and returns "0.00"; precision/scale overflow uses the same fallback.
• Technical root cause: legacy transformation compatibility was given precedence over semantic data integrity, so a validation failure is represented as a normal transformed value.
• Control failure: tests assert the zero fallback and documentation describes it as accepted behavior, so current CI positively protects the corruption semantics.
• Business impact: genuine zero and invalid input cannot be distinguished after ingestion; reconciliation, billing/finance-like downstream logic, data-quality monitoring, replay and audit cannot recover the original error from processed_data.

Distinct remedies

1. Preferred — fail closed on invalid amount. Treat malformed, blank, excessive-precision and extreme-scale AMOUNT as EtlRequestError.INVALID_RECORD (or a narrower stable amount error only if the public error taxonomy already has a justified compatible slot). Because all records are transformed before JDBC, the entire request remains atomic and performs zero writes.
2. Alternative — preserve original invalid value with an explicit typed quality flag. This would require a new typed target/data-quality contract and schema rather than the current text representation, so it is larger than the demonstrated defect.
3. Reject — keep mapping invalid values to 0.00. It destroys information and manufactures a legitimate numeric value.
4. Reject — log and continue. Logging does not restore data integrity and risks exposing business values.

TDD / acceptance

• add fail-first tests at the real EtlService.processData boundary requiring malformed, blank, excessive-precision and extreme-scale amounts to fail before JDBC;
• prove valid decimal inputs still use deterministic BigDecimal + HALF_UP scale 2 formatting;
• prove one invalid amount anywhere in a multi-record batch causes zero JDBC writes for the complete batch;
• preserve typed stable RFC 9457 error mapping without parser/amount text in the client response;
• no raw amount value, payload, principal, SQL or exception message is added to logs/metrics;
• update the bounded-batch product contract and CHANGELOG without promoting the change to implemented_on_develop before protected integration;
• run focused and full reactor tests, current dependency/SBOM/SAST/security gates, and non-vacuous owned-production coverage under the accepted [Quality gate] Make JaCoCo fail closed when production class selection is empty #162/fix(coverage): make JaCoCo production gate non-vacuous #164 control;
• require accepted literal-source evidence and applicable independent review before merge.

Compatibility / migration

This intentionally changes invalid-input behavior and is therefore an API/data-quality compatibility change, but it is safer than silently corrupting new data. Existing persisted 0.00 rows cannot be reliably distinguished from genuine zeros after the fact; canonical migration/data-governance documentation must record that historical limitation. No database migration is needed for the first fail-closed intake repair.

Relationships: #159 canonical documentation; #162/#164 coverage non-vacuity; #165 release acceptance.

[seonghobae]

Hosted RED evidence

Draft PR #199 test-only head 1b89e62d5f8b30c6dceb0d39f538d283238e7a8a reproduced the defect on the intended production boundary. CI run 31348838751 checked out GitHub synthetic merge e292fcd38b3f6af5d2efe9b332cc65ed1fdc8d0a (Merge 1b89e62... into 622e5e6...) and macOS job 93335761349 compiled production/tests successfully, then ran EtlServiceAmountIntegrityTest with 6 failures, 0 errors, 0 skipped. Every new case failed because production returned normally instead of throwing EtlRequestException; the multi-record case likewise proved the batch would proceed rather than fail before JDBC. Existing ETL tests up to that boundary remained green. This is valid merged-tree RED behavior evidence, not literal-source acceptance proof.

The smallest production correction has now been published on the same branch, followed by removal of the stale legacy zero-fallback expectations. Current candidate head is c0a3323713d726df842c8e4de10f53e57f7544da; its new checks are pending and are not counted as passing yet.

[seonghobae]

Additional production-boundary finding from the same amount-integrity audit

Protected EtlService.transformValue(...) returns early for null and any Jackson container node before dispatching on the field name. Consequently AMOUNT:null, AMOUNT:{...}, and AMOUNT:[...] bypass formatAmount(...) entirely and are serialized as ordinary transformed values (null / compact JSON). That is the same semantic corruption class as malformed text: the AMOUNT field ceases to be numeric without failing the record.

The current #199 candidate fixes malformed/scalar precision/scale behavior but does not yet prove this type boundary. Before #198 can close, add fail-first production-boundary tests for null/object/array AMOUNT; reject them with the same bounded INVALID_RECORD contract before JDBC while preserving supported textual and JSON-number scalar inputs. Do not solve this by stringifying containers or logging the raw value.