[Operability/Privacy] Separate local-debug observability from production-safe defaults

[seonghobae]

Buyer / operator gap

Protected develop@622e5e6c3d534f230c390f10e3832efadfc01825 applies developer-heavy observability settings in the ordinary service configuration rather than an explicit local/debug profile:

• ETL, CDC, and gateway each force management.tracing.sampling.probability: 1.0;
• ETL, CDC, and gateway each set com.xtrmetl: DEBUG;
• ETL, CDC, and gateway each set Spring Web logging to DEBUG;
• the same files are the configuration used outside the local Compose topology unless an operator remembers to override them.

The recent #170–#176 confidentiality work proves that raw diagnostics can contain row IDs, DDL, JDBC/provider exception text, credentials or other sensitive/high-cardinality values. Fixing individual log call sites is necessary but does not make blanket DEBUG + 100% tracing an acquisition-grade deployment default.

This is not a request to disable useful observability. The product needs an explicit profile/authority boundary: local diagnostics may be richer under an operator-selected profile, while ordinary production-safe defaults remain bounded, finite-cardinality, privacy-aware and cost-aware.

RCA

• Immediate cause: debugging and maximum trace sampling are hard-coded in the base application configuration of all three public services.
• Technical root cause: local-development diagnostics and deployable service defaults are not modeled as separate configuration profiles or an explicit externalized observability policy.
• Systemic/control cause: documentation/tests focus on whether tracing/logging exists, not on which profile owns sampling/log level, what data classes are allowed, or how cardinality/cost/privacy are bounded.
• Detection failure: confidentiality tests are mostly component-specific, so a newly added DEBUG statement or framework-generated diagnostic can still cross the ordinary log/trace boundary without violating a repository-wide contract.

Distinct remediation options / feasibility

Recommended — production-safe base + explicit local-debug overlay

• remove forced 100% sampling and package/framework DEBUG levels from the base deployable configuration;
• place intentionally verbose settings in an explicit local/dev profile or Compose/Replit-only override;
• keep sampling/log levels externally configurable without inventing credentials;
• define allowed ordinary-log/span attributes and stable finite-cardinality failure classifications;
• add source/profile contract tests so local diagnostics cannot silently become production defaults again.

This is operationally feasible with Spring profiles/externalized configuration and does not require a new service, secret, database, or cross-repository mutation.

Alternative — explicit environment-bound values in base config

Externalize every sampling/log-level value with conservative defaults. This is smaller, but weaker than a dedicated profile because one file still represents both local debugging and production behavior.

Reject

• setting sampling to an arbitrary new percentage without a documented policy/evidence model;
• disabling observability entirely;
• relying on regex masking of all logs/spans;
• keeping DEBUG/100% sampling in the base config and relying on operators to remember overrides;
• treating fix(cdc): prevent replication probe diagnostic leakage #170–fix(etl): keep connector lifecycle diagnostics out of logs #176's individual sanitizers as proof that framework/package DEBUG output is globally safe.

Acceptance

1. fail-first tests prove the protected base currently forces verbose/100%-sampled observability;
2. deployable base configuration no longer forces DEBUG package/framework logging or 100% trace sampling;
3. an explicit local/debug profile can restore intentionally verbose diagnostics when selected;
4. production/default behavior remains externally tunable and fail-safe if optional tracing backend is unavailable;
5. ordinary logs/spans exclude payloads, principals, row IDs, raw DDL/SQL, JDBC URLs, credentials/tokens, exception messages/stacks and other unbounded provider diagnostics unless a separately authorized restricted evidence channel is designed;
6. metrics/log/span dimensions use bounded cardinality and do not key on job IDs, row IDs, raw connector endpoints or tenant/customer names by default;
7. sampling/log policy and data classification are discoverable in Operability/Security/Threat Model/Architecture and linked from [Documentation] Close canonical architecture coverage and live traceability gaps #159 traceability;
8. local Compose/Replit behavior is tested separately from deployable defaults, including fix(ops): restore bundled Zipkin tracing transport #167 Zipkin transport and fix(supply-chain): retire unsafe Replit Zipkin bootstrap #169 repository-runtime ownership;
9. exact-source CI/security/non-vacuous coverage and review gates are regenerated before protected integration.

Live implementation state — 2026-08-10

The first bounded CDC slice is now active_pr in Draft PR #212, branch ops/cdc-safe-observability-defaults-622e5e6, on unchanged protected base develop@622e5e6c3d534f230c390f10e3832efadfc01825.

Fresh live source head is aa1afd3e5292a8bb4fb51f9a98a460d4cff50b7f. The branch has already advanced beyond its original test-only head and currently contains:

• a fail-first CDC base-profile contract;
• removal of forced CDC base management.tracing.sampling.probability: 1.0 and package/Spring Web DEBUG overrides;
• explicit application-local.yml restoring 100% sampling plus intended DEBUG levels only under the local profile.

All currently exposed workflow aggregates associated with that exact source head are terminal-success (CI 31358735259, Dependency Review 31358735145, SBOM 31358735244, SAST Semgrep 31358735181, Security Scan 31358735225). Those results remain merge-preview evidence under the current protected workflow/source-identity and scanner-completeness boundaries; aggregate green does not waive #121, #162/#164/#205, #196, review governance, or protected integration.

The #212 branch moved immediately before this issue reconciliation, so its source lane is treated as actively owned and must not be raced by another writer. ETL base observability remains path-overlapped by active schema-authority work (#184), and gateway application.yml remains owned by active JWT work (#142). Those two service slices therefore remain known_gap / defer_until_trigger until their overlapping branches stabilize or integrate.

Current classification

active_pr for the CDC slice via #212; known_gap / defer_until_trigger for ETL and gateway. Nothing is implemented_on_develop until the corresponding exact code integrates into protected history and protected acceptance is regenerated.

Relationship: #170–#176 own concrete diagnostic leaks; #166/#167 own Zipkin transport correctness; #168/#169 own repository runtime supply chain; #180 owns local host-port exposure; #159 owns canonical documentation and traceability.

[seonghobae]

2026-08-10 protected-base observability evidence

Fresh hosted CI on unrelated path-disjoint PR #214 (CI run 31359327945, macOS job 93364911226) checked GitHub synthetic integration source 0c4ea69074354bca4289cdded414c2966474d33a, whose product side is still protected develop@622e5e6c3d534f230c390f10e3832efadfc01825 for ETL observability. The ETL tests emitted ordinary EtlApiProblemHandler ERROR lines whose path={} value included concrete durable-job UUIDs, for example /api/etl/jobs/<uuid>, alongside the stable exception classification.

This is additional live evidence for acceptance item 6: durable job identifiers must not become default log dimensions merely because the request URI is convenient to log. It does not establish a new source owner or justify racing etl-service/src/main/resources/application.yml while #184 is active. The eventual ETL observability slice should treat route templates/stable operation codes separately from raw request-instance paths and should test finite-cardinality diagnostics at the real handler/logging boundary.

The same run also re-demonstrated raw CDC DDL/stack/row-ID diagnostics on protected source, but those exact source paths are already owned by #171/#174 and must not be duplicated here. This comment is dated evidence only; nothing becomes implemented_on_develop until protected integration.

[seonghobae]

Live implementation reconciliation (2026-08-12): the CDC slice is now implemented_on_develop via merged PR #303 at protected develop@b53a21eaa9a45141498af04427656afc1bd8be33; the current protected tip 502d1e2136f18b7dea3baa792559128dc87fec39 still contains the same CDC base/local-profile separation. The ETL slice is now active_pr in Draft #307 from exact protected develop@502d1e2136f18b7dea3baa792559128dc87fec39; test-only RED head 99e1c5d1bfb4f2dcecb907f0a93bdadc6b9bbf41 requires deployable ETL defaults to stop forcing trace sampling 1.0 and application/Spring Web DEBUG while preserving those settings in an explicit local profile. Current #307 workflows are queued and are not passing evidence. Gateway remains known_gap / defer_until_trigger while its active authentication/configuration lane owns the same configuration path. No CDC predecessor checks or #307 queued/synthetic evidence transfers to protected acceptance.