[Product gap] Eliminate production scaffold-only connector implementations

[seonghobae]

Gap

Protected develop@622e5e6c3d534f230c390f10e3832efadfc01825 still ships discoverable connector classes whose production methods deliberately refuse or no-op:

• AbstractScaffoldTargetConnector.write(...) always throws UnsupportedOperationException;
• DatabricksTargetConnector, SnowflakeTargetConnector, and QlikSenseTargetConnector inherit that refusal while appearing in connector discovery/configuration surfaces;
• AbstractScaffoldCdcSource.start(...) always throws UnsupportedOperationException and stop() is a no-op;
• MySQL and SQL Server CDC source descriptors inherit that scaffold behavior.

The existing catalog is honest about SCAFFOLD, which is safer than pretending the integrations work, but it still leaves buyer-visible production stubs in the shipped codebase. The product policy requires demo/stub-only production paths to be completed or removed from the production surface.

Product decision

For every scaffold connector, choose one of only two terminal outcomes:

1. productionize — implement a real, tested connector with operational support; or
2. remove from production discovery/code — retain design documentation/examples outside the production connector registry until implementation exists.

SCAFFOLD is an acceptable temporary migration state, not a permanent commercial endpoint.

Live execution state — 2026-08-09

Status taxonomy: all entries below are active_pr or planned; none is implemented_on_develop until protected integration.

Connector	Current bounded decision	Live work	Current blocking boundary
Qlik Sense	remove misleading row-sink registry/catalog/config binding; keep reference/design material only	PR #156, exact head 585b7f9439a9a5a3cf959bb10008d620424b4578	Current aggregate CI/dependency/SBOM/SAST/security is green on synthetic merge ec512c7f7e1a579eb20deabe877a8171038afc4d, but literal-source hard evidence, non-vacuous owned-production coverage (#162/#164), and qualifying independent approval remain absent.
MySQL CDC	remove nonfunctional Spring production discovery; retain explicit reference scaffold	PR #158, exact head 3b501f823fa0d3d93a5a95c714efdd79242a8a58	Its current Security Scan fails on inherited protected-base Jackson advisories, not the MySQL diff. Shared remediation is PR #160; after that integrates, #158 must be revalidated against the new protected base.
SQL Server CDC	remove nonfunctional Spring production discovery; retain explicit reference scaffold	PR #163, exact head 754cce280d71b1745c8514a661e304fd208ccfee	Current exact-head workflows are queued and therefore not passing; no formal approval exists.
Databricks	unresolved terminal outcome	planned	Do not open a competing implementation/removal slice until the already-active connector retirements stabilize/integrate and the production catalog is freshly re-read.
Snowflake	unresolved terminal outcome	planned	Same sequencing rule as Databricks.

Shared security PR #160 (2905c416679dfccf6ad48e7a7f056b22d476a553) has current aggregate-green CI, Dependency Review, SBOM, SAST, and Security Scan, and its Trivy merged-tree scan reports zero CRITICAL/HIGH/MEDIUM findings after moving Jackson to 2.21.5. It is still Draft because current protected workflows execute synthetic PR merge revisions, the inherited JaCoCo durable-job gate can pass with zero analyzed production classes, and no qualifying independent approval exists. No #160 evidence transfers to a connector PR after a base/head change.

This live table is sequencing evidence, not shipped product truth. Any head/base movement invalidates the corresponding exact-head facts above and requires a fresh issue/PR reconciliation.

Target connectors

Databricks

Before implementation, reverify current primary Databricks SQL/Statement Execution/JDBC documentation and select one supported production API. Implement bounded batch semantics, TLS/auth configuration, target identifier validation, transaction/idempotency behavior, retry classification, cancellation/timeouts, observability, and realistic integration tests against a controlled Databricks test environment. Never log PAT/OAuth credentials.

Snowflake

Reverify current Snowflake JDBC/SQL API guidance. Implement explicit warehouse/database/schema/table configuration, identifier safety, bounded batch ingestion, transaction/autocommit semantics, retry classification, query cancellation/timeouts, telemetry, and realistic integration tests. Do not treat successful config validation as write support.

Qlik Sense

The current bounded product decision is removal from the production row-sink surface, not invention of a fake warehouse write path. PR #156 removes Qlik from registry/catalog and Spring configuration binding while retaining only non-registered reference/design material. A future Qlik integration must model Qlik's actual supported reload/data-file/application orchestration boundary and obtain realistic provider evidence before production discovery returns.

CDC sources

MySQL / SQL Server

The current bounded decision for both sources is fail-closed removal from automatic Spring production discovery while retaining explicit reference scaffolds. PR #158 owns MySQL; PR #163 owns SQL Server.

A future productionization must use maintained Debezium source connectors through the existing source SPI or a refactored multi-engine factory. Each source requires:

• connector dependency/version pin;
• complete config validation with secret-safe errors;
• offset/schema-history semantics;
• start/stop/restart lifecycle;
• database-specific snapshot/log/binlog/CDC prerequisites;
• realistic container/integration fixture or supported external test environment;
• no skipped source-specific tests counted as support evidence;
• compatibility and recovery documentation.

TDD / quality gates

For each connector independently:

• RED test proves current scaffold refusal/discovery at the intended runtime boundary;
• production implementation or registry removal follows as the narrowest accepted remedy;
• production statement/branch coverage remains exact and non-vacuous for owned code;
• public API/Javadoc is complete;
• unit + integration + failure + timeout + retry + concurrency/idempotency tests as applicable;
• credentials are supplied only through environment/secret contracts and never committed;
• test environment uses realistic provider/database semantics, not a fake always-success client;
• catalog status becomes production-capable only after all implementation and operational gates pass;
• docs/PRD/TRD/Architecture/UML/API/Threat/Operability/Traceability/CHANGELOG updated truthfully;
• exact-source CI, dependency, SBOM, SAST/security, review and release evidence pass under the integrated protected controls;
• queued, pending, absent, stale-head, predecessor-head, status-only, synthetic-merge-only, or vacuous-coverage evidence is not passing.

Sequencing

Do not attempt all providers in one giant PR. Implement/remove one bounded connector at a time. Existing active connector PRs may proceed independently where their paths and writer leases do not conflict, but do not start Databricks/Snowflake merely to increase parallelism while the current production-surface retirements are unresolved. After each integration, refetch protected develop, the connector registry/catalog/configuration surfaces, exact remaining scaffold classes, checks/reviews, and this issue before selecting the next connector.

Completion definition

This issue closes only when no production registry/discovery/configuration path contains a connector whose only runtime behavior is scaffold validation + refusal/no-op. Experimental designs may remain in documentation or clearly isolated non-production reference code, but not as permanent shipped production stubs. Closure also requires canonical connector support documentation to agree with protected code and machine-checkable tests to prevent retired scaffold discovery from returning.

[seonghobae]

Authoritative feasibility pass (2026-08-09) narrows the connector sequence:

Databricks target

Current Databricks primary documentation provides the SQL Statement Execution API for authenticated SQL-warehouse execution, including synchronous/asynchronous execution and statement polling (https://docs.databricks.com/api/workspace/statementexecution; current tutorial updated 2026-07-20 at https://docs.databricks.com/aws/en/dev-tools/sql-execution-tutorial). This proves a supported remote execution surface exists, so productionize is technically plausible. However, a truthful merge requires a real controlled Databricks workspace/warehouse credential and integration target; repository-local mocks cannot prove the remote auth, timeout, warehouse, SQL, and rollback semantics. Until availability/scope of such a test credential is proven, live-provider acceptance is defer_until_trigger, not a reason to invent a secret.

Snowflake target

Current Snowflake JDBC documentation exposes maintained JDBC connection/authentication parameters, database/schema/warehouse/role selection, explicit login/network/query timeouts, OAuth/key-pair/WIF/PAT authentication options, and warns about transaction hazards when one session/connection is shared among threads (https://docs.snowflake.com/en/developer-guide/jdbc/jdbc-parameters, https://docs.snowflake.com/en/developer-guide/driver-connections). The 2026 JDBC release stream is active. This makes a Java JDBC connector a supported production route, but—as with Databricks—end-to-end acceptance needs a real least-privilege Snowflake test account/warehouse and must not be replaced by an always-success fake.

Qlik Sense target

Current Qlik Cloud primary APIs show that the natural integration boundary is not a transactional row-oriented TargetConnector.write(List<ChangeRecord>): Qlik provides Data Files APIs for uploading/updating flat files in spaces and Reloads APIs for triggering/cancelling app reloads (https://qlik.dev/apis/rest/data-files/, https://qlik.dev/apis/rest/reloads/). Qlik also publishes reload completion events. Therefore the existing QlikSenseTargetConnector extends AbstractScaffoldTargetConnector abstraction is likely the wrong product model. Preferred classification: execute_now design refactor/removal from the row-write target registry, then introduce a separately named file-publish/reload-orchestration integration only if buyer need justifies it. Do not implement row-by-row fake writes against Qlik merely to remove UnsupportedOperationException.

MySQL / SQL Server CDC

Debezium 3.4 officially ships both MySQL and SQL Server source connectors. MySQL requires binlog configuration and supports snapshots + continued binlog reading; SQL Server requires database/table CDC enablement and streams committed row-level changes. Current mightyETL dependency search finds no debezium-connector-mysql or debezium-connector-sqlserver dependency, confirming the discoverable source classes cannot start real engines today.

These two are more feasible to validate autonomously than SaaS warehouse targets because realistic database fixtures can be containerized. Recommended first productionization slice: MySQL Debezium source, provided the CI/runtime budget can run a real MySQL fixture and the implementation reuses the existing source SPI without breaking PostgreSQL. SQL Server follows only after its heavier CDC/server fixture and license/runtime requirements are proven practical.

Updated sequence

1. Remove/refactor Qlik from the row-write target abstraction unless a real row-write contract is proven.
2. Productionize MySQL CDC with real Debezium 3.4 + MySQL fixture, exact offset/schema-history/lifecycle tests.
3. Evaluate SQL Server CDC fixture feasibility, then productionize if practical.
4. Databricks and Snowflake remain implementable designs, but provider-live acceptance is gated on proving existing least-privilege test credentials/environments; do not create secrets speculatively.

This is feasibility evidence only; no connector is reclassified as production until its own TDD, integration, failure, security, operability, coverage, review, and exact-head gates pass.

[seonghobae]

Fresh connector-lifecycle reconciliation for SQL Server supersedes this issue body's older #163 queue snapshot.

PR #163 remains active_pr, Draft and mergeable on protected develop@622e5e6c3d534f230c390f10e3832efadfc01825, but its live source head is now d96d1c815511dffe8d48fa9d8b6eb0c92b353304 (not historical 754cce...). The current head adds the executable factory-level proof that configured sqlserver-debezium is unavailable and resolves as unknown_source_type, plus the requested Markdown-table fix. Its single CodeRabbit inline thread is resolved; formal review is still COMMENTED only, with no qualifying APPROVED.

Current-head aggregate workflows are terminal-success: CI 31317797409, Dependency Review 31317797366, CycloneDX SBOM 31317797360, SAST Semgrep 31317797356, and Security Scan 31317797370. These are strong source-local merged-tree results, not merge authorization. CI macOS job 93255755599 checked out synthetic merge da327fbbe3f542477a34581279afe8f704a283ff rather than literal source d96d1c..., and the same job still reported Analyzed bundle 'etl-service' with 0 classes followed by a vacuous coverage pass. Shared source-identity controls (#121/central scanner), non-vacuous coverage (#162/#164), and independent approval therefore remain separate non-passing prerequisites.

Connector table disposition remains unchanged in product semantics: SQL Server is being removed from automatic production discovery while retaining explicit reference-only material; it is not production-supported. Do not start Databricks/Snowflake merely because #163's feature-local tests are green. Re-evaluate those terminal decisions only after the currently active Qlik/MySQL/SQL Server retirement slices and shared controls stabilize/integrate, then refetch the production registry/configuration surface before choosing the next bounded connector action.

[seonghobae]

Live connector-truth reconciliation for SQL Server CDC: PR #163 has advanced beyond the issue table's historical 754cce280d71b1745c8514a661e304fd208ccfee to exact current head d96d1c815511dffe8d48fa9d8b6eb0c92b353304 on unchanged protected base develop@622e5e6c3d534f230c390f10e3832efadfc01825.

The old CodeRabbit review on 754cce... raised one material test-evidence gap: describeConfigured did not exercise an enabled sqlserver-debezium specification through the production-visible registry. Current source now adds CdcSourceFactoryTest.describeConfiguredMarksUnknownScaffoldAndRetiredSqlServerTypes, which feeds an enabled SQL Server spec through the default production-visible registry and asserts registered=false, scaffoldOnly=true, and error=unknown_source_type; the previous Markdown MD058 thread is resolved. This directly addresses the review finding without re-advertising the scaffold.

Fresh aggregate workflows on exact current head d96d1c8... are now complete and green: CI 31317797409, Dependency Review 31317797366, CycloneDX SBOM 31317797360, SAST Semgrep 31317797356, and Security Scan 31317797370. This supersedes the issue table's earlier queued statement. It still does not make #163 merge-ready: the branch is Draft; there is no qualifying independent non-author APPROVED review; protected PR workflows still need accepted literal-source identity rather than merely aggregate/synthetic-merge evidence; and shared non-vacuous owned-production coverage remains dependent on #162/#164. A new CodeRabbit re-review request for the advanced head was attempted, but the provider returned a temporary per-hour rate limit, so that exact action is defer_until_trigger rather than a reason to poll or block other work.

Keep SQL Server CDC status active_pr, not implemented_on_develop. No #163 source/ref mutation is performed from this loop because that branch moved under another writer earlier in the invocation.

[seonghobae]

2026-08-10 live connector-sequencing refresh

Fresh refs and workflows materially supersede the dated 2026-08-09 table above; these are execution evidence only, not implemented_on_develop truth.

• Qlik / PR fix(connectors): remove Qlik scaffold from production registry #156 — live head is now f04512c4cce2f0c1d33e74286cda2628050784b9 (the body’s 585b7f... is historical). Current aggregates are all green: CI 31319383359, Dependency Review 31319383410, SBOM 31319383346, SAST 31319383352, Security Scan 31319383355. No unresolved review thread was found on the fresh refresh. The branch moved under another writer in this invocation, so source/ref mutation is frozen here. Aggregate green still does not supply literal-source hard evidence under the current protected checkout semantics, non-vacuous shared coverage, or independent formal approval.
• SQL Server CDC / PR fix(cdc): remove SQL Server scaffold from production discovery #163 — live head is now d96d1c815511dffe8d48fa9d8b6eb0c92b353304 (the body’s 754cce... is historical). Current aggregates are all green: CI 31317797409, Dependency Review 31317797366, SBOM 31317797360, SAST 31317797356, Security Scan 31317797370. The former CodeRabbit Markdown finding is resolved; its remaining review submission is COMMENTED, not approval. This branch also moved under another writer, so source/ref mutation is frozen here.
• MySQL CDC / PR fix(cdc): remove MySQL scaffold from production discovery #158 — unchanged live head 3b501f823fa0d3d93a5a95c714efdd79242a8a58; its inherited Jackson failure remains a shared-baseline dependency on fix(security): move Jackson to fixed 2.21.5 baseline #160 rather than a MySQL-local defect.
• Shared Jackson / PR fix(security): move Jackson to fixed 2.21.5 baseline #160 — unchanged 2905c416679dfccf6ad48e7a7f056b22d476a553; current CI/dependency/SBOM/SAST/security aggregates remain green, but existing protected CI/Trivy source identity is synthetic-merge-only and formal approval is absent.
• Shared coverage / PR fix(coverage): make JaCoCo production gate non-vacuous #164 — live head f8fa8504f0cceb2f072befda09eef3d13169b5b4 now proves the non-vacuity repair itself works: CI analyzes 8 production classes, not zero. The real residual gate is now exact 100% coverage: CI 31317634945 fails with 7 missed instructions, 2 missed lines, and 2 missed branches while Dependency Review, SBOM, SAST and Security are green. That branch is under another active writer and remains source-frozen here.

Sequencing decision remains unchanged: do not start Databricks or Snowflake merely because Qlik/SQL Server aggregates are green. The current retirements are still unresolved on protected develop, and shared exact-source/coverage/review prerequisites are not integrated. After the active retirements and their shared controls integrate, refetch protected registry/catalog/configuration before choosing the next terminal connector outcome.

[seonghobae]

Live connector-lifecycle reconciliation — 2026-08-10

The issue body matrix contains historical heads. Fresh product/registry truth is:

Surface	Maturity	Current lane	Fresh boundary
Qlik target row-sink/config binding	active_pr	#156 @ f04512c4cce2f0c1d33e74286cda2628050784b9	Production registry/catalog/config-binding retirement is implemented on the active branch. All exposed aggregate workflows are terminal-success, but source identity, #196 scanner completeness, #162/#164 non-vacuous coverage and review governance remain independent.
MySQL CDC source discovery	active_pr	#158 @ 3b501f823fa0d3d93a5a95c714efdd79242a8a58	Nonfunctional scaffold is removed from Spring discovery. Security Scan is genuinely failed on inherited Jackson advisories; #160 owns that shared dependency repair.
SQL Server CDC source discovery	active_pr	#163 @ d96d1c815511dffe8d48fa9d8b6eb0c92b353304	Reference-only discovery retirement is implemented. Current aggregate CI/dependency/SBOM/SAST/Security are terminal-success, subject to the same source/completeness/coverage/review evidence limits.
Databricks target	planned	#153	Still a scaffold/refusal surface; no new competing implementation/removal branch should be opened merely to increase parallelism while current connector lanes remain active.
Snowflake target	planned	#153	Same sequencing rule as Databricks.
CDC live-target discovery vs canonical write(...) SPI	known_gap	#202	Kafka/JDBC product paths are real, but their registered Java target adapters remain non-executable through the generic canonical-write SPI. Do not mark live product paths as scaffold-only or silently change payload compatibility.
CDC registry duplicate-ID authority	active_pr	#201 @ 1ba5527effcc4605b44d53effd0dfb3be3618164	CDC source/target registries now fail closed on blank/duplicate IDs on the active branch; shared Jackson/security/evidence prerequisites remain.
ETL target registry duplicate-ID authority	known_gap	#203	Protected TargetConnectorRegistry still has silent replacement semantics. Source mutation is intentionally deferred while #156 owns the same registry/catalog surface.

Two distinctions are now important for the canonical connector support matrix:

1. product-path support != Java adapter executability ([Product/API] Separate live CDC target capability discovery from unwired canonical-write SPI stubs #202). A live raw Kafka/JDBC path must not be called unsupported merely because the canonical target adapter is unwired.
2. connector ID uniqueness is an authority invariant (fix(cdc): fail closed on duplicate connector registry IDs #201/[Connector integrity] Reject duplicate ETL target connector IDs after scaffold-retirement lane stabilizes #203). Registration order must never silently decide which implementation owns an advertised ID.

Keep SCAFFOLD temporary and truthful. Do not close #153 until protected code has no permanent production discovery/configuration path advertising a refusal/no-op connector, the remaining Databricks/Snowflake terminal decisions are executed, #202's capability/execution distinction is represented truthfully, and registry identity invariants cannot silently replace authority.