diff --git a/CHANGELOG.md b/CHANGELOG.md index e8c3e4db..35de7a07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Qlik Sense is no longer published as a mightyETL row-write target: it is removed from the production target registry/catalog, Spring `ConnectorProperties`, and the modern/legacy Qlik enable alias. `QlikSenseTargetConnector` remains only non-registered reference/design code until a separately reviewed Qlik reload/data-file integration is implemented and verified. - Production container builds now use digest-pinned Docker base images while retaining readable Maven/Temurin tags, preventing upstream tag movement from silently changing reviewed build inputs. - Durable `POST /api/etl/jobs` submissions now return RFC 9110 `202 Accepted`, a stable pending-job representation, `Location` status-monitor metadata, and explicit replay metadata without changing the synchronous `/api/etl/process` contract. The incomplete intake controller is fail-closed and requires explicit `xtrmetl.etl.jobs.intake-enabled=true` operator opt-in until worker execution and terminal payload clearing are implemented. - Concurrent requests using the same authenticated-principal-scoped semantic idempotency key now return immediate RFC 9457 `409 etl_idempotency_request_in_progress` responses through PostgreSQL `pg_try_advisory_xact_lock`; retries after completion still replay the committed response. @@ -28,14 +29,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - ETL problem-details client and operator contract: `docs/api/problem-details.md`. - Operator-configurable ETL admission limits under `mightyetl.etl.*` / `xtrmetl.etl.*`, backed by `ETL_MAX_PAYLOAD_BYTES` and `ETL_MAX_BATCH_RECORDS` environment variables with hard safety ceilings. - ETL transaction rollback integration coverage and the operator runbook `docs/etl/bounded-atomic-batches.md`. -- Connector scaffolds (contracts + docs only): Qlik Sense, Databricks, Snowflake under `docs/connectors/` and `etl-service` SPI stubs. +- Connector scaffold/reference contracts and docs: Databricks, Snowflake, and the non-registered Qlik design reference under `docs/connectors/` and `etl-service` SPI code. - Any-to-any CDC design notes and source SPI scaffold: `docs/cdc/any-to-any-cdc.md`, `cdc-service` SPI stubs. - CDC operations notes: `docs/cdc/ops-and-reliability.md`. - Product upgrade progress tracker: `docs/mightyETL-product-upgrade-progress.md`. - CDC status/sources API: `GET /api/cdc/status`, `GET /api/cdc/sources` (no secrets). - `DebeziumChangeRecordMapper` + `CanonicalChangeRecord` (mapper unit-tested; not on live publish path). - CDC target SPI registry (`kafka`, `jdbc-replica`) for any-to-any routing scaffold. -- `etl-service` `xtrmetl.connectors.*` disabled config keys for Databricks/Snowflake/Qlik. +- `etl-service` `xtrmetl.connectors.*` disabled config keys for the still-registered Databricks and Snowflake scaffolds. - Dual-read config aliases: `mightyetl.*` preferred → `xtrmetl.*` (`MightyEtlConfigAliasEnvironmentPostProcessor`). - Configurable replica tables (`xtrmetl.replica.tables`) for `(id,data)`-shaped tables. - Optional CDC canonical-map counters (`xtrmetl.cdc.canonical-map-enabled`). diff --git a/docs/connectors/qlik-sense.md b/docs/connectors/qlik-sense.md index 119dc050..48795274 100644 --- a/docs/connectors/qlik-sense.md +++ b/docs/connectors/qlik-sense.md @@ -1,37 +1,45 @@ -# Qlik Sense connector (scaffold) +# Qlik Sense integration (planned) ## Intent -Push or expose change data for Qlik Sense (SaaS or client-managed) so apps can reload against an up-to-date source. +Integrate mightyETL with Qlik Sense / Qlik Cloud only through a product boundary that matches Qlik's actual APIs and operating model. Until that boundary is implemented and verified, Qlik is not published as a mightyETL target connector or production connector configuration surface. ## Status | Item | State | |:-----|:------| -| Product claim | **Not supported in runtime** (`ConnectorStatus.SCAFFOLD`) | -| SPI class | `QlikSenseTargetConnector` | -| Config binding (`xtrmetl.connectors.qlik-sense.*`) | Yes | -| Required-key validation | Yes (tenant-url, api-key, app-id) | -| Catalog API fields | Yes (`GET /api/etl/connectors`) | -| Qlik REST / qrs API client | **No** | +| Product claim | **Not supported in runtime** | +| Production target registry | **Not registered** | +| Catalog API (`GET /api/etl/connectors`) | **Not advertised** | +| Production config binding (`xtrmetl.connectors.qlik-sense.*`) | **Not bound** | +| Modern/legacy config alias (`mightyetl.connectors.qlik-sense.enabled`) | **Not mirrored** | +| Reference SPI class | `QlikSenseTargetConnector` remains as non-registered design/reference code | +| Qlik reload API client | **No** | +| Qlik data-file upload/import client | **No** | | Direct QVD writer | **No** | -| Live write path | **Refused** (`UnsupportedOperationException`) | +| Live row-write path | **No** | -## Planned integration shapes (choose later) +## Product boundary -1. **Indirect (recommended first):** mightyETL lands data in Postgres/Snowflake/Databricks; Qlik connects to that warehouse. Connector only triggers **reload** via Qlik API. -2. **Direct:** REST bulk load into a Qlik data connection (product-specific; higher complexity). +The previous scaffold modeled Qlik as a `TargetConnector.write(...)` row sink even though it never implemented a Qlik client and always refused writes. That shape is no longer advertised through the production registry, `ConnectorProperties`, or the supported modern/legacy connector-enable alias list. -## Config keys +Current Qlik Cloud APIs expose product-specific operations such as application reload orchestration and data-file upload/import. A future mightyETL integration should therefore be designed explicitly around one of those supported boundaries instead of pretending that Qlik is a transactional warehouse row sink. -| Key | Required | Description | -|:----|:--------:|:------------| -| `tenant-url` | yes | Qlik Cloud tenant or Sense enterprise base URL | -| `api-key` | yes | Auth (**secret**) | -| `app-id` | yes | App to reload | -| `mode` | no | `reload-only` \| `push` (push not designed yet) | +## Planned integration shapes -## Limitations (honest) +1. **Warehouse + reload orchestration:** mightyETL lands data in a supported warehouse or database, then a separately designed Qlik integration triggers and observes an application reload. +2. **Data-file workflow:** mightyETL produces a governed file artifact and uses Qlik's supported data-file APIs before a reload, with explicit idempotency, authentication, tenancy, retry, audit, and cleanup contracts. -- No Qlik SDK dependency, no reload automation, no multi-tenant isolation. -- Prefer documenting Qlik → Postgres/Kafka as the supported path until a live client ships. +Neither path is implemented by the current `TargetConnector` SPI. A future implementation requires a separate PRD/TRD/API contract and realistic integration tests before production registration or configuration binding. + +## Historical reference keys + +The non-registered reference class documents historical design keys such as `tenant-url`, `api-key`, `app-id`, and optional `mode`. Those names are reference material only: Spring `ConnectorProperties` no longer binds a Qlik subtree and the product alias bridge no longer mirrors a Qlik enable flag. + +A future Qlik integration must define a new supported configuration contract from the selected API boundary rather than silently reviving these historical names. + +## Limitations + +- No Qlik API client or SDK integration is shipped. +- No reload automation, file upload/import workflow, QVD writer, or tenant-aware Qlik execution exists. +- Qlik must remain absent from production connector discovery and production config binding until an actual supported integration path has been implemented and verified. diff --git a/etl-service/src/main/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessor.java b/etl-service/src/main/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessor.java index 782958dc..f65405ec 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessor.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessor.java @@ -15,7 +15,8 @@ * Dual-read configuration bridge: {@code mightyetl.*} is preferred and mirrored to the legacy * {@code xtrmetl.*} namespace used by existing configuration-property consumers. * - *
Only explicitly supported product keys are mirrored. See + *
Only explicitly supported product keys are mirrored. Qlik Sense keys are deliberately absent: + * Qlik is not a published production row-write connector. See * {@code docs/rebrand-name-matrix.md} for compatibility boundaries.
*/ @Order(ConfigDataEnvironmentPostProcessor.ORDER + 1) @@ -46,12 +47,7 @@ public class MightyEtlConfigAliasEnvironmentPostProcessor implements Environment "connectors.snowflake.private-key", "connectors.snowflake.role", "connectors.snowflake.table", - "connectors.snowflake.merge-keys", - "connectors.qlik-sense.enabled", - "connectors.qlik-sense.tenant-url", - "connectors.qlik-sense.api-key", - "connectors.qlik-sense.app-id", - "connectors.qlik-sense.mode" + "connectors.snowflake.merge-keys" ); /** diff --git a/etl-service/src/main/java/com/xtrmetl/etl/connector/ConnectorProperties.java b/etl-service/src/main/java/com/xtrmetl/etl/connector/ConnectorProperties.java index ae07ae4e..41ddd2da 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/connector/ConnectorProperties.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/connector/ConnectorProperties.java @@ -6,8 +6,9 @@ import java.util.Map; /** - * Target connector enable flags and binding surface for warehouse/BI scaffolds. - * Enabling a scaffold without a real implementation is refused by {@link TargetConnectorDispatcher}. + * Target connector enable flags and binding surface for warehouse scaffolds that remain in the + * production registry. Enabling a scaffold without a real implementation is refused by + * {@link TargetConnectorDispatcher}. * *Prefix: {@code xtrmetl.connectors.*} (dual-read from {@code mightyetl.connectors.*}).
*/ @@ -16,7 +17,6 @@ public class ConnectorProperties { private final DatabricksProps databricks = new DatabricksProps(); private final SnowflakeProps snowflake = new SnowflakeProps(); - private final QlikSenseProps qlikSense = new QlikSenseProps(); public DatabricksProps getDatabricks() { return databricks; @@ -26,15 +26,10 @@ public SnowflakeProps getSnowflake() { return snowflake; } - public QlikSenseProps getQlikSense() { - return qlikSense; - } - public boolean isEnabled(String connectorId) { return switch (connectorId) { case "databricks" -> databricks.isEnabled(); case "snowflake" -> snowflake.isEnabled(); - case "qlik-sense" -> qlikSense.isEnabled(); default -> false; }; } @@ -47,7 +42,6 @@ public MapShips config contract + validation + catalog hooks. No Qlik REST client wired; - * {@link #write} always refuses (SCAFFOLD).
+ *This type is not registered in the production target registry and has no Spring-bound + * production configuration. It is retained only to preserve an explicit design reference while + * mightyETL has no supported Qlik row-write, reload, or data-file execution path. As a scaffold, + * {@link #write} always refuses execution.
+ * + *A production Qlik integration requires a separately reviewed reload/data-file contract, + * maintained client implementation, credential boundary, lifecycle semantics, operator guidance, + * and realistic integration evidence before this type may become production-discoverable.
* * @see docs/connectors/qlik-sense.md */ diff --git a/etl-service/src/main/java/com/xtrmetl/etl/connector/TargetConnectorRegistry.java b/etl-service/src/main/java/com/xtrmetl/etl/connector/TargetConnectorRegistry.java index 4e5390af..636dbdf7 100644 --- a/etl-service/src/main/java/com/xtrmetl/etl/connector/TargetConnectorRegistry.java +++ b/etl-service/src/main/java/com/xtrmetl/etl/connector/TargetConnectorRegistry.java @@ -19,7 +19,6 @@ public class TargetConnectorRegistry { public TargetConnectorRegistry() { register(new DatabricksTargetConnector()); register(new SnowflakeTargetConnector()); - register(new QlikSenseTargetConnector()); } public final void register(TargetConnector connector) { diff --git a/etl-service/src/test/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessorTest.java b/etl-service/src/test/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessorTest.java index 76ea2b40..47cc2a91 100644 --- a/etl-service/src/test/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessorTest.java +++ b/etl-service/src/test/java/com/xtrmetl/etl/config/MightyEtlConfigAliasEnvironmentPostProcessorTest.java @@ -46,12 +46,7 @@ class MightyEtlConfigAliasEnvironmentPostProcessorTest { Map.entry("connectors.snowflake.private-key", "test-private-key"), Map.entry("connectors.snowflake.role", "ETL_ROLE"), Map.entry("connectors.snowflake.table", "EVENTS"), - Map.entry("connectors.snowflake.merge-keys", "id"), - Map.entry("connectors.qlik-sense.enabled", "true"), - Map.entry("connectors.qlik-sense.tenant-url", "https://tenant.example"), - Map.entry("connectors.qlik-sense.api-key", "test-qlik-api-key"), - Map.entry("connectors.qlik-sense.app-id", "test-app-id"), - Map.entry("connectors.qlik-sense.mode", "reload-only") + Map.entry("connectors.snowflake.merge-keys", "id") ); @Test @@ -59,7 +54,6 @@ void aliasAllowListMatchesCompleteConnectorBindingSurface() { Set