Skip to content

feat(server): add CONNECTOR_DISCOVERY_ENABLED flag to disable registry polling#382

Open
antoniocali wants to merge 1 commit into
datazip-inc:masterfrom
antoniocali:fix/connector-discovery-flag
Open

feat(server): add CONNECTOR_DISCOVERY_ENABLED flag to disable registry polling#382
antoniocali wants to merge 1 commit into
datazip-inc:masterfrom
antoniocali:fix/connector-discovery-flag

Conversation

@antoniocali
Copy link
Copy Markdown

Summary

Closes part of #380.

Adds a new server config flag CONNECTOR_DISCOVERY_ENABLED (default true) that, when disabled, short-circuits utils.GetDriverImageTags and returns a configurable DEFAULT_CONNECTOR_VERSION instead of querying Docker Hub / ECR / GCR Artifact Registry. Default-on preserves the current behavior for every existing deployment.

This addresses the Fusion-only / air-gapped use case described in #380, where the UI hits Docker Hub on every /versions, /spec, and /test-connection call even though CDC connectors are unused. With discovery disabled, those endpoints return the configured version without an outbound HTTP request, eliminating both the user-facing slowness during rate-limited periods and the log noise.

This is suggestion (1) from the issue. Suggestion (5) (rewording the K8s warning) is bundled here since it's a one-liner and naturally points operators at the new flag. Suggestions (2) backoff, (3) DB spec cache, and (4) custom discovery registry are intentionally out of scope.

Changes

  • server/internal/appconfig/appconfig.go — two new fields on Config: ConnectorDiscoveryEnabled (bool) and DefaultConnectorVersion (string).
  • server/conf/app.yaml — defaults: CONNECTOR_DISCOVERY_ENABLED: true, DEFAULT_CONNECTOR_VERSION: "". Behavior unchanged unless an operator explicitly opts in.
  • server/internal/utils/docker_utils.goGetDriverImageTags short-circuits at the top of the function when discovery is disabled; returns (nil, "", error) with a clear message if DEFAULT_CONNECTOR_VERSION is unset.
  • server/internal/utils/docker_utils.go — rewords the Kubernetes "Cached fallback unavailable on Kubernetes (no Docker daemon)" warning at line 103 to be less alarming and to mention the new flag.

Helm chart

A separate docs-only PR to datazip-inc/olake-helm will surface the flag via olakeUI.env examples + a README note. No template change is required there because olakeUI.env is already a passthrough map.

Operator usage

# values.yaml
olakeUI:
  env:
    CONNECTOR_DISCOVERY_ENABLED: "false"
    DEFAULT_CONNECTOR_VERSION: "v0.3.18"

Test plan

  • go build ./... clean
  • go vet ./... clean
  • Manual: verify default (CONNECTOR_DISCOVERY_ENABLED=true) preserves existing /versions and /spec behavior
  • Manual: verify CONNECTOR_DISCOVERY_ENABLED=false with DEFAULT_CONNECTOR_VERSION=v0.3.18 makes /versions return ["v0.3.18"] without any outbound HTTP
  • Manual: verify CONNECTOR_DISCOVERY_ENABLED=false with empty DEFAULT_CONNECTOR_VERSION returns a clear error on /versions and /spec

Adds a server config flag (default true, so existing behavior is preserved)
that short-circuits container registry queries in GetDriverImageTags. When
disabled, /versions and /spec return DEFAULT_CONNECTOR_VERSION instead of
contacting Docker Hub / ECR / GCR. Intended for Fusion-only and air-gapped
deployments that don't run CDC connectors and are penalized by Docker Hub
rate limits (issue datazip-inc#380).

Also rewords the K8s fallback warning to point operators at the new flag
instead of implying a missing feature.
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented May 22, 2026

CLA assistant check
All committers have signed the CLA.

@nayanj98
Copy link
Copy Markdown

@antoniocali we are looking into this issue and your PR. Will have an update on it soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants