feat(server): add CONNECTOR_DISCOVERY_ENABLED flag to disable registry polling#382
Open
antoniocali wants to merge 1 commit into
Open
feat(server): add CONNECTOR_DISCOVERY_ENABLED flag to disable registry polling#382antoniocali wants to merge 1 commit into
antoniocali wants to merge 1 commit into
Conversation
Adds a server config flag (default true, so existing behavior is preserved) that short-circuits container registry queries in GetDriverImageTags. When disabled, /versions and /spec return DEFAULT_CONNECTOR_VERSION instead of contacting Docker Hub / ECR / GCR. Intended for Fusion-only and air-gapped deployments that don't run CDC connectors and are penalized by Docker Hub rate limits (issue datazip-inc#380). Also rewords the K8s fallback warning to point operators at the new flag instead of implying a missing feature.
2 tasks
|
@antoniocali we are looking into this issue and your PR. Will have an update on it soon. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes part of #380.
Adds a new server config flag
CONNECTOR_DISCOVERY_ENABLED(defaulttrue) that, when disabled, short-circuitsutils.GetDriverImageTagsand returns a configurableDEFAULT_CONNECTOR_VERSIONinstead of querying Docker Hub / ECR / GCR Artifact Registry. Default-on preserves the current behavior for every existing deployment.This addresses the Fusion-only / air-gapped use case described in #380, where the UI hits Docker Hub on every
/versions,/spec, and/test-connectioncall even though CDC connectors are unused. With discovery disabled, those endpoints return the configured version without an outbound HTTP request, eliminating both the user-facing slowness during rate-limited periods and the log noise.This is suggestion (1) from the issue. Suggestion (5) (rewording the K8s warning) is bundled here since it's a one-liner and naturally points operators at the new flag. Suggestions (2) backoff, (3) DB spec cache, and (4) custom discovery registry are intentionally out of scope.
Changes
server/internal/appconfig/appconfig.go— two new fields onConfig:ConnectorDiscoveryEnabled(bool) andDefaultConnectorVersion(string).server/conf/app.yaml— defaults:CONNECTOR_DISCOVERY_ENABLED: true,DEFAULT_CONNECTOR_VERSION: "". Behavior unchanged unless an operator explicitly opts in.server/internal/utils/docker_utils.go—GetDriverImageTagsshort-circuits at the top of the function when discovery is disabled; returns(nil, "", error)with a clear message ifDEFAULT_CONNECTOR_VERSIONis unset.server/internal/utils/docker_utils.go— rewords the Kubernetes "Cached fallback unavailable on Kubernetes (no Docker daemon)" warning at line 103 to be less alarming and to mention the new flag.Helm chart
A separate docs-only PR to
datazip-inc/olake-helmwill surface the flag viaolakeUI.envexamples + a README note. No template change is required there becauseolakeUI.envis already a passthrough map.Operator usage
Test plan
go build ./...cleango vet ./...cleanCONNECTOR_DISCOVERY_ENABLED=true) preserves existing/versionsand/specbehaviorCONNECTOR_DISCOVERY_ENABLED=falsewithDEFAULT_CONNECTOR_VERSION=v0.3.18makes/versionsreturn["v0.3.18"]without any outbound HTTPCONNECTOR_DISCOVERY_ENABLED=falsewith emptyDEFAULT_CONNECTOR_VERSIONreturns a clear error on/versionsand/spec