diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a9e985..1e6fa72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,42 @@ Each entry starts with a **plain-language summary** (what changed, in everyday words) before any technical detail — written so someone outside engineering can understand what shipped and why it matters. +## [0.7.18] - 2026-08-31 + +**In plain terms:** the client no longer talks gRPC, and the operator +no longer needs env vars for the things `deltix configure` can ask about +itself. + +### Changed + +- **`deltix configure` covers everything the operator needs.** The + wizard now prompts for the local Dolt SQL bind host AND port (default + `127.0.0.1:3306`) in addition to the server URL and TLS, and + persists both to `~/.deltix/config.json`. The previously required + `DELTIX_LOCAL_PORT=...` manual export on every `deltix start` is now + a one-time wizard answer. +- **Env vars are now CI / automation only.** Every connection-related + setting (`DELTIX_SERVER_URL`, `DELTIX_HTTP_TLS_CA_PATH`, + `DELTIX_HTTP_TLS_SERVER_NAME_OVERRIDE`, `DELTIX_LOCAL_HOST`, + `DELTIX_LOCAL_PORT`, `DELTIX_DOLT_BIN_PATH`) has a corresponding + field in `~/.deltix/config.json` that `deltix configure` writes. Env + vars still take precedence (for CI), but the human-facing flow is + now command-driven, not env-driven. + +### Removed + +- **gRPC transfer pipeline.** The client never used gRPC for data-plane + traffic — that was a legacy Fase 3 path behind + `DELTIX_ENABLE_GRPC_TRANSFER`. The client is now REST-only: the entire + `dataflow/` and `heartbeat/` bounded contexts, the + `acl/grpc-transfer-client.ts` and `acl/transfer-ticket-api-adapter.ts` + adapters, the `DELTIX_GRPC_*` and `DELTIX_HEARTBEAT_*` env vars, the + `grpcHost` / `grpcPort` / `grpcTlsCaPath` / `grpcTlsServerNameOverride` + fields on `StoredConfig`, the `@grpc/grpc-js` and `@grpc/proto-loader` + dependencies, and the associated tests are all gone. The + commit-based REST push/pull path (Fase 5.9) is the only path left, + which was already the recommended one. + ## [0.7.17] - 2026-08-31 **In plain terms:** three small fixes reported by the same operator in one diff --git a/README.md b/README.md index d6c1a14..cc257de 100644 --- a/README.md +++ b/README.md @@ -187,29 +187,27 @@ deltix push # first push ships the imported data active operator stays logged in for a week without prompting. Inactivity boot is at 7 days. - **TLS is always on.** `deltix configure` can fetch and pin the server's - self-signed certificate (trust-on-first-use) for both the REST and gRPC - endpoints; connecting over a bare IP works without disabling verification. - The `/status` probe used by `deltix version` threads the same CA + self-signed certificate (trust-on-first-use) for the REST endpoint; + connecting over a bare IP works without disabling verification. The + `/status` probe used by `deltix version` threads the same CA options through, so the probe succeeds whenever data calls do. -> The legacy whole-file gRPC transfer is retained only behind -> `DELTIX_ENABLE_GRPC_TRANSFER=1` for rollback while the commit-based path is -> confirmed, and is slated for removal. - --- ## Configuration -`deltix configure` writes defaults to `~/.deltix/config.json`. Any of these -environment variables override it: +`deltix configure` is the human-friendly path and the recommended way +to set up Deltix-Client — it prompts for the server URL, the local Dolt +SQL port, and TLS options, and persists the result to +`~/.deltix/config.json`. Env vars below are for CI / automation only; +`deltix configure` should be enough for every interactive use case. | Variable | Default | Purpose | |---|---|---| -| `DELTIX_SERVER_URL` | `http://127.0.0.1:9090` | REST control plane. | -| `DELTIX_GRPC_HOST` / `DELTIX_GRPC_PORT` | `127.0.0.1` / `50051` | Transfer engine (pull/legacy). | -| `DELTIX_HTTP_TLS_CA_PATH` / `DELTIX_GRPC_TLS_CA_PATH` | — | CA to trust a self-signed server. | -| `DELTIX_HTTP_TLS_SERVER_NAME_OVERRIDE` / `DELTIX_GRPC_TLS_SERVER_NAME_OVERRIDE` | — | SNI name when connecting by IP. | -| `DELTIX_LOCAL_HOST` / `DELTIX_LOCAL_PORT` | `127.0.0.1` / `3306` | Local Dolt SQL server. After the first explicit `DELTIX_LOCAL_PORT`, the value is persisted and you no longer need to export it. | +| `DELTIX_SERVER_URL` | `http://127.0.0.1:9090` | REST control plane. Persisted via `deltix configure`. | +| `DELTIX_HTTP_TLS_CA_PATH` | — | CA to trust a self-signed server. Persisted via `deltix configure`. | +| `DELTIX_HTTP_TLS_SERVER_NAME_OVERRIDE` | — | SNI name when connecting by IP. Persisted via `deltix configure`. | +| `DELTIX_LOCAL_HOST` / `DELTIX_LOCAL_PORT` | `127.0.0.1` / `3306` | Local Dolt SQL server. Both are persisted via `deltix configure`, so a host with a pre-installed MySQL/MariaDB on 3306 only types the new port once. | | `DELTIX_HOME` | `~/.deltix` | Root for local state + Dolt binary. | | `DELTIX_DOLT_VERSION` / `DELTIX_DOLT_BIN_PATH` | `2.3.1` / — | Pinned Dolt version / preinstalled binary. | | `DELTIX_IMPORT_URL` | — | Optional default for `deltix import --from`. | @@ -234,7 +232,6 @@ A modular monolith organized by **bounded contexts** under `src/contexts/*` | `import` | `deltix import` end-to-end: parse DSN, mask password, push masked prompt, schema-only / data-only, `--continue`, blob policies, JSON-column serialisation. | | `versioning-local` | Local Dolt operations: commit, push export, pull apply, merge, branches, `origin/*` tracking. Commits are tagged with the logged-in user (sanitised). | | `versioning` | REST parity with the server: repos, branches, merge, log/diff, roles, sync-prefs, push/pull-commits. | -| `dataflow`, `heartbeat` | Legacy gRPC transfer (behind the feature flag). | Full engineering rules (architecture, security, testing, logging) live in [`.github/copilot-instructions.md`](./.github/copilot-instructions.md). diff --git a/bun.lock b/bun.lock index 9b219f0..d0cba8b 100644 --- a/bun.lock +++ b/bun.lock @@ -5,8 +5,6 @@ "": { "name": "deltix-client", "dependencies": { - "@grpc/grpc-js": "1.14.4", - "@grpc/proto-loader": "0.8.1", "consola": "^3.4.2", "mysql2": "^3.24.2", "pino": "10.3.1", @@ -38,84 +36,40 @@ "@biomejs/cli-win32-x64": ["@biomejs/cli-win32-x64@2.5.10", "", { "os": "win32", "cpu": "x64" }, "sha512-M+2dgBsl3lXRiTfgPVc2p3anS4Tocojke4rzFLScZ2Y/wmF+36dRb1iHCLiyGqOzQGyTplZH1HnEYviiAqi3nA=="], - "@grpc/grpc-js": ["@grpc/grpc-js@1.14.4", "", { "dependencies": { "@grpc/proto-loader": "^0.8.0", "@js-sdsl/ordered-map": "^4.4.2" } }, "sha512-k9Dj3DV/itK9D06Y8f190Qgop7/Ui+D0njFV3LHMPwPT75DpXLQohE9Wmz0QElrJnzsjB7KPWiKJbOl7IPDArQ=="], - - "@grpc/proto-loader": ["@grpc/proto-loader@0.8.1", "", { "dependencies": { "lodash.camelcase": "^4.3.0", "long": "^5.0.0", "protobufjs": "^7.5.5", "yargs": "^17.7.2" }, "bin": { "proto-loader-gen-types": "build/bin/proto-loader-gen-types.js" } }, "sha512-wtF6h+DY6M3YaDBPAmvuuA6jV8Sif9MjtOI5euKFWRgCDl5PeDpPsHR9u2l6St5ceY8AZgoNDww5+HvEsXFsGg=="], - - "@js-sdsl/ordered-map": ["@js-sdsl/ordered-map@4.4.2", "", {}, "sha512-iUKgm52T8HOE/makSxjqoWhe95ZJA1/G1sYsGev2JDKUSS14KAgg1LHb+Ba+IPow0xflbnSkOsZcO08C7w1gYw=="], - "@pinojs/redact": ["@pinojs/redact@0.4.0", "", {}, "sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg=="], - "@protobufjs/aspromise": ["@protobufjs/aspromise@1.1.2", "", {}, "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ=="], - - "@protobufjs/base64": ["@protobufjs/base64@1.1.2", "", {}, "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg=="], - - "@protobufjs/codegen": ["@protobufjs/codegen@2.0.5", "", {}, "sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g=="], - - "@protobufjs/eventemitter": ["@protobufjs/eventemitter@1.1.1", "", {}, "sha512-vW1GmwMZNnL+gMRaovlh9yZX74kc+TTU3FObkkurpMaRtBfLP3ldjS9KQWlwZgraRE0+dheEEoAxdzcJQ8eXZg=="], - - "@protobufjs/fetch": ["@protobufjs/fetch@1.1.1", "", { "dependencies": { "@protobufjs/aspromise": "^1.1.1" } }, "sha512-GpptLrs57adMSuHi3VNj0mAF8dwh36LMaYF6XyJ6JMWlVsc+t42tm1HSEDmOs3A8fC9yyeisgLhsTVQokOZ0zw=="], - - "@protobufjs/float": ["@protobufjs/float@1.0.2", "", {}, "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ=="], - - "@protobufjs/path": ["@protobufjs/path@1.1.2", "", {}, "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA=="], - - "@protobufjs/pool": ["@protobufjs/pool@1.1.0", "", {}, "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw=="], - - "@protobufjs/utf8": ["@protobufjs/utf8@1.1.2", "", {}, "sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug=="], - "@types/bun": ["@types/bun@1.4.0", "", { "dependencies": { "bun-types": "1.4.0" } }, "sha512-K+lZULY23vRgK/CfTjFIV+tyifaNdSMlPh9j+6mQ/cLfpOznLyAuzgV/JQysyECpkBQLVMSyvjlr2fBUSA9wFQ=="], "@types/node": ["@types/node@26.3.0", "", { "dependencies": { "undici-types": "~8.3.0" } }, "sha512-L3fgrnchriRC2ExBflb8j4uZZURHZfQsmQeyVzhjcHW4kkwVyo8/0h1B2MVzMTrYUJYu6G7EWs14hW/L9putqw=="], - "ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "ansi-styles": ["ansi-styles@4.3.0", "", { "dependencies": { "color-convert": "^2.0.1" } }, "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg=="], - "atomic-sleep": ["atomic-sleep@1.0.0", "", {}, "sha512-kNOjDqAh7px0XWNI+4QbzoiR/nTkHAWNud2uvnJquD1/x5a7EQZMJT0AczqK0Qn67oY/TTQ1LbUKajZpp3I9tQ=="], "aws-ssl-profiles": ["aws-ssl-profiles@1.1.2", "", {}, "sha512-NZKeq9AfyQvEeNlN0zSYAaWrmBffJh3IELMZfRpJVWgrpEbtEpnjvzqBPf+mxoI287JohRDoa+/nsfqqiZmF6g=="], "bun-types": ["bun-types@1.4.0", "", { "dependencies": { "@types/node": "*" } }, "sha512-iIKw23BspnQQYd3prITOBxeUsxBHnwzX6YJfGMuNOZzeNcMmVqzIIVGRm1l69ogaPQmb4wB6BN8mA5bE9YuC5Q=="], - "cliui": ["cliui@8.0.1", "", { "dependencies": { "string-width": "^4.2.0", "strip-ansi": "^6.0.1", "wrap-ansi": "^7.0.0" } }, "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ=="], - - "color-convert": ["color-convert@2.0.1", "", { "dependencies": { "color-name": "~1.1.4" } }, "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ=="], - - "color-name": ["color-name@1.1.4", "", {}, "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="], - "colorette": ["colorette@2.0.20", "", {}, "sha512-IfEDxwoWIjkeXL1eXcDiow4UbKjhLdq6/EuSVR9GMN7KVH3r9gQ83e73hsz1Nd1T3ijd5xv1wcWRYO+D6kCI2w=="], "consola": ["consola@3.4.2", "", {}, "sha512-5IKcdX0nnYavi6G7TtOhwkYzyjfJlatbjMjuLSfE2kYT5pMDOilZ4OvMhi637CcDICTmz3wARPoyhqyX1Y+XvA=="], "dateformat": ["dateformat@4.6.3", "", {}, "sha512-2P0p0pFGzHS5EMnhdxQi7aJN+iMheud0UhG4dlE1DLAlvL8JHjJJTX/CSm4JXwV0Ka5nGk3zC5mcb5bUQUxxMA=="], - "emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], - "end-of-stream": ["end-of-stream@1.4.5", "", { "dependencies": { "once": "^1.4.0" } }, "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg=="], - "escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="], - "fast-copy": ["fast-copy@4.0.4", "", {}, "sha512-eVAiWVNPSEGIzDl5yPuLrx8fNMogScXvD9xp1Kzd41FjRIz2I3sSIcxsFeM5EzFfHAfobdvs8ZySffUopljvIA=="], "fast-safe-stringify": ["fast-safe-stringify@2.1.1", "", {}, "sha512-W+KJc2dmILlPplD/H4K9l9LcAHAfPtP6BY84uVLXQ6Evcz9Lcg33Y2z1IVblT6xdY54PXYVHEv+0Wpq8Io6zkA=="], "generate-function": ["generate-function@2.3.1", "", { "dependencies": { "is-property": "^1.0.2" } }, "sha512-eeB5GfMNeevm/GRYq20ShmsaGcmI81kIX2K9XQx5miC8KdHaC6Jm0qQ8ZNeGOi7wYB8OsdxKs+Y2oVuTFuVwKQ=="], - "get-caller-file": ["get-caller-file@2.0.5", "", {}, "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg=="], - "help-me": ["help-me@5.0.0", "", {}, "sha512-7xgomUX6ADmcYzFik0HzAxh/73YlKR9bmFzf51CZwR+b6YtzU2m0u49hQCqV6SvlqIqsaxovfwdvbnsw3b/zpg=="], "iconv-lite": ["iconv-lite@0.7.3", "", { "dependencies": { "safer-buffer": ">= 2.1.2 < 3.0.0" } }, "sha512-IKXpvIzjnC9XTAUbVBcMfGS0EPaIXtW6v+zr+RRp+hqULEpo0owZax6wyRwPOJbWbzjYspQwusTsfVr0ifh4uQ=="], - "is-fullwidth-code-point": ["is-fullwidth-code-point@3.0.0", "", {}, "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg=="], - "is-property": ["is-property@1.0.2", "", {}, "sha512-Ks/IoX00TtClbGQr4TWXemAnktAQvYB7HzcCxDGqEZU6oCmb2INHuOoKxbtR+HFkmYWBKv/dOZtGRiAjDhj92g=="], "joycon": ["joycon@3.1.1", "", {}, "sha512-34wB/Y7MW7bzjKRjUKTa46I2Z7eV62Rkhva+KkopW7Qvv/OSWBqvkSY7vusOPrNuZcUG3tApvdVgNB8POj3SPw=="], - "lodash.camelcase": ["lodash.camelcase@4.3.0", "", {}, "sha512-TwuEnCnxbc3rAvhf/LbG7tJUDzhqXyFnv3dtzLOPgCG/hODL7WFnsbwktkD7yUV0RrreP/l1PALq/YSg6VvjlA=="], - "long": ["long@5.3.2", "", {}, "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA=="], "lru.min": ["lru.min@1.1.4", "", {}, "sha512-DqC6n3QQ77zdFpCMASA1a3Jlb64Hv2N2DciFGkO/4L9+q/IpIAuRlKOvCXabtRW6cQf8usbmM6BE/TOPysCdIA=="], @@ -140,16 +94,12 @@ "process-warning": ["process-warning@5.1.0", "", {}, "sha512-jQSaVHsPgtyw60e1rQ/A+/ArPEj/S8pS/vFnyGa/gYFXrKk/6RuDkoqVDQ5NI5MmS01698ltlAk0NoDBNLujRw=="], - "protobufjs": ["protobufjs@7.6.5", "", { "dependencies": { "@protobufjs/aspromise": "^1.1.2", "@protobufjs/base64": "^1.1.2", "@protobufjs/codegen": "^2.0.5", "@protobufjs/eventemitter": "^1.1.1", "@protobufjs/fetch": "^1.1.1", "@protobufjs/float": "^1.0.2", "@protobufjs/path": "^1.1.2", "@protobufjs/pool": "^1.1.0", "@protobufjs/utf8": "^1.1.1", "@types/node": ">=13.7.0", "long": "^5.3.2" } }, "sha512-/FPD0nUc9jH6rfFjji9IBqOz4pcSE3CsT1m7Ep6Mdb0LxSUMj8hgl6GomOvZzpNpAqqGaXA0P3VSrZLFzIhQrw=="], - "pump": ["pump@3.0.4", "", { "dependencies": { "end-of-stream": "^1.1.0", "once": "^1.3.1" } }, "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA=="], "quick-format-unescaped": ["quick-format-unescaped@4.0.4", "", {}, "sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg=="], "real-require": ["real-require@0.2.0", "", {}, "sha512-57frrGM/OCTLqLOAh0mhVA9VBMHd+9U7Zb2THMGdBUoZVOtGbJzjxsYGDJ3A9AYYCP4hn6y1TVbaOfzWtm5GFg=="], - "require-directory": ["require-directory@2.1.1", "", {}, "sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q=="], - "safe-stable-stringify": ["safe-stable-stringify@2.5.0", "", {}, "sha512-b3rppTKm9T+PsVCBEOUR46GWI7fdOs00VKZ1+9c1EWDaDMvjQc6tUwuFyIprgGgTcWoVHSKrU8H31ZHA2e0RHA=="], "safer-buffer": ["safer-buffer@2.1.2", "", {}, "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="], @@ -162,26 +112,14 @@ "sql-escaper": ["sql-escaper@1.5.1", "", {}, "sha512-4toX5E1fQbBrpfXidaHnF0669nkAdETeIPTs2SUjxxD7RRIs9ICG4gtpmfc68JCEKehsdwLFqBu9VlQqZ1P1gg=="], - "string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], - - "strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - "strip-json-comments": ["strip-json-comments@5.0.3", "", {}, "sha512-1tB5mhVo7U+ETBKNf92xT4hrQa3pm0MZ0PQvuDnWgAAGHDsfp4lPSpiS6psrSiet87wyGPh9ft6wmhOMQ0hDiw=="], "thread-stream": ["thread-stream@4.2.0", "", { "dependencies": { "real-require": "^1.0.0" } }, "sha512-e2zZ96wSChazBsbENf/Pcm/4swHt2cEKQ92rhUjkL9GCKiTDJIaTBenjE/m9DXi0QBmTMDkFDdOomUy20A1tDQ=="], "undici-types": ["undici-types@8.3.0", "", {}, "sha512-j375ScV60dom+YkPFIfTLcOiPxkN/buHz5GobjLhixFuANaNs3C9l4GmrWqejgXWJ7BbJcFYpTEUkS1Ge8bpZQ=="], - "wrap-ansi": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="], - "wrappy": ["wrappy@1.0.2", "", {}, "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ=="], - "y18n": ["y18n@5.0.8", "", {}, "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA=="], - - "yargs": ["yargs@17.7.3", "", { "dependencies": { "cliui": "^8.0.1", "escalade": "^3.1.1", "get-caller-file": "^2.0.5", "require-directory": "^2.1.1", "string-width": "^4.2.3", "y18n": "^5.0.5", "yargs-parser": "^21.1.1" } }, "sha512-GZtjxm/J/4TSxuL3FNYjCmLktBTnIw/rVmKSIyKeYAZpmJB2ig9VauCC5xsa82GNKVKDAqpOn3KVzNt0zmrU0g=="], - - "yargs-parser": ["yargs-parser@21.1.1", "", {}, "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw=="], - "zod": ["zod@4.4.3", "", {}, "sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ=="], "thread-stream/real-require": ["real-require@1.0.0", "", {}, "sha512-P4nbQYQfePJxRSmY+v/KINxVucm4NF3p3s7pJveMTtom52FR4YGltUQLB8idDXwDDWW+eYrWDFbuzUnjoWHF7g=="], diff --git a/package.json b/package.json index 1dff0ee..b9cde4c 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "deltix-client", - "version": "0.7.17", + "version": "0.7.18", "private": true, "license": "MIT", "type": "module", @@ -19,8 +19,6 @@ "test:smoke": "bun test tests/smoke" }, "dependencies": { - "@grpc/grpc-js": "1.14.4", - "@grpc/proto-loader": "0.8.1", "consola": "^3.4.2", "mysql2": "^3.24.2", "pino": "10.3.1", diff --git a/src/acl/grpc-transfer-client.ts b/src/acl/grpc-transfer-client.ts deleted file mode 100644 index f3ccba1..0000000 --- a/src/acl/grpc-transfer-client.ts +++ /dev/null @@ -1,164 +0,0 @@ -/** - * ACL adapter wrapping the raw `@grpc/grpc-js` client for Deltix-Server's - * Fase 3 Transfer Engine (Push/Pull/Heartbeat), per `proto/transfer.proto` - * (kept in sync with the server repo's copy — this is a network contract, - * never shared code, per the MIT/BSL license separation rule). - * - * This is the ONLY module in the codebase allowed to know about grpc-js - * or the wire shape of the proto messages; `contexts/dataflow` and - * `contexts/heartbeat` only ever see the plain TS interfaces exported here. - * - * TLS is always required — the server has no plaintext code path, so this - * client never offers an insecure credentials option either. When - * `caCertPath` is omitted, the OS root store is trusted (appropriate for a - * real CA-signed server certificate); pass it explicitly to trust a - * self-signed dev/test certificate. - */ -import { readFileSync } from 'node:fs'; -import * as grpc from '@grpc/grpc-js'; -import * as protoLoader from '@grpc/proto-loader'; -// Bun's `with { type: 'file' }` import embeds the referenced file into a -// `bun build --compile` binary and resolves to a real on-disk path at -// runtime (inside Bun's virtual `/$bunfs/` fs for compiled binaries, or the -// real path during `bun run`/tests) -- a plain `join(import.meta.dir, ...)` -// path only works in dev, since compiled binaries have no `node_modules`/ -// source tree on disk to resolve relative paths against. -import PROTO_PATH from '../../proto/transfer.proto' with { type: 'file' }; - -export interface GrpcClientTlsConfig { - caCertPath?: string | undefined; - /** - * Overrides the TLS ServerName used for SNI/hostname verification. - * Node's TLS stack rejects IP-address ServerNames outright, so this MUST - * be set whenever `host` is an IP (e.g. `127.0.0.1`) and the server's - * certificate was issued for a DNS name (our self-signed dev/test certs - * use `CN=localhost`). - */ - serverNameOverride?: string | undefined; -} - -export interface PushSummary { - jobId: string; - checksum: string; - bytesReceived: number; -} - -interface TransferEngineClient extends grpc.Client { - push( - callback: (err: grpc.ServiceError | null, response?: unknown) => void, - ): grpc.ClientWritableStream; - pull(request: { ticketId: string; repo: string }): grpc.ClientReadableStream<{ data: Buffer }>; - heartbeat( - request: { ticketId: string }, - callback: (err: grpc.ServiceError | null, response?: { newExpiresAt: string }) => void, - ): void; -} - -function loadTransferEngineDefinition() { - const packageDefinition = protoLoader.loadSync(PROTO_PATH, { - keepCase: false, - longs: String, - enums: String, - defaults: true, - oneofs: true, - }); - const proto = grpc.loadPackageDefinition(packageDefinition) as unknown as { - deltix: { transfer: { v1: { TransferEngine: grpc.ServiceClientConstructor } } }; - }; - return proto.deltix.transfer.v1.TransferEngine; -} - -/** - * Normalizes the gRPC host into a form grpc-js will accept as a channel - * target. A stray surrounding/embedded whitespace character — typically a - * newline pasted into a `DELTIX_GRPC_HOST` env var or a config value — makes - * grpc-js reject the whole target with `Could not parse target name`, so we - * strip all whitespace. Exposed for testability. - */ -export function normalizeGrpcHost(host: string): string { - return host.replace(/\s+/g, ''); -} - -export class GrpcTransferClient { - private readonly client: TransferEngineClient; - - constructor(host: string, port: number, tls: GrpcClientTlsConfig = {}) { - const TransferEngine = loadTransferEngineDefinition(); - const rootCerts = tls.caCertPath ? readFileSync(tls.caCertPath) : null; - const credentials = grpc.credentials.createSsl(rootCerts); - const channelOptions: grpc.ChannelOptions = tls.serverNameOverride - ? { 'grpc.ssl_target_name_override': tls.serverNameOverride } - : {}; - // Defensively strip surrounding/embedded whitespace from the host: a - // stray newline (e.g. pasted into a DELTIX_GRPC_HOST env var or a config - // value) would otherwise make grpc-js reject the whole target with - // `Could not parse target name "host\n\n:port"`, which is never a - // legitimate hostname character. - const cleanedHost = normalizeGrpcHost(host); - if (cleanedHost === '') { - throw new Error('gRPC host must not be empty'); - } - this.client = new TransferEngine( - `${cleanedHost}:${port}`, - credentials, - channelOptions, - ) as unknown as TransferEngineClient; - } - - /** - * Streams a local file to the server as a Push transfer, authenticated - * by a previously-issued ticket. Resolves with the server's summary once - * the whole file has been sent and acknowledged. - */ - async push( - ticketId: string, - repo: string, - fileBytes: AsyncIterable | Uint8Array[], - ): Promise { - return new Promise((resolve, reject) => { - const call = this.client.push((err, response) => { - if (err) { - reject(err); - return; - } - const r = response as { jobId: string; checksum: string; bytesReceived: string }; - resolve({ jobId: r.jobId, checksum: r.checksum, bytesReceived: Number(r.bytesReceived) }); - }); - - call.write({ header: { ticketId, operation: 'push', repo } }); - - (async () => { - try { - for await (const chunk of fileBytes) { - call.write({ chunk: { data: Buffer.from(chunk) } }); - } - call.end(); - } catch (err) { - call.destroy(err instanceof Error ? err : new Error(String(err))); - } - })(); - }); - } - - /** Streams the server's staged/synced copy of `repo` back, chunk by chunk. */ - pullStream(ticketId: string, repo: string): grpc.ClientReadableStream<{ data: Buffer }> { - return this.client.pull({ ticketId, repo }); - } - - /** Renews the sliding-window expiry of an active ticket. Returns the new epoch-ms expiry. */ - async heartbeat(ticketId: string): Promise { - return new Promise((resolve, reject) => { - this.client.heartbeat({ ticketId }, (err, response) => { - if (err) { - reject(err); - return; - } - resolve(Number(response?.newExpiresAt ?? 0)); - }); - }); - } - - close(): void { - this.client.close(); - } -} diff --git a/src/acl/transfer-ticket-api-adapter.ts b/src/acl/transfer-ticket-api-adapter.ts deleted file mode 100644 index 6208fd9..0000000 --- a/src/acl/transfer-ticket-api-adapter.ts +++ /dev/null @@ -1,87 +0,0 @@ -/** - * ACL adapter translating Deltix-Client's dataflow context to - * Deltix-Server's Fase 3 ephemeral-ticket REST endpoints - * (`/api/v1/push/ticket`, `/api/v1/auth/session-close`). This is the only - * place that knows the server's ticket wire shape — the dataflow context - * works with its own local types. - * - * Ticket issuance always requires a fresh, valid access token (never the - * refresh token directly) — the caller is responsible for minting one via - * `AuthApiAdapter.refresh()` immediately before calling `issueTicket()`. - */ - -import { - TicketAuthenticationError, - TicketIssuanceError, - TicketNotFoundOrInactiveError, -} from '../contexts/dataflow/errors'; -import { ServerUnreachableError } from '../contexts/session/errors'; -import { buildFetchTlsOptions, type HttpTlsConfig } from '../shared/http-tls'; - -export interface IssuedTicket { - ticketId: string; - operation: 'push' | 'pull'; - repo: string; - expiresAt: number; -} - -export class TransferTicketApiAdapter { - constructor( - private readonly serverUrl: string, - private readonly tlsConfig: HttpTlsConfig = {}, - ) {} - - async issueTicket( - accessToken: string, - operation: 'push' | 'pull', - repo: string, - ): Promise { - const res = await this.request('/api/v1/push/ticket', accessToken, { operation, repo }); - - if (res.status === 401) { - throw new TicketAuthenticationError(); - } - if (!res.ok) { - throw new TicketIssuanceError(res.status); - } - - const body = (await res.json()) as { - ticketId: string; - operation: 'push' | 'pull'; - repo: string; - expiresAt: number; - }; - return body; - } - - async closeTicket(accessToken: string, ticketId: string): Promise { - const res = await this.request('/api/v1/auth/session-close', accessToken, { ticketId }); - - if (res.status === 401) { - throw new TicketAuthenticationError(); - } - if (res.status === 404) { - throw new TicketNotFoundOrInactiveError(); - } - if (!res.ok) { - throw new TicketIssuanceError(res.status); - } - } - - private async request(path: string, accessToken: string, body: unknown): Promise { - try { - const tls = buildFetchTlsOptions(this.tlsConfig); - return await fetch(`${this.serverUrl}${path}`, { - method: 'POST', - headers: { - 'content-type': 'application/json', - authorization: `Bearer ${accessToken}`, - }, - body: JSON.stringify(body), - ...(tls ? { tls } : {}), - }); - } catch (err) { - throw new ServerUnreachableError(err); - } - } -} diff --git a/src/cli/index.ts b/src/cli/index.ts index 4243c59..9446d7b 100644 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -260,25 +260,6 @@ async function runPull(args: string[]): Promise { const abort = args.includes('--abort'); const positional = args.filter((a) => !a.startsWith('--')); const repoArg = positional[0]; - const destFile = positional[1]; - - // Transitional legacy path: whole-file gRPC pull, behind - // DELTIX_ENABLE_GRPC_TRANSFER. Removed once the native commit-based pull is - // confirmed in production. - if (loadEnv().DELTIX_ENABLE_GRPC_TRANSFER && repoArg && destFile) { - const { createDataflowService } = await import('../contexts/dataflow'); - try { - const result = await createDataflowService().pull(repoArg, destFile); - printSuccess(`[legacy gRPC] Pull completed for ${repoArg}`, { - bytesReceived: result.bytesReceived, - checksum: result.checksum, - }); - return 0; - } catch (err) { - printError(`Pull failed (legacy gRPC): ${String(err)}`); - return 1; - } - } const identity = await resolveServerIdentity(repoArg); if (!identity) { @@ -997,20 +978,25 @@ function handleVersioningError(err: unknown, action: string): number { /** * Interactive one-time connection setup. Persists to `~/.deltix/config.json` - * so a first-time user isn't left to discover `DELTIX_GRPC_*`/ - * `DELTIX_HTTP_*` env vars on their own — in particular - * `DELTIX_GRPC_TLS_SERVER_NAME_OVERRIDE`, which is required whenever the - * server is reached by IP address (Node's TLS stack rejects IP addresses as - * SNI ServerNames outright). Env vars, when set, still always take - * precedence over this persisted config (see shared/env.ts's - * `applyPersistedConfigDefaults`). + * so a first-time user isn't left to discover `DELTIX_*` env vars on + * their own. Covers everything the client needs to talk to the server + * and bring up the local Dolt engine: + * + * - `serverUrl` — REST URL of Deltix-Server + * - `httpTlsCaPath` / `httpTlsServerNameOverride` + * — TLS trust for the REST endpoint + * - `localHost` / `localPort` — bind address for the local Dolt SQL server * - * When the server uses a self-signed certificate, offers to fetch it + * Env vars, when set, still take precedence over this persisted config + * (see shared/env.ts's `applyPersistedConfigDefaults`) — the wizard is + * the human-friendly path; env vars are the CI / automation path. + * + * When the server uses HTTPS with a self-signed cert, offers to fetch it * automatically (Trust-On-First-Use, like an SSH host key) instead of * requiring the operator to manually copy a `.crt` file off the server — - * the exact friction reported in production (missing path, `sudo` needing - * a TTY over SSH, etc). The fetched certificate's fingerprint is always - * shown for explicit confirmation before anything is trusted or saved. + * the exact friction reported in production. The fetched certificate's + * fingerprint is always shown for explicit confirmation before + * anything is trusted or saved. */ async function runConfigure(): Promise { printInfo('Deltix connection setup (Ctrl+C to cancel; press Enter to keep the default)'); @@ -1018,77 +1004,86 @@ async function runConfigure(): Promise { const serverUrl = await promptText('Deltix-Server REST URL', { default: 'http://127.0.0.1:9090', }); - const grpcHost = await promptText('Deltix-Server gRPC host (hostname or IP)', { - default: '127.0.0.1', - }); - const grpcPortRaw = await promptText('Deltix-Server gRPC port', { default: '50051' }); - const grpcPort = Number.parseInt(grpcPortRaw, 10); - const isIpAddress = /^(\d{1,3}\.){3}\d{1,3}$|:/.test(grpcHost); - let grpcTlsServerNameOverride: string | undefined; - let grpcTlsCaPath: string | undefined; + // Parse out the host from the REST URL so we can fetch its TLS cert and + // (when reached by bare IP) suggest a DNS name for SNI. + const parsed = new URL(serverUrl); + const host = parsed.hostname || '127.0.0.1'; + const port = parsed.port ? Number(parsed.port) : parsed.protocol === 'https:' ? 443 : 9090; + const isIpAddress = /^(\d{1,3}\.){3}\d{1,3}$|:/.test(host); - // DNS names the server's certificate is actually valid for — read from the - // fetched certificate so we can *suggest* the right server-name override - // instead of hard-coding one. This is what makes bare-IP servers usable by - // any company's clients without manual guesswork. + let httpTlsCaPath: string | undefined; + let httpTlsServerNameOverride: string | undefined; let autoSuggestedOverride: string | undefined; - const isHttps = serverUrl.trim().toLowerCase().startsWith('https://'); - if (isHttps) { + if (parsed.protocol === 'https:') { const wantsAutoFetch = await promptConfirm( 'Server uses HTTPS. Does it use a self-signed certificate that needs to be trusted ' + '(fetch it automatically instead of copying a .crt file by hand)?', { default: true }, ); if (wantsAutoFetch) { - const fetched = await autoFetchAndTrustCertificate(grpcHost, grpcPort); - grpcTlsCaPath = fetched?.path; + const fetched = await autoFetchAndTrustCertificate(host, port); + httpTlsCaPath = fetched?.path; autoSuggestedOverride = fetched?.dnsNames.find((name) => !/^(\d{1,3}\.){3}\d{1,3}$|:/.test(name)) ?? undefined; } - } - if (isIpAddress) { - // Fall back to a stable, sensible default when the certificate's SAN - // didn't reveal a DNS name (e.g. a pre-existing cert with only an IP). - const overrideDefault = autoSuggestedOverride ?? 'localhost'; - if (autoSuggestedOverride) { - printInfo( - `"${grpcHost}" is an IP address. TLS clients cannot verify a bare IP as a server name, ` + - `so this connection uses the DNS name the server's certificate identifies as — ` + - `suggested \`${autoSuggestedOverride}\` from the certificate.`, - ); - } else { - printInfo( - `"${grpcHost}" is an IP address. TLS requires a DNS-style server name for certificate ` + - 'verification (SNI), so you must provide the name the server certificate was issued for.', + if (isIpAddress) { + // Fall back to a stable, sensible default when the certificate's SAN + // didn't reveal a DNS name (e.g. a pre-existing cert with only an IP). + const overrideDefault = autoSuggestedOverride ?? 'localhost'; + if (autoSuggestedOverride) { + printInfo( + `"${host}" is an IP address. TLS clients cannot verify a bare IP as a server name, ` + + `so this connection uses the DNS name the server's certificate identifies as — ` + + `suggested \`${autoSuggestedOverride}\` from the certificate.`, + ); + } else { + printInfo( + `"${host}" is an IP address. TLS requires a DNS-style server name for certificate ` + + 'verification (SNI), so you must provide the name the server certificate was issued for.', + ); + } + httpTlsServerNameOverride = await promptText('TLS server name override', { + default: overrideDefault, + }); + } + + if (!httpTlsCaPath) { + const caPathAnswer = await promptText( + 'Path to a CA certificate to trust (leave blank if the server uses a publicly-trusted certificate)', + { default: '' }, ); + if (caPathAnswer.trim() !== '') httpTlsCaPath = caPathAnswer.trim(); } - grpcTlsServerNameOverride = await promptText('TLS server name override', { - default: overrideDefault, - }); } - if (!grpcTlsCaPath) { - const caPathAnswer = await promptText( - 'Path to a CA certificate to trust (leave blank if the server uses a publicly-trusted certificate)', - { default: '' }, - ); - if (caPathAnswer.trim() !== '') grpcTlsCaPath = caPathAnswer.trim(); - } + const localHost = await promptText('Local Dolt SQL bind host', { + default: '127.0.0.1', + }); + const localPortRaw = await promptText('Local Dolt SQL port (must be free)', { + default: '3306', + }); + const localPort = Number.parseInt(localPortRaw, 10); const store = new ConfigStore(defaultConfigPath); await store.save({ serverUrl, - grpcHost, - grpcPort: Number.isFinite(grpcPort) ? grpcPort : undefined, - grpcTlsCaPath, - grpcTlsServerNameOverride, + httpTlsCaPath, + httpTlsServerNameOverride, + localHost, + localPort: Number.isFinite(localPort) ? localPort : 3306, }); printSuccess(`Configuration saved to ${defaultConfigPath}`); - printKeyValues({ serverUrl, grpcHost, grpcPort, grpcTlsCaPath, grpcTlsServerNameOverride }); + printKeyValues({ + serverUrl, + localHost, + localPort: Number.isFinite(localPort) ? localPort : 3306, + httpTlsCaPath, + httpTlsServerNameOverride, + }); return 0; } diff --git a/src/contexts/config/config-store.ts b/src/contexts/config/config-store.ts index 484446f..42258ab 100644 --- a/src/contexts/config/config-store.ts +++ b/src/contexts/config/config-store.ts @@ -1,9 +1,9 @@ /** - * Persists connection settings (server URL, gRPC host/port, TLS trust - * options) to disk (default: `~/.deltix/config.json`), so a first-time user + * Persists connection settings (server URL, TLS trust options, local Dolt + * bind) to disk (default: `~/.deltix/config.json`), so a first-time user * doesn't have to hand-set env vars to connect to a non-default host — - * particularly the gRPC TLS options, which previously required knowing - * about `DELTIX_GRPC_TLS_SERVER_NAME_OVERRIDE` in advance to avoid the + * particularly the TLS options, which previously required knowing about + * `DELTIX_HTTP_TLS_SERVER_NAME_OVERRIDE` in advance to avoid the * `ERR_INVALID_ARG_VALUE` SNI-on-IP-address crash. * * Values here are only *defaults*: env vars, when explicitly set, always @@ -15,25 +15,22 @@ import { mkdir, readFile, writeFile } from 'node:fs/promises'; import { dirname } from 'node:path'; export interface StoredConfig { + /** REST URL of Deltix-Server (e.g. http://server:9090 or https://). */ serverUrl?: string; - grpcHost?: string; - grpcPort?: number; - grpcTlsCaPath?: string; - grpcTlsServerNameOverride?: string; - /** - * CA cert / SNI override for HTTP (REST) calls. Optional and independent - * from the gRPC fields above — set this only when the HTTP control plane - * presents a *different* certificate than the gRPC transfer engine. When - * omitted, `applyPersistedConfigDefaults()` falls back to the gRPC values, - * since both normally share the same self-signed certificate. - */ + /** CA to trust the server's HTTPS certificate against. */ httpTlsCaPath?: string; + /** SNI ServerName when connecting to the server by bare IP address. */ httpTlsServerNameOverride?: string; /** - * Port the local `dolt sql-server` (mysql-embedded context) binds to when - * `deltix start` launches it. Optional — defaults to 3306, and is only - * persisted when `deltix configure` (or manual config editing) sets it, - * e.g. to avoid colliding with a pre-installed MySQL service. + * Bind address for the local `dolt sql-server` (the mysql-embedded + * context). Defaults to 127.0.0.1; persisted via `deltix configure`. + */ + localHost?: string; + /** + * Port the local `dolt sql-server` binds to when `deltix start` launches + * it. Defaults to 3306; persisted via `deltix configure` so a host + * with a pre-installed MySQL on 3306 can pick a free port without + * exporting env vars by hand. */ localPort?: number; /** diff --git a/src/contexts/dataflow/README.md b/src/contexts/dataflow/README.md deleted file mode 100644 index 91316a1..0000000 --- a/src/contexts/dataflow/README.md +++ /dev/null @@ -1,20 +0,0 @@ -# Context: dataflow - -Orchestrates a full Push or Pull transfer against Deltix-Server's Fase 3 -gRPC Transfer Engine: - -1. Mint a fresh access token via the `session` context (never persisted to - disk — only the refresh token is stored). -2. Issue a scoped, single-use ephemeral ticket over REST - (`POST /api/v1/push/ticket`), authenticated by that access token. -3. Open a real TLS gRPC stream (`acl/grpc-transfer-client.ts`) authenticated - by the ticket, streaming the local file in Push, or the remote repo's - synced bytes in Pull. -4. Keep the ticket's sliding window alive for the duration of the transfer - via the `heartbeat` context. -5. Always close the ticket when the transfer finishes, success or failure. - -Depends on `session` (access tokens) and `heartbeat` (keep-alive loop) only -through their public `index.ts` barrels — never their internals. - -Only `index.ts` from this folder may be imported by other contexts (ACL boundary). diff --git a/src/contexts/dataflow/create-dataflow-service.ts b/src/contexts/dataflow/create-dataflow-service.ts deleted file mode 100644 index 743f86c..0000000 --- a/src/contexts/dataflow/create-dataflow-service.ts +++ /dev/null @@ -1,24 +0,0 @@ -import { GrpcTransferClient } from '../../acl/grpc-transfer-client'; -import { TransferTicketApiAdapter } from '../../acl/transfer-ticket-api-adapter'; -import { loadEnv } from '../../shared/env'; -import { createSessionService } from '../session'; -import { DataflowService } from './dataflow.service'; - -export function createDataflowService(): DataflowService { - const env = loadEnv(); - const sessionService = createSessionService(); - const ticketApi = new TransferTicketApiAdapter(env.DELTIX_SERVER_URL, { - caCertPath: env.DELTIX_HTTP_TLS_CA_PATH, - serverNameOverride: env.DELTIX_HTTP_TLS_SERVER_NAME_OVERRIDE, - }); - const grpcClient = new GrpcTransferClient(env.DELTIX_GRPC_HOST, env.DELTIX_GRPC_PORT, { - caCertPath: env.DELTIX_GRPC_TLS_CA_PATH, - serverNameOverride: env.DELTIX_GRPC_TLS_SERVER_NAME_OVERRIDE, - }); - return new DataflowService( - sessionService, - ticketApi, - grpcClient, - env.DELTIX_HEARTBEAT_INTERVAL_MS, - ); -} diff --git a/src/contexts/dataflow/dataflow.service.ts b/src/contexts/dataflow/dataflow.service.ts deleted file mode 100644 index 0cbb8ab..0000000 --- a/src/contexts/dataflow/dataflow.service.ts +++ /dev/null @@ -1,134 +0,0 @@ -/** - * Orchestrates a full Push or Pull transfer: mint a fresh access token via - * the session context -> issue a scoped ephemeral ticket over REST -> open - * a real TLS gRPC stream authenticated by that ticket -> keep the ticket's - * sliding window alive via the heartbeat context while bytes are in - * flight -> close the ticket once the transfer finishes (success or - * failure — a ticket must never be left open past its transfer). - * - * No business logic lives in the CLI command handlers — it all lives here, - * matching the `SessionService` pattern from Fase 2. - */ -import { createHash } from 'node:crypto'; -import { createReadStream, createWriteStream } from 'node:fs'; -import { stat } from 'node:fs/promises'; -import type { GrpcTransferClient, PushSummary } from '../../acl/grpc-transfer-client'; -import type { TransferTicketApiAdapter } from '../../acl/transfer-ticket-api-adapter'; -import { startHeartbeat } from '../heartbeat'; -import type { SessionService } from '../session'; -import { ChecksumMismatchError, LocalFileNotFoundError, TransferAbortedError } from './errors'; - -export interface PushResult { - jobId: string; - checksum: string; - bytesSent: number; -} - -export interface PullResult { - bytesReceived: number; - checksum: string; -} - -const CHUNK_SIZE = 64 * 1024; - -export class DataflowService { - constructor( - private readonly sessionService: SessionService, - private readonly ticketApi: TransferTicketApiAdapter, - private readonly grpcClient: GrpcTransferClient, - private readonly heartbeatIntervalMs: number, - ) {} - - async push(repo: string, localFilePath: string): Promise { - const fileInfo = await stat(localFilePath).catch(() => null); - if (!fileInfo?.isFile()) { - throw new LocalFileNotFoundError(localFilePath); - } - - const accessToken = await this.sessionService.mintAccessToken(); - const ticket = await this.ticketApi.issueTicket(accessToken, 'push', repo); - - const heartbeat = startHeartbeat( - () => this.grpcClient.heartbeat(ticket.ticketId), - this.heartbeatIntervalMs, - () => { - // A failed heartbeat is surfaced by the eventual gRPC call failure - // (the server fails-closed on an expired ticket) — nothing to do - // here beyond not crashing the interval loop itself. - }, - ); - - try { - const summary: PushSummary = await this.grpcClient.push( - ticket.ticketId, - repo, - readFileChunks(localFilePath), - ); - return { jobId: summary.jobId, checksum: summary.checksum, bytesSent: summary.bytesReceived }; - } catch (err) { - throw new TransferAbortedError(err instanceof Error ? err.message : String(err)); - } finally { - heartbeat.stop(); - await this.ticketApi.closeTicket(accessToken, ticket.ticketId).catch(() => { - // Best-effort close: the ticket also self-expires via its sliding - // window, so a failed close here is not a correctness issue. - }); - } - } - - async pull(repo: string, destinationFilePath: string): Promise { - const accessToken = await this.sessionService.mintAccessToken(); - const ticket = await this.ticketApi.issueTicket(accessToken, 'pull', repo); - - const heartbeat = startHeartbeat( - () => this.grpcClient.heartbeat(ticket.ticketId), - this.heartbeatIntervalMs, - () => {}, - ); - - try { - const stream = this.grpcClient.pullStream(ticket.ticketId, repo); - const hash = createHash('sha256'); - let bytesReceived = 0; - - await new Promise((resolve, reject) => { - const out = createWriteStream(destinationFilePath); - stream.on('data', (chunk: { data: Buffer }) => { - hash.update(chunk.data); - bytesReceived += chunk.data.length; - out.write(chunk.data); - }); - stream.on('end', () => out.end(resolve)); - stream.on('error', (err) => { - out.destroy(); - reject(err); - }); - }); - - return { bytesReceived, checksum: hash.digest('hex') }; - } catch (err) { - throw new TransferAbortedError(err instanceof Error ? err.message : String(err)); - } finally { - heartbeat.stop(); - await this.ticketApi.closeTicket(accessToken, ticket.ticketId).catch(() => {}); - } - } - - /** - * Verifies a just-completed pull's checksum against the server's - * PushSummary-equivalent (the caller supplies the expected checksum, - * e.g. from a prior manifest or the storage API's dead-letter listing). - */ - static assertChecksum(expected: string, actual: string): void { - if (expected !== actual) { - throw new ChecksumMismatchError(); - } - } -} - -async function* readFileChunks(path: string): AsyncGenerator { - const stream = createReadStream(path, { highWaterMark: CHUNK_SIZE }); - for await (const chunk of stream) { - yield chunk as Uint8Array; - } -} diff --git a/src/contexts/dataflow/errors.ts b/src/contexts/dataflow/errors.ts deleted file mode 100644 index 119bd5d..0000000 --- a/src/contexts/dataflow/errors.ts +++ /dev/null @@ -1,41 +0,0 @@ -export class TicketAuthenticationError extends Error { - constructor() { - super('Not authenticated. Run `deltix login` first.'); - this.name = 'TicketAuthenticationError'; - } -} - -export class TicketIssuanceError extends Error { - constructor(status: number) { - super(`Server rejected the ticket request (HTTP ${status})`); - this.name = 'TicketIssuanceError'; - } -} - -export class TicketNotFoundOrInactiveError extends Error { - constructor() { - super('Ticket not found or not active'); - this.name = 'TicketNotFoundOrInactiveError'; - } -} - -export class TransferAbortedError extends Error { - constructor(reason: string) { - super(`Transfer aborted: ${reason}`); - this.name = 'TransferAbortedError'; - } -} - -export class LocalFileNotFoundError extends Error { - constructor(path: string) { - super(`Local file not found: ${path}`); - this.name = 'LocalFileNotFoundError'; - } -} - -export class ChecksumMismatchError extends Error { - constructor() { - super('Checksum mismatch after transfer — data integrity could not be verified'); - this.name = 'ChecksumMismatchError'; - } -} diff --git a/src/contexts/dataflow/index.ts b/src/contexts/dataflow/index.ts deleted file mode 100644 index 8ce16c3..0000000 --- a/src/contexts/dataflow/index.ts +++ /dev/null @@ -1,20 +0,0 @@ -/** - * The "dataflow" bounded context: thin orchestration around Push/Pull - * transfers over the Fase 3 gRPC Transfer Engine, using an ephemeral - * ticket obtained from the `session` context's access token. - * - * This is the ONLY file other contexts/modules are allowed to import from - * (ACL boundary). Internals of this context must never be imported directly - * from outside. - */ -export { createDataflowService } from './create-dataflow-service'; -export type { PullResult, PushResult } from './dataflow.service'; -export { DataflowService } from './dataflow.service'; -export { - ChecksumMismatchError, - LocalFileNotFoundError, - TicketAuthenticationError, - TicketIssuanceError, - TicketNotFoundOrInactiveError, - TransferAbortedError, -} from './errors'; diff --git a/src/contexts/heartbeat/README.md b/src/contexts/heartbeat/README.md deleted file mode 100644 index 02a1e43..0000000 --- a/src/contexts/heartbeat/README.md +++ /dev/null @@ -1,10 +0,0 @@ -# Context: heartbeat - -Sliding-window keep-alive for gRPC transfer tickets: `startHeartbeat()` runs a -lightweight `setInterval` loop calling `renew()` (backed by -`GrpcTransferClient.heartbeat()`) roughly every `DELTIX_HEARTBEAT_INTERVAL_MS` -(default 30s), comfortably below the ticket's TTL (default 120s). Started by -`dataflow` at the beginning of a Push/Pull and always stopped when the -transfer finishes (success or failure) — never left running unattended. - -Only `index.ts` from this folder may be imported by other contexts (ACL boundary). diff --git a/src/contexts/heartbeat/heartbeat.service.ts b/src/contexts/heartbeat/heartbeat.service.ts deleted file mode 100644 index 5c08c6b..0000000 --- a/src/contexts/heartbeat/heartbeat.service.ts +++ /dev/null @@ -1,30 +0,0 @@ -/** - * Sliding-window heartbeat: keeps an active gRPC transfer ticket alive - * during a long-running Push/Pull by calling the server's Heartbeat RPC - * roughly every `intervalMs` (default 30s, comfortably below the ticket's - * TTL). Started when a transfer begins and stopped as soon as it finishes - * (success or failure) — never left running unattended. - * - * Deliberately decoupled from `dataflow`'s push/pull orchestration so it - * can be unit tested with a fake ticket-renewal function, without a live - * gRPC connection. - */ -export interface HeartbeatHandle { - stop(): void; -} - -export function startHeartbeat( - renew: () => Promise, - intervalMs: number, - onError: (err: unknown) => void, -): HeartbeatHandle { - const timer = setInterval(() => { - renew().catch(onError); - }, intervalMs); - // Never keep the CLI process alive just for the heartbeat timer. - timer.unref?.(); - - return { - stop: () => clearInterval(timer), - }; -} diff --git a/src/contexts/heartbeat/index.ts b/src/contexts/heartbeat/index.ts deleted file mode 100644 index b490b76..0000000 --- a/src/contexts/heartbeat/index.ts +++ /dev/null @@ -1,11 +0,0 @@ -/** - * The "heartbeat" bounded context: a background loop that pings the - * server's gRPC Heartbeat RPC roughly every 30s to keep the sliding-window - * transfer ticket alive during a long-running Push/Pull. - * - * This is the ONLY file other contexts/modules are allowed to import from - * (ACL boundary). Internals of this context must never be imported directly - * from outside. - */ -export type { HeartbeatHandle } from './heartbeat.service'; -export { startHeartbeat } from './heartbeat.service'; diff --git a/src/shared/env.ts b/src/shared/env.ts index 261d037..bc34ce7 100644 --- a/src/shared/env.ts +++ b/src/shared/env.ts @@ -12,39 +12,17 @@ const envSchema = z.object({ NODE_ENV: z.enum(['development', 'test', 'production']).default('development'), DELTIX_SERVER_URL: z.string().url().default('http://127.0.0.1:9090'), DELTIX_CREDENTIALS_PATH: z.string().min(1).optional(), - // Fase 3: gRPC transfer engine target. Host/port only — TLS is always - // required (no plaintext code path on the server), so the client never - // has an "insecure" option here either. - DELTIX_GRPC_HOST: z.string().min(1).default('127.0.0.1'), - DELTIX_GRPC_PORT: z.coerce.number().int().positive().default(50051), - // Path to a CA certificate to trust for the gRPC TLS connection. Required + // Path to a CA certificate to trust for HTTP (REST) calls to the control + // plane — login, push/pull ticket issuance, versioning API. Required // whenever the server uses a self-signed certificate (the default for // local/dev deployments) — omit only when the server's certificate is // already trusted by the OS root store (e.g. a real CA-signed cert). - DELTIX_GRPC_TLS_CA_PATH: z.string().min(1).optional(), - // Overrides SNI/TLS ServerName verification when DELTIX_GRPC_HOST is an - // IP address (Node's TLS stack rejects IP ServerNames outright). Must - // match the DNS name the server's certificate was issued for (our - // self-signed dev/test certs use CN=localhost). - DELTIX_GRPC_TLS_SERVER_NAME_OVERRIDE: z.string().min(1).optional(), - // Path to a CA certificate to trust for HTTP (REST) calls to the control - // plane — login, push/pull ticket issuance, versioning API. In most real - // deployments this is the *same* certificate as the gRPC one (the - // installer generates one self-signed cert used for both), so - // `applyPersistedConfigDefaults()` falls back to `DELTIX_GRPC_TLS_CA_PATH` - // when this is not set explicitly. Without it, any self-signed server - // certificate causes every HTTP call to fail with - // `TypeError: self signed certificate`. DELTIX_HTTP_TLS_CA_PATH: z.string().min(1).optional(), - // Overrides the TLS ServerName used for HTTP certificate verification, - // for the same reason as DELTIX_GRPC_TLS_SERVER_NAME_OVERRIDE (Node/Bun's - // TLS stack rejects IP-address ServerNames). Falls back to - // DELTIX_GRPC_TLS_SERVER_NAME_OVERRIDE when not set explicitly. + // Overrides the TLS ServerName used for HTTP certificate verification when + // the server URL uses a bare IP address (Bun/Node reject IP-address + // ServerNames). Must match the DNS name the server's certificate was + // issued for (our self-signed dev/test certs use CN=localhost). DELTIX_HTTP_TLS_SERVER_NAME_OVERRIDE: z.string().min(1).optional(), - // Heartbeat cadence while a Push/Pull is in flight, kept comfortably - // below the server's ticket TTL (default 120s) so the sliding window - // never lapses mid-transfer. - DELTIX_HEARTBEAT_INTERVAL_MS: z.coerce.number().int().positive().default(30_000), // Path to an existing `dolt` binary to use instead of the one the // binary-manager downloads/installs into `~/.deltix/bin/`. When set, the // binary-manager trusts it as-is (no download, no integrity re-verify it @@ -57,19 +35,11 @@ const envSchema = z.object({ // Defaults to `~/.deltix`; overridable so tests and CI can isolate state. DELTIX_HOME: z.string().min(1).optional(), // Host/port the local `dolt sql-server` binds to (mysql-embedded context). - // Loopback-only by default; 3306 is the conventional MySQL port, overridable - // when the host already runs a real MySQL service. + // Loopback-only by default; 3306 is the conventional MySQL port. Persisted + // via `deltix configure` so host names with a pre-installed MySQL/MariaDB + // can pick a free port without exporting env vars by hand. DELTIX_LOCAL_HOST: z.string().min(1).default('127.0.0.1'), DELTIX_LOCAL_PORT: z.coerce.number().int().positive().default(3306), - // Transitional feature flag (Fase 5.9). `deltix pull` is now commit-based - // over REST (native, git-like). When this is enabled AND a destination file - // is passed, `deltix pull` falls back to the legacy whole-file gRPC transfer - // instead. Off by default; kept only to allow an easy rollback/comparison - // until the native path is fully confirmed, then the gRPC pull path is removed. - DELTIX_ENABLE_GRPC_TRANSFER: z - .enum(['true', 'false', '1', '0']) - .default('false') - .transform((v) => v === 'true' || v === '1'), }); export type Env = z.infer; @@ -92,27 +62,17 @@ export function loadEnv(source: Record = Bun.env): E */ export function applyPersistedConfigDefaults(config: { serverUrl?: string; - grpcHost?: string; - grpcPort?: number; - grpcTlsCaPath?: string; - grpcTlsServerNameOverride?: string; httpTlsCaPath?: string; httpTlsServerNameOverride?: string; + localHost?: string; localPort?: number; localDoltBinPath?: string; }): void { const fallback: Record = { DELTIX_SERVER_URL: config.serverUrl, - DELTIX_GRPC_HOST: config.grpcHost, - DELTIX_GRPC_PORT: config.grpcPort?.toString(), - DELTIX_GRPC_TLS_CA_PATH: config.grpcTlsCaPath, - DELTIX_GRPC_TLS_SERVER_NAME_OVERRIDE: config.grpcTlsServerNameOverride, - // The HTTP control plane and gRPC transfer engine present the same - // certificate in most real deployments, so default the HTTP-specific - // values from their gRPC counterparts when not explicitly configured. - DELTIX_HTTP_TLS_CA_PATH: config.httpTlsCaPath ?? config.grpcTlsCaPath, - DELTIX_HTTP_TLS_SERVER_NAME_OVERRIDE: - config.httpTlsServerNameOverride ?? config.grpcTlsServerNameOverride, + DELTIX_HTTP_TLS_CA_PATH: config.httpTlsCaPath, + DELTIX_HTTP_TLS_SERVER_NAME_OVERRIDE: config.httpTlsServerNameOverride, + DELTIX_LOCAL_HOST: config.localHost, // mysql-embedded: the local Dolt SQL server's bind port and the Dolt // binary to use, so `deltix configure` can customise them once. DELTIX_LOCAL_PORT: config.localPort?.toString(), diff --git a/tests/integration/dataflow/push-pull.integration.test.ts b/tests/integration/dataflow/push-pull.integration.test.ts deleted file mode 100644 index 6411473..0000000 --- a/tests/integration/dataflow/push-pull.integration.test.ts +++ /dev/null @@ -1,165 +0,0 @@ -import { afterAll, beforeAll, describe, expect, it } from 'bun:test'; -import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; -import { tmpdir } from 'node:os'; -import { join } from 'node:path'; -import { $ } from 'bun'; -import { createDataflowService } from '../../../src/contexts/dataflow'; -import { createSessionService } from '../../../src/contexts/session'; -import { __resetEnvCacheForTests } from '../../../src/shared/env'; - -/** - * Real cross-repo end-to-end integration test: boots an ACTUAL Deltix-Server - * subprocess (black-box, only via its public HTTP/gRPC contract — never by - * importing its source, per the MIT/BSL license separation rule), logs in - * via the real `session` context, then drives the real `dataflow` context - * to Push a file over real TLS gRPC and Pull it back, verifying the round - * trip is byte-for-byte identical. This is the strongest evidence that - * Fase 3's client<->server integration actually works end-to-end. - * - * Skipped automatically if the sibling Deltix-Server checkout isn't present - * (e.g. a CI job that only checks out this repo). - */ -const SERVER_REPO_PATH = join(import.meta.dir, '..', '..', '..', '..', 'Deltix-Server'); -const SERVER_ENTRYPOINT = join(SERVER_REPO_PATH, 'src', 'index.ts'); -const serverAvailable = await Bun.file(SERVER_ENTRYPOINT).exists(); - -async function initTempDoltRepo(): Promise { - const repoPath = await mkdtemp(join(tmpdir(), 'deltix-dolt-repo-dataflow-it-')); - await $`dolt config --global --add user.name deltix-test`.quiet().nothrow(); - await $`dolt config --global --add user.email deltix-test@example.com`.quiet().nothrow(); - await $`dolt --data-dir ${repoPath} init`.quiet().nothrow(); - return repoPath; -} - -describe.if(serverAvailable)( - 'dataflow push/pull end-to-end (real Deltix-Server subprocess, real TLS gRPC)', - () => { - let repoPath: string; - let workDir: string; - let httpPort: number; - let grpcPort: number; - let serverProc: ReturnType; - - beforeAll(async () => { - repoPath = await initTempDoltRepo(); - workDir = await mkdtemp(join(tmpdir(), 'deltix-dataflow-it-work-')); - httpPort = 23000 + Math.floor(Math.random() * 2000); - grpcPort = 47000 + Math.floor(Math.random() * 2000); - - const keygen = await import( - join(SERVER_REPO_PATH, 'tests', 'fixtures', 'license-fixtures.ts') - ); - const { publicKeyBase64, privateKeyPem } = keygen.generateTestKeypair(); - const licenseKey = keygen.signLicensePayload(keygen.buildDefaultPayload(), privateKeyPem); - const { privateKeyPem: jwtPrivateKeyPem, publicKeyPem: jwtPublicKeyPem } = - keygen.generateTestJwtKeypairPem(); - const { hashPassword } = await import( - join(SERVER_REPO_PATH, 'src', 'contexts', 'auth', 'password-authenticator.ts') - ); - const localUsers = JSON.stringify([ - { username: 'alice', passwordHash: await hashPassword('s3cret-pass') }, - ]); - const { generateSelfSignedCert } = await import( - join(SERVER_REPO_PATH, 'tests', 'fixtures', 'tls-fixtures.ts') - ); - const { certPath, keyPath } = await generateSelfSignedCert(workDir); - - // Repo RBAC is enforced at ticket-issuance time on the server - // (transfer.router.ts): a caller needs `writer` for push and `reader` - // for pull. DELTIX_LOCAL_USERS carries no repo roles, so seed alice as - // a real user row (FK target for repo_roles) with admin on the repo - // this test exercises before boot. - const userDbPath = join(workDir, 'users.db'); - const { LibsqlUserStore } = await import( - join(SERVER_REPO_PATH, 'src', 'contexts', 'auth', 'libsql-user-store.ts') - ); - const userStore = new LibsqlUserStore(userDbPath); - await userStore.init(); - await userStore.create({ - username: 'alice', - passwordHash: await hashPassword('s3cret-pass'), - createdAt: Date.now(), - createdBy: 'test-seed', - active: true, - lastLoginAt: null, - }); - await userStore.upsertRepoRole({ - username: 'alice', - repoId: 'org/demo-repo', - role: 'admin', - grantedAt: Date.now(), - grantedBy: 'test-seed', - }); - - serverProc = Bun.spawn(['bun', 'run', SERVER_ENTRYPOINT], { - cwd: SERVER_REPO_PATH, - env: { - ...process.env, - DELTIX_LICENSE_PUBLIC_KEY: publicKeyBase64, - DELTIX_LICENSE_KEY: licenseKey, - DELTIX_DOLT_REPO_PATH: repoPath, - DELTIX_CLOCK_TOLERANCE_MS: '5000', - DELTIX_JWT_PRIVATE_KEY: jwtPrivateKeyPem, - DELTIX_JWT_PUBLIC_KEY: jwtPublicKeyPem, - DELTIX_LOCAL_USERS: localUsers, - DELTIX_USER_DB_PATH: userDbPath, - DELTIX_SESSION_DB_PATH: join(workDir, 'sessions.db'), - DELTIX_TICKET_DB_PATH: join(workDir, 'tickets.db'), - DELTIX_TRANSFER_JOB_DB_PATH: join(workDir, 'transfer-jobs.db'), - DELTIX_NAS_SIM_PATH: join(workDir, 'nas-sim'), - DELTIX_STAGING_ROOT_PATH: join(workDir, 'staging'), - DELTIX_NAS_SYNC_POLL_INTERVAL_MS: '300', - DELTIX_GRPC_PORT: String(grpcPort), - DELTIX_GRPC_TLS_CERT_PATH: certPath, - DELTIX_GRPC_TLS_KEY_PATH: keyPath, - HTTP_PORT: String(httpPort), - LOG_PRETTY: 'false', - }, - stdout: 'pipe', - stderr: 'pipe', - }); - - await new Promise((resolve) => setTimeout(resolve, 1500)); - - // Point the client at this real server + its self-signed cert. - __resetEnvCacheForTests(); - process.env.DELTIX_SERVER_URL = `http://127.0.0.1:${httpPort}`; - process.env.DELTIX_CREDENTIALS_PATH = join(workDir, 'credentials.json'); - process.env.DELTIX_GRPC_HOST = '127.0.0.1'; - process.env.DELTIX_GRPC_PORT = String(grpcPort); - process.env.DELTIX_GRPC_TLS_CA_PATH = certPath; - process.env.DELTIX_GRPC_TLS_SERVER_NAME_OVERRIDE = 'localhost'; - process.env.DELTIX_HEARTBEAT_INTERVAL_MS = '5000'; - - await createSessionService().login('alice', 's3cret-pass'); - }); - - afterAll(async () => { - serverProc.kill(); - await rm(repoPath, { recursive: true, force: true }); - await rm(workDir, { recursive: true, force: true }); - __resetEnvCacheForTests(); - }); - - it('pushes a local file, waits for NAS sync, then pulls it back byte-for-byte identical', async () => { - const localFilePath = join(workDir, 'local-repo.dolt'); - const pulledFilePath = join(workDir, 'pulled-repo.dolt'); - const fileContents = 'deltix repo bytes '.repeat(1000); - await writeFile(localFilePath, fileContents); - - const pushResult = await createDataflowService().push('org/demo-repo', localFilePath); - expect(pushResult.bytesSent).toBe(Buffer.byteLength(fileContents)); - expect(pushResult.checksum).toBeString(); - - // Give the real NAS sync worker (independent poll loop) time to - // promote the staged job before attempting the pull. - await new Promise((resolve) => setTimeout(resolve, 1500)); - - const pullResult = await createDataflowService().pull('org/demo-repo', pulledFilePath); - expect(pullResult.checksum).toBe(pushResult.checksum); - - const pulledContents = await readFile(pulledFilePath, 'utf8'); - expect(pulledContents).toBe(fileContents); - }, 20_000); - }, -); diff --git a/tests/unit/acl/grpc-transfer-client.test.ts b/tests/unit/acl/grpc-transfer-client.test.ts deleted file mode 100644 index dcf022b..0000000 --- a/tests/unit/acl/grpc-transfer-client.test.ts +++ /dev/null @@ -1,26 +0,0 @@ -import { describe, expect, it } from 'bun:test'; -import { normalizeGrpcHost } from '../../../src/acl/grpc-transfer-client'; - -describe('acl/grpc-transfer-client normalizeGrpcHost (unit)', () => { - it('keeps a clean hostname unchanged', () => { - expect(normalizeGrpcHost('10.1.10.129')).toBe('10.1.10.129'); - expect(normalizeGrpcHost('hbs-svr-pulse')).toBe('hbs-svr-pulse'); - }); - - it('strips surrounding whitespace', () => { - expect(normalizeGrpcHost(' 10.1.10.129 ')).toBe('10.1.10.129'); - }); - - it('strips embedded newlines that would otherwise break the grpc-js target', () => { - expect(normalizeGrpcHost('10.1.10.129\n\n')).toBe('10.1.10.129'); - expect(normalizeGrpcHost('10.1.10.129\n\n')).not.toContain('\n'); - }); - - it('strips mixed whitespace including carriage returns', () => { - expect(normalizeGrpcHost('10.1.10.129\r\n')).toBe('10.1.10.129'); - }); - - it('normalizes an all-whitespace host to an empty string', () => { - expect(normalizeGrpcHost(' \n ')).toBe(''); - }); -}); diff --git a/tests/unit/acl/transfer-ticket-api-adapter.test.ts b/tests/unit/acl/transfer-ticket-api-adapter.test.ts deleted file mode 100644 index 54267c5..0000000 --- a/tests/unit/acl/transfer-ticket-api-adapter.test.ts +++ /dev/null @@ -1,104 +0,0 @@ -import { afterEach, describe, expect, it } from 'bun:test'; -import { TransferTicketApiAdapter } from '../../../src/acl/transfer-ticket-api-adapter'; -import { - TicketAuthenticationError, - TicketIssuanceError, - TicketNotFoundOrInactiveError, -} from '../../../src/contexts/dataflow/errors'; -import { ServerUnreachableError } from '../../../src/contexts/session/errors'; - -describe('acl/transfer-ticket-api-adapter (unit, mocked fetch)', () => { - const originalFetch = globalThis.fetch; - - afterEach(() => { - globalThis.fetch = originalFetch; - }); - - it('issueTicket() posts operation/repo with a bearer token and returns the parsed ticket', async () => { - let capturedUrl: string | undefined; - let capturedAuth: string | null = null; - let capturedBody: unknown; - globalThis.fetch = (async (url: string, init: RequestInit) => { - capturedUrl = url; - capturedAuth = (init.headers as Record).authorization; - capturedBody = JSON.parse(init.body as string); - return new Response( - JSON.stringify({ ticketId: 't1', operation: 'push', repo: 'org/repo', expiresAt: 123 }), - { status: 201 }, - ); - }) as typeof fetch; - - const adapter = new TransferTicketApiAdapter('http://127.0.0.1:9090'); - const result = await adapter.issueTicket('access-token', 'push', 'org/repo'); - - expect(capturedUrl).toBe('http://127.0.0.1:9090/api/v1/push/ticket'); - expect(capturedAuth).toBe('Bearer access-token'); - expect(capturedBody).toEqual({ operation: 'push', repo: 'org/repo' }); - expect(result).toEqual({ ticketId: 't1', operation: 'push', repo: 'org/repo', expiresAt: 123 }); - }); - - it('issueTicket() throws TicketAuthenticationError on a 401 response', async () => { - globalThis.fetch = (async () => - new Response(JSON.stringify({ error: 'Unauthorized' }), { status: 401 })) as typeof fetch; - - const adapter = new TransferTicketApiAdapter('http://127.0.0.1:9090'); - - await expect(adapter.issueTicket('bad-token', 'push', 'org/repo')).rejects.toThrow( - TicketAuthenticationError, - ); - }); - - it('issueTicket() throws TicketIssuanceError on an unexpected non-2xx response', async () => { - globalThis.fetch = (async () => - new Response(JSON.stringify({ error: 'Invalid request body' }), { - status: 400, - })) as typeof fetch; - - const adapter = new TransferTicketApiAdapter('http://127.0.0.1:9090'); - - await expect(adapter.issueTicket('token', 'push', 'org/repo')).rejects.toThrow( - TicketIssuanceError, - ); - }); - - it('issueTicket() wraps a network failure in ServerUnreachableError', async () => { - globalThis.fetch = (async () => { - throw new Error('ECONNREFUSED'); - }) as typeof fetch; - - const adapter = new TransferTicketApiAdapter('http://127.0.0.1:9090'); - - await expect(adapter.issueTicket('token', 'push', 'org/repo')).rejects.toThrow( - ServerUnreachableError, - ); - }); - - it('closeTicket() posts the ticketId with a bearer token and resolves on 200', async () => { - let capturedUrl: string | undefined; - let capturedBody: unknown; - globalThis.fetch = (async (url: string, init: RequestInit) => { - capturedUrl = url; - capturedBody = JSON.parse(init.body as string); - return new Response(JSON.stringify({ ok: true }), { status: 200 }); - }) as typeof fetch; - - const adapter = new TransferTicketApiAdapter('http://127.0.0.1:9090'); - await adapter.closeTicket('token', 't1'); - - expect(capturedUrl).toBe('http://127.0.0.1:9090/api/v1/auth/session-close'); - expect(capturedBody).toEqual({ ticketId: 't1' }); - }); - - it('closeTicket() throws TicketNotFoundOrInactiveError on a 404 response', async () => { - globalThis.fetch = (async () => - new Response(JSON.stringify({ error: 'Ticket not found or not active' }), { - status: 404, - })) as typeof fetch; - - const adapter = new TransferTicketApiAdapter('http://127.0.0.1:9090'); - - await expect(adapter.closeTicket('token', 'unknown-ticket')).rejects.toThrow( - TicketNotFoundOrInactiveError, - ); - }); -}); diff --git a/tests/unit/dataflow/dataflow.service.test.ts b/tests/unit/dataflow/dataflow.service.test.ts deleted file mode 100644 index ca55fec..0000000 --- a/tests/unit/dataflow/dataflow.service.test.ts +++ /dev/null @@ -1,193 +0,0 @@ -import { describe, expect, it } from 'bun:test'; -import { EventEmitter } from 'node:events'; -import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; -import { tmpdir } from 'node:os'; -import { join } from 'node:path'; -import type { GrpcTransferClient } from '../../../src/acl/grpc-transfer-client'; -import type { TransferTicketApiAdapter } from '../../../src/acl/transfer-ticket-api-adapter'; -import { DataflowService } from '../../../src/contexts/dataflow/dataflow.service'; -import { - LocalFileNotFoundError, - TransferAbortedError, -} from '../../../src/contexts/dataflow/errors'; -import { NoActiveSessionError } from '../../../src/contexts/session/errors'; -import type { SessionService } from '../../../src/contexts/session/session.service'; - -function fakeSessionService(overrides: Partial = {}): SessionService { - return { - mintAccessToken: async () => 'fake-access-token', - ...overrides, - } as SessionService; -} - -function fakeTicketApi( - overrides: Partial = {}, -): TransferTicketApiAdapter { - return { - issueTicket: async (_token: string, operation: 'push' | 'pull', repo: string) => ({ - ticketId: 'ticket-1', - operation, - repo, - expiresAt: Date.now() + 120_000, - }), - closeTicket: async () => undefined, - ...overrides, - } as TransferTicketApiAdapter; -} - -function fakeGrpcClient(overrides: Partial = {}): GrpcTransferClient { - return { - push: async () => ({ jobId: 'job-1', checksum: 'abc123', bytesReceived: 42 }), - heartbeat: async () => Date.now() + 120_000, - pullStream: () => { - const emitter = new EventEmitter(); - queueMicrotask(() => { - emitter.emit('data', { data: Buffer.from('hello ') }); - emitter.emit('data', { data: Buffer.from('world') }); - emitter.emit('end'); - }); - return emitter as unknown as ReturnType; - }, - close: () => {}, - ...overrides, - } as GrpcTransferClient; -} - -describe('dataflow/dataflow.service (unit, fake adapters/clients)', () => { - it('push() rejects with LocalFileNotFoundError when the local file does not exist', async () => { - const service = new DataflowService( - fakeSessionService(), - fakeTicketApi(), - fakeGrpcClient(), - 30_000, - ); - - await expect(service.push('org/repo', '/nonexistent/path/file.bin')).rejects.toThrow( - LocalFileNotFoundError, - ); - }); - - it('push() mints a token, issues a ticket, streams the file, and closes the ticket on success', async () => { - const dir = await mkdtemp(join(tmpdir(), 'deltix-dataflow-push-')); - const filePath = join(dir, 'repo.dolt'); - await writeFile(filePath, 'some repo bytes'); - - let closedTicketId: string | undefined; - const service = new DataflowService( - fakeSessionService(), - fakeTicketApi({ - closeTicket: async (_token: string, ticketId: string) => { - closedTicketId = ticketId; - }, - }), - fakeGrpcClient(), - 30_000, - ); - - const result = await service.push('org/repo', filePath); - - expect(result).toEqual({ jobId: 'job-1', checksum: 'abc123', bytesSent: 42 }); - expect(closedTicketId).toBe('ticket-1'); - await rm(dir, { recursive: true, force: true }); - }); - - it('push() closes the ticket even when the gRPC call fails, and wraps the error', async () => { - const dir = await mkdtemp(join(tmpdir(), 'deltix-dataflow-push-fail-')); - const filePath = join(dir, 'repo.dolt'); - await writeFile(filePath, 'some repo bytes'); - - let ticketClosed = false; - const service = new DataflowService( - fakeSessionService(), - fakeTicketApi({ - closeTicket: async () => { - ticketClosed = true; - }, - }), - fakeGrpcClient({ - push: async () => { - throw new Error('connection dropped'); - }, - }), - 30_000, - ); - - await expect(service.push('org/repo', filePath)).rejects.toThrow(TransferAbortedError); - expect(ticketClosed).toBe(true); - await rm(dir, { recursive: true, force: true }); - }); - - it('push() propagates NoActiveSessionError when there is no local session (never issues a ticket)', async () => { - const dir = await mkdtemp(join(tmpdir(), 'deltix-dataflow-push-nosession-')); - const filePath = join(dir, 'repo.dolt'); - await writeFile(filePath, 'bytes'); - - let ticketIssued = false; - const service = new DataflowService( - fakeSessionService({ - mintAccessToken: async () => { - throw new NoActiveSessionError(); - }, - }), - fakeTicketApi({ - issueTicket: async () => { - ticketIssued = true; - return { ticketId: 'x', operation: 'push', repo: 'org/repo', expiresAt: 0 }; - }, - }), - fakeGrpcClient(), - 30_000, - ); - - await expect(service.push('org/repo', filePath)).rejects.toThrow(NoActiveSessionError); - expect(ticketIssued).toBe(false); - await rm(dir, { recursive: true, force: true }); - }); - - it('pull() streams the remote bytes to the destination file and returns the checksum', async () => { - const dir = await mkdtemp(join(tmpdir(), 'deltix-dataflow-pull-')); - const destinationPath = join(dir, 'downloaded.dolt'); - - const service = new DataflowService( - fakeSessionService(), - fakeTicketApi(), - fakeGrpcClient(), - 30_000, - ); - - const result = await service.pull('org/repo', destinationPath); - const written = await readFile(destinationPath, 'utf8'); - - expect(written).toBe('hello world'); - expect(result.bytesReceived).toBe('hello world'.length); - expect(result.checksum).toBeString(); - await rm(dir, { recursive: true, force: true }); - }); - - it('pull() closes the ticket even when the stream errors out', async () => { - const dir = await mkdtemp(join(tmpdir(), 'deltix-dataflow-pull-fail-')); - const destinationPath = join(dir, 'downloaded.dolt'); - - let ticketClosed = false; - const service = new DataflowService( - fakeSessionService(), - fakeTicketApi({ - closeTicket: async () => { - ticketClosed = true; - }, - }), - fakeGrpcClient({ - pullStream: () => { - const emitter = new EventEmitter(); - queueMicrotask(() => emitter.emit('error', new Error('stream reset'))); - return emitter as unknown as ReturnType; - }, - }), - 30_000, - ); - - await expect(service.pull('org/repo', destinationPath)).rejects.toThrow(TransferAbortedError); - expect(ticketClosed).toBe(true); - await rm(dir, { recursive: true, force: true }); - }); -}); diff --git a/tests/unit/heartbeat/heartbeat.service.test.ts b/tests/unit/heartbeat/heartbeat.service.test.ts deleted file mode 100644 index 8aef11e..0000000 --- a/tests/unit/heartbeat/heartbeat.service.test.ts +++ /dev/null @@ -1,57 +0,0 @@ -import { describe, expect, it } from 'bun:test'; -import { startHeartbeat } from '../../../src/contexts/heartbeat/heartbeat.service'; - -describe('heartbeat/heartbeat.service (unit, fake timers via short real intervals)', () => { - it('calls renew() repeatedly at roughly the configured interval until stopped', async () => { - let calls = 0; - const handle = startHeartbeat( - async () => { - calls += 1; - return Date.now(); - }, - 10, - () => {}, - ); - - await new Promise((resolve) => setTimeout(resolve, 55)); - handle.stop(); - const callsAtStop = calls; - - await new Promise((resolve) => setTimeout(resolve, 30)); - - expect(callsAtStop).toBeGreaterThanOrEqual(3); - // No further calls after stop() — the interval must actually be cleared. - expect(calls).toBe(callsAtStop); - }); - - it('invokes onError when renew() rejects, without crashing the loop', async () => { - let renewCalls = 0; - let errorCalls = 0; - const handle = startHeartbeat( - async () => { - renewCalls += 1; - throw new Error('ticket expired'); - }, - 10, - () => { - errorCalls += 1; - }, - ); - - await new Promise((resolve) => setTimeout(resolve, 55)); - handle.stop(); - - expect(renewCalls).toBeGreaterThanOrEqual(3); - expect(errorCalls).toBe(renewCalls); - }); - - it('stop() is idempotent and safe to call multiple times', () => { - const handle = startHeartbeat( - async () => 0, - 1000, - () => {}, - ); - handle.stop(); - expect(() => handle.stop()).not.toThrow(); - }); -}); diff --git a/tests/unit/shared/env.test.ts b/tests/unit/shared/env.test.ts index 5907d01..adc712e 100644 --- a/tests/unit/shared/env.test.ts +++ b/tests/unit/shared/env.test.ts @@ -62,23 +62,25 @@ describe('shared/env', () => { it('fills Bun.env from persisted config when unset', () => { applyPersistedConfigDefaults({ serverUrl: 'https://10.1.10.129:9090', - grpcHost: '10.1.10.129', - grpcPort: 50051, - grpcTlsServerNameOverride: 'localhost', + httpTlsCaPath: '/etc/deltix/server.crt', + httpTlsServerNameOverride: 'hbs-svr-pulse', + localHost: '127.0.0.1', + localPort: 3307, }); expect(Bun.env.DELTIX_SERVER_URL).toBe('https://10.1.10.129:9090'); - expect(Bun.env.DELTIX_GRPC_HOST).toBe('10.1.10.129'); - expect(Bun.env.DELTIX_GRPC_PORT).toBe('50051'); - expect(Bun.env.DELTIX_GRPC_TLS_SERVER_NAME_OVERRIDE).toBe('localhost'); + expect(Bun.env.DELTIX_HTTP_TLS_CA_PATH).toBe('/etc/deltix/server.crt'); + expect(Bun.env.DELTIX_HTTP_TLS_SERVER_NAME_OVERRIDE).toBe('hbs-svr-pulse'); + expect(Bun.env.DELTIX_LOCAL_HOST).toBe('127.0.0.1'); + expect(Bun.env.DELTIX_LOCAL_PORT).toBe('3307'); }); it('never overrides an already-set env var (env vars always win)', () => { - Bun.env.DELTIX_GRPC_HOST = 'explicit-host'; + Bun.env.DELTIX_LOCAL_PORT = '3307'; - applyPersistedConfigDefaults({ grpcHost: 'from-config' }); + applyPersistedConfigDefaults({ localPort: 9999 }); - expect(Bun.env.DELTIX_GRPC_HOST).toBe('explicit-host'); + expect(Bun.env.DELTIX_LOCAL_PORT).toBe('3307'); }); }); });