diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index ce742c22..2f2d7ca9 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -9,8 +9,8 @@ Database schema **diff & migration** tool. Compare a source schema against a tar generate dialect-native migration SQL, and deploy it. Fourteen SQL dialects: Postgres (and CockroachDB / YugabyteDB / Redshift), MySQL (and MariaDB / TiDB), SQL Server / Azure SQL, Oracle, Db2, SQLite, DuckDB, ClickHouse. MongoDB and Redis -carry connection settings only. Distributions: **CLI** (`foxschema` via npm/Homebrew) -and **Docker** (single amd64 image with Db2). +run a fail-closed SQL subset in the editor (no schema compare). Distributions: +**CLI** (`foxschema` via npm/Homebrew) and **Docker** (single amd64 image with Db2). ## Commands @@ -277,13 +277,21 @@ per-dialect gotchas live in `packages/sql/src/providers/DIALECTS.md` (tracked). Safe column change is a sibling registry, not a `SqlDialect` hook: `planColumnChange` (`modules/utilities/column-change.ts`) finds impact and blockers from the schema Fox already loaded; each family writes steps in -`providers//*.column-change.ts` (`column-change.registry.ts`). Engines not -listed answer "not yet" rather than SQL that would fail there. The Utils panel -computes the plan in the browser (`@foxschema/ui-shared`); Fox runs none of it. -A rename alone is in-place; a type change adds, copies in batches, switches -over, and drops last. PostgreSQL / YugabyteDB / CockroachDB, MySQL / MariaDB / -TiDB, and SQL Server / Azure SQL are in v1. SQLite needs a table rebuild; -Redshift's `ALTER TABLE` cannot express the steps. +`providers//*.column-change.ts` (`column-change.registry.ts`). All 14 SQL +engines are registered. Redis and MongoDB have no column-change entry; omit an +engine to answer "not yet" rather than SQL that would fail there. The Utils +panel (`ColumnChangePanel`) computes the plan in the browser +(`@foxschema/ui-shared`). **Copy** / **Open in editor** send the script; +**Run** and **Execute all** go through `POST /api/sql/execute` (Safe mode +confirms). A rename alone is in-place (`RENAME COLUMN` / `sp_rename`). A type +change is **one ALTER in place** by default (`alterTypeSteps`); tick +**Large table mode** for the add / batched-copy / switch-over path +(`rebuildSteps` + `progressSql`) on PostgreSQL, YugabyteDB, CockroachDB, +MySQL, MariaDB, TiDB, SQL Server and Azure SQL. The other six engines change +the column in place only, and refuse a change they cannot make (SQLite type +change, DuckDB while the table has an index, Redshift except lengthening a +VARCHAR, ClickHouse sorting-key columns). `columnChangeLargeTable()` hides +the checkbox where that path does not exist. Version-aware DDL: `SchemaProvider.detectVersion?()` → stored in Zustand as `targetServerVersion` → flows into `SchemaMapping` → dialect drop hooks use it. Oracle pre-23c @@ -396,6 +404,12 @@ Constraints: fence in text that does reach the database is two `--` lines around SQL, and the SQL is priced like any other (`-- @node\nDROP TABLE t\n-- @end` needs `editor.ddl`). +- **Routine DDL stays one statement.** `CREATE`/`ALTER` `FUNCTION`| + `PROCEDURE`|`TRIGGER` (and an anonymous `BEGIN … END`) keep inner `;` in + one editor cell. Qualification is read from the token stream: a `--` review + note that ends with `.` must not make the next `CREATE` look like + `s.begin` and split the trigger body. A split body is then `terminated` so + Run does not drag the next statement into it. - Adding a dialect: say how it lexes in `sql-readings.ts` (see `packages/sql/src/providers/DIALECTS.md` step 5). Missing there is safe but slow (every reading of every dialect). @@ -406,7 +420,7 @@ Constraints: 2. Register in `provider-settings.ts`, `adapter-registry.ts`, `provider-registry.ts`, `modules/dialect/registry.ts` (and `modules/access/user-sql.registry.ts` / `modules/access/access-sql.registry.ts` when the engine has account or GRANT SQL). Backup/restore commands: `.backup.ts` in `modules/utilities/backup.registry.ts`. A `history` query, where the engine records its backups, lets the panel list them; a server-side SQL backup can be run from the panel, a restore never is. - Safe column change: `.column-change.ts` in `modules/utilities/column-change.registry.ts`. Omit the engine to answer "not yet"; do not guess SQL. SQLite is a table rebuild (later); Redshift is not in v1. + Safe column change: `.column-change.ts` in `modules/utilities/column-change.registry.ts`. Omit the engine to answer "not yet"; do not guess SQL. Implement `alterTypeSteps` (in-place) and, if the engine can, `rebuildSteps` + `progressSql` (large-table mode). A change the engine cannot make belongs on `refuse`, not as SQL that fails there. SQLite still has no type-change path (table rebuild). 3. Add the dialect name to the `Dialect` union **and** the `DIALECTS` array in `packages/sql/src/providers/provider-settings.ts` (`dialect-registry.test.ts` fails until they match `PROVIDER_SETTINGS`). Nothing in `apps/web` needs diff --git a/docs/BACKEND_ARCHITECTURE.md b/docs/BACKEND_ARCHITECTURE.md index e5211ce0..76823af7 100644 --- a/docs/BACKEND_ARCHITECTURE.md +++ b/docs/BACKEND_ARCHITECTURE.md @@ -142,6 +142,14 @@ Every query emits one structured line: `component`, `operation`, `engine`, are the user's data. `target` is `host:port/database` — never the connection string, which carries the password. +**Driver-load errors shown in the UI do not include install paths.** +`packages/db/src/cores/driver-loader.ts` (`driverLoadReason`, +`redactInstallPaths`) strips Node's Require stack and absolute +`node_modules` paths from a missing-driver or ENOENT / dlopen message. The +browser sees the package-relative file (`@clickhouse/client/dist/index.js`) +or the filename, never `/Users/…/foxSchema/node_modules/…`. Action responses +for install / update / remove also omit `cwd` and raw npm output. + **Failures are logged at `debug`, not `error`** — the Fastify error boundary logs the failure once. Logging at each layer turns one failure into four lines. diff --git a/docs/CODE_MAP.md b/docs/CODE_MAP.md index 2a1cd11a..6ff49162 100644 --- a/docs/CODE_MAP.md +++ b/docs/CODE_MAP.md @@ -97,13 +97,13 @@ modules/ One folder per domain, named to match the frontend feature | Folder | What it covers | |---|---| | `dialect` | The `SqlDialect` contract, the registry, type mapping, capability flags | -| `sql-text` | Shared tokenizer (`sql-tokens.ts`), per-dialect readings (`sql-readings.ts`), statement splitting, and SQL templating. Classifiers (`sqlStatementCategories`, …) take the destination dialect and ask every way it can lex the text. Code fences that reach the database are `--` comments. | +| `sql-text` | Shared tokenizer (`sql-tokens.ts`), per-dialect readings (`sql-readings.ts`), statement splitting, and SQL templating. Classifiers (`sqlStatementCategories`, …) take the destination dialect and ask every way it can lex the text. Code fences that reach the database are `--` comments. CREATE / ALTER FUNCTION, PROCEDURE and TRIGGER bodies stay one statement (inner `;` do not split). | | `schema-diff` | Comparing two schemas, and browsing one. `normalizeDefinitionText` is shared with Lokee | | `migrations` | Generating DDL, ordering drops, validating a plan | | `lokee-weave` | Content-addressed schema versioning and revert | | `sql-editor` | FoxScript parsing, code cells, SELECT aliasing, the fail-closed SQL subset Redis and MongoDB run (`sql-subset.ts`) | | `access` | Permission intent, effective access, GRANT/REVOKE and account DDL (facades; emitters live in `providers/`) | -| `utilities` | DBA queries: server insights, index fragmentation (`isSafeIndexFragmentationCustomSql`: one read-only SELECT; Db2 defrag is `providers/db2/db2.index-fragmentation.ts`), backup/restore command builders, SQL Tracer, safe column change (`planColumnChange` in `column-change.ts`; per-family SQL in `providers//*.column-change.ts`, registered in `column-change.registry.ts` — engines not listed answer "not yet") | +| `utilities` | DBA queries: server insights, index fragmentation (`isSafeIndexFragmentationCustomSql`: one read-only SELECT; Db2 defrag is `providers/db2/db2.index-fragmentation.ts`), backup/restore command builders, SQL Tracer, safe column change (`planColumnChange` in `column-change.ts`; per-engine SQL in `providers//*.column-change.ts`, registered in `column-change.registry.ts` — all 14 SQL engines; omit one to answer "not yet"). A type change is `alterTypeSteps` in place by default; large-table mode is `rebuildSteps` + `progressSql` (`columnChangeLargeTable`). | `dialect` and `sql-text` are the foundations: the other folders build on them, never the reverse. @@ -207,8 +207,11 @@ app/ The application shell and settings screens. Home's Services card features/ The view registry: `featureRegistry.ts` (each view's rail button, who may see it, where someone goes who may not, what it adds to the top toolbar, - and its lazy entry). App.tsx, the toolbar, the activity rail - and prefetch all read it. + and its lazy entry). `NAV_FLAGS.workflow` is `false` in 0.3.1: + Workflow is off the rail and the command palette, and a stored + `activeView` for it is sent Home (`redirectFor`). The view and + `/api/workflow` stay; set the flag `true` to put it back. + App.tsx, the toolbar, the activity rail and prefetch all read it. features/ One folder per business domain. The files at its root are its public API; its folders (components/, state/, lib/, api/) are internal. @@ -291,7 +294,7 @@ the page-epoch guard, bookmarks and recents, SQL variables. | Calling an API endpoint | use `api` from `@/shared/api/client` — never `fetch` directly | | SQL for permissions or accounts | `packages/sql/src/modules/access/` (facade) + `packages/sql/src/providers//*.user-sql.ts` / `*.access-sql.ts`. Db2 OS-user docker steps: `buildDb2OsUserInstructions`. | | Backup / restore commands for an engine | `packages/sql/src/providers//*.backup.ts` registered in `modules/utilities/backup.registry.ts`. Saved folder/format defaults: `packages/server/src/features/backup/`. | -| Safe column-change SQL for an engine | `packages/sql/src/providers//*.column-change.ts` registered in `modules/utilities/column-change.registry.ts`. The facade (`planColumnChange`) finds impact, blockers, and lossy type-conversion warnings; omit the engine to answer "not yet" rather than guessed SQL. The panel is `apps/web/.../ColumnChangePanel.tsx`; Run / Execute all go through `POST /api/sql/execute`. | +| Safe column-change SQL for an engine | `packages/sql/src/providers//*.column-change.ts` registered in `modules/utilities/column-change.registry.ts`. The facade (`planColumnChange`) finds impact, blockers, and lossy type-conversion warnings; omit the engine to answer "not yet" rather than guessed SQL. Implement `alterTypeSteps` (in-place default) and optionally `rebuildSteps` + `progressSql` (large-table mode). A change the engine cannot make is `refuse`. The panel is `apps/web/.../ColumnChangePanel.tsx`; Run / Execute all go through `POST /api/sql/execute`. | | Home Services / engine reachability | `packages/server/src/features/system/service-status.ts` (`GET /api/system/services`). Types in `packages/shared/src/service-status.ts`. The card is `apps/web/.../app/shell/ServiceStatusCard.tsx`. | | Home Drivers inventory | `packages/server/src/features/system/drivers.service.ts` (`GET /api/system/drivers`, install/update/remove). Types in `packages/shared/src/driver-inventory.ts`. The card is `apps/web/.../app/shell/DriverInventoryCard.tsx`. | | A dialect capability the app must branch on | `packages/sql/src/modules/capabilities/` (`dialect-features.ts`) and `packages/sql/src/modules/dialect/` | diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 32975cef..4694d2fb 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -168,7 +168,9 @@ docker compose -f docker-compose.app.yml --profile workflow up -d --build `WORKFLOW_ENGINE_ENDPOINT` is the unsaved default only. An endpoint saved in Workflow settings still wins. Leave the profile off and the app runs as Fox Schema alone. Home → **Services** (`GET /api/system/services`) shows whether -this API can reach the engine; it does not start it. +this API can reach the engine; it does not start it. In **0.3.1** the +Workflow rail item is held back (`NAV_FLAGS.workflow`); the engine and this +proxy still run. | Variable | Set on | Default | Purpose | |----------|--------|---------|---------| diff --git a/docs/FEATURE_MAP.md b/docs/FEATURE_MAP.md index 07983587..296f992d 100644 --- a/docs/FEATURE_MAP.md +++ b/docs/FEATURE_MAP.md @@ -12,11 +12,11 @@ an interactive page (open it in a browser), and [the test and fix plan](plans/2026-10-25-feature-and-dialect-test-plan.md) sets the order every feature and dialect is tested in. -Measured on `main` at `66575674` (2026-11-02): 35 features, 18 server -feature modules, 159 HTTP routes (pinned by +Measured on `main` at `d7ae1cf2` (2026-11-09): 35 features, 18 server +feature modules, 163 HTTP routes (pinned by `packages/server/src/api/http-contract.test.ts`), 14 frontend features, 16 database engines, 28 metadata tables (31 migrations), 58 e2e specs in 35 -groups, ~550 unit-test files. Re-measure with the commands in +groups, ~532 unit-test files. Re-measure with the commands in [Keeping this page true](#keeping-this-page-true). - [System](#system) @@ -149,7 +149,7 @@ in its folders and its e2e specs (`apps/e2e/src/tests`). | `platform.workspaces` | Workspaces — rail button and menu, create, switch, archive, restore, settings, members and workspace roles (a personal workspace lends no more than its account holds), invites (accept / decline, 14 days), public discover / join, shared workspace scripts | Rail button under the Fox mark | `/api/workspaces` (20) | server `workspaces`, `platform/workspaces`; web `workspaces` | `workspace.members`, `workspace.settings`, `workspace.create` | `workspaces`, `workspace_members`, `workspace_invites`, `workspace_scripts` | server 5, web 3; e2e `workspaces`, `workspaces-sharing-limits` | | `platform.connections` | Connections & drivers — credential manager, connection form (password / Windows / LDAP / No password on Redis and MongoDB, SSL, pool, SQLite / DuckDB file picker), schema list, connection test, driver check and Install driver, Home Drivers inventory (list / version check / update / remove), cloud provider credentials | Creds (rail), connection modal, Home | `/api/connections` (4), `/api/connection/test`, `/api/db/test`, `/api/driver` (2), `/api/system/drivers` (4) | server `connections`, `system` (`drivers.service.ts`), `platform/connections`, `internal/driver-install`; web `connections`, `app/shell/DriverInventoryCard` | per workspace role; `admin.drivers` to install, update, or remove | `connections`, `cloud_provider_credentials` | server 1, web 1 | | `platform.secrets` | App secrets — vault entries, cloud secret providers, resolving a secret | Creds → secrets, editor Secrets panel | `/api/app-secrets` (9) | server `admin` (app-secrets mount) | `secrets.view/create/edit/delete` | `app_secrets`, `cloud_provider_credentials` | — | -| `platform.system` | Shell, health, updates — activity rail, top toolbar, command palette, home, **Services** card (API / Web UI / Workflow engine status; no start/stop yet), **Drivers** card (installed packages, version check, install / update / remove), backend-offline banner, activity, app info, config, update check / apply, email and update settings, send feedback | App shell, Prefs, Home | `/api/health`, `/api/config`, `/api/app-info`, `/api/activity`, `/api/updates` (2), `/api/system/services`, `/api/system/drivers` (4) | server `system` (`service-status.ts`, `drivers.service.ts`); web `app/shell` (`ServiceStatusCard`, `DriverInventoryCard`), `app/settings`, `feedback` | `admin.drivers` to change drivers | `app_settings` | server 2, web 3; e2e `smoke`, `lazy-loading`, `send-feedback`, `auto-error-catch` | +| `platform.system` | Shell, health, updates — activity rail, top toolbar, command palette, home, **Services** card (API / Web UI / Workflow engine status; no start/stop yet), **Drivers** card (installed packages, version check, install / update / remove), backend-offline banner, activity, app info, config, update check / apply, email and update settings, send feedback. **Workflow** is off the rail and the command palette in 0.3.1 (`NAV_FLAGS.workflow` in `featureRegistry.ts`; a stored last-view is sent Home) | App shell, Prefs, Home | `/api/health`, `/api/config`, `/api/app-info`, `/api/activity`, `/api/updates` (2), `/api/system/services`, `/api/system/drivers` (4) | server `system` (`service-status.ts`, `drivers.service.ts`); web `app/shell` (`ServiceStatusCard`, `DriverInventoryCard`), `app/features/featureRegistry.ts`, `app/settings`, `feedback` | `admin.drivers` to change drivers | `app_settings` | server 2, web 3; e2e `smoke`, `lazy-loading`, `send-feedback`, `auto-error-catch` | ### Schema @@ -193,7 +193,7 @@ in its folders and its e2e specs (`apps/e2e/src/tests`). | ID | Feature and sub-features | UI | API | Code | Perms | Tables | Tests | |---|---|---|---|---|---|---|---| -| `workflow.designer` | Designer — workflows list, canvas (pipes and edges), palette, inspector, triggers, SQL / script / HTTP / delimited editors, dependencies | Workflow (rail) | engine proxy under `/api/workflow` | web `workflow`; `packages/workflow-engine/src/definitions.ts` | `workflow.access`, `workflow.design` | engine's own store | web 12; e2e `workflow-designer-roundtrip` | +| `workflow.designer` | Designer — workflows list, canvas (pipes and edges), palette, inspector, triggers, SQL / script / HTTP / delimited editors, dependencies. Off the rail in 0.3.1 (`NAV_FLAGS.workflow`); the view and proxy stay | Workflow (rail, held back) | engine proxy under `/api/workflow` | web `workflow`; `packages/workflow-engine/src/definitions.ts` | `workflow.access`, `workflow.design` | engine's own store | web 12; e2e `workflow-designer-roundtrip` | | `workflow.pipes` | Pipes — db (SQL on every dialect, Postgres / MySQL sources), http (request, multi-http, transform, sink, cookies), notify (email, SMS, SMTP), control (loop, split, merge, goto, human gate, sub-workflow), utility (CSV, delimited, JSON, script, text, transforms, verify, files), trigger | Designer palette | (engine) | `packages/workflow-engine/src/pipes/*` | `workflow.design` | — | workflow-engine 87 | | `workflow.runs` | Runs — execution panel, cancel a live run, answer a human gate, variables, credentials | Workflow | engine proxy | web `RunsPanel`, `ExecutionPanel`, `VariablesPanel`, `CredentialsPanel` | `workflow.run` | engine's own store | (workflow tests) | | `workflow.engine` | Engine control plane — settings and health, saved-connection grants, internal resolve / engine-config routes (token-guarded) | Workflow → Engine | `/api/workflow` (5), `/api/workflow-internal` (2) | server `workflow`; `apps/workflow-server`; `packages/workflow-contract` | `workflow.admin` | `workflow_settings`, `workflow_connection_grants` | server 6, workflow-server 19; e2e `workflow-smoke` | @@ -560,15 +560,15 @@ that value matches the ObjectId a SELECT showed. | Layer | Where | Count | |---|---|---| -| Unit — dialect knowledge | `packages/sql` | 86 files (access 13, lokee-weave 15, sql-text 9, migrations 7, utilities 7, sql-editor 7, schema-diff 5, …) | -| Unit — drivers | `packages/db` | 26 (live `*.it.test.ts` behind `FOX_IT_DB=1`) | -| Unit — server | `packages/server` | 104 | -| Unit — web | `apps/web` | 179 (sql-editor 59, lokee-weave 14, access 12, workflow 12, …) | +| Unit — dialect knowledge | `packages/sql` | 89 files (access 13, lokee-weave 15, sql-text 9, migrations 7, utilities 9, sql-editor 7, schema-diff 5, …) | +| Unit — drivers | `packages/db` | 28 (live `*.it.test.ts` behind `FOX_IT_DB=1`) | +| Unit — server | `packages/server` | 111 | +| Unit — web | `apps/web` | 189 (sql-editor 65, lokee-weave 14, access 13, workflow 12, …) | | Unit — workflow | `packages/workflow-engine`, `apps/workflow-server` | 87 + 19 | -| Unit — CLI | `apps/cli` | 14 | +| Unit — CLI | `apps/cli` | 25 | | E2E (Playwright) | `apps/e2e/src/tests` | 58 spec files (`npm run test:e2e:all`) | -Known gaps and open work (2026-11-02, after #575 – #589) — each belongs on the planning board: +Known gaps and open work (2026-11-09, after 0.3.1) — each belongs on the planning board: | Feature | Gap | Kind | |---|---|---| @@ -579,6 +579,7 @@ Known gaps and open work (2026-11-02, after #575 – #589) — each belongs on t | `platform.workspaces` | Workspace ceiling caps by rank; a custom role mapping is not intersected with the account's permissions | improvement | | `platform.connections` | A requirepass-only Redis saved without its password is not prompted (the **No password** login method is shipped) | improvement | | `platform.system` | Home → Services is status only; start/stop the Workflow engine (`admin.services`) is later | feature | +| `workflow.designer` | Rail item and command-palette entry held back in 0.3.1 (`NAV_FLAGS.workflow === false`); turn the flag on to restore | feature | | `utilities.columns` | SQLite type change needs a table rebuild; DuckDB tables with an index are refused (no index columns in the model); Redshift only lengthens a VARCHAR; primary-key / inbound-FK / identity columns refused; large-table mode only on the 8 engines that have it | feature | ## Planning work diff --git a/docs/README.md b/docs/README.md index 634a71c1..1fefd1fa 100644 --- a/docs/README.md +++ b/docs/README.md @@ -74,7 +74,7 @@ Finished or in-progress design notes. Do not treat them as the map of the code. | Feature review test plan | [plans/2026-10-06-review-features-test-plan.md](plans/2026-10-06-review-features-test-plan.md) | | Feature and dialect test plan, and its first results | [plans/2026-10-25-feature-and-dialect-test-plan.md](plans/2026-10-25-feature-and-dialect-test-plan.md), [plans/2026-10-26-feature-and-dialect-test-results.md](plans/2026-10-26-feature-and-dialect-test-results.md) | | Service status on Home (status shipped; start/stop later) | [plans/2026-11-01-service-status.md](plans/2026-11-01-service-status.md) | -| Safe column change: plan and SQL for review (planner + Utils screen shipped) | [plans/2026-11-01-safe-column-change.md](plans/2026-11-01-safe-column-change.md) | +| Safe column change: plan and SQL for review (planner, Utils screen, in-place default, all 14 SQL engines) | [plans/2026-11-01-safe-column-change.md](plans/2026-11-01-safe-column-change.md) | ## Releases diff --git a/docs/feature-map.html b/docs/feature-map.html index 03e2f52f..932770fe 100644 --- a/docs/feature-map.html +++ b/docs/feature-map.html @@ -666,7 +666,7 @@

End-to-end suites