From daee6b3e5ac7a3511cba7eeb77d84c2c17764610 Mon Sep 17 00:00:00 2001 From: huyplb Date: Mon, 26 Oct 2026 15:56:20 -0600 Subject: [PATCH] docs(plans): results of the first feature and dialect test pass A snapshot of the planning board at the end of the 2026-10-25/26 round: every engine's result (all green except Oracle and MongoDB, both blocked by the environment), each feature's result and the manual walk still left, the PRs that fixed what the round found, and what is still open. Linked from the test plan and docs/README.md. Co-Authored-By: Claude Opus 5.5 --- docs/README.md | 1 + ...026-10-25-feature-and-dialect-test-plan.md | 2 + ...-10-26-feature-and-dialect-test-results.md | 103 ++++++++++++++++++ 3 files changed, 106 insertions(+) create mode 100644 docs/plans/2026-10-26-feature-and-dialect-test-results.md diff --git a/docs/README.md b/docs/README.md index c13b46be1..218ab5ee8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -71,6 +71,7 @@ Finished or in-progress design notes. Do not treat them as the map of the code. | Server beam | [plans/server-beam.md](plans/server-beam.md) | | Excel and Sheets import | [plans/excel-and-sheets-import.md](plans/excel-and-sheets-import.md) | | Feature review test plan | [plans/2026-10-06-review-features-test-plan.md](plans/2026-10-06-review-features-test-plan.md) | +| Feature and dialect test plan, and its first results | [plans/2026-10-25-feature-and-dialect-test-plan.md](plans/2026-10-25-feature-and-dialect-test-plan.md), [plans/2026-10-26-feature-and-dialect-test-results.md](plans/2026-10-26-feature-and-dialect-test-results.md) | ## Releases diff --git a/docs/plans/2026-10-25-feature-and-dialect-test-plan.md b/docs/plans/2026-10-25-feature-and-dialect-test-plan.md index badf6efa3..ecbaa0442 100644 --- a/docs/plans/2026-10-25-feature-and-dialect-test-plan.md +++ b/docs/plans/2026-10-25-feature-and-dialect-test-plan.md @@ -1,5 +1,7 @@ # Test and fix plan: every feature, every dialect (2026-10-25) +Results of the first round: [2026-10-26-feature-and-dialect-test-results.md](2026-10-26-feature-and-dialect-test-results.md). + Every feature in [FEATURE_MAP.md](../FEATURE_MAP.md) and every one of the 16 engines gets one **test-and-fix pass**, tracked as a work item on the planning board (kind *test*, one item per feature ID and per `dialect.*` ID). Bugs a pass diff --git a/docs/plans/2026-10-26-feature-and-dialect-test-results.md b/docs/plans/2026-10-26-feature-and-dialect-test-results.md new file mode 100644 index 000000000..d4199dba6 --- /dev/null +++ b/docs/plans/2026-10-26-feature-and-dialect-test-results.md @@ -0,0 +1,103 @@ +# Test and fix pass: results (2026-10-25 to 2026-10-26) + +The first round of the [test and fix plan](2026-10-25-feature-and-dialect-test-plan.md). +The planning board holds the live state, one card per feature and per engine. +This page is the snapshot at the end of the round, so it can be read without the +board. + +**Short version:** the automated suites are green on every engine except Oracle +and MongoDB, and both of those are blocked by the environment, not by Fox. No +feature has had its manual walk yet. Fixes found during the round shipped with +A/B tests in #546 and #550 to #564. + +## Engines + +All of them ran their `dialects/` e2e suite (11 tests each), unless the +note says otherwise. + +| Engine | Result | Notes | +|---|---|---| +| PostgreSQL | Pass | Plus compare-flow, execute-gate, schema-browse, history, revert, revert edges, compare-multi-target and the access suites | +| CockroachDB | Pass (10-26) | Utilities 4/4, database access 9/9, access assistant 4/4. The backup suite has no CockroachDB case | +| YugabyteDB | Pass (10-26) | Utilities 4/4, database access 9/9, access assistant 4/4 | +| Redshift | Pass | Against a local Postgres stand-in. Utilities 4/4 | +| MySQL | Pass | Access, accounts, backup and utilities suites as well | +| MariaDB | Pass | Utilities 4/4 | +| TiDB | Pass (10-26) | Utilities 4/4 including clone, database access 9/9, access assistant 4/4 | +| SQL Server | Pass | Access (with DENY), accounts, backup and utilities suites as well | +| Azure SQL | Pass | Against the SQL Server container | +| Db2 | Pass (10-26) | Utilities 4/4, database access 9/9, access assistant 4/4, after clearing the leftover `newrole` role ([runbook](../testing/RUNNING_E2E.md#failures-that-are-not-bugs)) | +| SQLite | Pass | Most feature suites run on SQLite too | +| DuckDB | Pass | | +| ClickHouse | Pass | Utilities 4/4 | +| Redis | Pass | redis-editor e2e 21/21, live `redis.it.test` 19/19 | +| Oracle | Blocked | The re-pulled image is a newer build than the one that created the data volume, so DDL fails with `ORA-01804`. Needs the volume recreated and reseeded, or the older image pinned | +| MongoDB | Blocked | The container runs. The live tests need the optional `mongodb` driver, which is not installed. There is no MongoDB e2e spec | + +The manual walk is still to do on every engine. + +## Features + +"Partial" means the automated suites passed and the manual walk is left. + +| Feature | Result | Manual walk left | +|---|---|---| +| `editor.run` | Partial | Shortcuts, Format SQL+JS, completions, column picker | +| `editor.datacompare` | Partial | Sync column, backup snapshot on apply, Preview SQL | +| `schema.compare` | Partial | Diff briefing, cross-dialect readiness, streamed progress | +| `schema.snapshots` | Partial | Graph, change chart, rename a version | +| `access.builder` | Partial | Object picker, stop at the first failure | +| `access.accounts` | Partial | By hand on each engine with accounts | +| `platform.auth` | Partial | SSO with real Google, Microsoft and GitHub apps; launch from `foxschema open`; email verification with real mail | +| `platform.workspaces` | Partial | Declining an invite, 14-day expiry, shared scripts | +| `platform.connections` | Partial | Windows and LDAP forms, SSL and pool options, cloud credentials, Install driver | +| `editor.code` | Partial | TypeScript cells, imports, Server Beam | +| `editor.results` | Partial | Column reorder, fullscreen | +| `editor.peek` | Pass | Date picker, generators | +| `editor.explorer` | Partial | Object source script, Install driver banner | +| `editor.library` | Partial | Bookmarks, samples, variables, secrets panels | +| `editor.files` | Partial | No e2e yet. CSV and Excel import into SQLite and a saved connection | +| `schema.applies` | Partial | The Applies list and rerun | +| `schema.git` | Pass | | +| `utilities.backup` | Pass | | +| `utilities.clone`, `.indexes`, `.insights`, `.dbaccess` | Partial | Green on every engine that ran | +| `utilities.trace` | New (#564) | SQL Tracer: e2e 2/2 (SQLite Fox activity, Postgres live) | +| `platform.admin` | Partial | Handing over admin, editing the role matrix | +| `platform.secrets` | Partial | No e2e yet. Vault entries, cloud providers, resolve | +| `platform.system` | Pass | | +| `access.inspector`, `access.report` | Partial | Report export. Oracle's FOX_READER case waits on Oracle | +| `workflow.designer` | Pass | | +| `workflow.engine`, `.pipes`, `.runs` | Partial | Engine settings and sinks; run history and retry | +| `cli` | Partial | `foxschema open`, shortcut, TUI | + +## Fixed during the round + +| PR | What | +|---|---| +| #546 | Run rail and Install driver follow-ups (15 review findings) | +| #550 | P0: sharing a personal workspace could lend more than the sharer holds. Restore archived workspaces | +| #552 | Redis keys as tables. A short `SELECT …;` was rewritten into broken SQL. Data peek WHERE on Redis and MongoDB | +| #553 | Data peek grid hidden by its filter rows | +| #554 | Demo role and demo mode | +| #555 | Emailed invite and reset links landed on sign-in | +| #557 | Redis and MongoDB offer No password as a login method | +| #558 | Force-migrate e2e | +| #559, #560 | Data Compare at 2,000 rows, one shared 2,000-op migrate cap | +| #561 | Read-only is a permission (`app.write`) | +| #564 | SQL Tracer | + +## Still open + +- **Oracle and MongoDB passes:** blocked as above. +- **Oracle FOX_READER:** the e2e account needs a dictionary grant in the seed (P3). +- **Code-cell timeout test:** flaky under load, because the 4 s budget includes + the worker's cold start (P3). +- **Opening a sample bookmark** unchecks every Destination (P3). +- **The workflow SQL step** warns about a missing password on SQLite and on Redis + without a password (P3). +- **Four e2e specs** still define their own `api()` helper instead of using + `helpers/accounts.ts` (P3). +- **Every manual walk** above. Four of them have their own cards on the board: + run rail (scheduled), compare and deploy, Redis in the editor, workspaces. +- **Later phases:** Data Compare across whole tables by key range; engine-native + profilers for the SQL Tracer.