Skip to content

ACE-082: prove the vetted statement is one the engine accepts — MySQL, SQL Server, DuckDB - #162

Open
sandeep-agami wants to merge 4 commits into
agami-governance-branchfrom
ACE-082-dialect-execution-matrix
Open

ACE-082: prove the vetted statement is one the engine accepts — MySQL, SQL Server, DuckDB#162
sandeep-agami wants to merge 4 commits into
agami-governance-branchfrom
ACE-082-dialect-execution-matrix

Conversation

@sandeep-agami

Copy link
Copy Markdown
Collaborator

Spec: ACE-082

Summary

ACE-079 made the guard read every statement in the datasource's own dialect, and proved the verdict on all eleven supported engines. But that proof runs a spy executor against no database and re-parses the vetted statement with sqlglot — it is sqlglot asserting about sqlglot. The other half of REQ-014 fidelity, the statement the guard vetted is a statement the engine actually accepts, was proven only on SQLite and PostgreSQL — both "-quoting, i.e. precisely the family where ACE-079's defect class cannot manifest. The fix was verified only where the bug could not occur.

This runs the same corpus, through the same tool, against real MySQL 8, real SQL Server 2022, and in-process DuckDB. With SQLite and Postgres that is five engines executing, covering all three identifier-quoting families — backtick, bracket, double-quote — which is the axis the defect turns on. Each of the six that remain verdict-only shares a quoting family with one that executes.

The divergence this exists to catch, reproduced on the pre-ACE-079 revision (8d63ccb):

SELECT TOP 5 [id], [amount] FROM [orders]
  BEFORE (generic parse, errors ignored) -> tables=[]        cols=['TOP']
  AFTER  (tsql, the declared engine)     -> tables=['orders'] cols=['amount','id']

No runtime source changed. git diff origin/agami-governance-branch -- packages/agami-core/src/ is empty, and two engines were dropped rather than break that rule (below).

Changes

The seam — the spec assumed a per-engine fixture seam existed; it did not. SQLite was seeded by harness.seed_sqlite, Postgres by a separate inline seeder in conftest.db_safety_env, each with its own type map and placeholder style. Both now go through one EngineFixture + seed_schema, so a DDL-spelling difference lives in a table and adding an engine is a row, not a second seeder that can drift.

New cases, pinned via the existing Case.engines:

  • a backtick-quoted governed query is allowed and executes on MySQL, while a backtick-quoted undeclared table is refused;
  • SELECT TOP n [col] FROM [tbl] is scoped correctly on SQL Server;
  • on MySQL a double-quoted statement is unscopable_sql, because there " is a string literal. The generic reading of that same text is a perfectly scopable query over orders — so this is the positive proof that reading in the engine's own grammar changes the verdict, not merely the parse. The two "-quoting cases that ran everywhere are now pinned to the engines where " actually quotes an identifier.

CI — three separate jobs, no needs: on integration-pg or on each other, so one flaky container cannot move the F9 done-bar. integration-pg's -k "db_path or role" selector is unchanged and matches zero new tests — now asserted by a test rather than assumed. No secrets, so they run on fork PRs.

Two pre-existing tests fixed (both latent, both surfaced by this work, neither weakened — each now checks strictly more):

  • test_no_false_refusal_on_valid_sql.py parsed every valid case under a hardcoded sqlite dialect despite its docstring claiming per-engine parsing. Now measures (case, engine) pairs: 15 → 23 checks, with all 15 originals retained.
  • test_resolve_guard_model.py loaded harness.py via spec_from_file_location without registering it in sys.modules, so adding any class that reads module globals at creation time broke it at import. Fixed to the documented idiom.

Decisions

  • CI does not depend on cloud credentials. Tier 3b (BigQuery dry-run, Snowflake EXPLAIN) is deferred. No credentials exist; the marginal evidence is low (Snowflake and Redshift are "-quoting, the family Postgres already covers); a public repo withholds secrets from fork PRs, making such a job structurally unrunnable for outside contributors; and it is a standing maintenance tax. If ever wanted, the right shape is a non-required workflow_dispatch/schedule job. Residual risk: real BigQuery and Snowflake grammar stays unproven by execution.
  • BigQuery emulator and local Spark were dropped as unreachable, not skipped by preference. _run_bigquery builds bigquery.Client(project=…) with no client_options/api_endpoint, so it cannot be pointed at an emulator; _run_databricks speaks to a workspace, not a local SparkSession. Reaching either needs a runtime source change this spec forbids, and going through a stub would assert exactly what ACE-079 already asserts. (The emulator is also SQLite-backed, so it would have proven parse/accept, not BigQuery semantics.)
  • One job per engine. Free (parallel, no needs:) and buys failure isolation: SQL Server is the heavy, slow-booting container and should not be able to mask the backtick evidence.
  • "Unavailable" includes a missing driver. Review reproduced a hole where importorskip ran before the AGAMI_IT_<ENGINE>_REQUIRED check, so a job whose driver failed to install skipped everything and exited 0 — green, proving nothing, in the job that owns that engine's only evidence. Closed and verified in both directions.

Checklist

  • dev.py check green — 2291 passed, 359 skipped, 2 xfailed
  • Zero files changed under packages/agami-core/src/
  • Live runs: MySQL 92 · SQL Server 90 · DuckDB 82 · Postgres (db_path + role floor) 88
  • Skip-vs-fail verified both directions, for an unreachable server and a missing driver
  • integration-pg selector unchanged; independence asserted by a test
  • Per-engine coverage tier stated in one place, including what is not proven
  • No test weakened or deleted; the two edited tests check strictly more
  • gitleaks clean; no customer names, internal hosts, or real credentials (CI literals are labelled non-secret sentinels for ephemeral localhost services)

sandeep-agami and others added 4 commits July 28, 2026 16:18
ACE-079 proved the guard's verdict on all eleven engines, but with a spy executor
and no database — the vetted statement was re-parsed by sqlglot, never accepted by
an engine. That leaves the fidelity half of REQ-014 proven only on SQLite and
Postgres, both `"`-quoting, i.e. the family where the defect class cannot manifest.

Adds a per-engine physical-fixture seam (`harness.EngineFixture` + `seed_schema`)
and refactors BOTH existing seeders onto it — SQLite's `seed_sqlite` and the
Postgres seeder that was inline in `conftest.db_safety_env` — so a DDL-spelling
difference lives in one table rather than a second hand-written seeder that can
drift. Adding an engine is now a row in `ENGINE_FIXTURES`.

Executes the corpus against MySQL 8 (backtick — its native quote), SQL Server 2022
(the only bracket-quoting engine) and DuckDB in-process. Together with Postgres and
SQLite that covers all three identifier-quoting families.

New cases pin the behaviour the dialect fix exists for:
  - a backtick-quoted governed query is allowed AND executes on MySQL;
  - `SELECT TOP n [col] FROM [tbl]` is scoped correctly on SQL Server — the exact
    shape a generic parse reduces to no tables with `TOP` mistaken for a column;
  - on MySQL a double-quoted statement is `unscopable_sql`, because there `"` is a
    string literal. The generic reading of that same text is a scopable query over
    `orders`, so this is the positive proof that reading in the engine's own grammar
    changes the verdict and not merely the parse.

The two `"`-quoting cases that ran everywhere are pinned to the engines where `"`
actually quotes an identifier, with MySQL's counterpart asserted separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three separate jobs with no `needs:` on integration-pg and none between themselves,
so one flaky engine container cannot turn the Postgres role-floor result — the F9
done-bar — green or red. integration-pg's selector is unchanged and matches none of
the new tests.

Each job sets AGAMI_IT_<ENGINE>_REQUIRED so an unreachable server FAILS instead of
skipping: pytest exits 0 when everything skips, and a silently-skipped job would
claim per-engine coverage while proving nothing. Verified both directions — with no
connection settings the engine tests skip cleanly, and with REQUIRED set against a
dead port they fail loudly.

No secrets are involved, so these run on fork PRs too. SQL Server accepts TCP before
it can serve queries, so readiness is an explicit query poll rather than a port check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d on

Two latent problems the new engines surfaced.

test_no_false_refusal_on_valid_sql parsed EVERY valid case under a hardcoded
sqlite dialect, though its own docstring says each statement is parsed "under its
declared engine". That held only while every case was SQLite-valid. `SELECT TOP n
[col] FROM [tbl]` is valid T-SQL and is not SQLite at all, so the corpus could not
carry a statement for another engine without a false failure. It now measures
(case, engine) pairs, checking a pinned case under each engine it is declared on
and only an unpinned case under the default — which is what the docstring already
claimed and what keeps the measurement honest as engines are added.

test_resolve_guard_model loaded tests/e2e/harness.py via spec_from_file_location
without registering it in sys.modules. A module executed detached that way cannot
resolve its own module globals, so adding any class that reads them at creation
time — @DataClass here — broke an unrelated test at import. Registering before
exec_module is the documented idiom and removes the trap rather than avoiding it.

Neither test is weakened: both now check strictly more than before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review found the AGAMI_IT_<ENGINE>_REQUIRED contract only half-closed, and
reproduced it: `pytest.importorskip("pymysql")` ran BEFORE the REQUIRED check, so
a missing driver skipped every test and pytest exited 0 — in the very job that
owns that engine's evidence. `integration-duckdb` was worse: no REQUIRED flag
existed at all, and these drivers are supplied only by `--with` flags on the CI
uvx line, so dropping one is the likeliest way to empty a job. Three comments in
the diff asserted this could not happen.

`_require_driver` now consults `_unavailable` first, and the DuckDB job sets
REQUIRED. Verified: missing driver + REQUIRED now exits 1; without REQUIRED it
still skips cleanly at exit 0.

Also from review:
- MySQL's `--health-cmd "mysqladmin ping"` could pass against the entrypoint's
  temporary init server, which accepts on the unix socket before the real server
  binds 3306 — a real flake, the same reason SQL Server already got an explicit
  poll. Probing `-h 127.0.0.1` stays refused until the port is really listening.
- Engine connections were opened outside the `try` that closes them, so a failure
  while seeding leaked one connection per test and the tail reported "too many
  connections" instead of the real DDL error.
- `assert_outcome`'s "ok" arm never checked that a result set came back, so a
  fixture seeded into the wrong database would still pass — which would have
  hollowed out exactly the cases named `*-executes`.
- The claim that integration-pg's `-k "db_path or role"` selects no engine test is
  now asserted rather than assumed; a future case note containing "role" would
  otherwise silently couple the F9 done-bar to an engine container.
- `EngineFixture.storage_type` / `.credential_type` were declared but never read,
  so the seam permitted the drift its docstring said it prevented. Both are now
  read at the call sites.
- SERIAL-ONLY note extended to the new fixtures; type hints on the new helpers;
  the coverage table no longer implies an executing engine is proven beyond the
  corpus's own cases.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant