ACE-082: prove the vetted statement is one the engine accepts — MySQL, SQL Server, DuckDB - #162
Open
sandeep-agami wants to merge 4 commits into
Open
Conversation
ACE-079 proved the guard's verdict on all eleven engines, but with a spy executor
and no database — the vetted statement was re-parsed by sqlglot, never accepted by
an engine. That leaves the fidelity half of REQ-014 proven only on SQLite and
Postgres, both `"`-quoting, i.e. the family where the defect class cannot manifest.
Adds a per-engine physical-fixture seam (`harness.EngineFixture` + `seed_schema`)
and refactors BOTH existing seeders onto it — SQLite's `seed_sqlite` and the
Postgres seeder that was inline in `conftest.db_safety_env` — so a DDL-spelling
difference lives in one table rather than a second hand-written seeder that can
drift. Adding an engine is now a row in `ENGINE_FIXTURES`.
Executes the corpus against MySQL 8 (backtick — its native quote), SQL Server 2022
(the only bracket-quoting engine) and DuckDB in-process. Together with Postgres and
SQLite that covers all three identifier-quoting families.
New cases pin the behaviour the dialect fix exists for:
- a backtick-quoted governed query is allowed AND executes on MySQL;
- `SELECT TOP n [col] FROM [tbl]` is scoped correctly on SQL Server — the exact
shape a generic parse reduces to no tables with `TOP` mistaken for a column;
- on MySQL a double-quoted statement is `unscopable_sql`, because there `"` is a
string literal. The generic reading of that same text is a scopable query over
`orders`, so this is the positive proof that reading in the engine's own grammar
changes the verdict and not merely the parse.
The two `"`-quoting cases that ran everywhere are pinned to the engines where `"`
actually quotes an identifier, with MySQL's counterpart asserted separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three separate jobs with no `needs:` on integration-pg and none between themselves, so one flaky engine container cannot turn the Postgres role-floor result — the F9 done-bar — green or red. integration-pg's selector is unchanged and matches none of the new tests. Each job sets AGAMI_IT_<ENGINE>_REQUIRED so an unreachable server FAILS instead of skipping: pytest exits 0 when everything skips, and a silently-skipped job would claim per-engine coverage while proving nothing. Verified both directions — with no connection settings the engine tests skip cleanly, and with REQUIRED set against a dead port they fail loudly. No secrets are involved, so these run on fork PRs too. SQL Server accepts TCP before it can serve queries, so readiness is an explicit query poll rather than a port check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d on Two latent problems the new engines surfaced. test_no_false_refusal_on_valid_sql parsed EVERY valid case under a hardcoded sqlite dialect, though its own docstring says each statement is parsed "under its declared engine". That held only while every case was SQLite-valid. `SELECT TOP n [col] FROM [tbl]` is valid T-SQL and is not SQLite at all, so the corpus could not carry a statement for another engine without a false failure. It now measures (case, engine) pairs, checking a pinned case under each engine it is declared on and only an unpinned case under the default — which is what the docstring already claimed and what keeps the measurement honest as engines are added. test_resolve_guard_model loaded tests/e2e/harness.py via spec_from_file_location without registering it in sys.modules. A module executed detached that way cannot resolve its own module globals, so adding any class that reads them at creation time — @DataClass here — broke an unrelated test at import. Registering before exec_module is the documented idiom and removes the trap rather than avoiding it. Neither test is weakened: both now check strictly more than before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review found the AGAMI_IT_<ENGINE>_REQUIRED contract only half-closed, and
reproduced it: `pytest.importorskip("pymysql")` ran BEFORE the REQUIRED check, so
a missing driver skipped every test and pytest exited 0 — in the very job that
owns that engine's evidence. `integration-duckdb` was worse: no REQUIRED flag
existed at all, and these drivers are supplied only by `--with` flags on the CI
uvx line, so dropping one is the likeliest way to empty a job. Three comments in
the diff asserted this could not happen.
`_require_driver` now consults `_unavailable` first, and the DuckDB job sets
REQUIRED. Verified: missing driver + REQUIRED now exits 1; without REQUIRED it
still skips cleanly at exit 0.
Also from review:
- MySQL's `--health-cmd "mysqladmin ping"` could pass against the entrypoint's
temporary init server, which accepts on the unix socket before the real server
binds 3306 — a real flake, the same reason SQL Server already got an explicit
poll. Probing `-h 127.0.0.1` stays refused until the port is really listening.
- Engine connections were opened outside the `try` that closes them, so a failure
while seeding leaked one connection per test and the tail reported "too many
connections" instead of the real DDL error.
- `assert_outcome`'s "ok" arm never checked that a result set came back, so a
fixture seeded into the wrong database would still pass — which would have
hollowed out exactly the cases named `*-executes`.
- The claim that integration-pg's `-k "db_path or role"` selects no engine test is
now asserted rather than assumed; a future case note containing "role" would
otherwise silently couple the F9 done-bar to an engine container.
- `EngineFixture.storage_type` / `.credential_type` were declared but never read,
so the seam permitted the drift its docstring said it prevented. Both are now
read at the call sites.
- SERIAL-ONLY note extended to the new fixtures; type hints on the new helpers;
the coverage table no longer implies an executing engine is proven beyond the
corpus's own cases.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Spec: ACE-082
Summary
ACE-079 made the guard read every statement in the datasource's own dialect, and proved the verdict on all eleven supported engines. But that proof runs a spy executor against no database and re-parses the vetted statement with sqlglot — it is sqlglot asserting about sqlglot. The other half of REQ-014 fidelity, the statement the guard vetted is a statement the engine actually accepts, was proven only on SQLite and PostgreSQL — both
"-quoting, i.e. precisely the family where ACE-079's defect class cannot manifest. The fix was verified only where the bug could not occur.This runs the same corpus, through the same tool, against real MySQL 8, real SQL Server 2022, and in-process DuckDB. With SQLite and Postgres that is five engines executing, covering all three identifier-quoting families — backtick, bracket, double-quote — which is the axis the defect turns on. Each of the six that remain verdict-only shares a quoting family with one that executes.
The divergence this exists to catch, reproduced on the pre-ACE-079 revision (
8d63ccb):No runtime source changed.
git diff origin/agami-governance-branch -- packages/agami-core/src/is empty, and two engines were dropped rather than break that rule (below).Changes
The seam — the spec assumed a per-engine fixture seam existed; it did not. SQLite was seeded by
harness.seed_sqlite, Postgres by a separate inline seeder inconftest.db_safety_env, each with its own type map and placeholder style. Both now go through oneEngineFixture+seed_schema, so a DDL-spelling difference lives in a table and adding an engine is a row, not a second seeder that can drift.New cases, pinned via the existing
Case.engines:SELECT TOP n [col] FROM [tbl]is scoped correctly on SQL Server;unscopable_sql, because there"is a string literal. The generic reading of that same text is a perfectly scopable query overorders— so this is the positive proof that reading in the engine's own grammar changes the verdict, not merely the parse. The two"-quoting cases that ran everywhere are now pinned to the engines where"actually quotes an identifier.CI — three separate jobs, no
needs:onintegration-pgor on each other, so one flaky container cannot move the F9 done-bar.integration-pg's-k "db_path or role"selector is unchanged and matches zero new tests — now asserted by a test rather than assumed. No secrets, so they run on fork PRs.Two pre-existing tests fixed (both latent, both surfaced by this work, neither weakened — each now checks strictly more):
test_no_false_refusal_on_valid_sql.pyparsed every valid case under a hardcodedsqlitedialect despite its docstring claiming per-engine parsing. Now measures (case, engine) pairs: 15 → 23 checks, with all 15 originals retained.test_resolve_guard_model.pyloadedharness.pyviaspec_from_file_locationwithout registering it insys.modules, so adding any class that reads module globals at creation time broke it at import. Fixed to the documented idiom.Decisions
EXPLAIN) is deferred. No credentials exist; the marginal evidence is low (Snowflake and Redshift are"-quoting, the family Postgres already covers); a public repo withholds secrets from fork PRs, making such a job structurally unrunnable for outside contributors; and it is a standing maintenance tax. If ever wanted, the right shape is a non-requiredworkflow_dispatch/schedulejob. Residual risk: real BigQuery and Snowflake grammar stays unproven by execution._run_bigquerybuildsbigquery.Client(project=…)with noclient_options/api_endpoint, so it cannot be pointed at an emulator;_run_databricksspeaks to a workspace, not a localSparkSession. Reaching either needs a runtime source change this spec forbids, and going through a stub would assert exactly what ACE-079 already asserts. (The emulator is also SQLite-backed, so it would have proven parse/accept, not BigQuery semantics.)needs:) and buys failure isolation: SQL Server is the heavy, slow-booting container and should not be able to mask the backtick evidence.importorskipran before theAGAMI_IT_<ENGINE>_REQUIREDcheck, so a job whose driver failed to install skipped everything and exited 0 — green, proving nothing, in the job that owns that engine's only evidence. Closed and verified in both directions.Checklist
dev.py checkgreen — 2291 passed, 359 skipped, 2 xfailedpackages/agami-core/src/integration-pgselector unchanged; independence asserted by a test