Feat/app UI i18n insights contracts#1262
Conversation
… label filtering, error/warning metrics Storage: move schemas, wheels volume, and Lakebase instance + logical database into databricks.yml with lifecycle.prevent_destroy; replace bootstrap_storage.sh with 'make app-bind' for adopting existing resources in workspaces created by the prior flow. Backend: introduce PgExecutor + Postgres migration runner for OLTP tables (rules, settings, RBAC, comments, schedules, scheduler bookkeeping); keep analytical tables (validation runs, profiling, quarantine, metrics) on Delta. Metrics: persist DQX observer's error_row_count / warning_row_count / input_row_count via dq_validation_runs.error_rows / warning_rows and fix Spark Connect Observation.get mutability bug that was overwriting total_rows with limit-pushed values. UI: rename 'Invalid' -> 'Errors', add 'Warnings' column to quarantine detail, replace 'Has invalid' filter with 'Has failures', surface label badges next to table names in rule selection and schedule editor, add label filtering to Execute Rules and Schedule settings. Migrations: fix FIELD_ALREADY_EXISTS idempotency in MigrationRunner so v3/v4/v5 ADD COLUMN migrations no-op on fresh deploys whose v1 baseline already includes warning_rows / warnings / error_rows. Misc: retention test coverage, custom metrics service, post-deploy grants script reuse, docs rewrite (DEPLOYMENT.md, README.md, CLAUDE.md, installation.mdx) for the new declarative storage model.
…, drop database_catalogs
The database_catalogs DAB resource is the only one that creates a logical Postgres database, but it also creates a Unity Catalog catalog as a side effect and therefore requires CREATE CATALOG on the metastore — a permission most app deployers don't hold. Drop it. Connect the app to the always-present databricks_postgres admin database instead; per-app isolation comes from the dedicated dqx_studio Postgres schema the app creates inside it on first start. The bundle stays fully declarative with no out-of-band bootstrap steps.
- databricks.yml: remove database_catalogs.lakebase_db and the lakebase_uc_catalog_name variable. Default lakebase_database_name to 'databricks_postgres'. App's database: binding now references ${var.lakebase_database_name} directly.
- Makefile: add BUNDLE_VARS forwarding to 'make app-deploy' so one-off CLI overrides (e.g. lakebase_instance_name during Lakebase's 7-day soft-delete name retention) don't require ad-hoc databricks.yml edits.
- bind_resources.sh: pass --auto-approve explicitly (newer databricks CLI versions reject piped 'yes' confirmation). Drop the lakebase_db bind step (no longer a bundle resource).
- Documentation updates across DEPLOYMENT.md, CLAUDE.md files, and installation.mdx to reflect the new layout: no separate logical-DB provisioning step, uninstall drops a Postgres schema instead of a database.
make fmt rewrites GitHub source URLs to match the version in __about__.py. CI merges this branch onto main (now at v0.14.0) and the lingering v0.13.0 refs added on this branch trip git diff --exit-code. Bump them to v0.14.0 explicitly.
The previous OltpExecutor = "SqlExecutor | PgExecutor" was just a string assignment, not a type alias — mypy/pyright treated it as str. Switch to Union["SqlExecutor", "PgExecutor"] (string forward refs because PgExecutor is TYPE_CHECKING-only) and use OltpExecutor | None for _pg_executor and the get/set helpers, so type checkers actually enforce the parity contract between the two executors.
…hardening, plus PgMigrationRunner unit tests and CI fixes
…nal:databrickslabs/dqx into feat/app-refactor-backend-add-lakebase
…tHub URLs to v0.14.0 in docs
…ing-your-own SQL warehouse
Per-run review status
- New ReviewStatusService + /api/v1/runs/{run_id}/review-status endpoints
(GET / PUT / DELETE + /history) with admin-managed catalogue via
/api/v1/config/run-review-statuses (e.g. Pending review → Acknowledged).
- Storage: dq_run_review_status (mutable current state) +
dq_run_review_status_history (append-only audit). Default value
surfaced virtually so dashboards/filters never see NULL.
- UI: RunReviewStatusPanel on the Run detail page next to the
comments thread; multi-select status filter on Runs History.
- Task runner emits the review-status field on completion; dryrun
and quarantine routes include effective status in their payloads.
Insights page + starter dashboard
- New /insights route embeds a Databricks AI/BI dashboard via iframe.
- Admin-configurable dashboard ID through config.py with
QX_DEFAULT_DASHBOARD_ID env fallback so the bundle can ship a
default without preventing customer overrides.
- Ships dashboards/dqx_quality_overview.lvdash.json as the starter.
DAB: bring-your-own SQL warehouse
- databricks.yml now supports two patterns per target: bundle-managed
(declared under resources.sql_warehouses.dqx_sql_warehouse) and
existing-warehouse-by-id (just set sql_warehouse_id).
- post_deploy_grants.sh hardened: CAN_USE grants to app + task-runner
SPs degrade to warnings when the deployer lacks CAN_MANAGE on a
shared warehouse, instead of failing the whole script.
Settings + plumbing
- app_settings_service expanded for review-status catalogue and
dashboard config; postgres migrations and models updated; DI wiring
in dependencies.py + app.py to expose the new service.
Docs
- New CUSTOMER_QA.md (positioning, security, lifecycle, ops Q&A).
- DEPLOYMENT.md / README.md refreshed for the new warehouse modes,
Insights page, and review-status flow.
…8n, and deployment refactor Backend: - Add Lakebase Postgres OLTP backend (pg_executor, pg migrations, token rotation) with Delta fallback; route rules/settings/RBAC/comments/schedules to OLTP - Add ODCS data-contract rule generation (contract_rules_service, /v1/contract) including natural-language (text) rules via the ChatDatabricks AI path - Surface reference-table checks (foreign_key, has_valid_schema) in single-table editor - Refactor app bootstrap, dependencies, sql_executor, and migrations runner Frontend: - Add i18n (en/es/it/pt-BR) with language selector - Add persistent Insights dashboard host (iframe no longer reloads on navigation) - Add from-contract and schema rule editors; relocate apx/ components to layout/ - Regenerate API client (orval) Deployment & docs: - Consolidate databricks.yml to a single canonical target; move Lakebase instance to base resources; document the 6 warehouse x Lakebase scenarios - Update DEPLOYMENT.md / DEVELOPMENT.md (CLI v1.4.0+, bring-your-own resources) - Add build_app.py / dev.py scripts and deploy-docs workflow Tests: add app scheduler, contract rules, discovery DDL, dryrun dispatch, i18n parity, lint policy, migration runner, pg executor, and quarantine filter suites
…pp-refactor-backend-add-lakebase
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1262 +/- ##
==========================================
- Coverage 92.56% 92.43% -0.13%
==========================================
Files 102 102
Lines 10075 10075
==========================================
- Hits 9326 9313 -13
- Misses 749 762 +13
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
✅ 772/772 passed, 1 flaky, 49 skipped, 5h59m42s total Flaky tests:
Running from acceptance #4942 |
|
✅ 194/194 passed, 2 skipped, 6h49m35s total Running from anomaly #1056 |
Additional findings — pre-existing (not introduced by this PR)While reviewing the backend changes I hit three issues in code this PR touches/neighbours. 1. Scheduler re-fires a schedule every tick if 2. Monthly 3. Non- |
mwojtyczka
left a comment
There was a problem hiding this comment.
Mostly looks good, left some comments. There are some pre-existing issues as well.
…eduler fixes - contract: route contract-generated rules through DQEngine.validate_checks and validate LLM output (function names via CHECK_FUNC_REGISTRY, SQL via is_sql_query_safe); surface validation_errors and stop relaying raw exception detail to callers - ai_rules_service/config: cap LLM calls with DQX_LLM_MAX_TOKENS - scheduler: advance next_run_at (with backoff) on _trigger_run failure to avoid tight retry loops; clamp monthly day_of_month to the real last day of month via calendar.monthrange - dryrun: map INTERNAL_ERROR/SKIPPED terminal states to FAILED - ui: internationalize RunReviewStatusPanel and config toasts; add matching keys to en/es/it/pt-BR locales - build: drop apx references from _align_wheel_version.py docstrings - tests: cover rule validation/LLM safety and scheduler clamp/backoff
…:databrickslabs/dqx into feat/app-ui-i18n-insights-contracts
Changes
This PR adds substantially richer
authoring UI, and overhauls the deployment story. At a high level:
Data-contract & AI rule generation
property-based, and natural-language) via a new
contract_rules_serviceand/v1/contractroute, including a dedicated "from contract" UI.type: text) expectations are generated through the app'sChatDatabricks Foundation Model path (no dspy/Spark needed in-container).
foreign_key,has_valid_schema) are now authored inthe single-table editor with a table picker; schema-validation gets its own
editor and mutual-exclusion handling for
expected_schemavsref_table.UI features
language selector and locale-parity test.
no longer reloads on every navigation; expandable labels UI; component
reorganisation (
apx/→layout/); regenerated API client (orval).Deployment & docs
postgres_roles, bind workflow) and removed duplicated content.Linked issues
Resolves #..
Tests
Added backend unit suites for the Lakebase executor, Postgres/Delta migration
runners, contract rule generation, discovery DDL, dry-run dispatch, app
scheduler lease, quarantine filtering, lint policy, and i18n locale parity.
make app-test(658 passing) andmake test(1160 passing) are green, alongwith
make fmtandmake app-check.Documentation and Demos