Skip to content

Feat/app UI i18n insights contracts#1262

Open
berrybluecode wants to merge 17 commits into
mainfrom
feat/app-ui-i18n-insights-contracts
Open

Feat/app UI i18n insights contracts#1262
berrybluecode wants to merge 17 commits into
mainfrom
feat/app-ui-i18n-insights-contracts

Conversation

@berrybluecode

@berrybluecode berrybluecode commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Changes

This PR adds substantially richer
authoring UI, and overhauls the deployment story. At a high level:

Data-contract & AI rule generation

  • Import ODCS v3.x data contracts and generate DQX rules (predefined,
    property-based, and natural-language) via a new contract_rules_service and
    /v1/contract route, including a dedicated "from contract" UI.
  • Natural-language (type: text) expectations are generated through the app's
    ChatDatabricks Foundation Model path (no dspy/Spark needed in-container).
  • Reference-table checks (foreign_key, has_valid_schema) are now authored in
    the single-table editor with a table picker; schema-validation gets its own
    editor and mutual-exclusion handling for expected_schema vs ref_table.

UI features

  • Internationalization (English, Spanish, Italian, Portuguese-BR) with a
    language selector and locale-parity test.
  • Insights dashboard is now hosted persistently so the embedded Lakeview iframe
    no longer reloads on every navigation; expandable labels UI; component
    reorganisation (apx/layout/); regenerated API client (orval).

Deployment & docs

  • Updated DEPLOYMENT.md / DEVELOPMENT.md (CLI v1.4.0+ gate, one-button
    postgres_roles, bind workflow) and removed duplicated content.

Linked issues

Resolves #..

Tests

Added backend unit suites for the Lakebase executor, Postgres/Delta migration
runners, contract rule generation, discovery DDL, dry-run dispatch, app
scheduler lease, quarantine filtering, lint policy, and i18n locale parity.
make app-test (658 passing) and make test (1160 passing) are green, along
with make fmt and make app-check.

  • manually tested
  • added unit tests
  • added integration tests
  • added end-to-end tests
  • added performance tests

Documentation and Demos

  • added/updated demos
  • added/updated docs
  • added/updated agent skills

… label filtering, error/warning metrics

Storage: move schemas, wheels volume, and Lakebase instance + logical database into databricks.yml with lifecycle.prevent_destroy; replace bootstrap_storage.sh with 'make app-bind' for adopting existing resources in workspaces created by the prior flow.

Backend: introduce PgExecutor + Postgres migration runner for OLTP tables (rules, settings, RBAC, comments, schedules, scheduler bookkeeping); keep analytical tables (validation runs, profiling, quarantine, metrics) on Delta.

Metrics: persist DQX observer's error_row_count / warning_row_count / input_row_count via dq_validation_runs.error_rows / warning_rows and fix Spark Connect Observation.get mutability bug that was overwriting total_rows with limit-pushed values.

UI: rename 'Invalid' -> 'Errors', add 'Warnings' column to quarantine detail, replace 'Has invalid' filter with 'Has failures', surface label badges next to table names in rule selection and schedule editor, add label filtering to Execute Rules and Schedule settings.

Migrations: fix FIELD_ALREADY_EXISTS idempotency in MigrationRunner so v3/v4/v5 ADD COLUMN migrations no-op on fresh deploys whose v1 baseline already includes warning_rows / warnings / error_rows.

Misc: retention test coverage, custom metrics service, post-deploy grants script reuse, docs rewrite (DEPLOYMENT.md, README.md, CLAUDE.md, installation.mdx) for the new declarative storage model.
…, drop database_catalogs

The database_catalogs DAB resource is the only one that creates a logical Postgres database, but it also creates a Unity Catalog catalog as a side effect and therefore requires CREATE CATALOG on the metastore — a permission most app deployers don't hold. Drop it. Connect the app to the always-present databricks_postgres admin database instead; per-app isolation comes from the dedicated dqx_studio Postgres schema the app creates inside it on first start. The bundle stays fully declarative with no out-of-band bootstrap steps.

- databricks.yml: remove database_catalogs.lakebase_db and the lakebase_uc_catalog_name variable. Default lakebase_database_name to 'databricks_postgres'. App's database: binding now references ${var.lakebase_database_name} directly.

- Makefile: add BUNDLE_VARS forwarding to 'make app-deploy' so one-off CLI overrides (e.g. lakebase_instance_name during Lakebase's 7-day soft-delete name retention) don't require ad-hoc databricks.yml edits.

- bind_resources.sh: pass --auto-approve explicitly (newer databricks CLI versions reject piped 'yes' confirmation). Drop the lakebase_db bind step (no longer a bundle resource).

- Documentation updates across DEPLOYMENT.md, CLAUDE.md files, and installation.mdx to reflect the new layout: no separate logical-DB provisioning step, uninstall drops a Postgres schema instead of a database.
make fmt rewrites GitHub source URLs to match the version in __about__.py. CI merges this branch onto main (now at v0.14.0) and the lingering v0.13.0 refs added on this branch trip git diff --exit-code. Bump them to v0.14.0 explicitly.
The previous OltpExecutor = "SqlExecutor | PgExecutor" was just a string assignment, not a type alias — mypy/pyright treated it as str. Switch to Union["SqlExecutor", "PgExecutor"] (string forward refs because PgExecutor is TYPE_CHECKING-only) and use OltpExecutor | None for _pg_executor and the get/set helpers, so type checkers actually enforce the parity contract between the two executors.
…hardening, plus PgMigrationRunner unit tests and CI fixes
…nal:databrickslabs/dqx into feat/app-refactor-backend-add-lakebase
…ing-your-own SQL warehouse

Per-run review status
- New ReviewStatusService + /api/v1/runs/{run_id}/review-status endpoints
  (GET / PUT / DELETE + /history) with admin-managed catalogue via
  /api/v1/config/run-review-statuses (e.g. Pending review → Acknowledged).
- Storage: dq_run_review_status (mutable current state) +
  dq_run_review_status_history (append-only audit). Default value
  surfaced virtually so dashboards/filters never see NULL.
- UI: RunReviewStatusPanel on the Run detail page next to the
  comments thread; multi-select status filter on Runs History.
- Task runner emits the review-status field on completion; dryrun
  and quarantine routes include effective status in their payloads.

Insights page + starter dashboard
- New /insights route embeds a Databricks AI/BI dashboard via iframe.
- Admin-configurable dashboard ID through config.py with
 QX_DEFAULT_DASHBOARD_ID env fallback so the bundle can ship a
  default without preventing customer overrides.
- Ships dashboards/dqx_quality_overview.lvdash.json as the starter.

DAB: bring-your-own SQL warehouse
- databricks.yml now supports two patterns per target: bundle-managed
  (declared under resources.sql_warehouses.dqx_sql_warehouse) and
  existing-warehouse-by-id (just set sql_warehouse_id).
- post_deploy_grants.sh hardened: CAN_USE grants to app + task-runner
  SPs degrade to warnings when the deployer lacks CAN_MANAGE on a
  shared warehouse, instead of failing the whole script.

Settings + plumbing
- app_settings_service expanded for review-status catalogue and
  dashboard config; postgres migrations and models updated; DI wiring
  in dependencies.py + app.py to expose the new service.

Docs
- New CUSTOMER_QA.md (positioning, security, lifecycle, ops Q&A).
- DEPLOYMENT.md / README.md refreshed for the new warehouse modes,
  Insights page, and review-status flow.
…8n, and deployment refactor

Backend:
- Add Lakebase Postgres OLTP backend (pg_executor, pg migrations, token rotation)
  with Delta fallback; route rules/settings/RBAC/comments/schedules to OLTP
- Add ODCS data-contract rule generation (contract_rules_service, /v1/contract)
  including natural-language (text) rules via the ChatDatabricks AI path
- Surface reference-table checks (foreign_key, has_valid_schema) in single-table editor
- Refactor app bootstrap, dependencies, sql_executor, and migrations runner

Frontend:
- Add i18n (en/es/it/pt-BR) with language selector
- Add persistent Insights dashboard host (iframe no longer reloads on navigation)
- Add from-contract and schema rule editors; relocate apx/ components to layout/
- Regenerate API client (orval)

Deployment & docs:
- Consolidate databricks.yml to a single canonical target; move Lakebase instance
  to base resources; document the 6 warehouse x Lakebase scenarios
- Update DEPLOYMENT.md / DEVELOPMENT.md (CLI v1.4.0+, bring-your-own resources)
- Add build_app.py / dev.py scripts and deploy-docs workflow

Tests: add app scheduler, contract rules, discovery DDL, dryrun dispatch, i18n
parity, lint policy, migration runner, pg executor, and quarantine filter suites
@codecov

codecov Bot commented Jun 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.43%. Comparing base (9980c98) to head (cddd404).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1262      +/-   ##
==========================================
- Coverage   92.56%   92.43%   -0.13%     
==========================================
  Files         102      102              
  Lines       10075    10075              
==========================================
- Hits         9326     9313      -13     
- Misses        749      762      +13     
Flag Coverage Δ
anomaly 54.37% <ø> (ø)
anomaly-serverless 54.38% <ø> (ø)
integration 50.54% <ø> (-0.01%) ⬇️
integration-serverless 50.40% <ø> (-0.23%) ⬇️
unit 57.02% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

✅ 772/772 passed, 1 flaky, 49 skipped, 5h59m42s total

Flaky tests:

  • 🤪 test_quality_checker (57.37s)

Running from acceptance #4942

@github-actions

github-actions Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

✅ 194/194 passed, 2 skipped, 6h49m35s total

Running from anomaly #1056

@mwojtyczka mwojtyczka added the under-review This PR is currently being reviewed by one of DQX maintainers. label Jun 23, 2026
Comment thread app/src/databricks_labs_dqx_app/backend/routes/v1/contract.py
Comment thread app/src/databricks_labs_dqx_app/ui/components/RunReviewStatusPanel.tsx Outdated
@mwojtyczka

Copy link
Copy Markdown
Contributor

Additional findings — pre-existing (not introduced by this PR)

While reviewing the backend changes I hit three issues in code this PR touches/neighbours. git blame puts all three at or before this branch's merge-base, so they're pre-existing, not regressions from this PR — flagging here for visibility; fine to fix in a follow-up rather than blocking this PR. (They're posted at PR level because they sit on unchanged lines, which GitHub won't let me anchor an inline comment to.)

1. Scheduler re-fires a schedule every tick if _trigger_run raises — scheduler_service.py:294-302.
next_run_at is only advanced to a future time at the _upsert_tracker call on line 301, which runs after _trigger_run returns. Per-table failures are caught inside _trigger_run and returned as errors, but _resolve_scope / _load_custom_metrics / DB access sit outside that inner try. If any raises, it propagates to the except at line 302 (log-only); since line 275 already wrote next_run_at = now - 1s, the schedule is "due" again on the next tick → a deterministic failure becomes a tight retry loop that keeps submitting jobs. Relevant now because this PR adds new code paths inside _trigger_run. Fix: advance next_run_at (or persist a failed status with backoff) in a finally/except.

2. Monthly day_of_month silently capped at 28 — scheduler_service.py:1089.
candidate = after.replace(day=min(dom, 28), ...) — a schedule configured for the 29th/30th/31st always fires on the 28th while the UI shows the configured day. Clamp to the actual last day of the target month instead, e.g. calendar.monthrange(year, month)[1].

3. Non-TERMINATED terminal states written verbatim as run status — dryrun.py:533-539.
When status.state is INTERNAL_ERROR or SKIPPED, update_run_status(..., status=status.state, ...) writes those raw strings into the run-history status column, outside the normal SUCCESS/FAILED/CANCELED/RUNNING vocabulary (the synthesized path at 505-513 correctly maps terminal → SUCCESS/FAILED). The row does update (not stuck RUNNING), but downstream UI/queries may not recognize these values. Consider mapping INTERNAL_ERROR/SKIPPEDFAILED.

Comment thread app/scripts/_align_wheel_version.py Outdated
Comment thread app/scripts/_align_wheel_version.py Outdated

@mwojtyczka mwojtyczka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good, left some comments. There are some pre-existing issues as well.

…eduler fixes

- contract: route contract-generated rules through DQEngine.validate_checks
  and validate LLM output (function names via CHECK_FUNC_REGISTRY, SQL via
  is_sql_query_safe); surface validation_errors and stop relaying raw
  exception detail to callers
- ai_rules_service/config: cap LLM calls with DQX_LLM_MAX_TOKENS
- scheduler: advance next_run_at (with backoff) on _trigger_run failure to
  avoid tight retry loops; clamp monthly day_of_month to the real last day
  of month via calendar.monthrange
- dryrun: map INTERNAL_ERROR/SKIPPED terminal states to FAILED
- ui: internationalize RunReviewStatusPanel and config toasts; add matching
  keys to en/es/it/pt-BR locales
- build: drop apx references from _align_wheel_version.py docstrings
- tests: cover rule validation/LLM safety and scheduler clamp/backoff
…:databrickslabs/dqx into feat/app-ui-i18n-insights-contracts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

under-review This PR is currently being reviewed by one of DQX maintainers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants