Migrate Configs and Rules to Pydantic

[fedeflowers]

Changes

Migrate DQX data models from dataclasses to Pydantic v2 for automatic
validation and simpler YAML serialization/deserialization.

Migrated to pydantic.BaseModel:

• checks_validator.py — ChecksValidationStatus (errors held in a real
  aliased field so it stays constructor-settable)
• config.py — 7 storage-config classes; __post_init__ →
  @model_validator(mode="after") with type(self) is ... guards for the
  multiple-inheritance composite; cached_property → property
• rule.py — DQRule, DQRowRule, DQDatasetRule, DQForEachColRule →
  frozen Pydantic models (ConfigDict(frozen=True, arbitrary_types_allowed=True)
  for Spark Callable/Column fields); class constants marked ClassVar
• pyproject.toml — add pydantic>=2.8.2

Intentionally left as dataclasses: WorkspaceConfig, RunConfig, and the
nested config tree, because databricks-labs-blueprint's Installation.load()
relies on dataclasses.is_dataclass() / __dataclass_fields__, which Pydantic
models don't satisfy.

Linked issues

Resolves #467

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

[mwojtyczka]

Generally looking good, left some comments.

We will need follow up PRs for pydantic to cover serialization and validation. That would bring a real value. It's fine to just have base-class swap in this PR. That's a reasonable incremental step; routing validation/serialization through Pydantic is a separate, higher-risk refactor that shouldn't bloat this one.

[fedeflowers]

Recap of changes:

1. engine.py — dataclasses.replace in _filter_for_original_columns_preselection

Added a small DQRule.replace() helper and call check.replace(check_func_kwargs=rule_kwargs) instead of model_copy(update=...). The reason for not using model_copy directly: it (a) does not re-run validators — DQRule's mode="after" validator validates attributes and initialises the name — and (b) shallow-copies the instance dict, so it carries over already-cached functools.cached_property values (rule_fingerprint, columns_as_string_expr). replace() rebuilds through the constructor, so validation re-runs and derived state is recomputed from the updated fields. Quick demonstration of the difference:

r = DQRowRule(name="x", check_func=is_not_null, column="id")
_ = r.rule_fingerprint  # populate the cached_property
r.model_copy(update={"column": "y"}).rule_fingerprint == r.rule_fingerprint  # True  (stale)
r.replace(column="y").rule_fingerprint            == r.rule_fingerprint       # False (recomputed)

2. checks_validator.py — _errors not constructor-settable

Good catch — PrivateAttr (and the interim aliased-field variant) were both more trouble than they were worth. Made errors a plain public field instead: errors: list[str] = [], which is constructor-settable as ChecksValidationStatus(errors=[...]); has_errors stays a @property. Updated the one call site (test_app_backend.py:1873) from _errors=[...] to errors=[...], so the test once again actually exercises the error path. Also added model_config = ConfigDict(extra="forbid") so a stray/unknown kwarg now raises ValidationError loudly instead of being silently dropped. This also removed the Field(...) that was tripping pylint's no-member on self.errors.append, so make lint is green again.

3. lakebase test — dataclasses.replace on LakebaseChecksStorageConfig

Rather than model_copy(update={"mode": ...}), I added a validating BaseChecksStorageConfig.replace() helper (mirroring DQRule.replace()) and switched the test to config.replace(mode=...), and dropped the import dataclasses. The reason for not using model_copy: it skips all validators, so an invalid override (e.g. a bad mode or a non-3-part location) would be accepted silently and only blow up later inside the save path. replace() rebuilds through the constructor so the storage-config validators re-run — verified:

c = LakebaseChecksStorageConfig(location="a.b.c", instance_name="inst")
c.model_copy(update={"mode": "BOGUS"}).mode  # 'BOGUS'           (accepted silently)
c.replace(mode="BOGUS")                       # raises InvalidConfigError

Also did a completeness sweep / added coverage:

• Grepped the repo for other dataclasses.replace/asdict on migrated models — the remaining occurrences all target classes still defined as @dataclass (WorkspaceConfig, workflow Task, DQProfile, anomaly ExplanationContext), so they're unaffected.
• Added unit tests for DQRule.replace() (override + field preservation, cached-fingerprint recomputation, validator re-run) and BaseChecksStorageConfig.replace() (override + field preservation, validator re-run).
• make lint green (10.00/10); affected unit tests pass (validator + storage configs + rule + app dry-run). Ran the affected integration paths: has_valid_schema (the check.replace preselection path) — 21 passed, 1 skipped; rule-fingerprint apply — 4 passed.

[grusin-db]

What is the planned approach for serialisation? that's the part where actually pydantic support matterns the most.

In dqx we have few serializers, we have yaml mode, where yamls need to be serialized to normal validators, also we have delta serializaer for storage, and probablt few more I am not aware/slipped my mind. This serializers needs considered before making changes in the models, so that we dont deadlock ourselves with potentially bad decisions.

[grusin-db]

Another topic, since we have now DQRules as pdyantic, have you tried serializing them into json_schema? there is .model_json_schema(), this should be included in testing, to ensure it passes.

It might - and probably will trip - on some internal non serializable types, so this needs to be verified to make sure it works e2e

[grusin-db]

In case of bad user input, all the errors will now be totally different, if pydantic model is throwing it will thrown its own ValidationError, which in turn will trip all validation error tests we have, hence that needs to be covered.

I don't see any tests changes regarding that. I.e. putting "blabla" into "int" field, would for sure throw pydantic validation error about "blabla" being not serializable to an int

[mwojtyczka]

What is the planned approach for serialisation? that's the part where actually pydantic support matterns the most.

In dqx we have few serializers, we have yaml mode, where yamls need to be serialized to normal validators, also we have delta serializaer for storage, and probablt few more I am not aware/slipped my mind. This serializers needs considered before making changes in the models, so that we dont deadlock ourselves with potentially bad decisions.

we will do this as a follow up PRs: serialization and validation

[mwojtyczka]

Another topic, since we have now DQRules as pdyantic, have you tried serializing them into json_schema? there is .model_json_schema(), this should be included in testing, to ensure it passes.

It might - and probably will trip - on some internal non serializable types, so this needs to be verified to make sure it works e2e

You are right:
DQRule.model_json_schema() → PydanticInvalidForJsonSchema: Cannot generate a JsonSchema for core_schema.CallableSchema
DQRowRule.model_json_schema() → same
FileChecksStorageConfig → OK
ChecksValidationStatus → OK

The Callable/Column fields need a custom json_schema / WithJsonSchema annotation — and a test

[mwojtyczka]

In case of bad user input, all the errors will now be totally different, if pydantic model is throwing it will thrown its own ValidationError, which in turn will trip all validation error tests we have, hence that needs to be covered.

I don't see any tests changes regarding that. I.e. putting "blabla" into "int" field, would for sure throw pydantic validation error about "blabla" being not serializable to an int

Yes, FileChecksStorageConfig raises pydantic_core.ValidationError. So programmatic construction with wrong-typed fields now surfaces Pydantic's error instead of DQX's own. The tests pass because the dict-level ChecksValidator still catches metadata errors before construction — but the model-construction path has no test coverage for the new error shape. I think DQX should catch/wrap ValidationError into its own error types for a consistent API rather than leak pydantic.ValidationError.

[mwojtyczka]

@fedeflowers I left comments as a follow up requests from @grusin-db , at least 2 of them are valid. Using pydantic for serializaiton can be done as a follow up PR. I kicked tests suite, it seems some tests are failing. Please check.