test(jobs): guard against lakets schema drift in workflow files#33
Merged
Conversation
Adds a static check (no live DB needed) asserting every `lakets.<object>`
referenced by databricks/workflows/*.py is actually defined by a
CREATE TABLE/VIEW/FUNCTION/AGGREGATE in sql/*.sql. Logger names
(getLogger("lakets.<job>")) are excluded.
This catches the class of bug that only surfaced at runtime: the
hypertable->chronotable rename left lakets._hypertable_registry references
in partition_manager.py / retention_job.py. The guard fails on any such
stale reference. GHA is disabled org-wide, so this runs in the existing
pytest suite rather than a live-run CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a static guard (no live DB, no GitHub Actions needed — GHA is disabled org-wide) that asserts every
lakets.<object>referenced bydatabricks/workflows/*.pyis actually defined by aCREATE TABLE/VIEW/FUNCTION/AGGREGATE/PROCEDUREinsql/*.sql. Logger names (getLogger("lakets.<job>")) are excluded.Why
partition_manager.pyandretention_job.pyreferencedlakets._hypertable_registry(andpr.hypertable_id/p_hypertable_id) long after the hypertable→chronotable rename. Nothing exercised those jobs, so the breakage only surfaced when run against a live Lakebase (PR #32). This guard fails fast on any such stale reference.Verified: injecting
lakets._hypertable_registryinto a source string makes the check report_hypertable_registryas undefined — i.e. it would have failed on the original bug.Test plan
pytest tests/test_python_patterns.py→ 13 passed (2 new)_hypertable_registryreference