feat: add biz-date based source collection date mode#19
Open
wgzhao wants to merge 3 commits into
Open
Conversation
Motivation:\nIntroduce source-level collection date policies so automatic scheduling can skip non-trading days while keeping manual backfill available.\n\nWhat changed:\n- Added CollectDateMode (DAILY/WEEKDAY/WEEKEND) to source model and schema.\n- Added SourceScheduleMatcher based on bizDate weekday/weekend evaluation.\n- Applied source date filtering to source-level scheduler, queue scans, override scans, and daily refresh/reset initialization path.\n- Updated SourceService schedule-change detection to include enabled and collectDateMode changes.\n- Added frontend form/list support for collectDateMode selection and display.\n- Added memory.md entry for this change set.\n\nValidation:\n- mvn -pl backend -DskipTests compile\n- npm run type-check
Append a new memory entry for branch creation, commit hash, PR URL, and validation status.
- add collect_date_mode to vw_etl_table_with_source view\n- map collectDateMode in VwEtlTableWithSource entity\n- compute expected/actual/missing days from scheduled dates by source mode\n- deduplicate daily stats by tid+biz_date\n- update memory.md with this change
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation / Background
Some data sources (e.g. securities/funds) should not run automatic collection on non-trading days. Existing source scheduling only supports time-of-day and assumes daily collection
What Changed
collectDateModewith enum values:DAILY,WEEKDAY,WEEKEND.bizDate.bizDate.enabledandcollectDateMode.scripts/schema.sqlto includeetl_source.collect_date_mode.memory.mdwith this change summaryDesign / Implementation Notes
bizDate.bizDatepolicy.Validation
mvn -pl backend -DskipTests compilenpm run type-check(infrontend/)Risks / Caveats / Follow-ups
collect_date_modecolumn toetl_sourcewith defaultDAILY.npm run build-only) was not executable in current environment due missingvitecommand; type-check passed.