Skip to content

feat: add biz-date based source collection date mode#19

Open
wgzhao wants to merge 3 commits into
masterfrom
feat/source-collect-date-mode
Open

feat: add biz-date based source collection date mode#19
wgzhao wants to merge 3 commits into
masterfrom
feat/source-collect-date-mode

Conversation

@wgzhao
Copy link
Copy Markdown
Owner

@wgzhao wgzhao commented May 4, 2026

Motivation / Background

Some data sources (e.g. securities/funds) should not run automatic collection on non-trading days. Existing source scheduling only supports time-of-day and assumes daily collection

What Changed

  • Added source-level date policy collectDateMode with enum values: DAILY, WEEKDAY, WEEKEND.
  • Added biz-date aware matcher to evaluate whether a source is collectible on current bizDate.
  • Updated source scheduler execution guard to skip when source policy does not match current bizDate.
  • Updated daily initialization flow to reset/refresh only matched source IDs instead of all sources.
  • Applied source-ID filtering to runnable task queries (including inherited source schedule and table override schedule scan path).
  • Updated source save/change detection to include enabled and collectDateMode.
  • Updated frontend source form and list to configure and display collection date mode.
  • Added schema change in scripts/schema.sql to include etl_source.collect_date_mode.
  • Added/updated memory.md with this change summary

Design / Implementation Notes

  • Workday definition follows current agreement: Monday-Friday on bizDate.
  • Manual single-task submit intentionally bypasses collect-date policy (backfill use case), with explicit why-comments added in code.
  • Automatic scheduling/filtering remains strict on bizDate policy.

Validation

  • mvn -pl backend -DskipTests compile
  • npm run type-check (in frontend/)

Risks / Caveats / Follow-ups

  • Database migration is required before deployment: add collect_date_mode column to etl_source with default DAILY.
  • Frontend full build (npm run build-only) was not executable in current environment due missing vite command; type-check passed.

wgzhao added 3 commits May 4, 2026 17:51
Motivation:\nIntroduce source-level collection date policies so automatic scheduling can skip non-trading days while keeping manual backfill available.\n\nWhat changed:\n- Added CollectDateMode (DAILY/WEEKDAY/WEEKEND) to source model and schema.\n- Added SourceScheduleMatcher based on bizDate weekday/weekend evaluation.\n- Applied source date filtering to source-level scheduler, queue scans, override scans, and daily refresh/reset initialization path.\n- Updated SourceService schedule-change detection to include enabled and collectDateMode changes.\n- Added frontend form/list support for collectDateMode selection and display.\n- Added memory.md entry for this change set.\n\nValidation:\n- mvn -pl backend -DskipTests compile\n- npm run type-check
Append a new memory entry for branch creation, commit hash, PR URL, and validation status.
- add collect_date_mode to vw_etl_table_with_source view\n- map collectDateMode in VwEtlTableWithSource entity\n- compute expected/actual/missing days from scheduled dates by source mode\n- deduplicate daily stats by tid+biz_date\n- update memory.md with this change
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant