Skip to content

feat: add build worker that executes a single job#321

Draft
Blackgaurd wants to merge 1 commit into04-17-feat_add_topological_scheduler_with_kahn_s_algorithmfrom
04-17-feat_add_build_worker_that_executes_a_single_job
Draft

feat: add build worker that executes a single job#321
Blackgaurd wants to merge 1 commit into04-17-feat_add_topological_scheduler_with_kahn_s_algorithmfrom
04-17-feat_add_build_worker_that_executes_a_single_job

Conversation

@Blackgaurd
Copy link
Copy Markdown
Collaborator

@Blackgaurd Blackgaurd commented Apr 17, 2026

Extracts per-dataset build logic from _build_recursive() into a standalone execute_job() function in service/worker.py.

What changed

  • New service/worker.py with execute_job(job, cancelled) -> JobResult
    • Generates valid timestamps, acquires per-dataset lock, checks existing timestamps
    • For each missing timestamp: fetches dep data (lookback vs single), runs builder subprocess, validates schema
    • Rows accumulated in memory, bulk-inserted only on full success (atomicity preserved)
    • Checks cancelled event between timestamps for early termination by failed siblings
    • Helper _fetch_dep_data() encapsulates dependency data fetching logic
  • New tests/service/test_worker.py with 8 tests covering:
    • All timestamps exist (skip), missing timestamps built + inserted
    • Builder failure mid-range (no partial insert), cancelled event stops early
    • Lookback dep uses `get_rows_range`, no-lookback uses `get_rows_timestamps`
    • Missing dep data returns failure, no valid timestamps returns failure

Why

The worker is the execution layer in the scheduler/queue/worker refactor. It handles the "how to build one dataset" concern, separated from the "what order to build" concern (scheduler) and the "orchestrate the full plan" concern (orchestrator, next PR).

Benefit

Clean separation of build execution from dependency resolution. The orchestrator (PR 5) will call execute_job() for each job in the topological plan.

🤖 Generated with Claude Code

Copy link
Copy Markdown
Collaborator Author

Blackgaurd commented Apr 17, 2026

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@Blackgaurd Blackgaurd force-pushed the 04-17-feat_add_build_worker_that_executes_a_single_job branch from f0f679d to d50c783 Compare April 25, 2026 04:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant