Skip to content

feat: a continuous trigger on a batch build warns at validate; scaffold-composer schedules only builds a scheduler starts - #737

Open
fas89 wants to merge 11 commits into
mainfrom
feat/continuous-batch-scaffold
Open

fas89 wants to merge 11 commits into
mainfrom
feat/continuous-batch-scaffold

Conversation

@fas89

@fas89 fas89 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

Follow-ups #734 documented instead of fixing: fluid validate, the build runners and fluid scaffold-composer did not know trigger.type: continuous.

A continuous trigger on a build forge-cli runs as a batch run is a validation warning (the user's decision: warn, not refuse). A continuous trigger says the job's platform keeps it running, so no scheduler starts the build; forge-cli's batch runners run a build to completion. Plain fluid validate passes and --strict fails:

 1. builds[0] (id 'customer360_build'): trigger.type continuous on engine 'dbt': forge-cli runs dbt builds as batch runs, which end, and its dbt runner refuses a continuous trigger: forge-cli never runs this build. Use trigger.type schedule or manual, or run the job on a platform that keeps it running.

fluid validate and fluid apply read one predicate (build_runners/base.py::_batch_runner_for), in the dispatcher's order:

Continuous build fluid apply --mode amend-and-build fluid validate
dbt, dbt-<adapter>, python, and everything else the python runner takes: no engine, a local spark / glue / custom script, an ingestion engine without pattern: acquisition the runner refuses it by name (exit 1) warns (names a missing pattern: acquisition)
inline SQL on local DuckDB or Snowflake; duckdb / airbyte / meltano / dlt acquisition runs it once warns
kafka-connect / debezium acquisition deploys the connector, which keeps running —
Flink (engine: flink, language: flink_sql) left to its platform (unchanged) —
spark / glue / custom that names a runtime.platform other than local; inline SQL on a platform with no embedded-SQL executor new: left to that platform like a Flink job (Run outside forge-cli: 1, run report skipped / runs_outside_forge); used to reach the python runner or the SQL executor and fail —
  • The streaming set is pinned against the runners' own declared_capabilities (RunnerCapability.STREAMING). A parity test runs the real dispatcher over every engine and pattern, and another checks that the python runner still receives every local script.
  • EMBEDDED_SQL_PLATFORMS is the one list the inline-SQL executor and the warning read.
  • The dbt and python runners print one refusal (refuse_continuous_trigger) with the warning's remedy, instead of Unknown trigger type: continuous / Supported types: manual, schedule. The dbt runner refuses before it resolves a profile.
  • A crash in the check is a warning that says the contract was not checked.

fluid scaffold-composer writes no DAG for a contract whose every build is continuous or runs outside forge-cli, unless it declares orchestration.tasks (the test fluid generate schedule applies, through the shared util/contract.has_only_unscheduled_builds), and says so. It names a DAG it wrote before (recognised by the template's tags=["FLUID"] and the dag_id; a same-named file it did not write is not claimed) and leaves it in place; the path prints whole and literally (soft_wrap, markup=False, emoji=False). Otherwise the DAG takes the cron of the first build a scheduler starts, not a Flink job's own cron.

Borrowed, not built

  • Spark UnsupportedOperationChecker.checkForContinuous: a continuous trigger is checked against what the engine can run before anything starts, and the refusal names what it cannot run.
  • Databricks INFINITE_STREAMING_TRIGGER_NOT_SUPPORTED: where a job cannot be kept running, the continuous trigger is refused by name, with the alternative.
  • In repo: is_unscheduled_build / is_continuous_build (feat: trigger.type continuous (0.7.6 preview) is never scheduled; --strict fails on a retargeted overlay patch #734), the dispatch predicates and _report_externally_run_build in build_runners/base.py, the runners' declared_capabilities, the (errors, warnings) validator shape of the Iceberg / governance gates, and generate schedule's "nothing to schedule". Strategy: adapt the pattern; no new dependency.

Validation

  • Tests: tests/test_continuous_trigger.py 26 → 107. Each round's fix was red first: the stale-DAG path fold, the :emoji:/[/] out-dir, the dispatcher parity (6 engine shapes), inline SQL on databricks, the explicit-tasks DAG, the foreign same-named file, the python-runner handoff (10 tests red on the round-2 commit). Every commit is green on its own.
  • Gates (local): ruff (incl. the dead-code gate), black 24.10.0, scripts/mypy_strict.py, import-linter (6 kept), license headers, product guardrail (0 failing). Affected suites (58 files/dirs, ~5.9k tests): failures equal clean main id-for-id in this venv (19 failed + 10 errors: dlt SDK, sqlglot, tmp-root naming). tests/perf: the module budget passes (1018 modules on both trees); the 3 wall-clock budgets fail on clean main too at load average ~145.
  • Live CLI (copies of examples/customer360 and examples/02-csv-to-data-product, plus python/spark/no-engine contracts):
    • continuous dbt: validate exit 0 with the warning, --strict exit 1, --quiet --strict names it, --format json lists it, the bundle report counts it; apply --mode amend-and-build refuses before any profile (exit 1).
    • continuous inline SQL: warned; plain apply and --mode amend-and-build each run it once.
    • continuous python / no engine: warned; refused, the script never runs.
    • continuous spark with a local script, and dlt without pattern: acquisition: warned (--strict exit 1), refused (exit 1), the script never runs. Continuous spark on runtime.platform: databricks: no warning, apply --mode amend-and-build exit 0, Run outside forge-cli: 1.
    • scaffold-composer: Flink-only or continuous-only contract → no DAG, no --out-dir; after a scheduled run wrote a DAG, the next run names it and keeps it; a foreign same-named file is not named; a 0.7.5 flink_sql + dbt contract takes the dbt cron (0 3, was the Flink 0 5); generate schedule unchanged.
  • ripwire --quality-delta: gating rows left are _run_contract_rules growing by one rule block in its existing pattern, and per-module test helpers flagged as duplication (house style).

Reviews

  • Security review (Opus): no HIGH or MEDIUM. Fixed: the no-DAG message claimed any same-named file as ours. Pre-existing, listed below: scaffold-composer overwrites a colliding file; the runners print the contract's trigger.type with Rich markup on.
  • Defect hunt (Opus, reproduced findings only): fixed the validate/dispatcher disagreement (spark/glue/custom, ingestion engines without the acquisition pattern, no engine), inline SQL on executor-less platforms, the explicit-tasks divergence, the run-once wording, the differing remedies, a terminal-width-dependent assertion.
  • Round 2 re-review (Opus): confirmed the round-1 fixes (6,624 mocked dispatches across 23 engines, 8 patterns and 9 platforms agreed with validate), and found that skipping every python-runner build as "outside forge-cli" reported success for dlt / local spark scripts that run when manual. Round 3 keeps them on the python runner (warned + refused) and skips only builds that name a platform of their own. A pre-security: fix injection/RCE/RLS findings (scheduler, IaC, MCP, build-runners, providers) #215 DAG file name is not matched; the CHANGELOG says so.
  • Round 3 verification (Opus): F1 and F2 confirmed fixed. Of 16,830 recorded dispatches, no non-continuous build changed runner against main, and every continuous build is warned exactly when a batch runner takes it. Two LOW wording items fixed: the CHANGELOG's platform exception excludes inline SQL, and the warning for spark / glue / custom names execution.runtime.platform.

Documentation

Not in this PR

  • fluid scaffold-composer reads only trigger.cron, not trigger.schedule (documented on its page); it overwrites a colliding file it did not write.
  • The dbt and python runners print the contract's trigger.type (the Trigger: and Unknown trigger type lines) with Rich markup on.
  • fluid verify still expects a dbt run record for a continuous dbt build, which fluid apply refuses.
  • With no dbt project or script found, a continuous dbt/python build is reported as skipped, not refused.
  • fluid validate workspace mode runs no contract rules (documented), so it does not show this warning either.
  • A DAG deployed before a product went all-continuous is not retired by stage 11 (upgrade note kept).

fas89 added 11 commits October 21, 2026 22:35
…runs as a batch run

A continuous trigger (schema 0.7.6) says the job's platform keeps it running,
so no scheduler starts the build. forge-cli runs dbt, python, inline-SQL and
duckdb, airbyte, meltano and dlt acquisition builds to completion: the dbt and
python runners refuse the trigger, so such a build never runs, and the others
run once per fluid apply. fluid validate now says so as a warning, so plain
validate passes and --strict fails:

  builds[0] (id 'transform'): trigger.type continuous on engine 'dbt':
  forge-cli runs dbt builds as batch runs, which end, and its dbt runner
  refuses a continuous trigger: forge-cli never runs this build. Use
  trigger.type schedule, or run the job on a platform that keeps it running.

The check follows run_builds_from_args' dispatch order, so it names the runner
that gets the build: a dbt build with inline SQL runs on DuckDB. The Kafka
Connect and Debezium connectors, whose runners declare
RunnerCapability.STREAMING, and builds that run outside forge-cli get no
warning, and neither do spark, glue and custom builds: forge-cli has no runner
of its own for them, and their platform may keep them running. A crash in the
check is a warning that says the contract was not checked for it.

Borrowed: Spark's UnsupportedOperationChecker.checkForContinuous checks a
continuous trigger against what the query can run before it starts, and
Databricks refuses Trigger.Continuous where it cannot keep a job running
(INFINITE_STREAMING_TRIGGER_NOT_SUPPORTED), naming the trigger.
…ntinuous by name

They printed "Unknown trigger type: continuous" and "Supported types: manual,
schedule", as for a typo. They now say that forge-cli runs the build as a
batch run, which ends, while a continuous trigger says the job's platform
keeps it running, and what to use instead. The exit code stays 1. The dbt
runner refuses before it resolves a profile or prepares the dbt command.
…d the cron of a scheduled build

fluid scaffold-composer wrote a DAG for any contract: one whose every build is
continuous or runs outside forge-cli got a nightly 0 2 * * * DAG, or the cron a
Flink job declares for its own platform. It now writes no DAG for such a
contract and says so, as fluid generate schedule does, and names a DAG it
wrote before, which it leaves in place. Otherwise the DAG takes the cron of
the first build a scheduler starts.

has_only_unscheduled_builds() moves to util/contract.py, so both commands
decide with one predicate.
Rich folded a path longer than the terminal across lines, so the path of a DAG
written before came out broken where it is meant to be copied. The line is
printed with soft_wrap, and the terminal wraps it.
…ous build

A defect scan found the warning and the dispatcher disagreeing. A continuous
spark, glue or custom build, or an ingestion engine declared without
pattern: acquisition, got no warning, while fluid apply --mode amend-and-build
handed it to the python runner, which refused it. Inline SQL on a platform
with no embedded-SQL executor, such as databricks, was warned as running once
per apply, while the executor refuses that platform.

Now one predicate decides both. A continuous build that no forge-cli runner
takes is left to its platform, as a Flink job is: the dispatcher prints that
there is nothing for forge-cli to run, counts it under "Run outside forge-cli"
and records it as skipped with reason runs_outside_forge, and validate gives
no warning. A build with no engine goes to the python runner, so it is warned
like a python build. EMBEDDED_SQL_PLATFORMS is the one list of the platforms
that run inline SQL, read by the executor and by the warning. A test runs the
dispatcher over every engine and pattern and checks that validate warns
exactly when a batch runner gets the build.

The dbt and python runners print one refusal, refuse_continuous_trigger, and
it gives the same remedy as the warning ("Use trigger.type schedule or
manual, ..."). The run-once warning says the build runs only when fluid apply
runs builds: plain fluid apply does not run an acquisition. The --strict
output assertion no longer depends on the terminal width.
…r declared tasks

The no-DAG message told the operator to delete any file at <out-dir>/<dag_id>.py,
and dag_id is a lossy clean-up of the contract id, so it could name another
team's DAG. It now names the file only when it carries the template's
tags=["FLUID"] and this dag_id, as the current template quotes it or as the
template before #215 did; a file it did not write is not mentioned.

A contract whose every build is unscheduled but which hand-declares
orchestration.tasks still gets its DAG, as fluid generate schedule still
writes files for one. The path prints with emoji=False beside markup=False, so
an :emoji: code in --out-dir prints as written.
…inuous or not

The previous commit left a continuous spark, glue or custom build, and an
ingestion engine declared without pattern: acquisition, to "its platform",
and fluid apply exited 0. But the python runner runs these builds' scripts
locally when the trigger is manual: a dlt pipeline.py or a local PySpark
script ran, and a connector without the acquisition pattern is never
deployed. Calling them outside forge-cli reported success with nothing
running.

The python runner keeps them again and refuses a continuous trigger by name,
and fluid validate now warns about each, naming an ingestion engine's
missing pattern. fluid apply leaves a continuous build to its platform only
where forge-cli does not run it: inline SQL on a platform with no
embedded-SQL executor, and a spark, glue or custom build that names a
runtime.platform other than local (PLATFORM_RUN_ENGINES). A test now checks
that the python runner receives those local scripts, which the parity test
alone could not catch: it passed when both sides were silent.
…platform; when scaffold-composer names an old DAG
…me its platform

The round-3 review found the warning did not say what leaves such a build to
its platform: execution.runtime.platform. Its remedy now ends "or run the job
on a platform that keeps it running, named in execution.runtime.platform."
The CHANGELOG says the platform exception holds for a build without inline
SQL: inline SQL is checked first, and Snowflake has an executor.
@github-actions github-actions Bot added cli Changes to the CLI surface or implementation tests Test coverage or test infrastructure changes needs-docs Pull request needs a linked docs update or justification labels Oct 21, 2026
@github-actions

Copy link
Copy Markdown

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

  • Link a docs PR — check the "Docs PR linked" box and paste the URL
  • Mark as no docs needed — check "No docs needed" with a justification
  • Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli Changes to the CLI surface or implementation tests Test coverage or test infrastructure changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant