Repository navigation
Conversation
…runs as a batch run A continuous trigger (schema 0.7.6) says the job's platform keeps it running, so no scheduler starts the build. forge-cli runs dbt, python, inline-SQL and duckdb, airbyte, meltano and dlt acquisition builds to completion: the dbt and python runners refuse the trigger, so such a build never runs, and the others run once per fluid apply. fluid validate now says so as a warning, so plain validate passes and --strict fails: builds[0] (id 'transform'): trigger.type continuous on engine 'dbt': forge-cli runs dbt builds as batch runs, which end, and its dbt runner refuses a continuous trigger: forge-cli never runs this build. Use trigger.type schedule, or run the job on a platform that keeps it running. The check follows run_builds_from_args' dispatch order, so it names the runner that gets the build: a dbt build with inline SQL runs on DuckDB. The Kafka Connect and Debezium connectors, whose runners declare RunnerCapability.STREAMING, and builds that run outside forge-cli get no warning, and neither do spark, glue and custom builds: forge-cli has no runner of its own for them, and their platform may keep them running. A crash in the check is a warning that says the contract was not checked for it. Borrowed: Spark's UnsupportedOperationChecker.checkForContinuous checks a continuous trigger against what the query can run before it starts, and Databricks refuses Trigger.Continuous where it cannot keep a job running (INFINITE_STREAMING_TRIGGER_NOT_SUPPORTED), naming the trigger.
…ntinuous by name They printed "Unknown trigger type: continuous" and "Supported types: manual, schedule", as for a typo. They now say that forge-cli runs the build as a batch run, which ends, while a continuous trigger says the job's platform keeps it running, and what to use instead. The exit code stays 1. The dbt runner refuses before it resolves a profile or prepares the dbt command.
…d the cron of a scheduled build fluid scaffold-composer wrote a DAG for any contract: one whose every build is continuous or runs outside forge-cli got a nightly 0 2 * * * DAG, or the cron a Flink job declares for its own platform. It now writes no DAG for such a contract and says so, as fluid generate schedule does, and names a DAG it wrote before, which it leaves in place. Otherwise the DAG takes the cron of the first build a scheduler starts. has_only_unscheduled_builds() moves to util/contract.py, so both commands decide with one predicate.
…d-composer schedules only scheduled builds
Rich folded a path longer than the terminal across lines, so the path of a DAG written before came out broken where it is meant to be copied. The line is printed with soft_wrap, and the terminal wraps it.
…ous build
A defect scan found the warning and the dispatcher disagreeing. A continuous
spark, glue or custom build, or an ingestion engine declared without
pattern: acquisition, got no warning, while fluid apply --mode amend-and-build
handed it to the python runner, which refused it. Inline SQL on a platform
with no embedded-SQL executor, such as databricks, was warned as running once
per apply, while the executor refuses that platform.
Now one predicate decides both. A continuous build that no forge-cli runner
takes is left to its platform, as a Flink job is: the dispatcher prints that
there is nothing for forge-cli to run, counts it under "Run outside forge-cli"
and records it as skipped with reason runs_outside_forge, and validate gives
no warning. A build with no engine goes to the python runner, so it is warned
like a python build. EMBEDDED_SQL_PLATFORMS is the one list of the platforms
that run inline SQL, read by the executor and by the warning. A test runs the
dispatcher over every engine and pattern and checks that validate warns
exactly when a batch runner gets the build.
The dbt and python runners print one refusal, refuse_continuous_trigger, and
it gives the same remedy as the warning ("Use trigger.type schedule or
manual, ..."). The run-once warning says the build runs only when fluid apply
runs builds: plain fluid apply does not run an acquisition. The --strict
output assertion no longer depends on the terminal width.
…r declared tasks The no-DAG message told the operator to delete any file at <out-dir>/<dag_id>.py, and dag_id is a lossy clean-up of the contract id, so it could name another team's DAG. It now names the file only when it carries the template's tags=["FLUID"] and this dag_id, as the current template quotes it or as the template before #215 did; a file it did not write is not mentioned. A contract whose every build is unscheduled but which hand-declares orchestration.tasks still gets its DAG, as fluid generate schedule still writes files for one. The path prints with emoji=False beside markup=False, so an :emoji: code in --out-dir prints as written.
…old-composer keeps DAGs for declared tasks
…inuous or not The previous commit left a continuous spark, glue or custom build, and an ingestion engine declared without pattern: acquisition, to "its platform", and fluid apply exited 0. But the python runner runs these builds' scripts locally when the trigger is manual: a dlt pipeline.py or a local PySpark script ran, and a connector without the acquisition pattern is never deployed. Calling them outside forge-cli reported success with nothing running. The python runner keeps them again and refuses a continuous trigger by name, and fluid validate now warns about each, naming an ingestion engine's missing pattern. fluid apply leaves a continuous build to its platform only where forge-cli does not run it: inline SQL on a platform with no embedded-SQL executor, and a spark, glue or custom build that names a runtime.platform other than local (PLATFORM_RUN_ENGINES). A test now checks that the python runner receives those local scripts, which the parity test alone could not catch: it passed when both sides were silent.
…platform; when scaffold-composer names an old DAG
…me its platform The round-3 review found the warning did not say what leaves such a build to its platform: execution.runtime.platform. Its remedy now ends "or run the job on a platform that keeps it running, named in execution.runtime.platform." The CHANGELOG says the platform exception holds for a build without inline SQL: inline SQL is checked first, and Snowflake has an executor.
📄 Documentation ReminderThis PR appears to be missing a documentation reference. Our docs live in a separate repo. Please update the PR description with one of:
See the Contributing Guide for details. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Follow-ups #734 documented instead of fixing:
fluid validate, the build runners andfluid scaffold-composerdid not knowtrigger.type: continuous.A continuous trigger on a build forge-cli runs as a batch run is a validation warning (the user's decision: warn, not refuse). A continuous trigger says the job's platform keeps it running, so no scheduler starts the build; forge-cli's batch runners run a build to completion. Plain
fluid validatepasses and--strictfails:fluid validateandfluid applyread one predicate (build_runners/base.py::_batch_runner_for), in the dispatcher's order:fluid apply --mode amend-and-buildfluid validatedbt,dbt-<adapter>,python, and everything else the python runner takes: no engine, a localspark/glue/customscript, an ingestion engine withoutpattern: acquisitionpattern: acquisition)duckdb/airbyte/meltano/dltacquisitionkafka-connect/debeziumacquisitionengine: flink,language: flink_sql)spark/glue/customthat names aruntime.platformother thanlocal; inline SQL on a platform with no embedded-SQL executorRun outside forge-cli: 1, run reportskipped/runs_outside_forge); used to reach the python runner or the SQL executor and faildeclared_capabilities(RunnerCapability.STREAMING). A parity test runs the real dispatcher over every engine and pattern, and another checks that the python runner still receives every local script.EMBEDDED_SQL_PLATFORMSis the one list the inline-SQL executor and the warning read.refuse_continuous_trigger) with the warning's remedy, instead ofUnknown trigger type: continuous/Supported types: manual, schedule. The dbt runner refuses before it resolves a profile.fluid scaffold-composerwrites no DAG for a contract whose every build is continuous or runs outside forge-cli, unless it declaresorchestration.tasks(the testfluid generate scheduleapplies, through the sharedutil/contract.has_only_unscheduled_builds), and says so. It names a DAG it wrote before (recognised by the template'stags=["FLUID"]and thedag_id; a same-named file it did not write is not claimed) and leaves it in place; the path prints whole and literally (soft_wrap,markup=False,emoji=False). Otherwise the DAG takes the cron of the first build a scheduler starts, not a Flink job's own cron.Borrowed, not built
UnsupportedOperationChecker.checkForContinuous: a continuous trigger is checked against what the engine can run before anything starts, and the refusal names what it cannot run.INFINITE_STREAMING_TRIGGER_NOT_SUPPORTED: where a job cannot be kept running, the continuous trigger is refused by name, with the alternative.is_unscheduled_build/is_continuous_build(feat: trigger.type continuous (0.7.6 preview) is never scheduled; --strict fails on a retargeted overlay patch #734), the dispatch predicates and_report_externally_run_buildinbuild_runners/base.py, the runners'declared_capabilities, the(errors, warnings)validator shape of the Iceberg / governance gates, andgenerate schedule's "nothing to schedule". Strategy: adapt the pattern; no new dependency.Validation
tests/test_continuous_trigger.py26 → 107. Each round's fix was red first: the stale-DAG path fold, the:emoji:/[/]out-dir, the dispatcher parity (6 engine shapes), inline SQL on databricks, the explicit-tasks DAG, the foreign same-named file, the python-runner handoff (10 tests red on the round-2 commit). Every commit is green on its own.scripts/mypy_strict.py, import-linter (6 kept), license headers, product guardrail (0 failing). Affected suites (58 files/dirs, ~5.9k tests): failures equal cleanmainid-for-id in this venv (19 failed + 10 errors: dlt SDK, sqlglot, tmp-root naming).tests/perf: the module budget passes (1018 modules on both trees); the 3 wall-clock budgets fail on cleanmaintoo at load average ~145.examples/customer360andexamples/02-csv-to-data-product, plus python/spark/no-engine contracts):validateexit 0 with the warning,--strictexit 1,--quiet --strictnames it,--format jsonlists it, the bundle report counts it;apply --mode amend-and-buildrefuses before any profile (exit 1).applyand--mode amend-and-buildeach run it once.pattern: acquisition: warned (--strictexit 1), refused (exit 1), the script never runs. Continuous spark onruntime.platform: databricks: no warning,apply --mode amend-and-buildexit 0,Run outside forge-cli: 1.scaffold-composer: Flink-only or continuous-only contract → no DAG, no--out-dir; after a scheduled run wrote a DAG, the next run names it and keeps it; a foreign same-named file is not named; a 0.7.5flink_sql+ dbt contract takes the dbt cron (0 3, was the Flink0 5);generate scheduleunchanged.ripwire --quality-delta: gating rows left are_run_contract_rulesgrowing by one rule block in its existing pattern, and per-module test helpers flagged as duplication (house style).Reviews
trigger.typewith Rich markup on.main, and every continuous build is warned exactly when a batch runner takes it. Two LOW wording items fixed: the CHANGELOG's platform exception excludes inline SQL, and the warning for spark / glue / custom namesexecution.runtime.platform.Documentation
Not in this PR
fluid scaffold-composerreads onlytrigger.cron, nottrigger.schedule(documented on its page); it overwrites a colliding file it did not write.trigger.type(theTrigger:andUnknown trigger typelines) with Rich markup on.fluid verifystill expects a dbt run record for a continuous dbt build, whichfluid applyrefuses.fluid validateworkspace mode runs no contract rules (documented), so it does not show this warning either.