Skip to content

feat(validate): fluid validate <bundle>.tgz --strict fails on a retargeted overlay patch the bundle recorded - #735

Open
fas89 wants to merge 7 commits into
mainfrom
feat/bundle-strict-overlay-warnings
Open

fas89 wants to merge 7 commits into
mainfrom
feat/bundle-strict-overlay-warnings

Conversation

@fas89

@fas89 fas89 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #734, which made fluid validate --env <env> --strict fail on an overlay patch that lands on another entry (overlay_patch_retargeted) but left the bundle path out. Pipeline stage 2 runs exactly fluid validate <bundle> --env <env> --strict, and a bundle is never re-overlaid, so validate on the frozen bundle could not see the warning: it was only logged by fluid bundle, and the bundle passed.

What changes

  • fluid bundle --env merges the overlay inside loader.collect_overlay_warnings(), says each collected warning once, and records it in MANIFEST.json under source.overlayWarnings. The key is written only when there is a warning. The source block is outside the merkle root, so the bundle digest is unchanged (the sample bundle keeps main's sha256:6238f9…). For --format tgz it prints overlay warnings: N recorded in the bundle; ....
  • fluid validate <bundle>.tgz reports each recorded warning as a finding ([warning] overlay: MANIFEST.json: ..., code OVERLAY-WARNING), so --strict fails on it. It shows in --format json and --report too.
  • The bundle path now prints what the contract-file path prints. A passing bundle lists its warnings, not only their count. --quiet used to print nothing at all; it now prints ERROR: lines, a recorded overlay warning as WARNING:, and under --strict each warning as ERROR (--strict):. When warnings alone fail --strict, it says Validation failed: warnings treated as errors (--strict).
  • A bundle without the record (one built before this change) has none recorded and validates as before.

Two fixes the pre-PR defect scan found, as separate commits:

  • A collector block that raises now logs what it took. A patch retargeted before another overlay entry failed the merge was otherwise never shown. That was a regression in the first cut of fluid bundle, and it was already the case in feat: trigger.type continuous (0.7.6 preview) is never scheduled; --strict fails on a retargeted overlay patch #734's fluid validate --env and fluid contract-validation --env. The fix is in collect_overlay_warnings, so it covers all three.
  • A hand-edited record cannot drive the terminal. Editing overlayWarnings keeps the digest. Escape sequences in it could move the cursor or forge lines, and a lone surrogate crashed validate before it wrote --report. Non-printable characters are now escaped as repr writes them, as the block's other values are quoted with !r. _cprint_literal also passes highlight=False, like _cprint_json, because Rich's highlighter is quadratic in a run of <.

Not in this PR

  • fluid contract-validation <bundle> and fluid test <bundle> do not read the record. The CHANGELOG says so and points at the contract file with --env. Generated pipelines run those commands on the contract file.
  • Other bundle findings still print MANIFEST- or contract-derived text unescaped, for example the expected digests in MANIFEST-TAMPER. That is already the case on main.

Documentation

forge_docs#158 updates cli/validate.md (the bundle bullet under --env, "Validating a bundle", and OVERLAY-WARNING), cli/bundle.md (source.overlayWarnings) and concepts/environments-and-overlays.md. It needs review.

Also fixed: the overlay CLI tests under a leaked debug level

tests/cli/conftest.py sets FLUID_LOG_LEVEL=DEBUG, and some xdist workers keep it. At DEBUG the loader also logs a warning the command reports, so #734's TestValidateStrict tests counted it twice (assert 2 == 1). That is how they failed on main's CI for 1320619b. The CLI subprocesses in tests/test_overlay_keyed_merge.py now drop FLUID_LOG_LEVEL, FLUID_DEBUG and FLUID_SHOW_REGISTRATION, as tests/output_ports already does. The file passes with and without FLUID_LOG_LEVEL=DEBUG set.

Borrowed

  • The record follows SLSA Provenance v1, which keeps the artifact digest (subject) apart from run metadata (runDetails). The MANIFEST source block already had that split, so the record is metadata about the build and not bundle content.
  • SARIF was considered and not used: it is a full diagnostics format, and this is one list of strings in an advisory block.

Test plan

  • tests/test_overlay_keyed_merge.py:
    • TestValidateBundleStrict (11 tests), the reader unit test and a collector test. Every test that can fail did fail against the code before its fix: 8 against main, and the 4 gate-fix tests against the first cut. The other 2 pass on main by design: they are additivity guards (a keyed overlay records nothing, and a bundle without the record still passes --strict).
  • Ran the real CLI:
    • fluid bundle contract.fluid.yaml --env prod --format tgz with a positional 0.7.5 overlay, then fluid validate runtime/bundle.tgz --env prod --strict (the stage-2 command), exits 1 and names the patch.
    • Also checked plain, --quiet, --quiet --strict, --format json, --report, an env mismatch next to the warning, --out - YAML (stdout stays clean), hand-edited overlayWarnings shapes, and fluid plan + fluid apply --dry-run on a bundle that carries the record.
  • ruff check fluid_build/ tests/, black --check (24.10.0), lint-imports (6 kept) and scripts/mypy_strict.py all pass.
  • Full suite (-n 6, ~50 min at load avg ~124): 27,059 passed. The 10 failures and 2 errors are all environmental or already on main:
    • 6 fail the same way on a clean canonical/main export: the dlt allow-list tests (4) and the duckdb sandbox temp-root tests (2).
    • 2 collection errors: bashlex is not installed in the local venv.
    • 3 are load: two wall-clock perf budgets, and a Hypothesis "input generation is slow" health check.
    • 1 dotenv test passes in isolation; it depends on xdist ordering.
    • The local venv's sqlglot install is broken (a stale sqlglotc compiled extension). For the run it was shadowed with a working pure-Python sqlglot through PYTHONPATH.

fas89 added 4 commits October 21, 2026 21:44
…erlay patch the bundle recorded

A bundle is never re-overlaid, so `fluid validate <bundle>.tgz` could not see
an overlay patch that landed on another entry (overlay_patch_retargeted): only
`fluid bundle --env` logged it, and pipeline stage 2, which runs exactly
`fluid validate <bundle> --env <env> --strict`, passed.

`fluid bundle` now merges the overlay inside collect_overlay_warnings(), says
each warning once, and records the messages in MANIFEST.json as
source.overlayWarnings, only when there are any. The source block is outside
the merkle root, so the digest still names the contract alone. Validate
reports each one as a warning finding (overlay, MANIFEST.json,
OVERLAY-WARNING), so --strict fails on it, and --format json and --report list
it. A bundle that records none, one built before this change included,
validates as before.

The bundle path now prints what the contract path prints. A passing bundle
lists its warnings, not only their count. --quiet, which printed nothing,
prints ERROR: lines, a WARNING: line for a recorded overlay warning, and under
--strict an ERROR (--strict): line for each warning. When warnings alone fail
--strict, it says so.

Borrowed: the split SLSA Provenance v1 makes between the artifact digest
(subject) and run metadata (runDetails). The MANIFEST source block already has
that shape. Not SARIF, which is a full diagnostics format, for one list of
strings in an advisory block.
…s what it took

The caller reports the warnings a block took only when the block completes,
and the loader logs them at DEBUG only. So when another overlay entry failed
the merge after a patch had been retargeted, the retargeted patch was never
shown. `fluid bundle --env` printed the merge error alone, where main printed
both, and `fluid validate --env` and `fluid contract-validation --env` lost it
the same way. The block now logs what it took, as the loader logs it, before
the error propagates.
…terminal

The record is outside the merkle root, so editing it keeps the bundle digest.
An entry carrying escape sequences could move the cursor, erase or forge
lines, or write the clipboard in a permissive terminal. A lone surrogate made
validate crash before it wrote --report. Each character that is not printable
is now escaped as repr writes it, as the block's other values are quoted with
!r.

_cprint_literal also passes highlight=False, as _cprint_json does. Rich's
highlighter takes time quadratic in a run of `<` (4 s for 10,000), and the
record put such a run on a passing bundle's output.
@github-actions github-actions Bot added cli Changes to the CLI surface or implementation tests Test coverage or test infrastructure changes needs-docs Pull request needs a linked docs update or justification labels Oct 21, 2026
@github-actions

Copy link
Copy Markdown

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

  • Link a docs PR — check the "Docs PR linked" box and paste the URL
  • Mark as no docs needed — check "No docs needed" with a justification
  • Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.

Comment thread tests/test_overlay_keyed_merge.py Fixed
Comment thread fluid_build/cli/bundle.py Fixed
…test.py leaks

tests/cli/conftest.py sets FLUID_LOG_LEVEL=DEBUG, and some xdist workers keep
it. The CLI subprocesses here inherited it, so the loader's DEBUG copy of a
warning the command reports was printed too. The tests that count the warning
once then failed with `assert 2 == 1`, as #734's TestValidateStrict did on
main's CI run for 1320619. The subprocesses now drop FLUID_LOG_LEVEL,
FLUID_DEBUG and FLUID_SHOW_REGISTRATION, as tests/output_ports already do,
with API keys as before.
fas89 added 2 commits October 21, 2026 22:12
CodeQL read the assertion after `with pytest.raises(...)` as unreachable,
because the block's last statement raises. The block now calls a helper that
merges and then raises, and the test asserts the error message too. It still
checks that a warning taken inside the failing block is logged.
@github-actions github-actions Bot removed the needs-docs Pull request needs a linked docs update or justification label Oct 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli Changes to the CLI surface or implementation tests Test coverage or test infrastructure changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant