Skip to content

ci: coverage on Python 3.14 (sys.monitoring) and Stage 1 tofu lanes in parallel - #770

Merged
fas89 merged 4 commits into
mainfrom
ci/coverage-sysmon-and-iac-lanes
Oct 30, 2026
Merged

fas89 merged 4 commits into
mainfrom
ci/coverage-sysmon-and-iac-lanes

Conversation

@fas89

@fas89 fas89 commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

What

CI on main slowed down after the fragment suite landed (#760). This PR changes two workflow files and the .secrets.baseline line number that moves with them. It makes no code changes and touches no required check.

  1. Coverage & Audit runs on Python 3.14 with COVERAGE_CORE=sysmon (ci.yml).
  2. iac-tests Stage 1 runs its seven tofu lanes, feat(databricks): Unity Catalog schemas, Delta tables and grants through OpenTofu (Phase 1) #769's Databricks lane included, as parallel matrix jobs, and runs its unit step under xdist (iac-tests.yml).

Why: measured

main ci.yml job durations over time (minutes):

main run Coverage & Audit Test (3.10–3.14)
2026-09-18 8 5–8
2026-10-15 14 7–11
2026-10-28, after #760 29–41 11–20

The junit files from the last main run (facb1dcf) show where the time goes:

  • Coverage inflates the suite's summed test time about 3.2×, from 42.6 to 137 minutes.
  • tests/fragments/test_split dominates. It takes 6.7 minutes without coverage and 31.3 minutes with it. Under --dist loadscope that one module sits on one worker, so it alone sets a floor of about 31 minutes on the coverage job.
  • The cause is the tracer core. With branch = true, coverage.py can use its sys.monitoring core only on Python 3.14+, where it is the default. On 3.12 it falls back to the C tracer (coverage.py config docs, [run] core).

Measured locally on tests/fragments/test_split.py (390 tests, one process):

plain with branch coverage
Python 3.12 (C tracer) 226s 546s (2.4×)
Python 3.14 (sysmon) 217s 220s (1.01×)

On 3.14 the debug output reads "Using sysmon because SYSMON_DEFAULT is set". On 3.12 it reads "Defaulting to ctrace core". xdist with sysmon and branch measurement works through pytest-cov 7.1.

iac-tests Stage 1 has run about 27 minutes on main for a month because its lanes run one after another in a single job. The lanes are independent: each runs its own test files and asserts its own junit (assert_lane_coverage.py). Run in parallel, Stage 1 takes as long as its longest lane, about 5.5 minutes, plus setup.

Details

  • Coverage job
    • COVERAGE_CORE: sysmon is set explicitly. With the core left to its 3.14 default, a setting sysmon can't serve (a plugin, dynamic contexts) makes coverage fall back silently. Set explicitly, coverage.py 7.17 still falls back but logs a no-sysmon warning (checked locally).
    • The timeout drops from 45 to 30 minutes. That's under the ~40 minutes a fallback would take, so a fallback fails the job instead of quietly slowing it.
    • Gate margins on main: core 82 vs 70, V1 85 vs 80, local 80 vs 50, cloud 69 vs 20.
    • The artifact is renamed coverage-results-py3.14. Nothing outside ci.yml uses the job or artifact name.
  • Stage 1 lanes
    • Each lane installs .[dev,test-emulators]. The unit job never installs the emulator extra, so it can't change what the unit step runs (the reason the old file installed it after that step).
    • Each lane shares one TF_PLUGIN_CACHE_DIR across its modules, because the tests honour an external one (env.setdefault). A lane is a single pytest process, so nothing writes the cache concurrently.
    • The unit step runs -n auto --dist loadscope under MALLOC_ARENA_MAX=2, as ci.yml's test matrix does for the same tests.
  • Check names. "Stage 1 — unit + tofu validate" stays. The lanes report as "Stage 1 — ". None of these is a required check.

Borrowed

  • Depend: coverage.py's sys.monitoring core (nedbat/coveragepy, 7.17.0 already installed).
  • Adapt: GitHub Actions matrix fan-out, and OpenTofu's TF_PLUGIN_CACHE_DIR.
  • Considered, not needed:
    • pytest-xdist --dist worksteal. It rebalances long modules, but once coverage costs almost nothing, test_split is no longer the long pole.
    • pytest-split shards with coverage combine. Heavier than switching the core.

Validation

actionlint is clean on both files. The lane shell logic (argument splitting, --require list, empty args) was checked in bash.

This PR's CI runs use the new workflows. Measured on 2026-10-30 against #769's CI, which ran main's ci.yml:

before: #769's CI this PR, run 1 this PR, run 2
Coverage & Audit 40m54s (3.12, C tracer) 21m26s (3.14, sysmon) 12m09s
ci.yml, whole run 41m 21m31s 20m39s
iac-tests 27m07s, Stage 1 alone 6m35s, whole workflow 6m39s

Coverage now takes about as long as one Test job, so the slowest Test job (~20 minutes) sets how long CI takes. Test jobs ranged from 11 to 25 minutes across these runs, with runner load.

#769

#769 merged first (97d0699) and added Databricks to Stage 1 as one more serial step. The merge from main (88ac7c6) makes it the seventh matrix entry, with the same test file, junit file and required-file label. .secrets.baseline takes main's copy, regenerated with detect-secrets 1.5.0.

fas89 added 2 commits October 30, 2026 17:29
…ys.monitoring

The Coverage & Audit job went from ~15 to 37-41 minutes after the
fragment suite landed (#760): with `branch = true`, coverage.py on 3.12
can only use the C tracer, and that tracer took tests/fragments/test_split
(390 in-process tests) from 6.7 to 31.3 minutes in CI. coverage.py's
sys.monitoring core ("sysmon") measures branches only on 3.14+, where it
is the default (https://coverage.readthedocs.io/en/latest/config.html).

Measured locally on that module: 3.12 226s plain, 546s with coverage;
3.14 217s plain, 220s with coverage. The job now runs on 3.14 with
COVERAGE_CORE=sysmon set (a fallback then at least logs no-sysmon), and
its timeout drops from 45 to 30, under the ~40 min a fallback would
take. Gate margins on main are 5 to 49 points (core 82/70, V1 85/80,
local 80/50, cloud 69/20). Nothing outside ci.yml uses the job or
artifact name; neither is a required check.
…t step under xdist

Stage 1 ran its six tofu lanes one after another in a single job, about
27 minutes on main for a month: the job took the sum of them. Each lane
runs its own files against moto or an in-process stand-in and asserts
its own junit, so each is now a matrix job, and Stage 1 takes the
longest lane plus setup. A lane shares one TF_PLUGIN_CACHE_DIR across its
modules (the tests honour an external one), one provider download per
lane instead of per module; a lane is one pytest process, so nothing
writes that cache concurrently.

The unit step ran ~7.4k tests in one process (five minutes); it now runs
with `-n auto --dist loadscope`, as ci.yml's test matrix runs the same
tests on every PR, under the same MALLOC_ARENA_MAX cap. The emulator
extra stays out of that job, so it still cannot change what the unit
step runs. Check names: "Stage 1 — unit + tofu validate" stays; the
lanes report as "Stage 1 — <lane>". None is a required check.
@github-actions github-actions Bot added ci Continuous integration and automation changes needs-docs Pull request needs a linked docs update or justification labels Oct 30, 2026
@github-actions

Copy link
Copy Markdown

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

  • Link a docs PR — check the "Docs PR linked" box and paste the URL
  • Mark as no docs needed — check "No docs needed" with a justification
  • Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.

Splitting Stage 1 into lanes moved `AWS_SECRET_ACCESS_KEY: test` from
line 258 to 318, so the changed-files detect-secrets step failed on the
stale line number. Regenerated with detect-secrets 1.5.0, the CI pin.
@github-actions github-actions Bot added the security Security-related changes or reports label Oct 30, 2026
#769 added Databricks to Stage 1 as one more serial step. Here it is the
seventh stage1-lanes matrix entry: the same file, junit and required-file
label, so it runs beside the other lanes instead of after them.

.secrets.baseline takes main's copy, regenerated with detect-secrets
1.5.0: the iac-tests.yml test credential moves from line 279 to 339.
@fas89
fas89 merged commit 8d8c64b into main Oct 30, 2026
51 checks passed
fas89 added a commit that referenced this pull request Oct 30, 2026
#771)

#770 cut the coverage job's timeout to 30 minutes so that a fallback to
the C tracer (~40 min) would fail instead of only slowing CI. GitHub
enforces that timeout server-side, and on 8d8c64b it cancelled a run
whose 30500 tests and coverage gates had passed at 21 minutes, after
GitHub stopped registering the job's step updates. 45 minutes leaves
room for that lag.

The fallback is caught by a step instead: coverage.py's --debug=core
logs the core it would use, and the step fails in a second unless it
is sysmon. Same COVERAGE_CORE and [tool.coverage.run] config as the run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Continuous integration and automation changes needs-docs Pull request needs a linked docs update or justification security Security-related changes or reports

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant