Repository navigation
ci: coverage on Python 3.14 (sys.monitoring) and Stage 1 tofu lanes in parallel - #770
Merged
Merged
Conversation
…ys.monitoring The Coverage & Audit job went from ~15 to 37-41 minutes after the fragment suite landed (#760): with `branch = true`, coverage.py on 3.12 can only use the C tracer, and that tracer took tests/fragments/test_split (390 in-process tests) from 6.7 to 31.3 minutes in CI. coverage.py's sys.monitoring core ("sysmon") measures branches only on 3.14+, where it is the default (https://coverage.readthedocs.io/en/latest/config.html). Measured locally on that module: 3.12 226s plain, 546s with coverage; 3.14 217s plain, 220s with coverage. The job now runs on 3.14 with COVERAGE_CORE=sysmon set (a fallback then at least logs no-sysmon), and its timeout drops from 45 to 30, under the ~40 min a fallback would take. Gate margins on main are 5 to 49 points (core 82/70, V1 85/80, local 80/50, cloud 69/20). Nothing outside ci.yml uses the job or artifact name; neither is a required check.
…t step under xdist Stage 1 ran its six tofu lanes one after another in a single job, about 27 minutes on main for a month: the job took the sum of them. Each lane runs its own files against moto or an in-process stand-in and asserts its own junit, so each is now a matrix job, and Stage 1 takes the longest lane plus setup. A lane shares one TF_PLUGIN_CACHE_DIR across its modules (the tests honour an external one), one provider download per lane instead of per module; a lane is one pytest process, so nothing writes that cache concurrently. The unit step ran ~7.4k tests in one process (five minutes); it now runs with `-n auto --dist loadscope`, as ci.yml's test matrix runs the same tests on every PR, under the same MALLOC_ARENA_MAX cap. The emulator extra stays out of that job, so it still cannot change what the unit step runs. Check names: "Stage 1 — unit + tofu validate" stays; the lanes report as "Stage 1 — <lane>". None is a required check.
📄 Documentation ReminderThis PR appears to be missing a documentation reference. Our docs live in a separate repo. Please update the PR description with one of:
See the Contributing Guide for details. |
Splitting Stage 1 into lanes moved `AWS_SECRET_ACCESS_KEY: test` from line 258 to 318, so the changed-files detect-secrets step failed on the stale line number. Regenerated with detect-secrets 1.5.0, the CI pin.
#769 added Databricks to Stage 1 as one more serial step. Here it is the seventh stage1-lanes matrix entry: the same file, junit and required-file label, so it runs beside the other lanes instead of after them. .secrets.baseline takes main's copy, regenerated with detect-secrets 1.5.0: the iac-tests.yml test credential moves from line 279 to 339.
fas89
added a commit
that referenced
this pull request
Oct 30, 2026
#771) #770 cut the coverage job's timeout to 30 minutes so that a fallback to the C tracer (~40 min) would fail instead of only slowing CI. GitHub enforces that timeout server-side, and on 8d8c64b it cancelled a run whose 30500 tests and coverage gates had passed at 21 minutes, after GitHub stopped registering the job's step updates. 45 minutes leaves room for that lag. The fallback is caught by a step instead: coverage.py's --debug=core logs the core it would use, and the step fails in a second unless it is sysmon. Same COVERAGE_CORE and [tool.coverage.run] config as the run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
CI on
mainslowed down after the fragment suite landed (#760). This PR changes two workflow files and the.secrets.baselineline number that moves with them. It makes no code changes and touches no required check.COVERAGE_CORE=sysmon(ci.yml).Why: measured
mainci.ymljob durations over time (minutes):The junit files from the last
mainrun (facb1dcf) show where the time goes:tests/fragments/test_splitdominates. It takes 6.7 minutes without coverage and 31.3 minutes with it. Under--dist loadscopethat one module sits on one worker, so it alone sets a floor of about 31 minutes on the coverage job.branch = true, coverage.py can use itssys.monitoringcore only on Python 3.14+, where it is the default. On 3.12 it falls back to the C tracer (coverage.py config docs,[run] core).Measured locally on
tests/fragments/test_split.py(390 tests, one process):On 3.14 the debug output reads "Using sysmon because SYSMON_DEFAULT is set". On 3.12 it reads "Defaulting to ctrace core". xdist with sysmon and branch measurement works through pytest-cov 7.1.
iac-tests Stage 1 has run about 27 minutes on
mainfor a month because its lanes run one after another in a single job. The lanes are independent: each runs its own test files and asserts its own junit (assert_lane_coverage.py). Run in parallel, Stage 1 takes as long as its longest lane, about 5.5 minutes, plus setup.Details
COVERAGE_CORE: sysmonis set explicitly. With the core left to its 3.14 default, a setting sysmon can't serve (a plugin, dynamic contexts) makes coverage fall back silently. Set explicitly, coverage.py 7.17 still falls back but logs ano-sysmonwarning (checked locally).main: core 82 vs 70, V1 85 vs 80, local 80 vs 50, cloud 69 vs 20.coverage-results-py3.14. Nothing outside ci.yml uses the job or artifact name..[dev,test-emulators]. The unit job never installs the emulator extra, so it can't change what the unit step runs (the reason the old file installed it after that step).TF_PLUGIN_CACHE_DIRacross its modules, because the tests honour an external one (env.setdefault). A lane is a single pytest process, so nothing writes the cache concurrently.-n auto --dist loadscopeunderMALLOC_ARENA_MAX=2, as ci.yml's test matrix does for the same tests.Borrowed
sys.monitoringcore (nedbat/coveragepy, 7.17.0 already installed).TF_PLUGIN_CACHE_DIR.--dist worksteal. It rebalances long modules, but once coverage costs almost nothing,test_splitis no longer the long pole.coverage combine. Heavier than switching the core.Validation
actionlintis clean on both files. The lane shell logic (argument splitting,--requirelist, empty args) was checked in bash.This PR's CI runs use the new workflows. Measured on 2026-10-30 against #769's CI, which ran
main'sci.yml:ci.yml, whole runCoverage now takes about as long as one Test job, so the slowest Test job (~20 minutes) sets how long CI takes. Test jobs ranged from 11 to 25 minutes across these runs, with runner load.
#769
#769 merged first (97d0699) and added Databricks to Stage 1 as one more serial step. The merge from
main(88ac7c6) makes it the seventh matrix entry, with the same test file, junit file and required-file label..secrets.baselinetakesmain's copy, regenerated with detect-secrets 1.5.0.