Shared state gets corrupted when running scenarios in parallel, flaking CI

[jeffcpullen]

Summary

Molecule can run a collection's scenarios in parallel with --workers. When those scenarios also share state — shared_state: true, which the parallel-testing guide recommends — every scenario running at the same time reads and writes the same small state file, and molecule does not protect that file against simultaneous access. The parallel runs collide on it, and two things go wrong:

1. A run crashes. One scenario reads the state file just as another is partway through rewriting it, gets a half-written file, and molecule aborts with a YAML parsing error. This is the intermittent test_workers failure on the py314-milestone CI job.
2. A run silently loses data. Each scenario rewrites the whole file from its own in-memory copy, so one scenario can overwrite a value another just wrote — no error, just a lost update.

This is not specific to Python 3.14; that version's timing exposes the crash more often, but the cause is the same everywhere.

Underneath, the state file is written non-atomically (it is truncated, then rewritten) and updated by reading the whole file, changing one field, and writing it all back — safe for a single process, unsafe for several at once. The exact code paths, and small scripts that reproduce both problems deterministically, are below.

Scope

The parallel-testing guide (docs/guides/parallel.md:30-32) tells maintainers to turn on shared_state: true when using --workers, and --workers is the recommended replacement for the deprecated --parallel. So the corruption happens on the configuration the docs point people to. It has to be turned on deliberately — --workers defaults to 1 — but that is exactly the step the guide describes for running a collection's scenarios in parallel.

Recommendation (phased)

1 — Immediate: make the state write atomic. Write to a temp file in the same directory and os.replace() onto state.yml, so a reader always sees a complete old-or-new document. This removes defect 1 (verified below: 104/9000 corrupt reads → 0/9000, no other change) and is safe to ship now — os.replace is atomic on POSIX and Windows, it changes only how the file is written (not its schema or semantics), and it can be scoped to the state write alone. It does not fix defect 2; it is a first step, not the design fix.

2 — The real fix: set up shared infrastructure once, then give each worker its own state. Molecule already runs the shared create and destroy once, centrally, for a parallel run. What's left racing is prepare (forced to run in every worker) and the per-scenario flags being written to the one shared file at the same time. The fix I'd suggest: stand the shared infrastructure up once before the parallel run begins, and let each worker keep its own state file for the phases that actually run in parallel (converge, idempotence, side_effect, verify). Then nothing is shared and written at the same time, and both problems go away. One real design decision sits inside this — who runs prepare — which is why I'm raising it here rather than sending a patch (see below).

3 — Long-term: per-group shared infrastructure. The original RFE also envisioned grouping scenarios that share infrastructure. That half was never built (today shared_state is a project-wide boolean). It's a larger RFE with real design questions, and it overlaps an open PR that adds --slice; noted here only so the short-term fix leaves room for it, not to scope it in.

---

How this came to be (feature history)

• RFE: Shared state between scenarios #4001 (RFE: Shared state between scenarios, closed) — the original request. It frames the goal as sharing infrastructure so scenarios can be split up without re-provisioning, and it already sketches the intended pattern for using shared state in parallel — provision once through a single scenario:

  Care will need to be taken by the user if running scenarios in parallel that share state as a race condition might occur during provisioning. The documented approach might suggest something like running molecule prepare prior to running parallel scenarios, effectively using the default scenario as the provisioner.

  That provision-once pattern is the direction of the step-2 fix below.

• Add support for shared inventory between scenarios #4443 (Add support for shared inventory between scenarios, merged) — shared inventory, closes RFE: Shared inventory directory #4002.

• Share state between molecule scenarios #4470 (Share state between molecule scenarios, merged) — adds SharedState backed by the shared ephemeral directory; closes RFE: Shared state between scenarios #4001. Shipped as a single boolean (shared_state / --shared-state), project-wide.

• feat: add --workers flag for concurrent scenario execution #4616 (add --workers for concurrent scenario execution, merged) — runs scenarios in parallel via a ProcessPoolExecutor. This is where the shared state.yml starts being read and written concurrently.

• feat: add --slice flag for worker scenario grouping #4617 (add --slice, open) — related, relevant to the long-term grouping question below.

• Reproducible in-tree: the test_workers integration suite runs molecule test --all --workers 2 against a fixture whose base config sets shared_state: true (tests/fixtures/integration/test_workers/extensions/molecule/config.yml:13, inherited by all six otherwise-empty scenario molecule.yml files). So molecule's own CI exercises exactly the shared_state + --workers combination.

Related:

• fix: report a clear error for unparseable molecule.yml instead of a traceback #4664 and fix: remove git-ignore-based scenario filter #4665 — where the py314-milestone flake was observed.
• shared_state does not share instance_config, inventory or ephemeral directory across scenarios #4588 — a different problem (shared state sharing too little: instance_config, inventory, ephemeral dir). This issue is the opposite: the state that is shared is written unsafely.

Reproduction

Environment: molecule 26.6.1.dev9, Python 3.14.6 (the py314-milestone interpreter; py3.14 timing makes the race hit more often, but it is not py3.14-specific).

Both scripts drive molecule's own functions — no mocks — and are deterministic on a normal multi-core machine.

Defect 1 — partial-read crash (script + transcript)

The script runs several processes that concurrently util.write_file(state.yml, safe_dump(...)) / util.safe_load_file(state.yml) on one shared path — exactly the pair State._write_state_file / State._load_file use.

import multiprocessing as mp, sys, tempfile
from collections import Counter
from pathlib import Path
from molecule import util

STATE = {"converged": False, "created": True, "driver": "default", "prepared": True,
         "molecule_yml_date_modified": None, "run_uuid": "0"*36, "is_parallel": True,
         **{f"note_{i:02d}": f"scenario detail: step {i} " + "y"*120 for i in range(40)}}

def writer(path, n):
    for _ in range(n):
        util.write_file(Path(path), util.safe_dump(STATE))

def reader(path, n):
    p, out = Path(path), Counter()
    for _ in range(n):
        try:
            data = util.safe_load_file(p)
        except Exception as exc:
            out[f"{type(exc).__name__}: {str(exc).splitlines()[0][:40]}"] += 1; continue
        if data is None: out["EMPTY_READ(None)"] += 1
        elif not isinstance(data, dict) or data.get("run_uuid") != STATE["run_uuid"]:
            out["TRUNCATED_DICT(missing keys)"] += 1
        else: out["ok"] += 1
    return out

def main():
    iters, workers = 3000, 6
    with tempfile.TemporaryDirectory() as d:
        path = str(Path(d) / "state.yml")
        util.write_file(Path(path), util.safe_dump(STATE))
        ctx = mp.get_context("fork"); outcomes = Counter()
        with ctx.Pool(workers) as pool:
            jobs = [("w" if i % 2 == 0 else "r",
                     pool.apply_async(writer if i % 2 == 0 else reader, (path, iters)))
                    for i in range(workers)]
            for kind, j in jobs:
                r = j.get()
                if kind == "r": outcomes.update(r)
    for k, v in outcomes.most_common(): print(f"  {k}  x{v}")
    corrupt = sum(v for k, v in outcomes.items() if k != "ok")
    print(f"{'CORRUPTION' if corrupt else 'clean'}: {corrupt} corrupt read(s)")
    return 1 if corrupt else 0

if __name__ == "__main__": raise SystemExit(main())

  ok  x8896
  TRUNCATED_DICT(missing keys)  x102
  MoleculeError: while scanning a quoted scalar  x2
CORRUPTION: 104 corrupt read(s)

104 of 9000 reads saw a partial document, including a genuine yaml.scanner.ScannerError — the same failure class as the CI could not find expected ':' (the exact scanner message varies with where the truncation lands).

With the atomic write applied (temp file + os.replace, described below), the identical run reports ok x9000 — 0 corrupt reads.

Defect 2 — silent lost update (script + transcript)

This models State's real write path: load the shared file once into an in-memory snapshot, mutate one key, write the whole snapshot back — never re-reading. Two workers start from the same seed and each set a different flag; the writes are deliberately serialized so there is no tearing at all.

import multiprocessing as mp, tempfile, time
from pathlib import Path
from molecule import util

def worker(path, key, barrier, delay):
    p = Path(path)
    snap = util.safe_load_file(p) or {}   # load once (as State.__init__ does)
    barrier.wait()                         # both snapshot the same seed
    time.sleep(delay)                      # force a write order; no torn write
    snap[key] = True                       # mutate one key in memory
    util.write_file(p, util.safe_dump(snap))  # write the WHOLE snapshot back

def main():
    with tempfile.TemporaryDirectory() as d:
        path = str(Path(d) / "state.yml")
        util.write_file(Path(path), util.safe_dump(
            {"created": False, "converged": False, "run_uuid": "seed"}))
        ctx = mp.get_context("fork"); b = ctx.Barrier(2)
        a = ctx.Process(target=worker, args=(path, "created", b, 0.0))
        c = ctx.Process(target=worker, args=(path, "converged", b, 0.2))
        a.start(); c.start(); a.join(); c.join()
        final = util.safe_load_file(Path(path))
    print(f"final: created={final['created']}  converged={final['converged']}")
    return 1 if not final["created"] else 0

if __name__ == "__main__": raise SystemExit(main())

final: created=False  converged=True

Worker A set created=True and wrote first; worker B set converged=True and wrote second. No write was torn — B's whole-snapshot write, taken before A's write existed, simply overwrote created back to False.

Defect 1 — non-atomic write

• Write path: State._write_state_file (state.py:234) → util.write_file → Path.write_text(content) (util.py:222). This truncates the file, then writes — a window in which a concurrent reader sees an empty or partial document.
• Read path: State._load_file (state.py:231) → util.safe_load_file → util.safe_load, which catches yaml.scanner.ScannerError and re-raises it as MoleculeError (util.py:285-286).

Defect 2 — whole-file overwrite from a stale snapshot

State loads the file into memory once in __init__ (state.py:114); every mutation goes through change_state → @marshal (state.py:81-83, 194-209), which sets one key in memory and dumps the entire in-memory dict back. A worker never re-reads before writing, so two workers each writing their own snapshot lose each other's changes.

In the current worker flows this is largely latent: prepare is force-run (so the cross-process prepared read is skipped), create/destroy are skipped in workers, and converged is written and read within one process. Today the lost update mostly surfaces as a molecule_yml_date_modified mismatch warning rather than a functional failure. It is still a correctness bug on the shared code path, which is why the atomic write (defect 1 only) is a first step and not the whole fix.

Why the atomic write is safe to ship on its own

• Atomicity is guaranteed on the same filesystem. The temp file is created in the state file's own directory, so os.replace is a same-filesystem rename — atomic on POSIX (rename(2)) and Windows. A reader opening the path gets either the old complete file or the new complete file, never a truncated one.
• Nothing else changes. The bytes written are identical (safe_dump output with the existing header); only the write mechanism changes. Readers, schema, and call sites are untouched.
• It can be scoped narrowly — to the state write (State._write_state_file, or a small util.atomic_write_file used only there) rather than to every write_file caller — keeping the blast radius to the file that has the race.
• Measured: the defect-1 repro goes from 104/9000 corrupt reads to 0/9000 with only this change.

Caveat: if a process is killed between creating the temp file and the rename, a stray temp file can be left in the directory. Using a recognizable prefix and unlinking on error limits that to a leftover temp file.

The design fix, and the one question I'd like maintainer input on

Beyond the crash, the durable fix is to stop concurrently mutating one shared state file under --workers. Per-scenario state already exists and is the default (State._get_state_file → scenario.ephemeral_directory/state.yml); SharedState is the opt-in that collapses every scenario onto one file. Provisioning is already delegated and run once centrally in the parallel path (create/destroy bracket the pool). So the shape I'd propose: provision once up front, then let each worker use its own per-scenario state.yml for the phases that run in parallel. No shared mutable state during the parallel window.

The decision that isn't mine to make is who runs prepare. prepare is overloaded: there is infra-level preparation of the shared instances (belongs with the single provisioner, run once) and per-scenario preparation of a scenario's own view (its own prepare.yml — a real, used feature). Today --workers sidesteps the distinction by force-running prepare everywhere. A correct fix has to separate the two: run the owner's infra prepare once, and still run each scenario's own prepare in its worker. I didn't want to encode one interpretation of that in a patch without maintainers weighing in — hence this issue rather than a PR for step 2.

The long-term picture (for context, not this issue)

The RFE's second use case — grouping scenarios that share infrastructure — was never implemented; shared_state is a project-wide boolean and the shared directory is keyed only on the project (scenario.py). Fully realizing it means a schema to associate a scenario with a named shared-infra group, keying the shared directory per group, and a per-group owner/lifecycle, with real design questions (group ownership of create/destroy, partial-group failure, interaction with --workers). It also overlaps an open PR adding --slice, which proposes another way to group scenarios for workers; the two would need to compose rather than duplicate. That's a separate, larger RFE — mentioned so the near-term fix does not preclude it.

Environment

• molecule 26.6.1.dev9
• Python 3.14.6
• Reproduced deterministically via the scripts above (they drive molecule.util / molecule.state directly).

[jeffcpullen]

Quick status so the remaining scope stays in one place.

Defect 1 is handled now, atomic write in #4667 and reload-before-write in #4694 (merged).

What's left is the shared-file contention under --workers. I'm working that as part of a broader fix rather than a standalone patch, so I'd hold off on a cross-process lock for now, it may well end up redundant. Happy to coordinate here if anyone wants to pick up a piece.

[jeffcpullen]

One root cause behind several distinct failures

This issue and its follow-ups have centered on state.yml corruption under --workers. In tracing the root cause I found that the same underlying problem also corrupts a scenario's inventory and config, not just state.yml, so the surface reported here is one of several.

The root cause is a single shared directory. When shared_state is on, every scenario's ephemeral working directory collapses to one path, so all scenarios read and write the same files. Most of the failures below are that collapse showing up on a different file. Giving each scenario its own working directory again resolves the whole set of directory collisions in one change. The state.yml concurrency failures are a separate family that the two merged state-file PRs already address. I have kept them in the tables so the full landscape and the boundary of the directory fix are both visible.

The files in a scenario's working directory

A scenario's ephemeral directory holds these files. Three are private to a scenario by design and only end up shared by accident of the collapsed path. Two are shared on purpose, because that sharing is what shared_state exists to provide.

File	What it is	Meant to be shared across scenarios?
hosts, group_vars/, host_vars/, ansible_inventory.yml (the inventory/ subtree)	The inventory the scenario's play runs against: which machines, and the variables for them	No
molecule.yml	The scenario's own configuration (platforms, provisioner settings)	No
ansible.cfg	The Ansible settings the scenario's ansible-playbook runs under	No
state.yml	Lifecycle flags (created, converged, prepared) that drive instance reuse	Yes, this is what lets a later scenario skip re-creating machines
instance_config.yml	The list of running machines and how to connect to them	Yes, this is the shared-instances feature

The failure modes

One row per distinct failure found under this issue, keyed by a short code for reference. Reading the columns: Fail means the failure occurs with that shared_state setting, Pass means it does not.

Code	Files it affects	shared_state = false	shared_state = true	What goes wrong
VARS-CLOBBER	group_vars/ host_vars/	Pass	Fail	Each scenario deletes and rewrites the shared inventory vars before every action, so one scenario's variables silently replace a sibling's and the sibling's data is lost. Deterministic, needs no --workers.
VARS-CRASH	group_vars/ host_vars/	Pass	Fail	The timing version of VARS-CLOBBER: two concurrent scenarios delete the same vars directory and one finds it already gone, so the run aborts with a FileNotFoundError. Needs --workers.
CFG-CROSSREAD	molecule.yml ansible.cfg	Pass	Fail	Each scenario rewrites the shared config before every action, last writer wins, so a scenario runs its whole action against a sibling's molecule.yml or ansible.cfg. The ansible.cfg case changes effective settings with no error.
HOSTS-WRONG	hosts ansible_inventory.yml	Pass	Fail	Each scenario rewrites the shared host list, last writer wins, so under concurrency a scenario runs its play against a sibling's machines. Silent, exit 0.
TEAR-READ	molecule.yml ansible.cfg hosts group_vars/ host_vars/ ansible_inventory.yml	Pass	Fail	These files are rewritten in place, so a concurrent reader can open one mid-write and see an empty or partial file. Subordinate to the last-writer-wins failures above, and mooted once each scenario has its own copy.
STATE-CRASH	state.yml	Pass	Fail	Fixed and shipped in #4667 (atomic write). A non-atomic state.yml write left a torn file that a concurrent reader crashed on.
STATE-LOST	state.yml	Pass	Fail	Fixed and shipped in #4694 (reload before write). A worker's stale in-memory snapshot overwrote a sibling's state.yml update, silently reverting a converged/prepared flag.
STATE-RACE	state.yml	Pass	Fail	Even with the two fixes above, two concurrent workers can still interleave between the reload and the write of the shared state.yml, because there is no cross-process lock. This is the residual that #4694's own body flags. state.yml stays shared by design, so the directory fix does not touch it, and it sits outside what closes this issue.

How far to take the fix

I really struggled trying to find the minimum viable solution here that resolves this issue in the cleanest most minimalistic way. I worked through five different variations of separating the shared files to find the safe boundary. I dug deep into the history of these files and the changes that have happened over the past decade. The handling of these directories has changed significantly at least five times. The irony of proposing yet another change is not lost on me, but it's unavoidable until there is a larger discussion around how parallel / shared infrastructure should be handled long term.

Failure mode	Nothing	Inventory	+ config (fix)	+ state	+ instance_config
VARS-CLOBBER	Fail	Pass	Pass	Pass	Pass
VARS-CRASH	Fail	Pass	Pass	Pass	Pass
CFG-CROSSREAD	Fail	Fail	Pass	Pass	Pass
HOSTS-WRONG	Fail	Pass	Pass	Pass	Pass
TEAR-READ	Fail	Fail	Pass	Pass	Pass
Keeps the shared-by-design files shared	Yes	Yes	Yes	No	No

Per-scenario isolation of inventory and config closes the directory-collision failures in #4666. I believe that fixing the method/mechanism of sharing state.yml and instance_config.yml is needed, but that is not a trivial change.

Test coverage of these modes

None of the directory modes was caught by a pre-existing test. Each slipped through because every relevant unit test is single-instance (one scenario acting on its own directory) and the one --workers integration test asserts only exit codes and recap strings against scenarios that differ by a marker string, so cross-contamination is invisible to it. The earlier fixes added tests for some modes and not others. The table is the state before and after the fix PR.

Reading the columns: Before is whether a test on main catches the mode. After is whether the fix PR ships a test that does. Each new directory test is red on main and green on the fix. The STATE-CRASH test passes on both, since #4667 already made the writer atomic, and it guards that binding.

Code	Before	After	Note
VARS-CLOBBER	No	Yes	Existing tests are single-instance. A deterministic last-writer-wins unit test now asserts a sibling's group_vars survive another scenario's action
VARS-CRASH	No	Yes	Covered by the same survival test plus a unit test that two scenarios get distinct directories under one shared root. A --workers integration test for the crash itself is not included
CFG-CROSSREAD	No	Yes	The existing test checks the file exists, not whose config wins. New tests assert ansible.cfg and molecule.yml survive a sibling's write
HOSTS-WRONG	No	Yes	The integration fixture cannot observe it. A deterministic unit test asserts the host list survives a sibling's action
TEAR-READ	No	No	Moot once each scenario is isolated, so the writers were left unconverted
STATE-CRASH	No	Yes	The atomic-write primitive was tested, but no test checked that the state.yml writer uses it. A new test binds the writer to it
STATE-LOST	Yes	Yes	Dedicated regression test shipped with #4694
STATE-RACE	No	No	Left open by design, so no regression test now

Adjacent issues, out of scope for #4666

I found these three while tracing the root cause. None is part of #4666.

Code	What goes wrong	Why it is separate
PIN-HOLE	Setting MOLECULE_EPHEMERAL_DIRECTORY pins every scenario to one literal directory regardless of shared_state, so on a pinned path every directory-collision failure above returns.	Pinning predates the shared-directory collapse and takes precedence over any per-scenario isolation, with or without shared_state. It was never the flaky-CI failure, and it stays a limitation. I considered but decided against adding a guard in this fix to close this issue.
TMP-RMTREE	Scenario teardown removes the parent of the ephemeral directory, which is now a cache root shared by every project and run on the machine, so the whole directory and its neighbors go.	--parallel only, independent of shared_state, reachable through molecule check --parallel.
PRUNE-GUARD	The prune step is skipped under shared_state on the success path but not on the failure path, so a failing scenario prunes the shared directory (it spares only state.yml, ansible.cfg, ansible_inventory.yml).	Low severity, since destroy runs first.