fix: shared_state - give each scenario its own working directory

docs/ansible-native.md

@@ -279,31 +279,36 @@ Shared state enables scenarios to share ephemeral state and testing resources.
 
 ### Configuration
 
+Enable shared state in a base `config.yml` that every scenario inherits, or in each scenario's own `molecule.yml`. Every scenario in the run must have it set. The [configuration reference](configuration.md#shared-state) lists the exact file locations and shows where the shared and per-scenario files are created on disk.
+
 ```yaml
+# config.yml, or each scenario's molecule.yml
 shared_state: true
 ```
 
 When enabled:
 
-- All scenarios share the same ephemeral state directory
+- All scenarios share one state directory (`state.yml` and `instance_config.yml`), and each scenario keeps its own inventory and configuration files
 - Default scenario manages testing resource lifecycle
 - Component scenarios access shared resources
 - State persists between scenario executions
 
 ### Resource Lifecycle Management
 
-**Default scenario** (testing resource management):
+**Default scenario** (testing resource management), in `molecule/default/molecule.yml`:
 
 ```yaml
+# molecule/default/molecule.yml
 scenario:
   test_sequence:
     - create
     - destroy
 ```
 
-**Component scenarios** (testing only):
+**Component scenarios** (testing only), one `molecule.yml` per scenario:
 
 ```yaml
+# molecule/<scenario-name>/molecule.yml
 scenario:
   test_sequence:
     - prepare

docs/configuration.md

@@ -41,32 +41,61 @@ By default, Molecule runs each scenario independently with its own isolated stat
 
 This is particularly useful for multi-scenario testing where one scenario manages testing resource lifecycle while other scenarios perform testing against those resources.
 
-To enable shared state, add `shared_state: true` to your configuration file:
-
-```yaml
----
-shared_state: true
-# ... rest of configuration
-```
-
 **Effects of enabling shared state:**
 
-- All scenarios share the same ephemeral state directory
+- All scenarios share one state directory (`state.yml` and `instance_config.yml`), and each scenario keeps its own inventory and configuration files
 - The default scenario handles create/destroy actions for all scenarios
 - Component scenarios can access resources created by the default scenario
 - Scenarios skip their own create/destroy actions when shared resources are managed elsewhere
 - Faster execution with single infrastructure lifecycle instead of per-scenario setup/teardown
 
-**Configuration locations:**
+### Where to set `shared_state` when you run several scenarios
+
+`molecule test --all` configures each scenario on its own, so `shared_state` must be true for every scenario in the run, not only the default one. There are two ways to do that.
 
-You can add this setting to:
+Set it once in a base `config.yml` that every scenario inherits. Molecule looks for a base config in these places:
 
-- `.config/molecule/config.yml` file in your `$HOME` directory (global default)
-- Base `config.yml` file at the project root (project default)
-- Collection molecule directory `extensions/molecule/config.yml`
-- Individual scenario `molecule.yml` files (scenario-specific override)
+- `.config/molecule/config.yml` in your `$HOME` directory (global default)
+- `config.yml` at the root of your project (project default)
+- `extensions/molecule/config.yml` in a collection
+
+```yaml
+# config.yml, inherited by every scenario
+shared_state: true
+```
+
+If you do not keep a base `config.yml`, put the same line in each scenario's own `molecule.yml`:
+
+```yaml
+# molecule/default/molecule.yml, and the same line in every other scenario's molecule.yml
+shared_state: true
+```
 
-**Alternative:** The `--shared-state` command-line flag can also enable this behavior temporarily, but configuration file approach is recommended for consistent usage.
+To turn the behavior on for a single run without editing any file, pass the flag. It applies only to that one invocation.
+
+```bash
+molecule test --all --shared-state
+```
+
+### Where the files live
+
+With `shared_state` enabled, one shared directory holds the state the scenarios have in common, and each scenario keeps its own directory for the files it must not share with its siblings.
+
+```text
+<shared ephemeral directory>/      # MOLECULE_SHARED_EPHEMERAL_DIRECTORY
+├── state.yml                      # shared, what has been created so far
+├── instance_config.yml            # shared, the instances every scenario connects to
+├── default/                       # this scenario's MOLECULE_EPHEMERAL_DIRECTORY
+│   ├── molecule.yml
+│   ├── ansible.cfg
+│   └── inventory/
+└── role1/                         # another scenario's MOLECULE_EPHEMERAL_DIRECTORY
+    ├── molecule.yml
+    ├── ansible.cfg
+    └── inventory/
+```
+
+These are generated paths, so read them at run time rather than hard-coding them. Run a command with `--debug` to print the resolved directories, or read them inside a playbook from the `MOLECULE_SHARED_EPHEMERAL_DIRECTORY` and `MOLECULE_EPHEMERAL_DIRECTORY` variables described below.
 
 ## Variable Substitution
 
@@ -108,34 +137,53 @@ will read variables when rendering `molecule.yml`. See command usage.
 
 Following are the environment variables available in `molecule.yml`:
 
+!!! note
+
+    The ephemeral paths below are generated for each run. Molecule creates them
+    under its cache directory, by default `~/.ansible/tmp/`, or `$ANSIBLE_HOME/tmp/`
+    when `ANSIBLE_HOME` is set, with a
+    generated name of the form `molecule.<hash>.<scenario-name>`, and
+    `molecule.<hash>` for the shared directory. Treat the
+    examples as the shape of the value, not a literal you can hard-code, and run
+    a command with `--debug` to print the resolved paths. Inside a playbook the
+    same directories are available as the lowercase variables
+    `molecule_ephemeral_directory` and `molecule_shared_ephemeral_directory`.
+
 MOLECULE_DEBUG
 
 : If debug is turned on or off
 
 MOLECULE_FILE
 
-: Path to molecule config file, usually
-`~/.cache/molecule/<role-name>/<scenario-name>/molecule.yml`
+: Path to the generated molecule config file, `molecule.yml` inside the
+scenario's ephemeral directory
 
 MOLECULE_ENV_FILE
 
 : Path to molecule environment file, usually `<role_path>/.env.yml`
 
 MOLECULE_STATE_FILE
 
-: The path to molecule state file contains the state of the instances
-(created, converged, etc.). Usually
-`~/.cache/molecule/<role-name>/<scenario-name>/state.yml`
+: The molecule state file, which holds the state of the instances (created,
+converged, etc.). It is `state.yml` inside the scenario's ephemeral directory,
+or inside the shared ephemeral directory when `shared_state` is enabled
 
 MOLECULE_INVENTORY_FILE
 
-: Path to generated inventory file, usually
-`~/.cache/molecule/<role-name>/<scenario-name>/inventory/ansible_inventory.yml`
+: Path to the generated inventory file,
+`inventory/ansible_inventory.yml` inside the scenario's ephemeral directory
 
 MOLECULE_EPHEMERAL_DIRECTORY
 
-: Path to generated directory, usually
-`~/.cache/molecule/<role-name>/<scenario-name>`
+: The scenario's generated working directory. See the note above for its
+location
+
+MOLECULE_SHARED_EPHEMERAL_DIRECTORY
+
+: Path to the directory all scenarios share when `shared_state` is enabled,
+where `state.yml` and `instance_config.yml` live. Each scenario's own
+`MOLECULE_EPHEMERAL_DIRECTORY` is created beneath it. Without `shared_state`,
+the same value as `MOLECULE_EPHEMERAL_DIRECTORY`.
 
 MOLECULE_SCENARIO_DIRECTORY
 
@@ -148,9 +196,10 @@ MOLECULE_PROJECT_DIRECTORY
 
 MOLECULE_INSTANCE_CONFIG
 
-: Path to the instance config file, contains instance name,
-connection, user, port, etc. (populated from driver). Usually
-`~/.cache/molecule/<role-name>/<scenario-name>/instance_config.yml`
+: Path to the instance config file, which holds instance name, connection,
+user, port, etc. (populated from the driver). It is `instance_config.yml`
+inside the scenario's ephemeral directory, or inside the shared ephemeral
+directory when `shared_state` is enabled
 
 MOLECULE_ANSIBLE_ARGS_STRICT_MODE
 

docs/getting-started-collections.md

@@ -450,7 +450,7 @@ With `shared_state` enabled, the **default scenario becomes the lifecycle manage
 
 - **Default scenario handles create/destroy**: The default scenario's `create` and `destroy` actions manage the infrastructure lifecycle for ALL scenarios
 - **Component scenarios skip create/destroy**: Individual scenarios (role1, role2, role3) only run their test sequence (prepare, converge, verify, etc.) - they do not create or destroy their own resources
-- **Shared ephemeral state**: All scenarios share the same state directory, allowing them to access resources created by the default scenario
+- **Shared ephemeral state**: All scenarios share one state directory (`state.yml` and `instance_config.yml`), allowing them to access resources created by the default scenario, and each scenario keeps its own inventory and configuration files
 
 **Why this approach is required for this configuration:**
 

docs/philosophy.md

@@ -415,8 +415,14 @@ This pattern directly demonstrates several [testing framework requirements](#ess
 - **Multi-platform support**: Abstract enterprise infrastructure complexity while enabling test-specific instance definitions
 - **Extensibility and integration**: Clean integration with existing enterprise toolchains and governance policies
 
-**Multi-scenario/multi-action data sharing**
-When using native inventory patterns, teams often need to share host-specific data between different Molecule actions (create, converge, verify, destroy). This is especially valuable when using `--shared-state`, where the `default` scenario's create action provisions infrastructure and other scenarios need access to resource-specific data captured during that initial provisioning. A simple and effective approach uses temporary files to pass data from one action to subsequent actions:
+**Sharing one environment across scenarios**
+Real environments are built once and used many times, and a test suite should work the same way. Standing up infrastructure, preparing data, and connecting services is costly, so repeating that work for every check is slow and unlike how the system really runs.
+
+Molecule lets a single setup serve the whole suite. One part of a run creates the shared environment and records what it produced. The checks that follow run against that same environment and build on those results instead of starting over, and when the run ends the environment is torn down once.
+
+Molecule calls this shared state. One scenario, the default, creates and destroys the environment for the whole run, and the others skip setup and teardown to test against it. Whatever the default scenario captures while provisioning is kept where every scenario can reach it, so a later scenario reads what an earlier one produced rather than rediscovering it.
+
+The following playbooks show one way to do this, writing data as the default scenario provisions, reading it back in a later scenario, and removing it at teardown:
 
 ```yaml
 # Example: Sharing infrastructure and host-specific data between actions
@@ -426,7 +432,7 @@ When using native inventory patterns, teams often need to share host-specific da
   hosts: localhost
   gather_facts: false
   vars:
-    execution_vars: "{% raw %}{{ molecule_ephemeral_directory }}{% endraw %}/execution_vars/"
+    execution_vars: "{% raw %}{{ molecule_shared_ephemeral_directory }}{% endraw %}/execution_vars/"
   tasks:
     - name: Ensure execution vars directory exists
       ansible.builtin.file:
@@ -455,7 +461,7 @@ When using native inventory patterns, teams often need to share host-specific da
   hosts: molecule
   gather_facts: false
   vars:
-    execution_vars: "{% raw %}{{ molecule_ephemeral_directory }}{% endraw %}/execution_vars/"
+    execution_vars: "{% raw %}{{ molecule_shared_ephemeral_directory }}{% endraw %}/execution_vars/"
   vars_files:
     - "{% raw %}{{ execution_vars }}{% endraw %}host_{% raw %}{{ inventory_hostname }}{% endraw %}.yml"
   tasks:
@@ -469,7 +475,7 @@ When using native inventory patterns, teams often need to share host-specific da
   hosts: localhost
   gather_facts: false
   vars:
-    execution_vars: "{% raw %}{{ molecule_ephemeral_directory }}{% endraw %}/execution_vars/"
+    execution_vars: "{% raw %}{{ molecule_shared_ephemeral_directory }}{% endraw %}/execution_vars/"
   tasks:
     - name: Destroying resources
       ansible.builtin.debug:

src/molecule/config.py

@@ -124,6 +124,11 @@ def __init__(
         self.command_args: CommandArgs = command_args if command_args is not None else {}
         self.ansible_args = ansible_args
         self.config_data = self._get_config()
+        # Apply the CLI shared_state override before env first evaluates the
+        # scenario's ephemeral directory and the state file path (both cached
+        # on first use); _reget_config replaces config_data, so the override
+        # is applied again below.
+        self._apply_cli_overrides()
         self._action: str | None = None
         self._run_uuid = str(uuid4())
         self.project_directory = os.getenv(
@@ -379,6 +384,11 @@ def env(self) -> dict[str, str]:
             "MOLECULE_STATE_FILE": self.state.state_file,
             "MOLECULE_INVENTORY_FILE": self.provisioner.inventory_file,  # type: ignore[union-attr]
             "MOLECULE_EPHEMERAL_DIRECTORY": self.scenario.ephemeral_directory,
+            "MOLECULE_SHARED_EPHEMERAL_DIRECTORY": (
+                self.scenario.shared_ephemeral_directory
+                if self.shared_state
+                else self.scenario.ephemeral_directory
+            ),
             "MOLECULE_SCENARIO_DIRECTORY": self.scenario.directory,
             "MOLECULE_PROJECT_DIRECTORY": self.project_directory,
             "MOLECULE_INSTANCE_CONFIG": self.driver.instance_config,

src/molecule/driver/base.py

@@ -169,9 +169,19 @@ def options(self) -> DriverOptions:
     def instance_config(self) -> str:
         """Instance config file location.
 
+        When shared_state is enabled, this file lives at the shared ephemeral
+        directory so sibling scenarios read the same instance list.
+
         Returns:
             Path to instance_config.yml.
         """
+        if self._config.shared_state:
+            return str(
+                Path(
+                    self._config.scenario.shared_ephemeral_directory,
+                    "instance_config.yml",
+                ),
+            )
         return str(
             Path(
                 self._config.scenario.ephemeral_directory,

src/molecule/provisioner/ansible.py

@@ -246,6 +246,7 @@ def inventory(self) -> dict[str, Any]:
         molecule_vars = {
             "molecule_file": "{{ lookup('env', 'MOLECULE_FILE') }}",
             "molecule_ephemeral_directory": "{{ lookup('env', 'MOLECULE_EPHEMERAL_DIRECTORY') }}",
+            "molecule_shared_ephemeral_directory": "{{ lookup('env', 'MOLECULE_SHARED_EPHEMERAL_DIRECTORY') }}",
             "molecule_scenario_directory": "{{ lookup('env', 'MOLECULE_SCENARIO_DIRECTORY') }}",
             "molecule_yml": "{{ lookup('file', molecule_file) | from_yaml }}",
             "molecule_instance_config": "{{ lookup('env', 'MOLECULE_INSTANCE_CONFIG') }}",

src/molecule/scenario.py

@@ -127,8 +127,9 @@ def directory(self) -> str:
     def ephemeral_directory(self) -> str:
         """Acquire the ephemeral directory.
 
-        When shared_state is enabled, returns the shared ephemeral directory
-        so all scenarios use the same working directory.
+        When shared_state is enabled and MOLECULE_EPHEMERAL_DIRECTORY is unset,
+        returns a per-scenario directory nested under the shared ephemeral
+        directory.
 
         Returns:
             The ephemeral directory for this scenario.
@@ -149,7 +150,7 @@ def ephemeral_directory(self) -> str:
             path = self.config.runtime.cache_dir / "tmp" / project_scenario_directory
 
             if self.config.shared_state:
-                path = Path(self.shared_ephemeral_directory)
+                path = Path(self.shared_ephemeral_directory) / safe_name
         else:
             path = Path(os.getenv("MOLECULE_EPHEMERAL_DIRECTORY", ""))
         path.mkdir(parents=True, exist_ok=True)

src/molecule/state.py

@@ -238,4 +238,14 @@ def _write_state_file(self) -> None:
         util.atomic_write_file(self.state_file, util.safe_dump(self._data))
 
     def _get_state_file(self) -> Path:
+        """Resolve the path to the scenario's state file.
+
+        When shared_state is enabled, the state file lives at the shared
+        ephemeral directory so scenarios share one run state.
+
+        Returns:
+            Path to state.yml.
+        """
+        if self._config.shared_state:
+            return Path(self._config.scenario.shared_ephemeral_directory) / "state.yml"
         return Path(self._config.scenario.ephemeral_directory) / "state.yml"