Use correct name for main directory

README.md

@@ -86,3 +86,9 @@ $ pytest tests/e2e_tests -s --include-core-tests
 ```
 
 Currently, CI tests only run unit and integration tests. There's an open task in django-simple-deploy to remove the dependence on poetry and pipenv for running tests. When that is implemented, e2e tests can run much more easily in CI.
+
+Documentation
+---
+
+For more information, see the [docs](docs/) directory.
+

docs/README.md

@@ -0,0 +1,8 @@
+dsd-plugin-generator documentation
+---
+
+I'll probably make a full documentation site on read the docs before long for this project, but for now the most important information is in the main README and here.
+
+- [Plugin structure](plugin_structure.md): This page describes the structure of a plugin, and the naming conventions used.
+- [End to end tests](e2e_tests.md): This page describes how the end to end tests are set up, and how they can be used for development purposes as well as for testing purposes.
+- 

docs/e2e_tests.md

@@ -0,0 +1,77 @@
+End to end tests
+===
+
+Currently, plugins can only be tested when they're installed into a django-simple-deploy test environment.
+
+To manage this, the end to end tests for this project do the following:
+
+- Get a temp dir from pytest.
+- Make a shallow clone of django-simple-deploy into the temp dir.
+- For each plugin test:
+    - Generate the plugin using `generate_plugin()`.
+    - Install the plugin to the temp django-simple-deploy env.
+    - Run django-simple-deploy's tests, which in turn run the plugin's tests.
+    - Uninstall the plugin from the django-simple-deploy env.
+
+CLI args
+---
+
+There are three CLI args that can help with e2e testing:
+
+`-s`
+---
+
+This is a pytest flag that streams output as the test runs. Since these are long running tests, it's almost always useful to include this flag. Otherwise it can look like the tests are hanging.
+
+`--run-core-tests`
+---
+
+This flag runs the full set of django-simple-deploy tests, for every test plugin. This takes a lot longer. The default behavior is to only use django-simple-deploy to run the plugin's tests.
+
+`--setup-plugins-only`
+---
+
+When we run this project's e2e tests, we're actually calling pytest within a temp django-simple-deploy env, once for every test plugin. That means the temp dirs such as `pytest-237` get garbage collected even before the test finishes running. That makes it difficult to see what kind of environment is being built.
+
+The `--setup-plugins-only` flag prevents any tests from being run. We just set up the temp dev environment for django-simple-deploy, and then generate the full series of test plugins. You can then drop into the latest temp pytest directory, and poke around the full test environment.
+
+Here's what that looks like:
+
+```sh
+$ pytest tests/e2e_tests -s --setup-plugins-only
+...
+Building e2e test env at: /private/.../pytest-239/e2e_new_plugin_test0
+...
+    Built django-simple-deploy...
+    Built dsd-newfly...
+    Built dsd-newplatform...
+    Built dsd-my-plugin...
+...
+```
+
+In just a few seconds, you have a development environment for django-simple-deploy, alongside a set of plugins generated by this project.
+
+You can cd to this folder, and do anything you want with these resources. Here's how to drop into that folder, see the plugins, install one to the django-simple-deploy environment, and run the new platform's integration tests:
+
+```sh
+$ cd /private/.../pytest-239/e2e_new_plugin_test0 
+e2e_new_plugin_test0$ ls -l
+django-simple-deploy
+dsd-my-plugin
+dsd-newfly
+dsd-newplatform
+e2e_new_plugin_test0$ cd django-simple-deploy 
+django-simple-deploy$ source .venv/bin/activate
+(.venv) django-simple-deploy$ uv pip install -e ../dsd-newplatform 
+      Built dsd-newplatform
+...
+(.venv) django-simple-deploy$ pytest -k newplatform
+========== test session starts ========================
+collected 77 items / 59 deselected / 18 selected
+test_newplatform_config.py ..................    [100%]
+========== 18 passed, 59 deselected in 3.65s ==========
+```
+
+This is really helpful for maintaining the test suite, and for development work.
+
+**Note:** This can get a little flaky, because we're working in a temp environment that pytest will destroy when more resources are created. If you want to keep working with these resources, you can copy the entire `e2e_new_plugin_test0` directory to a more stable location. You'll have to rebuild the `django-simple-deploy/.venv` environment, but you'll have a useful, stable development environment to work with. You'll be able to run any tests you want from that environment.

docs/plugin_structure.md

@@ -0,0 +1,67 @@
+Plugin project structure
+===
+
+This document describes the overall project structure, and naming conventions within the project.
+
+Example plugin
+---
+
+Consider a plugin that should be built from the following user input:
+
+```sh
+$ python generate_plugin.py
+What platform are you targeting? (Example: Fly.io) GreenHost
+What's the name of your plugin package? (Example: dsd-flyio) dsd-greenhost-high-traffic
+Will your plugin support the --automate-all CLI arg? (yes/no) y
+What name do you want to appear in the LICENSE file? Eric
+```
+
+This is a plugin that will support high-traffic sites on a platform called GreenHost.
+
+Example project structure
+---
+
+Here's the project structure for the plugin that's generated:
+
+```sh
+$ tree -L 3 dsd-greenhost-high-traffic
+dsd-greenhost-high-traffic
+├── CHANGELOG.md
+├── LICENSE
+├── MANIFEST.in
+├── README.md
+├── developer_resources
+│   └── README.md
+├── dsd_greenhost_high_traffic
+│   ├── __init__.py
+│   ├── deploy.py
+│   ├── deploy_messages.py
+│   ├── platform_deployer.py
+│   ├── plugin_config.py
+│   └── templates
+│       ├── dockerfile_example
+│       └── settings.py
+├── pyproject.toml
+└── tests
+    ├── conftest.py
+    ├── e2e_tests
+    │   ├── __init__.py
+    │   ├── test_deployment.py
+    │   └── utils.py
+    └── integration_tests
+        ├── reference_files
+        └── test_greenhost_config.py
+
+8 directories, 18 files
+```
+
+Notes
+---
+
+Here's what's important to note:
+
+- The outer project directory is the name of the plugin package.
+- The main project directory is the plugin package name, with hyphens replaced by underscores.
+- The main integration test for configuration changes is named `test_<platform_name_lower>_config.py`.
+
+Currently, plugin package names need to start with `dsd-`. This requirement will be removed before long.

tests/e2e_tests/conftest.py

@@ -1,23 +1,103 @@
 """Configuration for e2e test runs."""
 
+import json
+import subprocess
+import shlex
 from dataclasses import dataclass
 
 import pytest
 
+from tests.e2e_tests.utils import e2e_utils
 
+
+# --- Custom CLI args ---
 def pytest_addoption(parser):
     parser.addoption(
-        "--include-core-tests",
+        "--run-core-tests",
         action="store_true",
         help="Run the full set of core django-simple-deploy tests for each new plugin.",
     )
+    parser.addoption(
+        # Useful for troubleshooting this test env. May also be useful for poking around
+        # the full set of test plugins.
+        # In a full test run, this test runs many django-simple-deploy pytest calls. Each of
+        # those leads to a pytest-___ temp dir. That can make it difficult or impossible
+        # to find the dir associated with a specific failing test.
+        "--setup-plugins-only",
+        action="store_true",
+        help="Build full set of test dev env with test plugins, but don't run any tests."
+    )
 
 @dataclass
 class CLIOptions:
-    include_core_tests: bool=False
+    run_core_tests: bool=False
+    setup_plugins_only: bool=False
 
 @pytest.fixture(scope="session")
 def cli_options(request):
     return CLIOptions(
-        include_core_tests=request.config.getoption("--include-core-tests"),
+        run_core_tests=request.config.getoption("--run-core-tests"),
+        setup_plugins_only=request.config.getoption("--setup-plugins-only"),
     )
+
+
+# --- Fixtures ---
+
+@pytest.fixture(scope="module")
+def get_dev_env(tmp_path_factory, cli_options):
+    """Set up an env where plugins can be generated and tested within django-simple-deploy.
+
+    - Set up a temp dir.
+    - Install dev env for django-simple-deploy.
+    - We can generate plugins, with this temp dir as target location.
+    - Then install newly-generated plugins to dsd dev env, and run tests from dsd.
+    """
+    # Make the temp directory for the dsd development env.
+    tmp_path = tmp_path_factory.mktemp("e2e_new_plugin_test")
+    print(f"\nBuilding e2e test env at: {tmp_path.as_posix()}")
+
+    # Clone django-simple-deploy in temp env.
+    path_dsd = tmp_path / "django-simple-deploy"
+    cmd = f"git clone https://github.com/django-simple-deploy/django-simple-deploy.git {path_dsd.as_posix()} --depth 1"
+    cmd_parts = shlex.split(cmd)
+    subprocess.run(cmd_parts)
+
+    # Build a venv in the django-simple-deploy temp dir.
+    venv_dir = path_dsd / ".venv"
+    cmd = f"uv venv {venv_dir}"
+    cmd_parts = shlex.split(cmd)
+    subprocess.run(cmd_parts)
+
+    # Make an editable install of django-simple-deploy in its environment.
+    path_to_python = venv_dir / "bin" / "python"
+    cmd = f'uv pip install --python {path_to_python} -e "{path_dsd.as_posix()}[dev]"'
+    cmd_parts = shlex.split(cmd)
+    subprocess.run(cmd_parts)
+
+    if not cli_options.setup_plugins_only:
+        # Run core tests without a plugin installed.
+        e2e_utils.run_dsd_core_tests(path_dsd, path_to_python, cli_options)
+
+    return tmp_path, path_to_python, path_dsd
+
+@pytest.fixture(scope="function", autouse=True)
+def clear_plugins(get_dev_env):
+    """Remove any plugins from dev env.
+
+    Most tests install a plugin to the dev env. Remove each plugin after its test runs.
+    """
+    dev_env_dir, path_to_python, path_dsd = get_dev_env
+
+    # Yield to let test function run, then clear any plugins that were installed.
+    yield
+
+    cmd = f"uv pip list --python {path_to_python} --format=json"
+    cmd_parts = shlex.split(cmd)
+    package_dicts_str = subprocess.run(cmd_parts, capture_output=True).stdout
+    package_dicts = json.loads(package_dicts_str)
+    package_names = [pd["name"] for pd in package_dicts if pd["name"].startswith("dsd-")]
+
+    for pkg_name in package_names:
+        cmd = f"uv pip uninstall {pkg_name} --python {path_to_python}"
+        cmd_parts = shlex.split(cmd)
+        subprocess.run(cmd_parts)