📖 [Docs]: Add best practices for shared test infrastructure

[Marius Storhaug (MariusStorhaug)]

The README now includes actionable guidance on how to structure BeforeAll.ps1 and AfterAll.ps1 scripts for modules that run integration tests across multiple OS runners. This helps module authors avoid common pitfalls like rate limits, naming collisions, and stale resources from failed runs.

• Fixes 📖 [Docs]: Add best practices for shared test infrastructure #286

Best practices for shared test infrastructure

A new section under "Setup and Teardown Scripts" covers four key patterns:

• Deterministic naming with $env:GITHUB_RUN_ID — use the stable run ID for resource names instead of [guid]::NewGuid() or Get-Random, so test files on different runners can reference shared resources by name.
• Stale resource cleanup — remove leftover resources matching the naming prefix before creating new ones, to handle incomplete teardown from previous failed runs.
• Tests reference, don't create — test files should fetch shared resources by their deterministic name rather than provisioning their own.
• Naming conventions — a table of recommended patterns for shared resources, secrets, variables, and environments, including guidance for multi-context scenarios.

[Copilot]

Pull request overview

This PR adds documentation for best practices when using shared test infrastructure with BeforeAll.ps1 and AfterAll.ps1 setup/teardown scripts. The guidance aims to help module authors structure their test infrastructure to avoid rate limits, naming collisions, and stale resources across parallel test runs on multiple operating systems.

Changes:

• Added "Best practices for shared test infrastructure" section under "Setup and Teardown Scripts"
• Documented patterns for deterministic naming using $env:GITHUB_RUN_ID
• Provided guidance on stale resource cleanup and resource referencing patterns
• Included naming convention table for different resource types

[Copilot]

The naming patterns in this table assume OS-specific resources (e.g., Test-{OS}-{RunID}), but this conflicts with the architectural reality that BeforeAll.ps1 runs only on ubuntu-latest. The patterns should either:

1. Remove the {OS} component if resources are shared across all test matrix OSes
2. Clarify that BeforeAll.ps1 must explicitly create resources for each target OS in the test matrix (Windows, Linux, macOS)

Additionally, the table includes patterns for "Secret / variable" and "Environment" that use underscores for secrets/variables ({TestName}_{OS}_{RunID}) and hyphens for environments ({TestName}-{OS}-{RunID}). Consider explaining why different separators are used, or if this is based on naming constraints of the actual GitHub APIs for secrets vs environments.

[Copilot]

The cleanup example uses Get-Resources -Filter "Test-$os-*" which assumes an OS-specific filter. However, since BeforeAll.ps1 runs only on ubuntu-latest, $os will always be "Linux". This means:

1. If previous runs used different OSes or if the pattern was intended to clean up resources from all OSes, this filter is too narrow
2. Consider using a more general filter like "Test-*" if cleaning up all stale test resources, or document that only Linux resources need cleanup since BeforeAll always runs on ubuntu-latest

[Copilot]

The statement "Tests run in parallel across multiple OS runners" is accurate, but the next sentence "provision shared infrastructure once in BeforeAll.ps1" should clarify that BeforeAll.ps1 runs once on ubuntu-latest, not once per OS. This is important context for understanding why the OS-specific naming patterns throughout the rest of the section are problematic.

Consider revising to: "Tests run in parallel across multiple OS runners. To avoid rate limits or conflicts from excessive resource creation, provision shared infrastructure once in BeforeAll.ps1 (which runs on ubuntu-latest before the test matrix) and tear it down in AfterAll.ps1. Individual test files should consume the shared infrastructure instead of creating their own."

Suggested change

	provision shared infrastructure once in `BeforeAll.ps1` and tear it down in `AfterAll.ps1`. Individual test files
	should consume the shared infrastructure instead of creating their own.
	provision shared infrastructure once in `BeforeAll.ps1` (which runs on `ubuntu-latest` before the test matrix) and tear it
	down in `AfterAll.ps1`. Individual test files should consume the shared infrastructure instead of creating their own.

[Copilot]

The guidance to use $env:RUNNER_OS in BeforeAll.ps1 is misleading. According to the workflow configuration (.github/workflows/BeforeAll-ModuleLocal.yml), BeforeAll runs only once on ubuntu-latest before the test matrix jobs start. This means $env:RUNNER_OS will always be "Linux" in BeforeAll.ps1, regardless of which OS runners the tests will eventually run on.

If the intent is to create OS-specific resources for each test runner (Windows, Linux, macOS), the BeforeAll.ps1 script would need to create resources for all target OSes explicitly, not rely on $env:RUNNER_OS. Alternatively, if a single shared resource works for all OSes, then the naming pattern should not include {OS} at all.

Consider clarifying:

1. Whether shared resources are truly OS-specific or shared across all test matrix OSes
2. If OS-specific, how BeforeAll.ps1 should create resources for each target OS (perhaps by iterating over the test matrix configuration)
3. Update examples to reflect the actual execution model where BeforeAll runs once on ubuntu-latest

[Copilot]

This example shows using $env:RUNNER_OS inside test files, which is correct since test files run on each matrix OS runner. However, this contradicts the earlier guidance in the BeforeAll.ps1 section that also uses $env:RUNNER_OS.

If BeforeAll.ps1 creates a resource named Test-Linux-1234 (because it runs on ubuntu-latest), but a test file running on Windows looks for Test-Windows-1234 (using its own $env:RUNNER_OS), the resource names won't match.

The naming strategy needs to be consistent between BeforeAll.ps1 and the test files. Either:

1. BeforeAll.ps1 creates a shared resource without OS in the name (e.g., Test-1234), and test files reference it the same way
2. BeforeAll.ps1 creates OS-specific resources for each target OS, and test files use their actual $env:RUNNER_OS to find their specific resource

Clarify the intended pattern to ensure test files can actually find the resources created by BeforeAll.ps1.