Skip to content

📖 [Docs]: Add best practices for shared test infrastructure #285

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Marius Storhaug (MariusStorhaug) merged 2 commits into from
Feb 18, 2026
Merged

📖 [Docs]: Add best practices for shared test infrastructure #285

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5812ab5
Add best practices for shared test infrastructure to README
MariusStorhaug Feb 17, 2026
9a25d6f
Merge branch 'main' into docs/add-test-infrastructure-best-practices
MariusStorhaug Feb 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,70 @@ Write-Host "Cleaning up test environment..."
Write-Host "Cleanup completed!"
```

##### Best practices for shared test infrastructure

Tests run in parallel across multiple OS runners. To avoid rate limits or conflicts from excessive resource creation,
provision shared infrastructure once in `BeforeAll.ps1` and tear it down in `AfterAll.ps1`. Individual test files
should consume the shared infrastructure instead of creating their own.
Comment on lines +211 to +212
Copy link

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The statement "Tests run in parallel across multiple OS runners" is accurate, but the next sentence "provision shared infrastructure once in BeforeAll.ps1" should clarify that BeforeAll.ps1 runs once on ubuntu-latest, not once per OS. This is important context for understanding why the OS-specific naming patterns throughout the rest of the section are problematic.

Consider revising to: "Tests run in parallel across multiple OS runners. To avoid rate limits or conflicts from excessive resource creation, provision shared infrastructure once in BeforeAll.ps1 (which runs on ubuntu-latest before the test matrix) and tear it down in AfterAll.ps1. Individual test files should consume the shared infrastructure instead of creating their own."

Suggested change
provision shared infrastructure once in `BeforeAll.ps1` and tear it down in `AfterAll.ps1`. Individual test files
should consume the shared infrastructure instead of creating their own.
provision shared infrastructure once in `BeforeAll.ps1` (which runs on `ubuntu-latest` before the test matrix) and tear it
down in `AfterAll.ps1`. Individual test files should consume the shared infrastructure instead of creating their own.

Copilot uses AI. Check for mistakes.

###### Use deterministic naming with `$env:GITHUB_RUN_ID`

Use `$env:GITHUB_RUN_ID` (stable per workflow run, shared across OS runners) to build deterministic resource names.
This lets test files reference shared resources by name without passing state between jobs.

```powershell
# BeforeAll.ps1
$os = $env:RUNNER_OS
$id = $env:GITHUB_RUN_ID
$resourceName = "Test-$os-$id"
```

Do **not** use `[guid]::NewGuid()` or `Get-Random` for shared resource names — these produce different values on
each runner and cannot be referenced by other jobs.
Comment on lines +214 to +227
Copy link

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The guidance to use $env:RUNNER_OS in BeforeAll.ps1 is misleading. According to the workflow configuration (.github/workflows/BeforeAll-ModuleLocal.yml), BeforeAll runs only once on ubuntu-latest before the test matrix jobs start. This means $env:RUNNER_OS will always be "Linux" in BeforeAll.ps1, regardless of which OS runners the tests will eventually run on.

If the intent is to create OS-specific resources for each test runner (Windows, Linux, macOS), the BeforeAll.ps1 script would need to create resources for all target OSes explicitly, not rely on $env:RUNNER_OS. Alternatively, if a single shared resource works for all OSes, then the naming pattern should not include {OS} at all.

Consider clarifying:

  1. Whether shared resources are truly OS-specific or shared across all test matrix OSes
  2. If OS-specific, how BeforeAll.ps1 should create resources for each target OS (perhaps by iterating over the test matrix configuration)
  3. Update examples to reflect the actual execution model where BeforeAll runs once on ubuntu-latest

Copilot uses AI. Check for mistakes.

###### Clean up stale resources from previous failed runs

If a previous workflow run failed before teardown completed, stale resources may remain. Start `BeforeAll.ps1` by
removing any resources matching your naming prefix before creating new ones:

```powershell
# Remove stale resources from previous failed runs
Get-Resources -Filter "Test-$os-*" | Remove-Resource

# Create fresh shared resources
New-Resource -Name "Test-$os-$id"
```
Comment on lines +234 to +240
Copy link

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cleanup example uses Get-Resources -Filter "Test-$os-*" which assumes an OS-specific filter. However, since BeforeAll.ps1 runs only on ubuntu-latest, $os will always be "Linux". This means:

  1. If previous runs used different OSes or if the pattern was intended to clean up resources from all OSes, this filter is too narrow
  2. Consider using a more general filter like "Test-*" if cleaning up all stale test resources, or document that only Linux resources need cleanup since BeforeAll always runs on ubuntu-latest

Copilot uses AI. Check for mistakes.

###### Tests reference shared resources — they do not create them

Test files should fetch the shared resource by its deterministic name, not create new resources:

```powershell
# Inside a test file
BeforeAll {
$os = $env:RUNNER_OS
$id = $env:GITHUB_RUN_ID
$resource = Get-Resource -Name "Test-$os-$id"
}
Comment on lines +246 to +252
Copy link

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example shows using $env:RUNNER_OS inside test files, which is correct since test files run on each matrix OS runner. However, this contradicts the earlier guidance in the BeforeAll.ps1 section that also uses $env:RUNNER_OS.

If BeforeAll.ps1 creates a resource named Test-Linux-1234 (because it runs on ubuntu-latest), but a test file running on Windows looks for Test-Windows-1234 (using its own $env:RUNNER_OS), the resource names won't match.

The naming strategy needs to be consistent between BeforeAll.ps1 and the test files. Either:

  1. BeforeAll.ps1 creates a shared resource without OS in the name (e.g., Test-1234), and test files reference it the same way
  2. BeforeAll.ps1 creates OS-specific resources for each target OS, and test files use their actual $env:RUNNER_OS to find their specific resource

Clarify the intended pattern to ensure test files can actually find the resources created by BeforeAll.ps1.

Copilot uses AI. Check for mistakes.
```

Test-specific ephemeral resources (for example, secrets, variables, or temporary items) can still be created and
cleaned up within each test file. Only long-lived or expensive resources should be shared.

###### Naming conventions

Use a consistent naming scheme so that resources are easy to identify and clean up. A recommended pattern:

| Resource | Pattern | Example |
|-------------------|---------------------------------------|----------------------------|
| Shared resource | `Test-{OS}-{RunID}` | `Test-Linux-1234` |
| Extra resource | `Test-{OS}-{RunID}-{N}` | `Test-Linux-1234-1` |
| Secret / variable | `{TestName}_{OS}_{RunID}` | `Secrets_Linux_1234` |
| Environment | `{TestName}-{OS}-{RunID}` | `Secrets-Linux-1234` |
Comment on lines +262 to +267
Copy link

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The naming patterns in this table assume OS-specific resources (e.g., Test-{OS}-{RunID}), but this conflicts with the architectural reality that BeforeAll.ps1 runs only on ubuntu-latest. The patterns should either:

  1. Remove the {OS} component if resources are shared across all test matrix OSes
  2. Clarify that BeforeAll.ps1 must explicitly create resources for each target OS in the test matrix (Windows, Linux, macOS)

Additionally, the table includes patterns for "Secret / variable" and "Environment" that use underscores for secrets/variables ({TestName}_{OS}_{RunID}) and hyphens for environments ({TestName}-{OS}-{RunID}). Consider explaining why different separators are used, or if this is based on naming constraints of the actual GitHub APIs for secrets vs environments.

Copilot uses AI. Check for mistakes.

When tests use multiple authentication contexts that share the same runner, include a token or context identifier in
the name to avoid collisions (for example, `Test-{OS}-{ContextID}-{RunID}`).


#### Module tests

Expand Down
Loading