Fix catastrophic CI flake: thread-safe FileWatcher subscriber lists

[phil-scott-78]

Symptom

CI intermittently fails the ubuntu Test job with a catastrophic (process-terminating) exception, while all test assemblies report Passed!:

[FATAL ERROR] System.InvalidOperationException
Catastrophic failure: ... Collection was modified; enumeration operation may not execute.

It's flaky (same commit passed on the feature branch, failed once on main) and Linux-only. A rerun clears it — but it will keep recurring.

Root cause

FileWatcher holds its subscriber callbacks in two plain List<T> fields:

• NotifySubscribers enumerates them on the debounce timer's ThreadPool thread (fired ~100 ms after a file event).
• SubscribeToChanges appends to them lock-free from other threads — AuditRunner subscribes at host start, and RedirectContentService subscribes lazily on the first request.

Under WebApplicationFactory/TestServer (env Testing), the watcher is armed unconditionally over a real inotify FileSystemWatcher on the content root. On Linux, an inotify event during startup fires NotifySubscribers right as those late subscribers are still .Add()-ing. The List is mutated mid-foreach → Collection was modified thrown on the unobserved timer thread → the whole test process is torn down.

The per-callback try/catch in NotifySubscribers wraps each invocation, not the foreach, so the enumerator's MoveNext() throw escapes it entirely.

Note: this is not related to the WASM-boot-manifest PR that happened to catch the flake — that change's build code doesn't even run in the host tests, and there's no WASM render mode anywhere in the suite. The race is long-standing; the merge just rolled the dice.

Fix

Switch both subscriber lists to ImmutableList<T>, appended atomically via ImmutableInterlocked.Update. NotifySubscribers snapshots each field into a local once, so a concurrent registration swaps the field reference without disturbing the in-flight enumeration. Lock-free on both paths, and readers always see a consistent immutable set. Subscriber ordering (insertion order) is preserved.

Test

SubscribeDuringNotification_DoesNotCorruptEnumeration reproduces the exact fault deterministically: a subscriber registers another subscriber from inside a notification, and the debounce is fired off a FakeTimeProvider so the enumeration runs on the test thread.

• On the buggy code it fails with the identical System.InvalidOperationException : Collection was modified; enumeration operation may not execute. (verified by reverting the fix locally).
• With the fix it passes; the trailing subscriber still fires.

Verification

• Pennington.Tests: 1047 passed (incl. the new test).
• Pennington.IntegrationTests: 67 passed, 4 skipped.
• Core builds clean, 0 warnings.

[github-actions]

🛰️ Docs preview: https://pr-65.pennington-dev.pages.dev

Rebuilt on every push to this PR; torn down when it closes.