AB#294553 Data egress to LDS

[ajs-uk-dev]

Summary

• Adds a new admin section (/admin/egress) where a CYPMD ops user pulls the scrutiny decisions
  for one checking window's Add and Remove amendment requests, preprocesses the approved ones into
  LDS-spec CSV files, persists them, and transfers them to the LDS storage account.
• Zendesk supplies only the per-ticket decision; every record value comes from CYPMD's own data.
  Runs are persisted at pull time (so "save" is just leaving the page) with a database-enforced
  one-active-run-per-window-and-output-type rule, an eight-step server-reported preprocessing
  pipeline that is all-or-nothing, and a create-only, compensating, audited transfer.
• New learners' column set is FLAGGED pending the LDS spec v2.4 workbook — see
  docs/data-egress.md and the PR notes for every other FLAGGED item.

Test plan

• Unit: 5,415 passed, 0 failed
• Integration (Testcontainers Postgres + Azurite): 838 passed, 0 failed
• E2E (full suite, in container): 218 passed, 0 failed, 2 skipped (pre-existing, unrelated)
• dotnet ef migrations has-pending-model-changes: none
• Walked the live app: Pull (including the S5 error markup), Results, Preprocessing (both the
  streamed and no-JS paths, and Abandon from that page), Summary, Complete, the "already being
  processed" and "already transferred" refusals, resume-from-list, and — in the closing pass —
  the Abandon-ordering fix in both directions (owned vs. foreign-stamped blob) and the
  PreprocessingFailed redirect, against a real Azurite account with the image rebuilt from
  this branch's final tip
• git status --porcelain clean after discarding rewritten E2E snapshots

An independent Opus review (.superpowers/plans/2026-09-14-294553-opus-review.md) found one
Blocker (B1: the web host defaulted to the dev Zendesk fake) and four Must-fixes in the transfer/
lock state machine (M1-M4), plus ten Should-fixes, a handful of nits, and six questions for BA/LDS.
All Blocker/Must-fix/Should-fix findings and the named nits are fixed in 17 follow-up commits on
this branch (770eeea3..8565cb49). A second-pass review of those fixes found one further
Must-fix (R1: AbandonAsync swept a run's blobs before writing its Abandoned status, which could
let a concurrent transfer's just-uploaded files be deleted, or — the more serious direction — let
the run end up Transferred with a Succeeded audit row and no files actually in LDS; fixed by
writing the status first and sweeping only once that write has landed) plus seven optional nits
(one of which, the PreprocessingFailed refusal message, is user-visible — it no longer leaks the
raw C# status name, e.g. "This run cannot be preprocessed because its stage is Preprocessing
failed." instead of "...is PreprocessingFailed..."). A final whole-branch review of that closing
work found no blocking issues and two small follow-ups, both logged in PR notes §7 item 9 rather
than fixed here (a narrower crash-window orphan-blob case the R1 fix's own trade-off introduces,
and a pre-existing ownership-check asymmetry in the transfer's own failure-compensation path) —
see PR notes §9 for the full finding-by-finding record across all three review passes. The
PreprocessingStream state-mutating-GET pattern was left as-is (an accepted precedent elsewhere in
this codebase) but is worth calling out in review: the JS closes the EventSource on a terminal/
error event, so the browser's automatic reconnect never restarts the server-side pipeline.

Closes #244

[ajs-uk-dev]

LDS spec v2.4 alignment pushed — see docs/data-egress.md §6 and the Q7–Q14 questions for LDS/BA

[ajs-uk-dev]

Response to the 18 Sep review

Fixed in 6503e28a, restored diff in e812ee00, main merged in 4a101cb4.

1. CSV formula injection — fixed, differently from the suggestion. LdsSpecValidator now refuses a Surname or Forename beginning with =, +, - or @ (the only free-text cells; typed by the school in the Add journey) with the reason "must not begin with =, +, - or @". A hyphen inside a name still passes. Rejecting at preprocessing keeps the file spec-pure — LDS expects raw values and no real name begins with one of these — and the ops user sees which record to look at, instead of an apostrophe being silently written into a name. Pinned by three new LdsSpecValidatorTests facts; docs/data-egress.md step 7 updated.

2. Abandon-vs-transfer race — consciously accepted, and the mechanism is not quite as described. AbandonAsync is a compare-and-set (Status != Transferred → Abandoned), and the service sweeps blobs only after that write reports a row changed (R1). If Abandon wins mid-upload, TransferAsync's final MarkTransferredAsync(expected: Transferring) returns 0 rows, it compensates its own uploads, and the run ends Abandoned, no files, no Succeeded audit — a consistent end state, covered by EgressTransferServiceTests. The only exposure is a verified file being visible for the few seconds before it is swept, and the existing failed-transfer compensation path has exactly the same window. Refusing Abandon on Transferring would bring back the stuck-run problem M1 exists for (the pair lock has no expiry; a pod restart mid-upload would need a DBA). An age-based guard ("refuse Abandon unless Transferring for more than N minutes") is the narrowing I would take if wanted — it needs a TransferringAtUtc column and a migration, so it is a follow-up rather than a pre-merge change.

3. Early config gating — fixed. The Pull page and the Summary show a govuk-warning-text (egress-storage-not-configured) when ConnectionStrings:EgressStorage is absent. Pull, preprocess, preview and download are deliberately not blocked — they are useful without a target account; only Transfer refuses, as before. Pinned in EgressControllerTests and EgressViewSourceTests.

4. Sex mapping — verified, no change. All three Add_*.json flows store the radio option value (F/M/U), not the label; the journey answer that reaches EgressRecordBuilder is that value, and the dev seeder mirrors it. Female cannot arrive.

5. Batch semantics — kept as is. A Zendesk failure (after the Polly retries) throws EgressTicketSourceException and the pull is refused before any run is created (EgressStartResult.PullFailed); a single missing ticket is not a failure — that record reads as "Ticket not found" and is discarded at the approval filter. Partial-with-NotFound on a transient blip would let an operator transfer a set that silently lacks approved records, which is worse than a one-click retry of the pull.

6. Header-only file for one empty output — open question for LDS. Already in the documented gaps (docs/data-egress.md §10). Today the operator can untick the empty output type before pulling; skipping empty outputs at transfer is a small change if LDS says header-only files are unwelcome.

7. Diff hygiene — half done. The 34 search-ux PNGs are gone from the diff (e812ee00) — they were reviewer screenshots that every full E2E run regenerates, not an intended change. I have not rebased: the branch carries five merges from main and the repo allows squash merge, so squashing at merge time gives the same single-commit view without rewriting shared history.

Verified on 4a101cb4: .slnx Release build 0 errors / 117 warnings (unchanged), unit 5,619/0, integration 870/0, no pending EF model changes. Review-app E2E is running now.