[Bug]: waiting_for_user becomes visible before pending input exists

[dennisonbertram]

Summary

The runner exposed waiting_for_user before AskUserQuestion pending input was readable. Review then found the same lifecycle could be broken by notifier deadlines, quick accepted answers, stale status writes, process-local checkpoint resolution, callback-omitting third-party brokers, transient persistence failures, and SSE replay gaps. This issue owns the complete harness invariant required by TUI/native clients: once waiting is visible, input is readable and the wait/resume transcript advances exactly once.

Acceptance criteria

• A run cannot expose waiting_for_user or emit run.waiting_for_user until PendingInput(runID) succeeds.
• The invariant holds for in-memory, checkpoint-backed, and callback-omitting brokers that expose Pending.
• Callback and fallback observer share once-on-success publication: immediate UpdateRun/AppendEvent failures retry, an observer already publishing drains to success/deadline, and exactly one wait/resume is visible.
• An accepted answer cannot emit resume or return from broker recovery before pending publication completes; notifier cancellation cannot turn accepted input into a synthetic timeout.
• Run status persistence is serialized per run, honors notifier context, and delayed nonterminal writes cannot replace terminal status.
• Checkpoint resolution is pending-only and single-winner across Service instances sharing a store; unrelated records are not serialized and waiting callers honor context.
• Cross-Service waiters observe remote durable resolution, remain registered across transient polling read failures, and local resolution notification is not overtaken.
• Failed strict waiting-event appends do not fan out or enter replay, roll back the sequence allocation, and preserve contiguous run SSE IDs plus duplicate-free Last-Event-ID replay.
• Intentional redaction suppression completes publication without retrying; ordinary nonterminal events retain best-effort persistence behavior.
• Invalid, denied, cancelled, timed-out, and lost-race paths remain bounded and return stable no-pending/conflict semantics.
• Full normal, race, and coverage gates pass.

In scope

• AskUserQuestion pending registration, notifier/observer, status, event, timeout, and accepted-answer ordering.
• In-memory and checkpoint broker semantics.
• Atomic Memory/SQLite pending resolution, per-record Service coordination, and cross-Service waiter observation.
• Strict waiting-event persistence/replay semantics and run SSE cursor continuity.
• Harness/server regressions and required plan, impact map, and logs.

Out of scope

• Question schema or answer UX changes.
• Cron/callback scheduling semantics.
• A global change to ordinary event persistence behavior.

Current-architecture search evidence

• internal/harness/runner_step_engine.go owns AskUserQuestion execution and lifecycle publication.
• internal/harness/ask_user_broker.go and internal/harness/checkpoint_brokers.go register and resolve pending input.
• internal/checkpoints/service.go coordinates waits/resolution; Memory and SQLite stores own durable single-winner semantics.
• internal/harness/runner.go and runner_event_journal.go own status/event persistence and replay ordering.
• internal/server/http_runs.go treats run event sequence as the replay cursor for Last-Event-ID.
• Hosted race run 30584671976 reproduced PendingInput: no pending input after observing waiting_for_user.

Cross-surface impact

• Harness lifecycle/API: strengthens pending/status/event atomicity and durable resolution.
• TUI/native GUI: clients can immediately render/submit a visible question and observe one ordered wait/resume conversation.
• Persistence: adds atomic pending-only store resolution and strict persistence only for the waiting lifecycle event.
• Concurrency: per-record checkpoint resolution, per-run status serialization, context-aware notifier work, fallback Pending observation, and opportunistic cross-Service polling.
• SSE/reconnect: failed strict publication cannot create a visible sequence gap or duplicate replay.
• Security/privacy: no new question or answer data is persisted or logged; StorageModeNone remains authoritative.
• Cron/callback: no scheduler change; this unblocks their end-to-end promotion chain.

Test-first plan

1. Preserve each review finding as a deterministic red regression before its fix.
2. Publish only after readable pending registration and preserve accepted-answer ordering.
3. Make checkpoint resolution atomic across stores/Service instances and make remote wait observation resilient.
4. Serialize status/event publication, add callback-omission fallback, and retry transient waiting publication once-on-success.
5. Preserve SSE cursor continuity, redaction semantics, and ordinary-event baseline behavior.
6. Run repeated focused normal/race stress, affected-package normal/race, and unchanged ./scripts/test-regression.sh.

Rollout / rollback

Merge before the cron/callback repair chain. No schema migration or feature flag is required. Revert if exact-head hosted checks, AskUserQuestion UI flow, or SSE reconnect behavior regress.

Evidence

• https://github.com/dennisonbertram/go-code/actions/runs/30584671976/job/91013125023
• Blocks PR Stabilize provider API-key capture test #1053 and transitively PR Keep pinned runs outside completed retention quota #1051.