Automate full merge train scheduler

[cbusillo]

Objective

Replace manual one-pass operation with a scheduler that advances active trains through safe repeated passes. The scheduler should pace GitHub reads/mutations, respect active train locks, and continue work automatically when checks complete or blockers clear.

Finish Line

Automate full merge train scheduler

Current Status

State: Manual dispatch now proves the full batch flow end to end: candidate plan/build/observe plus landing plan/land. The missing piece is unattended scheduler/controller automation that advances those phases without manual record-id copying.
Next action: Implement scheduler/controller state advancement: find latest active candidate/landing records, run the next phase, stop safely on pending/fail/stale, and summarize the active record IDs in the workflow UI.
Blocked by: No service primitive blocker; this is now scheduler orchestration and UX work.
Last verified: 2026-05-14 via live codex-skills proof. Manual runs: candidate plan 25835435391, build 25835453873, observe 25835478586, landing plan 25836427035, land 25836447245.

Scope

• In: scheduler admission, locks, polling cadence, backoff, re-read rules, per-train concurrency, stale run recovery, and workflow/runtime configuration.
• Out: UI beyond read-model data needed by feedback/operator work.

Acceptance Criteria

• Only one scheduler mutates a train at a time for each repository + base_branch.
• Scheduler can resume after interrupted runs using persisted records.
• Polling/backoff avoids unnecessary GitHub/API load while still reacting after checks complete.
• Scheduled dry-run and mutate modes are separately controllable through DB-backed policy/records or explicit workflow inputs, not hardcoded repos.
• Every automatic transition writes a run record and queue-state change.

Relationships

• Blocked by queue records and speculative validation.
• Related to Inventory self-hosted runner lanes #413 and Control self-hosted runner lanes #414.

Validation

• Unit tests for admission, locking, backoff, stale run recovery, and concurrency.
• Workflow dispatch dry-runs for at least two consecutive scheduler passes.
• Live scheduled dry-run before scheduled mutate is enabled.

Decisions

• Keep one mutation per fresh evidence boundary unless the full-train state machine explicitly proves a chained transition safe.

Open Questions

• Should train scheduling live in GitHub Actions initially, or move into a long-running Launchplane worker later?