Skip to content

ops: #1947 phases 2.5/3/5 are unrunnable — gptoss_rerun.sh, requant_sweep.sh and slots_sweep.sh were never committed and did not survive the rebuild #2985

Description

@Xore

The 2026-09-03/04 host rebuild wiped /mnt-1/benchmarks/. Three scratch scripts that the #1947 phase chain execs are gone and cannot be recovered:

script run by phase
gptoss_rerun.sh chain2b.sh 2.5 — re-measure empty-answer models under the harmony-aware harness (#2279)
requant_sweep.sh chain2b.sh (exec) 3 — self-requantisation sweep (#2245)
slots_sweep.sh chain3.sh (exec) 5 — sessions + Rev·Deck slots across the roster

Not in the hermes mirror, not in /mnt-1/benchmarks/scripts-copy/, not anywhere on the host, and never committed to this repo. sweep_extra.sh and preseed_ollama_config_blob.sh were committed under #2728 for precisely this reason ("so a script driving multi-hundred-GB pulls and real benchmark runs is reviewable and isn't one rm away from being lost") — these three were missed.

The 2026-09-04 resume launched chain2b.sh and chain3.sh anyway. They did not fail; they waited. chain2b.sh polls up to 14 days for EXTRA_COMPLETE, chain3.sh polls up to 14 days for a REQUANT_SWEEP_COMPLETE marker nothing can write. Both reported as running processes for five hours while being structurally incapable of doing anything.

Already done (2026-09-05)

  • chain2b.sh / chain3.sh now abort at launch when their downstream script is absent, instead of queueing behind an unwritable marker.
  • resume_phases.sh launches only the waiters whose scripts exist, and prints SKIP chain2b/SKIP chain3 otherwise.
  • Recorded in /mnt-1/benchmarks/STATE-2026-09-05-fix-stage.md.

Phase 2 (sweep_extra.sh) is unaffected and running.

Scope

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingopsDeployment, runners, observability, host access

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions