Schema-v2 cutover: canonical records, deterministic bridge, migration, progress manifests, adaptive and backtracking harnesses

.gitignore

@@ -39,7 +39,7 @@ htmlcov/
 gunicorn.ctl
 
 # Database
-metrics.db
+metrics*.db
 instance
 site/traces/
 

PLAN.md

@@ -0,0 +1,207 @@
+# LLM Quest Harness Engineering Plan
+
+## Goal
+
+Make long-running quest runs replayable, resumable, progress-aware, and attributable to an explicit harness treatment. Keep the QM environment authoritative. Backtracking is available only through a named experimental harness.
+
+## Decisions
+
+- Replace the existing implicit `AgentState` step record with a versioned transition record. No runtime reader accepts the old shape.
+- Persist the full engine saving state and the exact transition timestamp needed by `performJump`.
+- Initialize run metadata before execution. Do not patch identity/configuration after JSON export.
+- Use one-time migration for old SQLite and `run_summary.json` records. Migrated records remain analyzable but are marked non-resumable when legacy data cannot prove the executed action, engine saving, or timestamp.
+- Keep terminal outcome authoritative. Progress is a separate monotonic diagnostic derived from state-based milestone manifests.
+- Resume restores and verifies recorded state before any new model call.
+- Backtracking is a harness action with an explicit restore budget. It is not an evaluator convenience silently added to existing harnesses.
+- Adaptive reasoning changes prompt depth only when a progress stall or repeated state triggers it. It does not imply unsupported provider-specific reasoning controls.
+- Replace legacy template/memory inference maps with canonical harness specifications and treatment signatures.
+
+## Canonical contracts
+
+### Quest action
+
+A quest action is one of:
+
+- `choose`: one-based choice index plus the recorded `performed_at_ms` used by the engine transition.
+- `restore`: checkpoint index selected by the backtracking harness.
+
+### Quest snapshot
+
+A snapshot contains:
+
+- location ID;
+- observation text;
+- available choices with stable IDs and text;
+- parameter-state text;
+- reward, termination state, and authoritative engine game state;
+- full JSON-serializable engine saving;
+- a deterministic digest over canonical state fields.
+
+### Quest transition
+
+A transition contains:
+
+- monotonic transition index;
+- before snapshot;
+- executed quest action;
+- after snapshot;
+- agent response and usage;
+- progress state after execution;
+- provenance and replay status.
+
+Terminal state is the `after` snapshot of the final executed transition. It is not a synthetic decision row.
+
+### Run record v2
+
+`run_summary.json` and SQLite contain the same logical record:
+
+- `schema_version: 2`;
+- run identity and timestamps;
+- quest path, checksum, language, and engine revision;
+- canonical treatment description and signature;
+- resume lineage;
+- outcome, usage, metrics, and final snapshot;
+- explicit transitions.
+
+## Existing-record mapping
+
+The migration command accepts a legacy JSON file/directory or SQLite database and writes a separate v2 destination.
+
+For each legacy decision row:
+
+1. Build `before` from its location, observation, and choices.
+2. Use the next legacy row as `after` only when it is structurally the next observed state.
+3. Use `final_state` for the last `after` when available.
+4. Treat the logger-only terminal pseudo-step as terminal state, not as an executed action.
+5. Prefer SQLite `steps.action` over compact JSON `llm_decision.choice`, because old compact JSON could persist the model proposal rather than the runner-clamped action.
+6. Mark missing parameter state, engine saving, timestamp, choice ID, or post-state as unavailable. Never invent values.
+7. Set migrated transition provenance to `legacy_mapped` and replay status to `unavailable` unless all deterministic inputs exist.
+8. Recompute usage and metrics from mapped transitions.
+9. Derive the treatment from legacy agent config and the canonical harness registry. Unknown configurations receive an explicit `unknown` component, not a compatibility alias.
+
+The runtime, analyzers, reports, and site consume only v2 after cutover.
+
+## Implementation sequence
+
+### 1. Data model and engine protocol
+
+- Add action, snapshot, progress, transition, treatment, and run-record schema types.
+- Extend `QMBridgeState` with authoritative game state and full engine saving.
+- Change the TypeScript bridge protocol to accept structured choose/restore commands.
+- Pass the recorded transition timestamp to `performJump` instead of generating it inside the bridge.
+- Add exact `loadSaving` support and snapshot digest verification.
+- Replace `AgentState` callbacks and trajectory storage with canonical transitions.
+
+Acceptance:
+
+- A choose transition round-trips through JSON without losing choice IDs, saving state, timestamp, or executed action.
+- Loading a saving reproduces the recorded digest.
+- No synthetic terminal decision is emitted.
+
+### 2. Persistence cutover and migration
+
+- Replace legacy SQLite tables with v2 run and transition tables.
+- Remove column-addition, old-column fallback, post-run config patching, and random-run JSON suppression.
+- Export one v2 `run_summary.json` for every player type.
+- Update analyzers, reports, leaderboard generation, replay scripts, trace import/export, and CLI inspection to v2.
+- Add `scripts/migrate_records.py --source PATH --output PATH` for JSON trees and SQLite databases.
+- Update human/web trace export to retain restore events rather than erasing undone transitions.
+
+Acceptance:
+
+- A legacy fixture maps deterministically to the expected v2 record.
+- Unknowable fields are marked unavailable and the record is rejected for resume.
+- Current runs write no legacy fields or tables.
+
+### 3. Replay and resume
+
+- Add transition replay that executes recorded choose timestamps and restore actions and compares each resulting digest.
+- Add `llm-quest run --resume-from PATH`.
+- Resume loads quest and treatment from the record, verifies quest checksum and engine revision, rebuilds harness memory from transitions, restores the active checkpoint, then makes the next agent call.
+- Add a resumable `TRUNCATED` outcome for explicit step limits.
+- Link resumed runs to their source run and preserve prior transitions.
+
+Acceptance:
+
+- Replay of an unchanged record verifies every transition.
+- Any quest checksum, action, timestamp, or state mutation fails before model inference.
+- A truncated run resumes and reaches the same state as an uninterrupted deterministic run.
+
+### 4. Structured progress
+
+- Add a validated YAML progress manifest covering quest milestone predicates.
+- Match milestones against location ID, parameter-state contents, and engine game state.
+- Track current progress, maximum progress, newly reached milestones, and stalled transition count.
+- Include a state-based Boat manifest as the executable example.
+- Expose `progress_manifest` in benchmark configuration and recover it from a resume record.
+
+Acceptance:
+
+- Progress is monotonic even after restore.
+- Terminal success yields 100 percent.
+- Missing manifests fall back to terminal-only progress without guessing story advancement.
+
+### 5. Canonical harness treatments
+
+- Replace the class-only harness registry with specifications that declare prompt, memory, tools, loop, and reasoning policy.
+- Generate treatment signatures from canonical JSON plus model and material knobs.
+- Remove legacy template/memory compatibility maps and config-key shims.
+- Persist the full treatment before the first transition.
+
+Acceptance:
+
+- Materially different harness configurations have different signatures.
+- Equivalent configurations serialize and hash identically.
+- Reports group by treatment components without inferring them from names.
+
+### 6. Backtracking harness
+
+- Add the `backtracking` harness and a prompt/action parser that can choose or restore a checkpoint.
+- Add `restore_limit`, valid only for this harness.
+- Track restore attempts, accepted restores, restored distance, and progress recovered.
+- Preserve the full chronological transition log while maintaining a separate active branch checkpoint stack.
+
+Acceptance:
+
+- Restore loads the exact recorded saving and truncates only the active branch.
+- Restore events remain visible in persisted transitions and human/web traces.
+- Existing harnesses cannot emit restore actions.
+
+### 7. Adaptive reasoning harness
+
+- Add the `adaptive_reasoning` harness.
+- Use concise reasoning by default and a deeper planning prompt after repeated state or progress stall.
+- Add `adaptive_stall_steps`, valid only for this harness.
+- Persist the reasoning mode used on every transition.
+
+Acceptance:
+
+- Routine states use concise mode.
+- The configured trigger switches the next decision to deep mode.
+- Recovery resets the stall trigger without losing history.
+
+### 8. Verification and documentation
+
+- Update README, architecture, specification, CLI help/examples, configs, and public trace documentation.
+- Run the complete Python suite and JavaScript build.
+- Run a random-policy Boat quest end to end and inspect its v2 record.
+- Migrate a legacy JSON and SQLite fixture.
+- Truncate and resume a deterministic Boat run.
+- Exercise a restore action against the real QM bridge.
+
+## Non-goals
+
+- Supporting legacy runtime schemas or legacy configuration keys after migration.
+- Replacing the QM engine with Gymnasium, OpenEnv, Inspect, or Harbor.
+- Adding remote environment services, RL training, model graders, or automatic milestone generation.
+- Claiming progress metrics are comparable across quests without curated manifests.
+- Enabling backtracking for existing public harnesses.
+
+## Risks and controls
+
+- Engine randomness: record and replay the exact `performed_at_ms` and full saving.
+- Legacy ambiguity: mark it; do not infer a resumable state.
+- Resume contamination: verify quest checksum, engine revision, treatment, and state before inference.
+- Bundled treatment effects: persist component-level treatment data and keep new harnesses separately named.
+- Cost growth: adaptive deep mode is trigger-bound; backtracking has an explicit restore limit.
+- Public metric drift: keep terminal outcome primary and progress diagnostic until manifests are reviewed.

README.md

@@ -6,6 +6,8 @@
 
 Benchmark for evaluating LLM context scaffolds on interactive fiction quests. Measures how prompt context, compact memory, tools, and planning loops affect sequential decision-making across models and tasks.
 
+Every run is recorded as exact environment transitions (schema v2), making runs replayable, resumable after truncation by a step limit, and attributable to an explicit treatment configuration.
+
 **[Project Site](https://yourconscience.github.io/llm_quest_benchmark/)** | **[Leaderboard](https://yourconscience.github.io/llm_quest_benchmark/index.html)** | **[About / Write-up](https://yourconscience.github.io/llm_quest_benchmark/about.html)**
 
 See the [About page](https://yourconscience.github.io/llm_quest_benchmark/about.html) for the project narrative, taxonomy, metrics, caveats, and model selection rationale.
@@ -54,11 +56,25 @@ uv run llm-quest run --quest quests/Boat.qm --model gemini-3-flash-preview --tim
 # Run benchmark matrix
 uv run llm-quest benchmark --config configs/benchmarks/memory_full_transcript.yaml
 
+# Run with a curated progress manifest and an explicit step limit
+uv run llm-quest run --quest quests/Boat.qm --model gemini-3-flash-preview \
+  --progress-manifest configs/progress/Boat.yaml --max-steps 20
+
+# Continue a TRUNCATED run; quest and treatment come from the record
+uv run llm-quest run --resume-from results/<agent_id>/<quest>/run_<id>/run_summary.json
+
 # Generate report from benchmark results
 uv run llm-quest benchmark-report --benchmark-id <id> --output report.md
 
 # Analyze a single run
-uv run llm-quest analyze-run --run-summary results/<harness>/<quest>/run_<id>/run_summary.json
+uv run llm-quest analyze-run --run-summary results/<agent_id>/<quest>/run_<id>/run_summary.json
+
+# Convert pre-v2 records (one-time; the only legacy reader)
+uv run scripts/migrate_records.py --source results/ --output results_v2/
+uv run scripts/migrate_records.py --source metrics.db --output metrics_v2.db
+
+# Verify recorded runs against the real engine
+uv run scripts/replay_runs.py --results-dir results/
 
 # Play as human in terminal
 uv run llm-quest play --quest quests/Boat.qm
@@ -111,7 +127,9 @@ Provider-specific keys in `.env`:
 - `llm_quest_benchmark/players/` - Non-LLM player primitives (`human`, `random_choice`)
 - `llm_quest_benchmark/prompt_templates/` - Jinja2 prompt templates for the public context-scaffold taxonomy
 - `llm_quest_benchmark/executors/` - CLI, benchmark orchestration, TS bridge
+- `llm_quest_benchmark/schemas/records.py` - schema-v2 run record types
 - `configs/benchmarks/` - YAML benchmark configurations
+- `configs/progress/` - curated quest progress manifests
 - `quests/` - Quest files (downloaded via `download_quests.sh`)
 - `space-rangers-quest/` - TypeScript quest engine (submodule)
 - `docs/ARCHITECTURE.md` - Runtime architecture and taxonomy mapping

configs/benchmarks/exp8_recovery_harnesses.yaml

@@ -0,0 +1,41 @@
+# Exp 8: recovery harnesses (experimental, not part of the public taxonomy).
+#
+# Compares the compact-memory baseline against two recovery treatments:
+#   backtracking       - may restore a recorded checkpoint, bounded by restore_limit
+#   adaptive_reasoning - deepens its prompt after a repeated state or progress stall
+#
+# restore_limit is valid only for backtracking; adaptive_stall_steps only for
+# adaptive_reasoning. Any other pairing is rejected at config load.
+
+name: exp8_recovery_harnesses
+
+quests:
+  - quests/Boat.qm
+
+agents:
+  - model: gpt-5-mini
+    harness: memo_compact
+    temperature: 0.4
+    skip_single: true
+    runs: 5
+
+  - model: gpt-5-mini
+    harness: backtracking
+    temperature: 0.4
+    skip_single: true
+    restore_limit: 3
+    runs: 5
+
+  - model: gpt-5-mini
+    harness: adaptive_reasoning
+    temperature: 0.4
+    skip_single: true
+    adaptive_stall_steps: 3
+    runs: 5
+
+debug: false
+quest_timeout: 300
+max_steps: 60
+max_workers: 2
+output_dir: results/benchmarks
+progress_manifest: configs/progress/Boat.yaml

configs/progress/Boat.yaml

@@ -0,0 +1,66 @@
+# Curated progress manifest for quests/Boat.qm (four gods river crossing).
+#
+# Milestones are state predicates over the engine's location id and rendered
+# parameter state. Gods standing on the right bank render as "xxx - <Name>(N)";
+# gods still on the left bank render as "<Name> (N) - xxx".
+#
+# Progress is a diagnostic. Terminal success always reports 100 percent
+# regardless of which milestones matched.
+quest: Boat
+version: 1
+milestones:
+  - id: mission_accepted
+    percent: 10
+    description: Accepted the expedition contract and landed on Bonnasis.
+    match:
+      location_id: ["2"]
+
+  - id: legend_heard
+    percent: 20
+    description: Heard the legend of the four sons of the elder god.
+    match:
+      location_id: ["3"]
+
+  - id: ceremony_explained
+    percent: 30
+    description: Learned the ceremony rules that constrain the crossing.
+    match:
+      location_id: ["4", "5"]
+
+  - id: crossing_started
+    percent: 40
+    description: Entered the boat puzzle with the crossing clock running.
+    match:
+      location_id: ["6", "7", "8", "9", "10", "14"]
+
+  - id: one_god_across
+    percent: 55
+    description: At least one god stands on the right bank.
+    match:
+      params_pattern:
+        pattern: "^xxx - (Ах|Бах|Вау|Гэ)"
+        min_count: 1
+
+  - id: two_gods_across
+    percent: 70
+    description: At least two gods stand on the right bank.
+    match:
+      params_pattern:
+        pattern: "^xxx - (Ах|Бах|Вау|Гэ)"
+        min_count: 2
+
+  - id: three_gods_across
+    percent: 85
+    description: At least three gods stand on the right bank.
+    match:
+      params_pattern:
+        pattern: "^xxx - (Ах|Бах|Вау|Гэ)"
+        min_count: 3
+
+  - id: all_gods_across
+    percent: 90
+    description: All four gods stand on the right bank.
+    match:
+      params_pattern:
+        pattern: "^xxx - (Ах|Бах|Вау|Гэ)"
+        min_count: 4

configs/test/test_benchmark.yaml

@@ -10,3 +10,5 @@ debug: true
 quest_timeout: 60
 max_workers: 2
 output_dir: results/benchmarks
+# Curated state-based milestones; progress stays diagnostic, outcome stays authoritative.
+progress_manifest: configs/progress/Boat.yaml