diff --git a/.gitignore b/.gitignore index 3edf86e..4d7dfee 100644 --- a/.gitignore +++ b/.gitignore @@ -39,7 +39,7 @@ htmlcov/ gunicorn.ctl # Database -metrics.db +metrics*.db instance site/traces/ diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..6fea7fb --- /dev/null +++ b/PLAN.md @@ -0,0 +1,207 @@ +# LLM Quest Harness Engineering Plan + +## Goal + +Make long-running quest runs replayable, resumable, progress-aware, and attributable to an explicit harness treatment. Keep the QM environment authoritative. Backtracking is available only through a named experimental harness. + +## Decisions + +- Replace the existing implicit `AgentState` step record with a versioned transition record. No runtime reader accepts the old shape. +- Persist the full engine saving state and the exact transition timestamp needed by `performJump`. +- Initialize run metadata before execution. Do not patch identity/configuration after JSON export. +- Use one-time migration for old SQLite and `run_summary.json` records. Migrated records remain analyzable but are marked non-resumable when legacy data cannot prove the executed action, engine saving, or timestamp. +- Keep terminal outcome authoritative. Progress is a separate monotonic diagnostic derived from state-based milestone manifests. +- Resume restores and verifies recorded state before any new model call. +- Backtracking is a harness action with an explicit restore budget. It is not an evaluator convenience silently added to existing harnesses. +- Adaptive reasoning changes prompt depth only when a progress stall or repeated state triggers it. It does not imply unsupported provider-specific reasoning controls. +- Replace legacy template/memory inference maps with canonical harness specifications and treatment signatures. + +## Canonical contracts + +### Quest action + +A quest action is one of: + +- `choose`: one-based choice index plus the recorded `performed_at_ms` used by the engine transition. +- `restore`: checkpoint index selected by the backtracking harness. + +### Quest snapshot + +A snapshot contains: + +- location ID; +- observation text; +- available choices with stable IDs and text; +- parameter-state text; +- reward, termination state, and authoritative engine game state; +- full JSON-serializable engine saving; +- a deterministic digest over canonical state fields. + +### Quest transition + +A transition contains: + +- monotonic transition index; +- before snapshot; +- executed quest action; +- after snapshot; +- agent response and usage; +- progress state after execution; +- provenance and replay status. + +Terminal state is the `after` snapshot of the final executed transition. It is not a synthetic decision row. + +### Run record v2 + +`run_summary.json` and SQLite contain the same logical record: + +- `schema_version: 2`; +- run identity and timestamps; +- quest path, checksum, language, and engine revision; +- canonical treatment description and signature; +- resume lineage; +- outcome, usage, metrics, and final snapshot; +- explicit transitions. + +## Existing-record mapping + +The migration command accepts a legacy JSON file/directory or SQLite database and writes a separate v2 destination. + +For each legacy decision row: + +1. Build `before` from its location, observation, and choices. +2. Use the next legacy row as `after` only when it is structurally the next observed state. +3. Use `final_state` for the last `after` when available. +4. Treat the logger-only terminal pseudo-step as terminal state, not as an executed action. +5. Prefer SQLite `steps.action` over compact JSON `llm_decision.choice`, because old compact JSON could persist the model proposal rather than the runner-clamped action. +6. Mark missing parameter state, engine saving, timestamp, choice ID, or post-state as unavailable. Never invent values. +7. Set migrated transition provenance to `legacy_mapped` and replay status to `unavailable` unless all deterministic inputs exist. +8. Recompute usage and metrics from mapped transitions. +9. Derive the treatment from legacy agent config and the canonical harness registry. Unknown configurations receive an explicit `unknown` component, not a compatibility alias. + +The runtime, analyzers, reports, and site consume only v2 after cutover. + +## Implementation sequence + +### 1. Data model and engine protocol + +- Add action, snapshot, progress, transition, treatment, and run-record schema types. +- Extend `QMBridgeState` with authoritative game state and full engine saving. +- Change the TypeScript bridge protocol to accept structured choose/restore commands. +- Pass the recorded transition timestamp to `performJump` instead of generating it inside the bridge. +- Add exact `loadSaving` support and snapshot digest verification. +- Replace `AgentState` callbacks and trajectory storage with canonical transitions. + +Acceptance: + +- A choose transition round-trips through JSON without losing choice IDs, saving state, timestamp, or executed action. +- Loading a saving reproduces the recorded digest. +- No synthetic terminal decision is emitted. + +### 2. Persistence cutover and migration + +- Replace legacy SQLite tables with v2 run and transition tables. +- Remove column-addition, old-column fallback, post-run config patching, and random-run JSON suppression. +- Export one v2 `run_summary.json` for every player type. +- Update analyzers, reports, leaderboard generation, replay scripts, trace import/export, and CLI inspection to v2. +- Add `scripts/migrate_records.py --source PATH --output PATH` for JSON trees and SQLite databases. +- Update human/web trace export to retain restore events rather than erasing undone transitions. + +Acceptance: + +- A legacy fixture maps deterministically to the expected v2 record. +- Unknowable fields are marked unavailable and the record is rejected for resume. +- Current runs write no legacy fields or tables. + +### 3. Replay and resume + +- Add transition replay that executes recorded choose timestamps and restore actions and compares each resulting digest. +- Add `llm-quest run --resume-from PATH`. +- Resume loads quest and treatment from the record, verifies quest checksum and engine revision, rebuilds harness memory from transitions, restores the active checkpoint, then makes the next agent call. +- Add a resumable `TRUNCATED` outcome for explicit step limits. +- Link resumed runs to their source run and preserve prior transitions. + +Acceptance: + +- Replay of an unchanged record verifies every transition. +- Any quest checksum, action, timestamp, or state mutation fails before model inference. +- A truncated run resumes and reaches the same state as an uninterrupted deterministic run. + +### 4. Structured progress + +- Add a validated YAML progress manifest covering quest milestone predicates. +- Match milestones against location ID, parameter-state contents, and engine game state. +- Track current progress, maximum progress, newly reached milestones, and stalled transition count. +- Include a state-based Boat manifest as the executable example. +- Expose `progress_manifest` in benchmark configuration and recover it from a resume record. + +Acceptance: + +- Progress is monotonic even after restore. +- Terminal success yields 100 percent. +- Missing manifests fall back to terminal-only progress without guessing story advancement. + +### 5. Canonical harness treatments + +- Replace the class-only harness registry with specifications that declare prompt, memory, tools, loop, and reasoning policy. +- Generate treatment signatures from canonical JSON plus model and material knobs. +- Remove legacy template/memory compatibility maps and config-key shims. +- Persist the full treatment before the first transition. + +Acceptance: + +- Materially different harness configurations have different signatures. +- Equivalent configurations serialize and hash identically. +- Reports group by treatment components without inferring them from names. + +### 6. Backtracking harness + +- Add the `backtracking` harness and a prompt/action parser that can choose or restore a checkpoint. +- Add `restore_limit`, valid only for this harness. +- Track restore attempts, accepted restores, restored distance, and progress recovered. +- Preserve the full chronological transition log while maintaining a separate active branch checkpoint stack. + +Acceptance: + +- Restore loads the exact recorded saving and truncates only the active branch. +- Restore events remain visible in persisted transitions and human/web traces. +- Existing harnesses cannot emit restore actions. + +### 7. Adaptive reasoning harness + +- Add the `adaptive_reasoning` harness. +- Use concise reasoning by default and a deeper planning prompt after repeated state or progress stall. +- Add `adaptive_stall_steps`, valid only for this harness. +- Persist the reasoning mode used on every transition. + +Acceptance: + +- Routine states use concise mode. +- The configured trigger switches the next decision to deep mode. +- Recovery resets the stall trigger without losing history. + +### 8. Verification and documentation + +- Update README, architecture, specification, CLI help/examples, configs, and public trace documentation. +- Run the complete Python suite and JavaScript build. +- Run a random-policy Boat quest end to end and inspect its v2 record. +- Migrate a legacy JSON and SQLite fixture. +- Truncate and resume a deterministic Boat run. +- Exercise a restore action against the real QM bridge. + +## Non-goals + +- Supporting legacy runtime schemas or legacy configuration keys after migration. +- Replacing the QM engine with Gymnasium, OpenEnv, Inspect, or Harbor. +- Adding remote environment services, RL training, model graders, or automatic milestone generation. +- Claiming progress metrics are comparable across quests without curated manifests. +- Enabling backtracking for existing public harnesses. + +## Risks and controls + +- Engine randomness: record and replay the exact `performed_at_ms` and full saving. +- Legacy ambiguity: mark it; do not infer a resumable state. +- Resume contamination: verify quest checksum, engine revision, treatment, and state before inference. +- Bundled treatment effects: persist component-level treatment data and keep new harnesses separately named. +- Cost growth: adaptive deep mode is trigger-bound; backtracking has an explicit restore limit. +- Public metric drift: keep terminal outcome primary and progress diagnostic until manifests are reviewed. diff --git a/README.md b/README.md index 013fb57..d3fe4c3 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ Benchmark for evaluating LLM context scaffolds on interactive fiction quests. Measures how prompt context, compact memory, tools, and planning loops affect sequential decision-making across models and tasks. +Every run is recorded as exact environment transitions (schema v2), making runs replayable, resumable after truncation by a step limit, and attributable to an explicit treatment configuration. + **[Project Site](https://yourconscience.github.io/llm_quest_benchmark/)** | **[Leaderboard](https://yourconscience.github.io/llm_quest_benchmark/index.html)** | **[About / Write-up](https://yourconscience.github.io/llm_quest_benchmark/about.html)** See the [About page](https://yourconscience.github.io/llm_quest_benchmark/about.html) for the project narrative, taxonomy, metrics, caveats, and model selection rationale. @@ -54,11 +56,25 @@ uv run llm-quest run --quest quests/Boat.qm --model gemini-3-flash-preview --tim # Run benchmark matrix uv run llm-quest benchmark --config configs/benchmarks/memory_full_transcript.yaml +# Run with a curated progress manifest and an explicit step limit +uv run llm-quest run --quest quests/Boat.qm --model gemini-3-flash-preview \ + --progress-manifest configs/progress/Boat.yaml --max-steps 20 + +# Continue a TRUNCATED run; quest and treatment come from the record +uv run llm-quest run --resume-from results///run_/run_summary.json + # Generate report from benchmark results uv run llm-quest benchmark-report --benchmark-id --output report.md # Analyze a single run -uv run llm-quest analyze-run --run-summary results///run_/run_summary.json +uv run llm-quest analyze-run --run-summary results///run_/run_summary.json + +# Convert pre-v2 records (one-time; the only legacy reader) +uv run scripts/migrate_records.py --source results/ --output results_v2/ +uv run scripts/migrate_records.py --source metrics.db --output metrics_v2.db + +# Verify recorded runs against the real engine +uv run scripts/replay_runs.py --results-dir results/ # Play as human in terminal uv run llm-quest play --quest quests/Boat.qm @@ -111,7 +127,9 @@ Provider-specific keys in `.env`: - `llm_quest_benchmark/players/` - Non-LLM player primitives (`human`, `random_choice`) - `llm_quest_benchmark/prompt_templates/` - Jinja2 prompt templates for the public context-scaffold taxonomy - `llm_quest_benchmark/executors/` - CLI, benchmark orchestration, TS bridge +- `llm_quest_benchmark/schemas/records.py` - schema-v2 run record types - `configs/benchmarks/` - YAML benchmark configurations +- `configs/progress/` - curated quest progress manifests - `quests/` - Quest files (downloaded via `download_quests.sh`) - `space-rangers-quest/` - TypeScript quest engine (submodule) - `docs/ARCHITECTURE.md` - Runtime architecture and taxonomy mapping diff --git a/configs/benchmarks/exp8_recovery_harnesses.yaml b/configs/benchmarks/exp8_recovery_harnesses.yaml new file mode 100644 index 0000000..4104bb8 --- /dev/null +++ b/configs/benchmarks/exp8_recovery_harnesses.yaml @@ -0,0 +1,41 @@ +# Exp 8: recovery harnesses (experimental, not part of the public taxonomy). +# +# Compares the compact-memory baseline against two recovery treatments: +# backtracking - may restore a recorded checkpoint, bounded by restore_limit +# adaptive_reasoning - deepens its prompt after a repeated state or progress stall +# +# restore_limit is valid only for backtracking; adaptive_stall_steps only for +# adaptive_reasoning. Any other pairing is rejected at config load. + +name: exp8_recovery_harnesses + +quests: + - quests/Boat.qm + +agents: + - model: gpt-5-mini + harness: memo_compact + temperature: 0.4 + skip_single: true + runs: 5 + + - model: gpt-5-mini + harness: backtracking + temperature: 0.4 + skip_single: true + restore_limit: 3 + runs: 5 + + - model: gpt-5-mini + harness: adaptive_reasoning + temperature: 0.4 + skip_single: true + adaptive_stall_steps: 3 + runs: 5 + +debug: false +quest_timeout: 300 +max_steps: 60 +max_workers: 2 +output_dir: results/benchmarks +progress_manifest: configs/progress/Boat.yaml diff --git a/configs/progress/Boat.yaml b/configs/progress/Boat.yaml new file mode 100644 index 0000000..64e1d6d --- /dev/null +++ b/configs/progress/Boat.yaml @@ -0,0 +1,66 @@ +# Curated progress manifest for quests/Boat.qm (four gods river crossing). +# +# Milestones are state predicates over the engine's location id and rendered +# parameter state. Gods standing on the right bank render as "xxx - (N)"; +# gods still on the left bank render as " (N) - xxx". +# +# Progress is a diagnostic. Terminal success always reports 100 percent +# regardless of which milestones matched. +quest: Boat +version: 1 +milestones: + - id: mission_accepted + percent: 10 + description: Accepted the expedition contract and landed on Bonnasis. + match: + location_id: ["2"] + + - id: legend_heard + percent: 20 + description: Heard the legend of the four sons of the elder god. + match: + location_id: ["3"] + + - id: ceremony_explained + percent: 30 + description: Learned the ceremony rules that constrain the crossing. + match: + location_id: ["4", "5"] + + - id: crossing_started + percent: 40 + description: Entered the boat puzzle with the crossing clock running. + match: + location_id: ["6", "7", "8", "9", "10", "14"] + + - id: one_god_across + percent: 55 + description: At least one god stands on the right bank. + match: + params_pattern: + pattern: "^xxx - (Ах|Бах|Вау|Гэ)" + min_count: 1 + + - id: two_gods_across + percent: 70 + description: At least two gods stand on the right bank. + match: + params_pattern: + pattern: "^xxx - (Ах|Бах|Вау|Гэ)" + min_count: 2 + + - id: three_gods_across + percent: 85 + description: At least three gods stand on the right bank. + match: + params_pattern: + pattern: "^xxx - (Ах|Бах|Вау|Гэ)" + min_count: 3 + + - id: all_gods_across + percent: 90 + description: All four gods stand on the right bank. + match: + params_pattern: + pattern: "^xxx - (Ах|Бах|Вау|Гэ)" + min_count: 4 diff --git a/configs/test/test_benchmark.yaml b/configs/test/test_benchmark.yaml index b3321d9..78e6da5 100644 --- a/configs/test/test_benchmark.yaml +++ b/configs/test/test_benchmark.yaml @@ -10,3 +10,5 @@ debug: true quest_timeout: 60 max_workers: 2 output_dir: results/benchmarks +# Curated state-based milestones; progress stays diagnostic, outcome stays authoritative. +progress_manifest: configs/progress/Boat.yaml diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 60b2982..dc14576 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -11,9 +11,10 @@ The runtime loop is: 1. Parse or step quest state via the TypeScript engine bridge. 2. Build harness context from current state, available choices, and memory. -3. Get a choice from a human, random policy, or LLM-backed harness. -4. Apply the choice, log the step, and detect the terminal outcome. -5. Persist run metrics and run summaries. +3. Get a quest action from a human, random policy, or LLM-backed harness. +4. Execute the action with a recorded timestamp, record the canonical + transition, update progress, and detect the terminal outcome. +5. Persist the schema-v2 run record to SQLite and `run_summary.json`. ## Harness Engineering Framing @@ -48,7 +49,23 @@ planning choices change behavior. ### 2. Environment Layer - `llm_quest_benchmark/environments/qm.py`: Wraps the bridge into Python - environment semantics (`reset`, `step`, terminal detection). + environment semantics (`reset`, `step`, `restore`, terminal detection). Every + call returns a canonical `QuestSnapshot` carrying the full engine saving and a + deterministic digest. + +### 2a. Record Layer + +- `llm_quest_benchmark/schemas/records.py`: `QuestAction`, `QuestSnapshot`, + `ProgressState`, `QuestTransition`, and `RunRecord` (schema v2). These are the + only transition/run types accepted at runtime. +- `llm_quest_benchmark/core/provenance.py`: quest checksum and engine revision + used to gate replay and resume. +- `llm_quest_benchmark/core/progress.py`: validated YAML progress manifests and + the monotonic progress tracker. +- `llm_quest_benchmark/core/replay.py`: transition replay, environment + verification, and resume assembly. +- `llm_quest_benchmark/core/migration.py`: the only legacy reader, used by + `scripts/migrate_records.py` (one-time conversion tool, not a CLI command). ### 3. Harness Layer @@ -60,13 +77,18 @@ planning choices change behavior. - `llm_quest_benchmark/harnesses/tools.py`: Calculator, scratchpad, and quest history helpers used by tool harnesses. - `llm_quest_benchmark/harnesses/trajectory.py`: `Trajectory`, a run-local - retrieval/index view holding references to canonical executed `AgentState` - objects for `programmatic_memory`, with bounded deterministic `read`/`search`. - `QuestRunner` emits the same object to the harness, callbacks, and - `QuestLogger`; `QuestLogger` serializes the persisted `run_summary.json` - trace. + retrieval/index view holding references to canonical executed + `QuestTransition` objects for `programmatic_memory`, with bounded + deterministic `read`/`search`. `QuestRunner` emits the same object to the + harness, callbacks, and `QuestLogger`; `QuestLogger` serializes the persisted + `run_summary.json` record. +- `llm_quest_benchmark/harnesses/specs.py`: canonical harness specifications + (prompt, memory, tools, loop, reasoning) and treatment signatures. +- `llm_quest_benchmark/harnesses/backtracking.py`, + `llm_quest_benchmark/harnesses/adaptive.py`: experimental harnesses for + checkpoint restore and adaptive reasoning depth. - `llm_quest_benchmark/harnesses/factory.py`: `create_harness()` and the - canonical harness registry. + implementation classes behind each specification. - `llm_quest_benchmark/players/human.py`, `llm_quest_benchmark/players/random.py`: Non-LLM `QuestPlayer` implementations preserved for interactive and random baselines. @@ -88,12 +110,13 @@ and benchmark configuration parsing do not require API keys. - `llm_quest_benchmark/core/analyzer.py`: Post-run analysis and benchmark summaries. - `llm_quest_benchmark/core/benchmark_report.py`: Markdown report generator. -- `llm_quest_benchmark/core/logging.py`: Quest logger with per-run metrics - (`repetition_rate`, `bad_decision_rate`). +- `llm_quest_benchmark/core/logging.py`: Quest logger that writes complete run + metadata before execution, persists each transition, and computes per-run + metrics (`repetition_rate`, `bad_decision_rate`, restore statistics). - `llm_quest_benchmark/executors/benchmark.py`: Benchmark orchestration with parallel workers. - `llm_quest_benchmark/executors/cli/commands.py`: CLI commands (`run`, `play`, - `analyze`, `analyze-run`, `benchmark`, `benchmark-report`, + `analyze`, `analyze-run`, `benchmark`, `benchmark-report`, `leaderboard`, `download-quests`, `cleanup`). ### 6. Prompt Templates @@ -112,18 +135,30 @@ and benchmark configuration parsing do not require API keys. compact memory, optionally with hints. - `programmatic_memory.jinja`: Tool prompt for bounded recent context plus full-fidelity `history_read`/`history_search` retrieval, no compaction. + - `backtracking.jinja`: Choose-or-restore prompt listing restorable + checkpoints and the remaining restore budget. + - `adaptive_reasoning.jinja`: Concise prompt that expands into an explicit + planning prompt when a stall or repeated state triggers deep mode. ## Persistence -- `metrics.db`: Benchmark/run metrics for CLI workflows. -- `results///run_/run_summary.json`: Step trace, - per-step decisions, and aggregated token/cost usage. +Both stores hold the same logical schema-v2 record. + +- `metrics.db`: `runs` (identity, quest provenance, treatment, outcome, usage, + metrics, progress, final snapshot) and `transitions` (before snapshot, + executed action, after snapshot, response, usage, progress, provenance, + replay status, reasoning mode). There is no in-place upgrade from the + pre-v2 schema; convert once with `scripts/migrate_records.py`. +- `results///run_/run_summary.json`: the same record as + JSON, written for every player type including random policies. ## Configuration - `.env` (copied from `.env.template`): Provider API keys. - `configs/benchmarks/`: Benchmark YAML configs defining model × harness × - quest matrices. + quest matrices, optionally with `progress_manifest`. +- `configs/progress/`: Curated YAML progress manifests (`Boat.yaml` is the + executable example). ## Public Taxonomy (Benchmark Dimension) @@ -138,6 +173,11 @@ and benchmark configuration parsing do not require API keys. | Tools + hints + compact memory | `tool_hinted` | `tool_augmented_hints.jinja` | `CompactionMemory` | calculator, scratchpad, quest history | tool-select-then-act | | Planner loop | `planner` | `planner.jinja` | `CompactionMemory` | none | plan-maintain-act | +Every run persists the full treatment (prompt, memory, tools, loop, reasoning, +model, temperature, system prompt, material knobs) plus a `t2_` signature +derived from that canonical data. Reports group by those components rather than +inferring them from harness names. + The harness names above are canonical snake_case identifiers used in YAML configs, the CLI, result artifacts, and documentation. Public labels can be friendlier, but experiment records should preserve the canonical names so runs @@ -148,15 +188,24 @@ remain comparable. | Label | Harness name | Template | Memory | Tools | Loop | |---|---|---|---|---|---| | Programmatic memory (experimental) | `programmatic_memory` | `programmatic_memory.jinja` | `DefaultMemory` | calculator, scratchpad, history_read, history_search | tool-select-then-act | +| Backtracking (experimental) | `backtracking` | `backtracking.jinja` | `CompactionMemory` | none | choose-or-restore | +| Adaptive reasoning (experimental) | `adaptive_reasoning` | `adaptive_reasoning.jinja` | `DefaultMemory` | none | adaptive-depth | + +`backtracking` is the only harness that may restore a recorded checkpoint. Its +`restore_limit` knob is rejected for any other harness, restores stay visible as +transitions, and only the active checkpoint branch is truncated. +`adaptive_reasoning` starts concise and switches to a deeper planning prompt +after a repeated state or an `adaptive_stall_steps` progress stall; the reasoning +mode used is recorded on every transition. `programmatic_memory` is an experimental treatment (see `docs/PROGRAMMATIC_MEMORY_PROPOSAL.md`): it replaces `tool_compact`'s clipped `quest_history` keyword search with bounded deterministic reads/searches over a -run-local `Trajectory` view of canonical executed `AgentState` objects, and +run-local `Trajectory` view of canonical executed `QuestTransition` objects, and replaces `CompactionMemory` with `DefaultMemory` so no LLM compaction and no full transcript run in the background. `QuestRunner` constructs each -`AgentState` once after an action executes, then sends that exact object to the -harness retrieval view, callbacks, and `QuestLogger` for persistence. It reuses +`QuestTransition` once after an action executes, then sends that exact object to +the harness retrieval view, callbacks, and `QuestLogger` for persistence. It reuses `ToolCompactHarness`'s tool-select-then-act call budget, so the model gets at most one retrieval call before its final action, matching `tool_compact`. `DefaultMemory` is the single bounded recent-context source in its select-turn diff --git a/docs/PROGRAMMATIC_MEMORY_PROPOSAL.md b/docs/PROGRAMMATIC_MEMORY_PROPOSAL.md index 4509a98..a1edba4 100644 --- a/docs/PROGRAMMATIC_MEMORY_PROPOSAL.md +++ b/docs/PROGRAMMATIC_MEMORY_PROPOSAL.md @@ -2,6 +2,13 @@ Status: implemented, unbenchmarked +Historical note: this proposal was written against the pre-v2 record schema. +Its `AgentState` step object is now `QuestTransition`, and `run_summary.json` +is a schema-v2 record. The design contract is unchanged: `QuestRunner` builds +one canonical object per executed action and hands that same object to the +harness retrieval view, callbacks, and `QuestLogger`. See `docs/SPEC.md` and +`docs/ARCHITECTURE.md` for the current schema. + Delivery Sequence steps 1-4 below are complete: the append-only trajectory, the `programmatic_memory` harness, a passing fake-provider deterministic quest smoke, and `configs/benchmarks/programmatic_memory_pilot.yaml`. Steps 5-6 (run diff --git a/docs/SPEC.md b/docs/SPEC.md index 5897566..d0d12fc 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -1,155 +1,227 @@ -# SPEC: LLM-Quest Current State - -This document is a current-state project specification, not a roadmap promise. -For the public narrative and interpretation of results, use the project -[About page](../site/about.html) as the main story surface. - -## Purpose - -LLM Quest Benchmark evaluates how LLMs make sequential choices in Space -Rangers text quests. The benchmark varies the agent harness around a model -while holding the quest environment and result logging consistent. A harness is -the wrapper that decides what context the model sees and how its response is -converted into an action: prompt template, memory strategy, tools, and loop -shape. - -The core question is practical: which kinds of context help, hurt, or expose -state-tracking failures during 10-50 turn interactive fiction tasks? - -The current public result should be read as a selective-intervention story, not -as a claim that larger context wrappers are universally better. Minimal prompts -remain a strong baseline on easier local quests; heavier context scaffolds are -most interesting when they recover specific stateful failures that the baseline -cannot solve. - -## Current Public Scope - -The public leaderboard is a curated comparable slice, not the full raw -experiment history. It currently reports: - -- 6 primary publication models. -- 15 comparable quest IDs with coverage across all six primary models. -- 1,584 published leaderboard runs. -- Exploratory, one-model, and partial-coverage runs excluded from the public - comparison slice unless they support direct comparison. - -Raw benchmark artifacts and experiment notes remain useful for follow-up -analysis, but the public slice is the authoritative comparison surface. - -## Current Taxonomy - -Use these labels for current public descriptions of benchmark harnesses: - -| Label | Harness name | Template | Memory | Tools / loop | -|---|---|---|---|---| -| Minimal prompt | `minimal` | `stub.jinja` | `DefaultMemory` | no tools, react loop | -| Short-context reasoning | `reasoning_recent` | `reasoning.jinja` | `DefaultMemory` | no tools, react loop | -| Compact memory / memo | `memo_compact` | `stateful_compact.jinja` | `CompactionMemory` | no tools, react loop | -| Prompt hints | `hinted_compact` | `stateful_compact_hints.jinja` | `CompactionMemory` | no tools, react loop | -| Tools + compact memory | `tool_compact` | `tool_augmented.jinja` | `CompactionMemory` | calculator, scratchpad, quest history | -| Tools + hints + compact memory | `tool_hinted` | `tool_augmented_hints.jinja` | `CompactionMemory` | calculator, scratchpad, quest history | -| Planner loop | `planner` | `planner.jinja` | `CompactionMemory` | plan-maintain-act loop | - -Older internal experiment labels are historical and should not be presented as -the current public taxonomy. - -## Experimental Harnesses (Not Yet Public) - -`programmatic_memory` (see `docs/PROGRAMMATIC_MEMORY_PROPOSAL.md`) is -implemented and registered but has no published benchmark runs, so it is -deliberately excluded from the Current Taxonomy table above. It pairs -`DefaultMemory` (recent bounded context, no compaction, no full transcript) -with a run-local `Trajectory` retrieval view over canonical executed -`AgentState` objects and two bounded deterministic retrieval tools, -`history_read` and `history_search`, capped at one retrieval call per decision. -`QuestRunner` creates each `AgentState` once after an action executes, passes -that object to the harness, callbacks, and `QuestLogger`, and the logger -serializes it into the persisted trace; the retrieval view creates no second -step representation or artifact. The pilot benchmark -(`configs/benchmarks/programmatic_memory_pilot.yaml`) is an exploratory -bundled-harness comparison of whether the full `programmatic_memory` harness -outperforms `tool_compact`'s clipped keyword search or `memo_compact`'s -LLM-compacted summary on long/stateful quests, not an isolated test of -retrieval alone: relative to those baselines it also removes compaction and -changes the tool/prompt path, so an observed effect cannot be attributed to -retrieval specifically. It should not be treated as a public result until that -benchmark matrix has run and been reported. - -## Current Interpretation - -The strongest pattern so far is that bigger scaffolds are not automatically -better. A concise 20-word memo produced a useful sweet spot: it improved over -no-memo and full-transcript baselines, while longer or more structured memo -variants regressed. The likely mechanism is selective pressure: the short memo -forces the harness to preserve only state that matters for future decisions. - -Tools and hints showed a synergy effect. Prompt hints alone hurt, and tools -alone were modest, but tools plus hints improved outcomes because the hints -pointed the model toward quantities and quest mechanics while the calculator, -scratchpad, and history search gave it ways to act on those signals. - -Verbosity is a recurring failure mode. Some newer or larger models timed out -more often because they spent too much of the quest budget generating long step -responses. For sequential decision tasks, a harness that elicits concise, -actionable state updates can outperform one that invites broad reasoning. - -## Implemented Runtime - -- Quest execution uses the TypeScript `space-rangers-quest` submodule through - the Python bridge in `llm_quest_benchmark/executors/ts_bridge/`. -- Environment state is exposed through `llm_quest_benchmark/environments/qm.py`. -- Agent harnesses live under `llm_quest_benchmark/harnesses/` and are selected - by canonical snake_case harness names. -- Provider calls are normalized in `llm_quest_benchmark/llm/client.py` with - OpenAI-compatible, Anthropic, Google, and DeepSeek adapters. -- Benchmark execution is CLI + YAML driven through `uv run llm-quest ...`. -- Static public results are generated into `site/leaderboard.json` and rendered - by `site/index.html`. - -## Metrics - -Current public metrics include success rate, average steps, token/cost -statistics, and repetition rate. Repetition is interpreted as a diagnostic -signal for loopiness or context loss, not as a solved predictor of success. - -Aggregate success-rate rankings should be interpreted alongside the Per Quest -view. A mode can rank well overall by solving easier quests while still failing -harder stateful, search-heavy, or navigation-heavy quests. - -Progress-style metrics and richer quest difficulty annotations remain future -work unless present in generated result artifacts. - -## Data and Distribution - -Quest files are downloaded with `download_quests.sh` from the Space Rangers -community archive and are not redistributed as benchmark source data. The -repository includes benchmark code, configs, tests, static site assets, and the -curated public leaderboard JSON. +# SPEC: Replayable Harness Evaluation + +## Goal + +LLM Quest Benchmark evaluates the model-harness pair on sequential Space +Rangers choices while holding the QM environment fixed. A run must be: + +- attributable to an explicit prompt, memory, tool, loop, and reasoning + treatment; +- recorded as exact environment transitions; +- resumable when its engine state is complete and verified; +- diagnosable through terminal outcome, curated progress, and transcript + metrics. + +The current research question remains: which harness interventions recover +state-tracking, planning, exploration, and memory failures during long-running +interactive fiction? + +## User-visible behavior + +### Run records + +Every human, random, or LLM run writes a schema-v2 `run_summary.json` and the +same logical data to SQLite. Each transition records: + +- the complete state before the action; +- the executed choose or restore action; +- the complete state after the action; +- the agent response and usage; +- progress and replay provenance. + +The final environment state is the final transition's `after` state. Terminal +states are not represented as fake agent decisions. + +### Replay and resume + +`llm-quest run --resume-from PATH` loads the quest and treatment from a v2 run, +verifies the quest checksum, engine revision, recorded transitions, and active +checkpoint, then continues without losing harness memory. + +Explicit step limits produce `TRUNCATED`, which is resumable. Divergent or +legacy-mapped records fail before a new model call. + +### Existing records + +`scripts/migrate_records.py --source PATH --output PATH` maps legacy +`run_summary.json` trees and SQLite databases into schema v2. Migration is the +only legacy reader. + +Migration never fabricates missing action, timestamp, parameter, saving, or +post-state data. Records lacking deterministic state remain analyzable and are +marked non-resumable. + +### Progress + +A benchmark may declare `progress_manifest`, a validated YAML file of +quest-specific state predicates and percentages. Runtime progress includes: + +- current and maximum progress; +- newly reached milestones; +- transitions since the last new milestone. + +Progress is monotonic across restores. Terminal success is always 100 percent. +Without a manifest, progress is terminal-only. + +### Harness treatments + +The harness registry declares the material components of each treatment: + +- prompt; +- memory; +- tools; +- loop; +- reasoning policy. + +The persisted treatment signature is derived from canonical component data, +model, temperature, and material harness knobs. Reports use this data directly +and do not infer components from harness names. + +### Backtracking + +The experimental `backtracking` harness may either choose a current option or +restore a recorded checkpoint. `restore_limit` applies only to this harness. +Restores retain chronological history and truncate only the active branch. +Existing harnesses cannot restore. + +### Adaptive reasoning + +The experimental `adaptive_reasoning` harness uses concise reasoning by +default. It switches to a deeper planning prompt after repeated state or a +configured progress stall. `adaptive_stall_steps` applies only to this harness, +and every transition records the reasoning mode used. + +## Acceptance tests + +1. A real Boat transition round-trips without losing choice IDs, full engine + saving, timestamp, executed action, or state digest. +2. Exact engine saving restore reproduces the recorded digest. +3. Replay detects any changed quest, action, timestamp, or resulting state + before model inference. +4. A truncated deterministic run resumes to the same state as an uninterrupted + run. +5. A legacy JSON fixture and SQLite fixture map into v2; ambiguous fields are + explicitly unavailable and prevent resume. +6. All player types persist v2 JSON; random runs are not suppressed. +7. Progress milestones are state-based, monotonic, and separate from terminal + outcome. +8. A real restore returns to the selected checkpoint and remains visible as a + transition. +9. Backtracking and adaptive-only knobs are rejected for other harnesses. +10. Treatment signatures are stable for equivalent configs and distinct for + material differences. +11. Existing analyzers, reports, leaderboard generation, replay scripts, and + web/human traces consume only v2. +12. The Python suite, JavaScript build, random Boat smoke, migration smoke, + resume smoke, and restore smoke pass. + +## Constraints + +- Python 3.11 is the supported local runtime for the locked dependency set. +- The TypeScript `space-rangers-quest` engine remains authoritative for game + state and win/fail outcome. +- The environment remains unchanged for public harness comparisons. +- Backtracking is an explicit experimental capability, not evaluator behavior. +- Deterministic replay uses the full engine saving and the original transition + timestamp passed to `performJump`. +- Existing public outcome metrics remain comparable; progress is diagnostic + until manifests are curated. +- No legacy config aliases, database fallbacks, dual record writers, or runtime + schema adapters remain after cutover. ## Non-goals -- Claiming that any context scaffold is universally best. Results are jagged by - quest and model. -- Treating exploratory or partial-coverage runs as public comparison data. -- Adding a production web service; the benchmark remains CLI/YAML first with a - static publication site. -- Changing quest authoring or the upstream quest format. - -## Reproducibility Entry Points - -```bash -uv sync --extra dev -pnpm install -uv run llm-quest --help -uv run llm-quest benchmark --config configs/benchmarks/memory_full_transcript.yaml -pnpm run build -``` - -Provider API keys are required for real LLM runs. Tests and static validation -should run without external credentials in a prepared checkout. - -Reproducible benchmark rows depend on recording the quest file, model/provider -ID, harness name, run ID, outcome, and run summaries with usage/metrics. -Harness responses are parsed into a chosen action plus optional -analysis/reasoning so action validity, terminal outcome, steps, tokens/cost, -and repetition diagnostics can be regenerated from stored artifacts. +- OpenEnv, Gymnasium, Inspect, Harbor, remote environment, or RL integration. +- Automatically generating or model-grading progress milestones. +- Making every historical record resumable. +- Treating more context, more reasoning, or backtracking as universally better. +- Changing quest authoring or the upstream `.qm` format. + +## Dependencies and integrations + +- `space-rangers-quest` supplies parser, state transition, saving, and outcome + semantics through the TypeScript bridge. +- YAML benchmark configuration supplies an optional progress manifest. +- SQLite and `run_summary.json` store the same schema-v2 logical record. +- Existing static reports and the web player remain publication surfaces after + their v2 cutover. + +## Risks + +- Legacy compact JSON can contain the model-proposed action rather than the + runner-executed action. Migration prefers SQLite and otherwise marks the + action unverified. +- Legacy records omit full engine saving and transition timestamps, so most are + not resumable. +- Dynamic quest behavior can diverge if timestamps are regenerated. The bridge + therefore receives the recorded timestamp. +- Restore can inflate apparent capability. It has a separate harness name, + explicit budget, and restore metrics. +- Adaptive reasoning can hide extra inference cost. Reasoning mode and usage + remain visible per transition. + +## Codebase notes + +- Detailed implementation order and mapping rules live in `PLAN.md`. +- `QMPlayerEnv` and `QMBridge` own environment snapshots and saving restore. +- `QuestRunner` owns active checkpoints, replay verification, and transition + construction. +- `QuestLogger` persists completed transitions and run metadata initialized + before execution. +- Harness specifications and treatment signatures are canonical in the harness + registry/configuration layer. + +## Outcome / Deviations + +Implemented as a single schema-v2 cutover. No runtime reader accepts a pre-v2 +record; `scripts/migrate_records.py` is the only legacy reader. + +Delivered: + +- Canonical `QuestAction`, `QuestSnapshot`, `ProgressState`, `QuestTransition`, + and `RunRecord` types in `llm_quest_benchmark/schemas/records.py`. +- Structured TypeScript bridge protocol (`state` / `jump` / `load`) carrying the + full engine saving; `performedAtMs` is supplied by the caller and never + generated inside the bridge. +- `QMPlayerEnv.restore()` with digest verification, and a snapshot digest over + location, observation, choices, parameter state, terminal state, and saving. +- v2 SQLite (`runs`, `transitions`) plus one `run_summary.json` per run for + every player type, with run metadata written before execution. +- `scripts/migrate_records.py` for legacy JSON trees and SQLite databases. +- `llm-quest run --resume-from`, replay verification, and the resumable + `TRUNCATED` outcome. +- Validated YAML progress manifests with a state-based `configs/progress/Boat.yaml`. +- Canonical harness specifications and `t2_` treatment signatures. +- Experimental `backtracking` and `adaptive_reasoning` harnesses. + +Implementation details: + +1. **Progress fields.** `ProgressState.current` is the monotonic achieved + percentage and `maximum` is the manifest ceiling. `scored` distinguishes a + curated manifest from terminal-only progress. +2. **Deterministic engine inputs.** The TypeScript player starts from a stable + quest-derived seed. The runner derives `performedAtMs` from the quest + checksum and transition index, then persists it. Independent runs with the + same actions therefore reproduce the same state, and replay compares exact + snapshot digests. +3. **Web trace timestamps.** The browser player's engine timestamp is not + observable from the page. Human web traces record `performed_at_ms: null` + and `replay_status: unavailable`; they still retain full savings, abandoned + branches, and explicit restore transitions. +4. **`progress_recovered`.** This diagnostic is the monotonic progress gained + after the last accepted restore. Individual restore effects remain + recoverable from transitions. +5. **Leaderboard labels.** Friendly mode labels are presentation derived from + canonical treatment components. Persisted identity and grouping use the + treatment signature; no runtime reader accepts legacy record shapes. + +Verified: + +- `uv run pytest`: 370 passed, 4 skipped. +- `uv run ruff check .`: passed. +- `pnpm run build`: passed. +- Existing SQLite data migrated: 8 runs and 73 transitions; all correctly + marked non-resumable because legacy records lack deterministic engine state. +- A real random-policy Boat run wrote a nested schema-v2 record, truncated at + four transitions, replayed those transitions exactly, resumed to six, and + preserved lineage and progress. diff --git a/llm_quest_benchmark/core/analyzer.py b/llm_quest_benchmark/core/analyzer.py index 9e3a7b1..7b165cc 100644 --- a/llm_quest_benchmark/core/analyzer.py +++ b/llm_quest_benchmark/core/analyzer.py @@ -1,24 +1,69 @@ -"""Quest run analyzer for metrics analysis""" +"""Quest run analyzer for schema-v2 metrics analysis""" import json import sqlite3 from pathlib import Path from typing import Any -from llm_quest_benchmark.core.logging import LogManager +from llm_quest_benchmark.core.logging import LogManager, verify_v2_schema from llm_quest_benchmark.renderers.benchmark_result import BenchmarkResultRenderer +from llm_quest_benchmark.schemas.records import QuestTransition # Initialize logging log_manager = LogManager() log = log_manager.get_logger() +def _json_field(value: Any, default: Any = None) -> Any: + if not value: + return default + try: + return json.loads(value) + except (json.JSONDecodeError, TypeError): + return default + + +def load_transitions(conn: sqlite3.Connection, run_id: int) -> list[QuestTransition]: + """Load canonical transitions for one run.""" + rows = conn.execute( + """ + SELECT transition_index, before_state, action, after_state, response, + usage, progress, provenance, replay_status, reasoning_mode + FROM transitions + WHERE run_id = ? + ORDER BY transition_index + """, + (run_id,), + ).fetchall() + + transitions = [] + for row in rows: + (index, before, action, after, response, usage, progress, provenance, replay_status, reasoning_mode) = row + transitions.append( + QuestTransition.from_dict( + { + "index": index, + "before": _json_field(before, {}), + "action": _json_field(action, {}), + "after": _json_field(after, {}), + "response": _json_field(response), + "usage": _json_field(usage, {}), + "progress": _json_field(progress, {}), + "provenance": provenance, + "replay_status": replay_status, + "reasoning_mode": reasoning_mode, + } + ) + ) + return transitions + + def analyze_quest_run( quest_name: str, db_path: Path, debug: bool = False, ) -> dict[str, Any]: - """Analyze metrics for a specific quest from database. + """Analyze metrics for a specific quest from the schema-v2 database. Args: quest_name: Name of the quest to analyze @@ -35,77 +80,70 @@ def analyze_quest_run( try: conn = sqlite3.connect(db_path) - cursor = conn.cursor() - - # Get all runs for this quest - cursor.execute( - """ - SELECT id, start_time, end_time, model, template, outcome, reward - FROM runs - WHERE quest_name = ? - ORDER BY start_time DESC - """, - (quest_name,), - ) - runs = cursor.fetchall() - - if not runs: - raise ValueError(f"No runs found for quest: {quest_name}") - - # Prepare analysis results - results = { - "quest_name": quest_name, - "total_runs": len(runs), - "outcomes": {"SUCCESS": 0, "FAILURE": 0}, - "runs": [], - } - - # Process each run - for run in runs: - run_id, start_time, end_time, model, template, outcome, reward = run - results["outcomes"][outcome] = results["outcomes"].get(outcome, 0) + 1 - - # Get steps for this run - cursor.execute( + try: + if not verify_v2_schema(conn): + raise ValueError(f"Metrics database has no schema-v2 runs table: {db_path}") + runs = conn.execute( """ - SELECT step, observation, choices, action, reward, llm_response - FROM steps - WHERE run_id = ? - ORDER BY step - """, - (run_id,), - ) - steps = cursor.fetchall() - - run_data = { - "start_time": start_time, - "end_time": end_time, - "model": model, - "template": template, - "outcome": outcome, - "reward": reward, - "steps": [], + SELECT id, start_time, end_time, agent_id, treatment, treatment_signature, + outcome, reward, transcript_diagnostics, progress + FROM runs + WHERE quest_name = ? + ORDER BY start_time DESC + """, + (quest_name,), + ).fetchall() + + if not runs: + raise ValueError(f"No runs found for quest: {quest_name}") + + results: dict[str, Any] = { + "quest_name": quest_name, + "total_runs": len(runs), + "outcomes": {"SUCCESS": 0, "FAILURE": 0}, + "runs": [], } - for step in steps: - step_num, obs, choices_json, action, step_reward, llm_response = step - choices = json.loads(choices_json) - run_data["steps"].append( + for run in runs: + ( + run_id, + start_time, + end_time, + agent_id, + treatment_json, + treatment_signature, + outcome, + reward, + diagnostics_json, + progress_json, + ) = run + results["outcomes"][outcome] = results["outcomes"].get(outcome, 0) + 1 + treatment = _json_field(treatment_json, {}) + transitions = load_transitions(conn, run_id) + + results["runs"].append( { - "step": step_num, - "observation": obs, - "choices": choices, - "action": action, - "reward": step_reward, - "llm_response": llm_response, + "id": run_id, + "start_time": start_time, + "end_time": end_time, + "agent_id": agent_id, + "model": treatment.get("model", "unknown"), + "harness": treatment.get("harness", "unknown"), + "treatment_signature": treatment_signature, + "outcome": outcome, + "reward": reward, + "transcript_diagnostics": _json_field(diagnostics_json, {}), + "progress": _json_field(progress_json, {}), + "transitions": [t.to_dict() for t in transitions], } ) - results["runs"].append(run_data) - - conn.close() - return results + return results + finally: + conn.close() + except ValueError: + raise except Exception as e: log.exception(f"Error analyzing quest run: {e}") raise ValueError(f"Error analyzing quest run: {e}") @@ -113,14 +151,14 @@ def analyze_quest_run( def analyze_benchmark( db_path: Path, - benchmark_name: str | None = None, + benchmark_id: str | None = None, debug: bool = False, ) -> dict[str, Any]: - """Analyze benchmark results from database. + """Analyze benchmark results from the schema-v2 database. Args: db_path: Path to SQLite database - benchmark_name: Optional name of benchmark to analyze + benchmark_id: Optional benchmark id to analyze debug: Enable debug logging and output Returns: @@ -133,92 +171,114 @@ def analyze_benchmark( try: conn = sqlite3.connect(db_path) - cursor = conn.cursor() - - # Base query conditions - where_clause = "WHERE 1=1" - params = [] - if benchmark_name: - where_clause += " AND benchmark_name = ?" - params.append(benchmark_name) - - # Get overall statistics - cursor.execute( - f""" - SELECT - COUNT(*) as total_runs, - COUNT(CASE WHEN outcome = 'SUCCESS' THEN 1 END) as successes, - COUNT(CASE WHEN outcome = 'FAILURE' THEN 1 END) as failures, - AVG(CASE WHEN outcome = 'SUCCESS' THEN reward END) as avg_success_reward - FROM runs - {where_clause} - """, - params, - ) - stats = cursor.fetchone() - total_runs, successes, failures, avg_success_reward = stats - - if total_runs == 0: - raise ValueError(f"No benchmark data found{' for ' + benchmark_name if benchmark_name else ''}") - - # Get per-model statistics - cursor.execute( - f""" - SELECT - model, - COUNT(*) as runs, - COUNT(CASE WHEN outcome = 'SUCCESS' THEN 1 END) as successes, - AVG(CASE WHEN outcome = 'SUCCESS' THEN reward END) as avg_reward - FROM runs - {where_clause} - GROUP BY model - """, - params, - ) - model_stats = cursor.fetchall() - - # Get per-quest statistics - cursor.execute( - f""" - SELECT - quest_name, - COUNT(*) as runs, - COUNT(CASE WHEN outcome = 'SUCCESS' THEN 1 END) as successes - FROM runs - {where_clause} - GROUP BY quest_name - """, - params, - ) - quest_stats = cursor.fetchall() - - # Prepare results - results = { - "summary": { - "total_runs": total_runs, - "success_rate": (successes / total_runs * 100) if total_runs > 0 else 0, - "avg_success_reward": avg_success_reward or 0, - "outcomes": {"SUCCESS": successes, "FAILURE": failures}, - }, - "models": [ - { - "name": model, - "runs": runs, - "success_rate": (successes / runs * 100) if runs > 0 else 0, - "avg_reward": avg_reward or 0, - } - for model, runs, successes, avg_reward in model_stats - ], - "quests": [ - {"name": quest, "runs": runs, "success_rate": (successes / runs * 100) if runs > 0 else 0} - for quest, runs, successes in quest_stats - ], - } - - if benchmark_name: - results["benchmark_name"] = benchmark_name + try: + if not verify_v2_schema(conn): + raise ValueError(f"Metrics database has no schema-v2 runs table: {db_path}") + where_clause = "WHERE 1=1" + params: list[Any] = [] + if benchmark_id: + where_clause += " AND benchmark_id = ?" + params.append(benchmark_id) + + stats = conn.execute( + f""" + SELECT + COUNT(*) as total_runs, + COUNT(CASE WHEN outcome = 'SUCCESS' THEN 1 END) as successes, + COUNT(CASE WHEN outcome = 'FAILURE' THEN 1 END) as failures, + AVG(CASE WHEN outcome = 'SUCCESS' THEN reward END) as avg_success_reward + FROM runs + {where_clause} + """, + params, + ).fetchone() + total_runs, successes, failures, avg_success_reward = stats + + if total_runs == 0: + raise ValueError(f"No benchmark data found{' for ' + benchmark_id if benchmark_id else ''}") + + model_stats = conn.execute( + f""" + SELECT + json_extract(treatment, '$.model') as model, + COUNT(*) as runs, + COUNT(CASE WHEN outcome = 'SUCCESS' THEN 1 END) as successes, + AVG(CASE WHEN outcome = 'SUCCESS' THEN reward END) as avg_reward + FROM runs + {where_clause} + GROUP BY model + """, + params, + ).fetchall() + + quest_stats = conn.execute( + f""" + SELECT + quest_name, + COUNT(*) as runs, + COUNT(CASE WHEN outcome = 'SUCCESS' THEN 1 END) as successes + FROM runs + {where_clause} + GROUP BY quest_name + """, + params, + ).fetchall() + + treatment_stats = conn.execute( + f""" + SELECT + treatment_signature, + json_extract(treatment, '$.harness') as harness, + json_extract(treatment, '$.memory') as memory, + json_extract(treatment, '$.loop') as loop, + json_extract(treatment, '$.reasoning') as reasoning, + COUNT(*) as runs, + COUNT(CASE WHEN outcome = 'SUCCESS' THEN 1 END) as successes + FROM runs + {where_clause} + GROUP BY treatment_signature + """, + params, + ).fetchall() + + results = { + "summary": { + "total_runs": total_runs, + "success_rate": (successes / total_runs * 100) if total_runs > 0 else 0, + "avg_success_reward": avg_success_reward or 0, + "outcomes": {"SUCCESS": successes, "FAILURE": failures}, + }, + "models": [ + { + "name": model or "unknown", + "runs": runs, + "success_rate": (model_successes / runs * 100) if runs > 0 else 0, + "avg_reward": avg_reward or 0, + } + for model, runs, model_successes, avg_reward in model_stats + ], + "quests": [ + {"name": quest, "runs": runs, "success_rate": (quest_successes / runs * 100) if runs > 0 else 0} + for quest, runs, quest_successes in quest_stats + ], + "treatments": [ + { + "signature": signature, + "harness": harness, + "memory": memory, + "loop": loop, + "reasoning": reasoning, + "runs": runs, + "success_rate": (treatment_successes / runs * 100) if runs > 0 else 0, + } + for signature, harness, memory, loop, reasoning, runs, treatment_successes in treatment_stats + ], + } - conn.close() + if benchmark_id: + results["benchmark_id"] = benchmark_id + finally: + conn.close() # Create renderer and display results renderer = BenchmarkResultRenderer(debug=debug) @@ -226,6 +286,8 @@ def analyze_benchmark( return results + except ValueError: + raise except Exception as e: log.exception(f"Error analyzing benchmark: {e}") raise ValueError(f"Error analyzing benchmark: {e}") diff --git a/llm_quest_benchmark/core/benchmark_report.py b/llm_quest_benchmark/core/benchmark_report.py index 2ead67e..36f3fa5 100644 --- a/llm_quest_benchmark/core/benchmark_report.py +++ b/llm_quest_benchmark/core/benchmark_report.py @@ -3,12 +3,17 @@ from __future__ import annotations import json +import logging from collections import Counter, defaultdict from dataclasses import dataclass from datetime import datetime from pathlib import Path from typing import Any +from llm_quest_benchmark.schemas.records import QuestTransition, RunRecord + +log = logging.getLogger(__name__) + @dataclass class RunInsight: @@ -18,6 +23,8 @@ class RunInsight: run_id: int model: str harness: str + treatment_signature: str + progress: float quest_name: str outcome: str duration: float @@ -51,35 +58,29 @@ def _shorten(text: str | None, limit: int = 180) -> str | None: return clean[: limit - 3] + "..." -def _extract_model(run_row: dict[str, Any]) -> str: - model = None - raw_cfg = run_row.get("agent_config") - if isinstance(raw_cfg, dict): - model = raw_cfg.get("model") - elif isinstance(raw_cfg, str): +def _treatment(run_row: dict[str, Any]) -> dict[str, Any]: + """Read the canonical treatment recorded on a schema-v2 run row.""" + raw = run_row.get("treatment") + if isinstance(raw, dict): + return raw + if isinstance(raw, str) and raw: try: - model = json.loads(raw_cfg).get("model") + parsed = json.loads(raw) except json.JSONDecodeError: - model = None + return {} + return parsed if isinstance(parsed, dict) else {} + return {} + + +def _extract_model(run_row: dict[str, Any]) -> str: + model = _treatment(run_row).get("model") if model: return str(model) - - agent_id = str(run_row.get("agent_id") or "") - if agent_id.startswith("llm_"): - return agent_id[len("llm_") :] - return agent_id or "unknown" + return str(run_row.get("agent_id") or "unknown") def _extract_harness(run_row: dict[str, Any]) -> str: - raw_cfg = run_row.get("agent_config") - harness = None - if isinstance(raw_cfg, dict): - harness = raw_cfg.get("harness") - elif isinstance(raw_cfg, str): - try: - harness = json.loads(raw_cfg).get("harness") - except json.JSONDecodeError: - harness = None + harness = _treatment(run_row).get("harness") return str(harness) if harness else "" @@ -109,7 +110,7 @@ def _group_label(model: str, harness: str, harnesses_by_model: dict[str, set[str def _extract_last_decision( - steps: list[dict[str, Any]], + transitions: list[QuestTransition], ) -> tuple[str | None, str | None, str | None, str | None, int, int]: decision_steps = 0 default_decisions = 0 @@ -118,27 +119,25 @@ def _extract_last_decision( last_analysis = None last_observation = None - for step in steps: - if not isinstance(step, dict): - continue - choices = step.get("choices") - if not isinstance(choices, dict) or len(choices) <= 1: + for transition in transitions: + choices = transition.before.choices + if len(choices) <= 1: continue decision_steps += 1 - llm_decision = step.get("llm_decision") if isinstance(step.get("llm_decision"), dict) else {} - if bool(llm_decision.get("is_default", False)): + response = transition.response + if response is not None and response.is_default: default_decisions += 1 - choice_map = llm_decision.get("choice") - selected = None - if isinstance(choice_map, dict) and choice_map: - idx, text = next(iter(choice_map.items())) - selected = f"{idx}: {text}" + if transition.action.is_restore: + last_choice = f"restore checkpoint {transition.action.checkpoint_index}" + else: + index = transition.action.choice_index + text = choices[index - 1]["text"] if index and 1 <= index <= len(choices) else "" + last_choice = f"{index}: {text}" - last_choice = selected - last_reasoning = _shorten(llm_decision.get("reasoning")) - last_analysis = _shorten(llm_decision.get("analysis")) - last_observation = _shorten(step.get("observation"), 220) + last_reasoning = _shorten(response.reasoning if response else None) + last_analysis = _shorten(response.analysis if response else None) + last_observation = _shorten(transition.before.observation, 220) return ( last_choice, @@ -168,17 +167,29 @@ def _parse_run_insight(benchmark_id: str, run_row: dict[str, Any]) -> RunInsight # run_summary can drift for timeout/error cases when background execution finishes later. outcome = str(run_row.get("outcome") or "UNKNOWN") duration = float(run_row.get("run_duration") or 0.0) - usage = {} - steps: list[dict[str, Any]] = [] + usage = run_row.get("usage") if isinstance(run_row.get("usage"), dict) else {} + progress = run_row.get("progress") if isinstance(run_row.get("progress"), dict) else {} + transitions: list[QuestTransition] = [] if isinstance(run_summary, dict): - usage = run_summary.get("usage") if isinstance(run_summary.get("usage"), dict) else {} - loaded_steps = run_summary.get("steps") - if isinstance(loaded_steps, list): - steps = [s for s in loaded_steps if isinstance(s, dict)] + try: + record = RunRecord.from_dict(run_summary) + except (ValueError, TypeError) as exc: + log.warning( + "Skipping unparseable run summary for run %s (%s); DB row still counts toward totals: %s", + run_id, + summary_path, + exc, + ) + run_summary = None + if isinstance(run_summary, dict): + record = RunRecord.from_dict(run_summary) + usage = usage or record.usage + progress = progress or record.progress.to_dict() + transitions = record.transitions selected_choice, selected_reasoning, selected_analysis, selected_observation, decision_steps, default_decisions = ( - _extract_last_decision(steps) + _extract_last_decision(transitions) ) prompt_tokens = int(usage.get("prompt_tokens") or 0) @@ -194,6 +205,8 @@ def _parse_run_insight(benchmark_id: str, run_row: dict[str, Any]) -> RunInsight run_id=run_id, model=_extract_model(run_row), harness=_extract_harness(run_row), + treatment_signature=str(run_row.get("treatment_signature") or _treatment(run_row).get("signature") or ""), + progress=float(progress.get("current") or 0.0), quest_name=str(run_row.get("quest_name") or "unknown"), outcome=outcome, duration=duration, @@ -315,6 +328,8 @@ def _format_model_summary( "tokens": total_tokens, "cost": total_cost if priced_runs else None, "default_rate": (default_steps / decision_steps * 100.0) if decision_steps else 0.0, + "avg_progress": (sum(r.progress for r in rows) / len(rows)) if rows else 0.0, + "treatment_signature": rows[0].treatment_signature if rows else "", } return model_summary @@ -396,15 +411,16 @@ def render_benchmark_report( sections.append(f"### {breakdown_label} Breakdown") sections.append("") sections.append( - f"| {breakdown_label} | Runs | Success | Failure | Timeout | Error | Success Rate | Tokens | " - "Est. Cost (USD) | Default Decision Rate |" + f"| {breakdown_label} | Treatment | Runs | Success | Failure | Timeout | Error | Success Rate | " + "Avg Progress | Tokens | Est. Cost (USD) | Default Decision Rate |" ) - sections.append("|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|") + sections.append("|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|") for group, row in model_summary.items(): cost = "n/a" if row["cost"] is None else f"{row['cost']:.6f}" sections.append( - f"| {group} | {row['runs']} | {row['success']} | {row['failure']} | {row['timeout']} | " - f"{row['error']} | {row['success_rate']:.1f}% | {row['tokens']} | {cost} | {row['default_rate']:.1f}% |" + f"| {group} | `{row['treatment_signature']}` | {row['runs']} | {row['success']} | {row['failure']} | " + f"{row['timeout']} | {row['error']} | {row['success_rate']:.1f}% | {row['avg_progress']:.1f}% | " + f"{row['tokens']} | {cost} | {row['default_rate']:.1f}% |" ) sections.append("") diff --git a/llm_quest_benchmark/core/leaderboard.py b/llm_quest_benchmark/core/leaderboard.py index 450d40b..3eda79f 100644 --- a/llm_quest_benchmark/core/leaderboard.py +++ b/llm_quest_benchmark/core/leaderboard.py @@ -12,6 +12,7 @@ from llm_quest_benchmark.core.quest_lang import canonical_quest_id from llm_quest_benchmark.llm.client import parse_model_name +from llm_quest_benchmark.schemas.records import RunRecord TAXONOMY_MODES = { "minimal_prompt": "Minimal prompt", @@ -21,18 +22,11 @@ "prompt_hints": "Prompt hints", "tools_compact_memory": "Tools + compact memory", "tools_hints_compact_memory": "Tools + hints + compact memory", + "tools_programmatic_memory": "Tools + programmatic memory", "planner_loop": "Planner loop", -} - -TEMPLATE_TO_MODE = { - "stub": ("minimal_prompt", TAXONOMY_MODES["minimal_prompt"]), - "strategic": ("short_context_reasoning", TAXONOMY_MODES["short_context_reasoning"]), - "stateful_compact": ("compact_memory_memo", TAXONOMY_MODES["compact_memory_memo"]), - "light_hints": ("prompt_hints", TAXONOMY_MODES["prompt_hints"]), - "stateful_compact_hints": ("prompt_hints", TAXONOMY_MODES["prompt_hints"]), - "planner": ("planner_loop", TAXONOMY_MODES["planner_loop"]), - "tool_augmented": ("tools_compact_memory", TAXONOMY_MODES["tools_compact_memory"]), - "tool_augmented_hints": ("tools_hints_compact_memory", TAXONOMY_MODES["tools_hints_compact_memory"]), + "backtracking_loop": "Backtracking loop", + "adaptive_reasoning": "Adaptive reasoning", + "unknown": "Unknown treatment", } RETIRED_BENCHMARK_NAMES = { @@ -49,21 +43,6 @@ "memo_structured", } -RETIRED_TEMPLATE_IDS = { - "memo_cot", - "memo_extended", - "memo_structured", -} - -REASONING_STYLE_TEMPLATES = { - "reasoning", - "strategic", - "loop_aware_reasoning", - "objective_guard", - "consequence_scan", - "consequence_scan_subgoal", -} - MODE_ORDER = list(TAXONOMY_MODES) MODEL_ALIASES = { @@ -89,49 +68,68 @@ def _load_json(path: Path) -> dict[str, Any] | None: return json.load(f) -def _strip_template_suffix(template_name: str) -> str: - return Path(template_name or "").stem - +def _mode_from_treatment(treatment: dict[str, Any]) -> tuple[str, str]: + """Group runs by declared treatment components, never by harness name.""" + prompt = str(treatment.get("prompt") or "") + memory = str(treatment.get("memory") or "") + loop = str(treatment.get("loop") or "") + reasoning = str(treatment.get("reasoning") or "") + has_tools = bool(treatment.get("tools")) + hinted = Path(prompt).stem.endswith("_hints") + + if loop == "choose_or_restore": + mode_id = "backtracking_loop" + elif loop == "adaptive_depth": + mode_id = "adaptive_reasoning" + elif loop == "plan_act": + mode_id = "planner_loop" + elif loop == "tool_select_act" or has_tools: + if hinted: + mode_id = "tools_hints_compact_memory" + elif memory == "compaction": + mode_id = "tools_compact_memory" + else: + mode_id = "tools_programmatic_memory" + elif hinted: + mode_id = "prompt_hints" + elif memory == "full_transcript": + mode_id = "full_history_reasoning" + elif memory == "compaction": + mode_id = "compact_memory_memo" + elif memory == "recent_window": + mode_id = "minimal_prompt" if reasoning == "none" else "short_context_reasoning" + else: + mode_id = "unknown" -def _mode_from_template(template_name: str, memory_mode: str | None = None) -> tuple[str, str]: - template_id = _strip_template_suffix(template_name) - if template_id in REASONING_STYLE_TEMPLATES: - if memory_mode == "full_transcript": - return "full_history_reasoning", TAXONOMY_MODES["full_history_reasoning"] - if memory_mode == "compaction": - return "compact_memory_memo", TAXONOMY_MODES["compact_memory_memo"] - return "short_context_reasoning", TAXONOMY_MODES["short_context_reasoning"] - return TEMPLATE_TO_MODE.get(template_id, (template_id or "unknown", template_id or "unknown")) + return mode_id, TAXONOMY_MODES[mode_id] def _is_retired_result( source_name: str | None, benchmark_id: str | None, result_row: dict[str, Any], - agent_config: dict[str, Any], - template_name: str, + treatment: dict[str, Any], ) -> bool: source_names = {str(value) for value in (source_name, benchmark_id) if value} if source_names & RETIRED_BENCHMARK_NAMES: return True - harness = str(result_row.get("harness") or agent_config.get("harness") or "") - if harness in RETIRED_HARNESSES: - return True - - template_id = _strip_template_suffix(template_name) - return template_id in RETIRED_TEMPLATE_IDS + harness = str(result_row.get("harness") or treatment.get("harness") or "") + return harness in RETIRED_HARNESSES -def _agent_config(db_run: dict[str, Any]) -> dict[str, Any]: - raw_config = db_run.get("agent_config") - if not isinstance(raw_config, str) or not raw_config: - return {} - try: - parsed = json.loads(raw_config) - except json.JSONDecodeError: - return {} - return parsed if isinstance(parsed, dict) else {} +def _run_treatment(db_run: dict[str, Any]) -> dict[str, Any]: + """Read the canonical treatment recorded on a schema-v2 run row.""" + raw = db_run.get("treatment") + if isinstance(raw, dict): + return raw + if isinstance(raw, str) and raw: + try: + parsed = json.loads(raw) + except json.JSONDecodeError: + return {} + return parsed if isinstance(parsed, dict) else {} + return {} def _combine_numeric_tokens(parts: list[str]) -> list[str]: @@ -358,43 +356,43 @@ def generate_leaderboard( continue model = str(result_row.get("model") or "unknown") - template = str(result_row.get("template") or "") quest_path = str(result_row.get("quest") or "") raw_quest_id = _quest_id_from_path(quest_path) quest_id = canonical_quest_id(raw_quest_id) source_lang = _detect_quest_lang(quest_path) outcome = str(result_row.get("outcome") or "UNKNOWN") - # TODO: cost tracking is broken - run_summary.json usage data is not populated by OpenRouter runs - # Correlate with db_runs to get run ID for metrics. + # Correlate with db_runs to get the run id for usage and metrics. usage: dict[str, Any] = {} metrics: dict[str, Any] = {} - config: dict[str, Any] = {} + progress: dict[str, Any] = {} + treatment: dict[str, Any] = result_row.get("treatment") or {} db_run = _db_run_for_result(result_row, i, db_runs, db_runs_by_id, db_run_queues, used_db_run_ids) if db_run is not None: - config = _agent_config(db_run) + treatment = _run_treatment(db_run) or treatment + usage = _dict_field(db_run, "usage") + metrics = _dict_field(db_run, "transcript_diagnostics") + progress = _dict_field(db_run, "progress") run_id = db_run.get("id") quest_name = db_run.get("quest_name") agent_id = db_run.get("agent_id") - if run_id is not None and quest_name and agent_id: + if (not usage or not metrics) and run_id is not None and quest_name and agent_id: run_path = Path("results") / str(agent_id) / str(quest_name) / f"run_{run_id}" / "run_summary.json" - run_summary = _load_json(run_path) or {} - usage = _dict_field(run_summary, "usage") - metrics = _dict_field(run_summary, "metrics") - - template_from_config = str(config.get("action_template") or "") - if template_from_config: - template = template_from_config - memory_mode = config.get("memory_mode") or result_row.get("memory_mode") + run_summary = _load_json(run_path) + if isinstance(run_summary, dict): + record = RunRecord.from_dict(run_summary) + usage = usage or record.usage + metrics = metrics or record.transcript_diagnostics + progress = progress or record.progress.to_dict() + if _is_retired_result( str(source_name) if source_name else None, str(benchmark_id) if benchmark_id else None, result_row, - config, - template, + treatment, ): continue - mode_id, mode_label = _mode_from_template(template, str(memory_mode) if memory_mode is not None else None) + mode_id, mode_label = _mode_from_treatment(treatment) try: spec = parse_model_name(model) @@ -416,6 +414,7 @@ def generate_leaderboard( "total_tokens": float(usage.get("total_tokens") or 0), "estimated_cost_usd": float(usage.get("estimated_cost_usd") or 0), "repetition_rate": float(metrics.get("repetition_rate") or 0), + "progress": float(progress.get("current") or 0), } ) if display_id not in model_entries: @@ -460,6 +459,7 @@ def generate_leaderboard( "avg_tokens": _mean(row["total_tokens"] for row in rows), "avg_cost_usd": _mean(row["estimated_cost_usd"] for row in rows), "repetition_rate": _mean(row["repetition_rate"] for row in rows), + "avg_progress": _mean(row["progress"] for row in rows), } ) diff --git a/llm_quest_benchmark/core/logging.py b/llm_quest_benchmark/core/logging.py index f4aa7b6..7443684 100644 --- a/llm_quest_benchmark/core/logging.py +++ b/llm_quest_benchmark/core/logging.py @@ -1,4 +1,4 @@ -"""Unified logging module for LLM Quest Benchmark""" +"""Unified logging and schema-v2 run persistence for LLM Quest Benchmark""" import json import logging @@ -9,21 +9,98 @@ from pathlib import Path from typing import Any -from llm_quest_benchmark.schemas import AgentState +from llm_quest_benchmark.schemas.records import ( + SCHEMA_VERSION, + ProgressState, + QuestSnapshot, + QuestTransition, + ResumeLineage, + RunRecord, +) # Configure logging logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s") # Constants -DEFAULT_DB_PATH = "metrics.db" -RESULTS_DIR = Path("results") +# +# Both stores are process-wide defaults resolved from the environment at import +# time, so a benchmark worker started with ``spawn`` inherits the same isolated +# database and results tree as its parent instead of silently writing to the +# working directory. +DB_PATH_ENV_VAR = "LLM_QUEST_DB_PATH" +RESULTS_DIR_ENV_VAR = "LLM_QUEST_RESULTS_DIR" + +DEFAULT_DB_PATH = os.environ.get(DB_PATH_ENV_VAR) or "metrics.db" +RESULTS_DIR = Path(os.environ.get(RESULTS_DIR_ENV_VAR) or "results") + +LEGACY_DB_MESSAGE = ( + "metrics database uses the pre-v2 schema. Convert it with: " + "scripts/migrate_records.py --source --output " +) + + +def default_db_path() -> str: + """Metrics database used whenever a caller supplies no explicit path. + + Frozen at import time from ``LLM_QUEST_DB_PATH``; subprocesses can redirect + via the environment variable, in-process overrides cannot. + """ + return DEFAULT_DB_PATH + + +RUNS_TABLE_SQL = """ + CREATE TABLE IF NOT EXISTS runs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + schema_version INTEGER NOT NULL, + quest_file TEXT NOT NULL, + quest_name TEXT NOT NULL, + quest_checksum TEXT NOT NULL, + quest_language TEXT NOT NULL, + engine_revision TEXT NOT NULL, + agent_id TEXT NOT NULL, + treatment TEXT NOT NULL, + treatment_signature TEXT NOT NULL, + benchmark_id TEXT, + lineage TEXT, + start_time TIMESTAMP NOT NULL, + end_time TIMESTAMP, + run_duration REAL, + outcome TEXT, + reward REAL, + usage TEXT, + transcript_diagnostics TEXT, + progress TEXT, + terminal_snapshot TEXT + ) +""" + +TRANSITIONS_TABLE_SQL = """ + CREATE TABLE IF NOT EXISTS transitions ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + run_id INTEGER NOT NULL, + transition_index INTEGER NOT NULL, + before_state TEXT NOT NULL, + action TEXT NOT NULL, + after_state TEXT NOT NULL, + response TEXT, + usage TEXT, + progress TEXT, + provenance TEXT NOT NULL, + replay_status TEXT NOT NULL, + reasoning_mode TEXT, + FOREIGN KEY (run_id) REFERENCES runs (id) + ) +""" + +# SQLite stores the same logical record as run_summary.json, flattened: each +# column is one field of a canonical domain. class LogManager: """Manages logging configuration""" - def __init__(self): - self.logger = logging.getLogger("llm_quest") + def __init__(self, name: str = "llm_quest"): + self.logger = logging.getLogger("llm_quest" if name == "llm_quest" else f"llm_quest.{name}") def setup(self, debug: bool = False): """Setup logging configuration""" @@ -39,190 +116,97 @@ def get_logger(self): return self.logger -class QuestLogger: - """Logs quest runs to SQLite database and exports to JSON when complete""" - - DEFAULT_REPETITION_WINDOW = 5 - - @staticmethod - def _safe_json_load(json_str, default=None): - """Safely load JSON with error handling and repair attempts""" - if not json_str: - return default +def verify_v2_schema(conn: sqlite3.Connection) -> bool: + """Return whether a v2 ``runs`` table exists, refusing a legacy database. - try: - return json.loads(json_str) - except json.JSONDecodeError: - # Try to repair damaged JSON - try: - from json_repair import repair_json - - repaired = repair_json(json_str) - return json.loads(repaired) - except ImportError: - # Log warning about missing json-repair - import logging - - logging.getLogger("quest_logger").warning( - "json-repair module not available - some JSON may not parse correctly" - ) + A database with no ``runs`` table is simply empty, not legacy; readers use + that to distinguish "nothing recorded yet" from "needs migration". + """ + cursor = conn.cursor() + cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='runs'") + if cursor.fetchone() is None: + return False - # Manual repair attempt - try: - # If it starts with a string that looks like a dict - if json_str.strip().startswith("{"): - # Extract everything between the first { and the last } - clean_str = json_str[json_str.find("{") : json_str.rfind("}") + 1] - return json.loads(clean_str) - except Exception: - pass + cursor.execute("PRAGMA table_info(runs)") + columns = {row[1] for row in cursor.fetchall()} + if "schema_version" not in columns: + raise RuntimeError(LEGACY_DB_MESSAGE) + return True - # Return default if all repair attempts failed - return default - @staticmethod - def _safe_int(value) -> int | None: - """Best-effort conversion to int.""" - try: - if value is None: - return None - return int(value) - except (TypeError, ValueError): - return None - - @staticmethod - def _choices_map(choices: list[dict[str, Any]]) -> dict[str, str]: - """Map quest choices to a compact indexed text dictionary.""" - return {str(idx): choice.get("text", "") for idx, choice in enumerate(choices, start=1)} +TRANSITION_INSERT_SQL = """ + INSERT INTO transitions ( + run_id, transition_index, before_state, action, after_state, + response, usage, progress, provenance, replay_status, reasoning_mode + ) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """ - @staticmethod - def _selected_choice_map(choices_map: dict[str, str], action_index: int | None) -> dict[str, str] | None: - """Return selected choice as {index: text} if action is valid.""" - if action_index is None: - return None - key = str(action_index) - if key not in choices_map: - return None - return {key: choices_map[key]} - - def _calculate_run_metrics( - self, - step_rows: list[tuple[Any, ...]], - outcome: str | None, - ) -> dict[str, Any]: - """Compute simple run-level behavior metrics from raw step rows.""" - recent_actions: list[int] = [] - repetition_count = 0 - window = self._repetition_window - for _, _, _, _, action, _ in step_rows: - action_index = self._safe_int(action) - if action_index is None: - continue - if action_index in recent_actions: - repetition_count += 1 - recent_actions.append(action_index) - recent_actions = recent_actions[-window:] +def insert_transition_row(conn: sqlite3.Connection, run_id: int, transition: QuestTransition) -> None: + """Single writer for schema-v2 transition rows (runtime logger and migration).""" + payload = transition.to_dict() + conn.execute( + TRANSITION_INSERT_SQL, + ( + run_id, + transition.index, + json.dumps(payload["before"], ensure_ascii=False), + json.dumps(payload["action"], ensure_ascii=False), + json.dumps(payload["after"], ensure_ascii=False), + json.dumps(payload["response"], ensure_ascii=False) if payload["response"] else None, + json.dumps(payload["usage"], ensure_ascii=False), + json.dumps(payload["progress"], ensure_ascii=False), + transition.provenance, + transition.replay_status, + transition.reasoning_mode, + ), + ) + + +def ensure_v2_schema(conn: sqlite3.Connection) -> None: + """Create the v2 tables, refusing to touch a legacy database. + + There is no in-place upgrade path: legacy databases are converted once by + ``scripts/migrate_records.py`` into a fresh v2 destination. + """ + verify_v2_schema(conn) + cursor = conn.cursor() + cursor.execute(RUNS_TABLE_SQL) + cursor.execute(TRANSITIONS_TABLE_SQL) + conn.commit() - total_steps = len(step_rows) - bad_decision_count = 1 if outcome == "FAILURE" and total_steps > 0 else 0 - return { - "total_steps": total_steps, - "repetition_window": window, - "repetition_count": repetition_count, - "repetition_rate": (repetition_count / total_steps) if total_steps else 0.0, - "bad_decision_count": bad_decision_count, - "bad_decision_rate": (bad_decision_count / total_steps) if total_steps else 0.0, - } +class QuestLogger: + """Persists schema-v2 runs to SQLite and exports one run_summary.json. - def _format_step_export( - self, - step_num: int, - location_id: str, - observation: str, - choices: list[dict[str, Any]], - action: Any, - llm_response: dict[str, Any] | None, - ) -> dict[str, Any]: - """Format an exported step in compact analysis-friendly form.""" - choices_map = self._choices_map(choices) - parsed_action_index = self._safe_int(action) - analysis = None - reasoning = None - memo = None - tool_calls = None - tool_results = None - is_default = True - parse_mode = None - prompt_tokens = 0 - completion_tokens = 0 - total_tokens = 0 - estimated_cost_usd = None - - if isinstance(llm_response, dict): - parsed_action_index = self._safe_int( - llm_response.get("action") or llm_response.get("result") or llm_response.get("choice") or action - ) - analysis = llm_response.get("analysis") - reasoning = llm_response.get("reasoning") - memo = llm_response.get("memo") - tool_calls = llm_response.get("tool_calls") - tool_results = llm_response.get("tool_results") - is_default = bool(llm_response.get("is_default", False)) - parse_mode = llm_response.get("parse_mode") - prompt_tokens = int(llm_response.get("prompt_tokens") or 0) - completion_tokens = int(llm_response.get("completion_tokens") or 0) - total_tokens = int(llm_response.get("total_tokens") or (prompt_tokens + completion_tokens)) - if llm_response.get("estimated_cost_usd") is not None: - estimated_cost_usd = float(llm_response.get("estimated_cost_usd")) - - llm_decision = { - "analysis": analysis, - "reasoning": reasoning, - "memo": memo, - "tool_calls": tool_calls, - "tool_results": tool_results, - "is_default": is_default, - "parse_mode": parse_mode, - "choice": self._selected_choice_map(choices_map, parsed_action_index), - "prompt_tokens": prompt_tokens, - "completion_tokens": completion_tokens, - "total_tokens": total_tokens, - "estimated_cost_usd": estimated_cost_usd, - } + Run identity, quest provenance, and the harness treatment are written + before the first transition, so nothing is patched into the record after + the JSON export. + """ - return { - "step": step_num, - "location_id": location_id, - "observation": observation, - "choices": choices_map, - "llm_decision": llm_decision, - } + DEFAULT_REPETITION_WINDOW = 5 # Thread-local storage for database connections _local = threading.local() # Track all instances for cleanup _instances = [] - def __init__(self, db_path: str = DEFAULT_DB_PATH, debug: bool = False, agent: str | None = None): + def __init__(self, db_path: str | None = None, debug: bool = False, agent: str | None = None): """Initialize the quest logger. Args: - db_path: Path to SQLite database + db_path: Path to SQLite database; defaults to DEFAULT_DB_PATH at call time debug: Enable debug logging - agent: Agent identifier + agent: Agent identifier used for the results directory """ - self.db_path = db_path + self.db_path = db_path or default_db_path() self.debug = debug self.agent = agent self._repetition_window = self.DEFAULT_REPETITION_WINDOW self.current_run_id = None - self.quest_file = None - self.steps = [] - self.run_outcome = None - self.end_time = None - self.final_state = None + self.record: RunRecord | None = None + self.start_time: datetime | None = None self._finalize_lock = threading.Lock() self._finalized = False @@ -264,190 +248,129 @@ def _shutdown_handler(signal=None, frame=None): self.logger.debug("Skipping signal handlers in non-main thread") def _init_connection(self): - """Initialize a thread-local database connection""" - # Create a new connection for this thread if it doesn't exist - if not hasattr(self._local, "conn") or self._local.conn is None: - # Reducing debug logging - pass # Skip thread connection logging - self._local.conn = sqlite3.connect(self.db_path) - self._local.cursor = self._local.conn.cursor() - - # Create tables if they don't exist - self._create_tables() - - def _create_tables(self): - """Create database tables if they don't exist""" - # Check if the runs table already exists - self._local.cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='runs'") - table_exists = self._local.cursor.fetchone() is not None - - if table_exists: - # Check if we need to add columns to existing table - self._local.cursor.execute("PRAGMA table_info(runs)") - columns = [column[1] for column in self._local.cursor.fetchall()] - - # Add missing columns - if "outcome" not in columns: - self._local.cursor.execute("ALTER TABLE runs ADD COLUMN outcome TEXT") - if "reward" not in columns: - self._local.cursor.execute("ALTER TABLE runs ADD COLUMN reward REAL") - if "run_duration" not in columns: - self._local.cursor.execute("ALTER TABLE runs ADD COLUMN run_duration REAL") - if "benchmark_id" not in columns: - self._local.cursor.execute("ALTER TABLE runs ADD COLUMN benchmark_id TEXT") - else: - # Create the runs table if it doesn't exist - self._local.cursor.execute(""" - CREATE TABLE IF NOT EXISTS runs ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - quest_file TEXT, - quest_name TEXT, - start_time TIMESTAMP, - end_time TIMESTAMP, - agent_id TEXT, - agent_config TEXT, - outcome TEXT, - reward REAL, - run_duration REAL, - benchmark_id TEXT - ) - """) - - # Create steps table - self._local.cursor.execute(""" - CREATE TABLE IF NOT EXISTS steps ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - run_id INTEGER, - step INTEGER, - location_id TEXT, - observation TEXT, - choices TEXT, - action TEXT, - llm_response TEXT, - FOREIGN KEY (run_id) REFERENCES runs (id) - ) - """) + """Initialize a thread-local database connection. - self._local.conn.commit() + The connection cache is keyed by database path: two loggers in the same + thread pointing at different databases must never share a handle. + """ + conn = getattr(self._local, "conn", None) + if conn is not None and getattr(self._local, "db_path", None) != self.db_path: + conn.close() + conn = None + self._local.conn = None + self._local.cursor = None - def set_quest_file(self, quest_file: str): - """Set the quest file and create a new run record. + if conn is None: + connection = sqlite3.connect(self.db_path) + try: + ensure_v2_schema(connection) + except Exception: + connection.close() + raise + self._local.conn = connection + self._local.cursor = connection.cursor() + self._local.db_path = self.db_path - Args: - quest_file: Path to quest file - """ - # Ensure we have a connection for this thread + def start_run( + self, + *, + quest_file: str, + quest_name: str, + quest_checksum: str, + quest_language: str, + engine_revision: str, + agent_id: str, + treatment: dict[str, Any], + benchmark_id: str | None = None, + lineage: ResumeLineage | None = None, + ) -> int: + """Create the run row with complete metadata and return its id.""" self._init_connection() - self.quest_file = quest_file - self.steps = [] + self.agent = agent_id self.start_time = datetime.utcnow() - self.end_time = None - self.run_outcome = None - self.final_state = None self._finalized = False - try: - # Extract quest name from path (filename without extension) - quest_name = Path(quest_file).stem - - # Create run record with both quest_file and quest_name - self._local.cursor.execute( - """ - INSERT INTO runs (quest_file, quest_name, start_time, agent_id) - VALUES (?, ?, ?, ?) - """, - (quest_file, quest_name, self.start_time, self.agent), + self._local.cursor.execute( + """ + INSERT INTO runs ( + schema_version, quest_file, quest_name, quest_checksum, quest_language, + engine_revision, agent_id, treatment, treatment_signature, benchmark_id, + lineage, start_time ) - self._local.conn.commit() - except sqlite3.OperationalError as e: - if "no such column: quest_file" in str(e): - # Fallback for older schema without quest_file column - self.logger.warning("quest_file column not found in database, using quest_name instead") - - # Extract quest name from path (filename without extension) - quest_name = Path(quest_file).stem - - self._local.cursor.execute( - """ - INSERT INTO runs (quest_name, start_time, agent_id) - VALUES (?, ?, ?) - """, - (quest_name, self.start_time, self.agent), - ) - self._local.conn.commit() - else: - raise - - # Get the run ID + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + SCHEMA_VERSION, + quest_file, + quest_name, + quest_checksum, + quest_language, + engine_revision, + agent_id, + json.dumps(treatment, ensure_ascii=False), + str(treatment.get("signature") or ""), + benchmark_id, + json.dumps(lineage.to_dict(), ensure_ascii=False) if lineage else None, + self.start_time, + ), + ) + self._local.conn.commit() self.current_run_id = self._local.cursor.lastrowid - # Skip logging run record ID to reduce output - pass - def log_step(self, agent_state: AgentState): - """Log a step to the database. + self.record = RunRecord( + run_id=self.current_run_id, + quest_file=quest_file, + quest_name=quest_name, + quest_checksum=quest_checksum, + quest_language=quest_language, + engine_revision=engine_revision, + agent_id=agent_id, + treatment=treatment, + started_at=self.start_time.isoformat(), + benchmark_id=benchmark_id, + lineage=lineage, + ) + return self.current_run_id - Args: - agent_state: Agent state to log - """ - # Ensure we have a connection for this thread - self._init_connection() + def log_transition(self, transition: QuestTransition) -> None: + """Persist one executed transition.""" + if self.record is None or self.current_run_id is None: + self.logger.warning("Cannot log transition, no active run") + return - self.steps.append(agent_state) + self._init_connection() + self.record.transitions.append(transition) if self.debug: - self.logger.debug(self.format_step_for_console(agent_state)) + self.logger.debug(self.format_transition_for_console(transition)) try: - # Format choices as JSON for storage - choices_json = json.dumps(agent_state.choices, ensure_ascii=False) - - # Store all step data together including action and llm_response if available - if agent_state.observation: - llm_response_json = None - if agent_state.llm_response is not None: - llm_response_json = json.dumps( - agent_state.llm_response.to_dict(), - ensure_ascii=False, - ) - - self._local.cursor.execute( - """ - INSERT INTO steps (run_id, step, location_id, observation, choices, action, llm_response) - VALUES (?, ?, ?, ?, ?, ?, ?) - """, - ( - self.current_run_id, - agent_state.step, - agent_state.location_id, - agent_state.observation, - choices_json, - agent_state.action, - llm_response_json, - ), - ) - self._local.conn.commit() - + insert_transition_row(self._local.conn, self.current_run_id, transition) + self._local.conn.commit() except Exception as e: - self.logger.error(f"Error logging step: {e}") + self.logger.error(f"Error logging transition: {e}") - def set_quest_outcome( + def adopt_transitions(self, transitions: list[QuestTransition]) -> None: + """Persist prior transitions carried over into a resumed run.""" + for transition in transitions: + self.log_transition(transition) + + def finish_run( self, outcome: str, reward: float = 0.0, - benchmark_id: str = None, - final_state: dict[str, Any] | None = None, - ): - """Set the quest outcome and finalize the run. - - Args: - outcome: Quest outcome (SUCCESS, FAILURE, etc.) - reward: Final reward value - benchmark_id: Optional benchmark ID to associate with this run - final_state: Optional final environment state snapshot for export + terminal_snapshot: QuestSnapshot | None = None, + progress: ProgressState | None = None, + diagnostics: dict[str, Any] | None = None, + ) -> None: + """Record the terminal outcome and export the v2 run summary. + + First write wins: late outcome updates from a background timeout race + cannot overwrite the outcome already recorded for this run. """ with self._finalize_lock: - if not self.current_run_id: + if self.record is None or not self.current_run_id: self.logger.warning("Cannot set outcome, no active run") return if self._finalized: @@ -455,201 +378,156 @@ def set_quest_outcome( "Ignoring late outcome update for run %s: %s (already finalized as %s)", self.current_run_id, outcome, - self.run_outcome, + self.record.outcome, ) return - self.run_outcome = outcome - self.final_state = final_state - self.end_time = datetime.utcnow() - run_duration = (self.end_time - self.start_time).total_seconds() + end_time = datetime.utcnow() + record = self.record + record.outcome = outcome + record.reward = reward + record.terminal_snapshot = terminal_snapshot + record.ended_at = end_time.isoformat() + record.run_duration = (end_time - self.start_time).total_seconds() if self.start_time else None + record.usage = self.aggregate_usage(record.transitions) + record.transcript_diagnostics = self.calculate_metrics(record.transitions, outcome, self._repetition_window) + if diagnostics: + record.transcript_diagnostics.update(diagnostics) + if progress is not None: + record.progress = progress try: - # Update the run record with outcome and end time - if benchmark_id: - self.logger.debug(f"Setting quest outcome with benchmark_id: {benchmark_id}") - self._local.cursor.execute( - """ - UPDATE runs - SET outcome = ?, end_time = ?, reward = ?, run_duration = ?, benchmark_id = ? - WHERE id = ? - """, - (outcome, self.end_time, reward, run_duration, benchmark_id, self.current_run_id), - ) - else: - self._local.cursor.execute( - """ - UPDATE runs - SET outcome = ?, end_time = ?, reward = ?, run_duration = ? - WHERE id = ? + self._local.cursor.execute( + """ + UPDATE runs + SET end_time = ?, run_duration = ?, outcome = ?, reward = ?, + usage = ?, transcript_diagnostics = ?, progress = ?, terminal_snapshot = ? + WHERE id = ? """, - (outcome, self.end_time, reward, run_duration, self.current_run_id), - ) + ( + end_time, + record.run_duration, + outcome, + reward, + json.dumps(record.usage, ensure_ascii=False), + json.dumps(record.transcript_diagnostics, ensure_ascii=False), + json.dumps(record.progress.to_dict(), ensure_ascii=False), + json.dumps(terminal_snapshot.to_dict(), ensure_ascii=False) if terminal_snapshot else None, + self.current_run_id, + ), + ) self._local.conn.commit() - # Export the run to JSON self._export_run_to_json() self._finalized = True - except Exception as e: self.logger.error(f"Error setting quest outcome: {e}") + @staticmethod + def aggregate_usage(transitions: list[QuestTransition]) -> dict[str, Any]: + """Sum token usage and cost across transitions.""" + prompt_tokens = completion_tokens = total_tokens = priced_steps = 0 + estimated_cost = 0.0 + for transition in transitions: + usage = transition.usage or {} + prompt = int(usage.get("prompt_tokens") or 0) + completion = int(usage.get("completion_tokens") or 0) + prompt_tokens += prompt + completion_tokens += completion + total_tokens += int(usage.get("total_tokens") or (prompt + completion)) + if usage.get("estimated_cost_usd") is not None: + estimated_cost += float(usage["estimated_cost_usd"]) + priced_steps += 1 + return { + "prompt_tokens": prompt_tokens, + "completion_tokens": completion_tokens, + "total_tokens": total_tokens, + "estimated_cost_usd": round(estimated_cost, 8) if priced_steps > 0 else None, + "priced_steps": priced_steps, + } + + @staticmethod + def calculate_metrics( + transitions: list[QuestTransition], + outcome: str | None, + window: int = DEFAULT_REPETITION_WINDOW, + ) -> dict[str, Any]: + """Compute run-level behaviour metrics from canonical transitions.""" + recent_actions: list[int] = [] + repetition_count = 0 + choose_count = 0 + restore_count = 0 + default_decisions = 0 + reasoning_modes: dict[str, int] = {} + + for transition in transitions: + if transition.reasoning_mode: + reasoning_modes[transition.reasoning_mode] = reasoning_modes.get(transition.reasoning_mode, 0) + 1 + if transition.action.is_restore: + restore_count += 1 + continue + choose_count += 1 + if transition.response is not None and transition.response.is_default: + default_decisions += 1 + index = transition.action.choice_index + if index is None: + continue + if index in recent_actions: + repetition_count += 1 + recent_actions.append(index) + recent_actions = recent_actions[-window:] + + bad_decision_count = 1 if outcome == "FAILURE" and choose_count > 0 else 0 + return { + "total_steps": choose_count, + "total_transitions": len(transitions), + "choose_transitions": choose_count, + "restore_transitions": restore_count, + "repetition_window": window, + "repetition_count": repetition_count, + "repetition_rate": (repetition_count / choose_count) if choose_count else 0.0, + "bad_decision_count": bad_decision_count, + "bad_decision_rate": (bad_decision_count / choose_count) if choose_count else 0.0, + "default_decisions": default_decisions, + "reasoning_modes": reasoning_modes, + } + def _export_run_to_json(self): """Export the complete run to a single run_summary.json file.""" - if not self.agent or not self.quest_file or not self.current_run_id: - return - if self.agent.startswith("random"): - # Keep random-agent runs in DB for diagnostics, but avoid result-dir clutter. + if self.record is None or not self.agent or not self.current_run_id: return try: - # Create agent directory if it doesn't exist - agent_dir = RESULTS_DIR / self.agent - agent_dir.mkdir(parents=True, exist_ok=True) - - # Create quest directory if it doesn't exist - quest_name = Path(self.quest_file).stem - quest_dir = agent_dir / quest_name - quest_dir.mkdir(exist_ok=True) - - # Create run directory if it doesn't exist - run_dir = quest_dir / f"run_{self.current_run_id}" - run_dir.mkdir(exist_ok=True) - - # Fetch complete run data from database - run_data = self._get_run_data() + run_dir = RESULTS_DIR / self.agent / self.record.quest_name / f"run_{self.current_run_id}" + run_dir.mkdir(parents=True, exist_ok=True) - # Save complete run summary run_summary_file = run_dir / "run_summary.json" with open(run_summary_file, "w", encoding="utf-8") as f: - json.dump(run_data, f, indent=2, ensure_ascii=False) + json.dump(self.record.to_dict(), f, indent=2, ensure_ascii=False) self.logger.debug(f"Exported run data to {run_summary_file}") - except Exception as e: self.logger.error(f"Error exporting run to JSON: {e}") - def _get_run_data(self) -> dict[str, Any]: - """Get complete run data from the database for the current run. - - Returns: - Dict containing run and step data - """ - # Ensure we have a connection for this thread - self._init_connection() - - # Get run data - self._local.cursor.execute( - """ - SELECT quest_file, quest_name, start_time, end_time, agent_id, - agent_config, outcome, reward, run_duration, benchmark_id - FROM runs - WHERE id = ? - """, - (self.current_run_id,), - ) - - run = self._local.cursor.fetchone() - if not run: - return {"error": f"Run with ID {self.current_run_id} not found"} - - ( - quest_file, - quest_name, - start_time, - end_time, - agent_id, - agent_config, - outcome, - reward, - run_duration, - benchmark_id, - ) = run - - # Get steps for this run - self._local.cursor.execute( - """ - SELECT step, location_id, observation, choices, action, llm_response - FROM steps - WHERE run_id = ? - ORDER BY step - """, - (self.current_run_id,), + @staticmethod + def format_transition_for_console(transition: QuestTransition) -> str: + """Format a transition for console output.""" + choices_str = "\n".join(f"{i + 1}. {choice['text']}" for i, choice in enumerate(transition.before.choices)) + if transition.action.is_restore: + action_str = f"restore checkpoint {transition.action.checkpoint_index}" + else: + action_str = f"choose {transition.action.choice_index}" + return ( + f"Transition {transition.index}:\n" + f"Observation: {transition.before.observation}\n" + f"Choices:\n{choices_str}\n" + f"Action: {action_str}" ) - step_rows = self._local.cursor.fetchall() - steps = [] - usage_prompt_tokens = 0 - usage_completion_tokens = 0 - usage_total_tokens = 0 - usage_estimated_cost = 0.0 - usage_priced_steps = 0 - for step_data in step_rows: - step_num, location_id, obs, choices_json, action, llm_response = step_data - parsed_choices = self._safe_json_load(choices_json, []) - parsed_response = self._safe_json_load(llm_response) - if isinstance(parsed_response, dict): - prompt_tokens = int(parsed_response.get("prompt_tokens") or 0) - completion_tokens = int(parsed_response.get("completion_tokens") or 0) - total_tokens = int(parsed_response.get("total_tokens") or (prompt_tokens + completion_tokens)) - usage_prompt_tokens += prompt_tokens - usage_completion_tokens += completion_tokens - usage_total_tokens += total_tokens - if parsed_response.get("estimated_cost_usd") is not None: - usage_estimated_cost += float(parsed_response.get("estimated_cost_usd")) - usage_priced_steps += 1 - steps.append( - self._format_step_export( - step_num=step_num, - location_id=location_id, - observation=obs, - choices=parsed_choices if isinstance(parsed_choices, list) else [], - action=action, - llm_response=parsed_response if isinstance(parsed_response, dict) else None, - ) - ) - - metrics = self._calculate_run_metrics(step_rows, outcome) - - return { - "run_id": self.current_run_id, - "quest_file": quest_file, - "quest_name": quest_name, - "start_time": start_time, - "end_time": end_time, - "agent_id": agent_id, - "agent_config": self._safe_json_load(agent_config), - "outcome": outcome, - "reward": reward, - "run_duration": run_duration, - "benchmark_id": benchmark_id, - "final_state": self.final_state, - "usage": { - "prompt_tokens": usage_prompt_tokens, - "completion_tokens": usage_completion_tokens, - "total_tokens": usage_total_tokens, - "estimated_cost_usd": (round(usage_estimated_cost, 8) if usage_priced_steps > 0 else None), - "priced_steps": usage_priced_steps, - }, - "metrics": metrics, - "steps": steps, - } - - def format_step_for_console(self, agent_state: AgentState) -> str: - """Format step for console output. - - Args: - agent_state: Agent state to format - - Returns: - Formatted step string - """ - choices_str = "\n".join([f"{i + 1}. {choice['text']}" for i, choice in enumerate(agent_state.choices)]) - return f"Step {agent_state.step}:\nObservation: {agent_state.observation}\nChoices:\n{choices_str}\nAction: {agent_state.action}" - def close(self): """Close the database connection for this thread""" if hasattr(self._local, "conn") and self._local.conn: self._local.conn.close() self._local.conn = None self._local.cursor = None + self._local.db_path = None diff --git a/llm_quest_benchmark/core/migration.py b/llm_quest_benchmark/core/migration.py new file mode 100644 index 0000000..48ecfaa --- /dev/null +++ b/llm_quest_benchmark/core/migration.py @@ -0,0 +1,574 @@ +"""One-time migration of legacy records into schema v2. + +This module is the only legacy reader in the codebase. It never fabricates a +missing action, timestamp, parameter state, engine saving, or post-state: +unknowable fields are marked unavailable and the resulting transitions are +rejected for resume. +""" + +from __future__ import annotations + +import json +import sqlite3 +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +from llm_quest_benchmark.core.logging import QuestLogger, ensure_v2_schema, insert_transition_row +from llm_quest_benchmark.core.provenance import quest_checksum +from llm_quest_benchmark.harnesses.specs import HARNESS_SPECS, HarnessTreatment, build_treatment +from llm_quest_benchmark.schemas.records import ( + PROVENANCE_LEGACY_MAPPED, + REPLAY_UNAVAILABLE, + SCHEMA_VERSION, + ProgressState, + QuestAction, + QuestSnapshot, + QuestTransition, + RunRecord, +) +from llm_quest_benchmark.schemas.response import LLMResponse + +UNAVAILABLE = "unavailable" +UNKNOWN = "unknown" +LEGACY_RUN_COLUMNS = ( + "id", + "quest_file", + "quest_name", + "start_time", + "end_time", + "agent_id", + "agent_config", + "outcome", + "reward", + "run_duration", + "benchmark_id", +) + + +@dataclass +class MigrationReport: + """Summary of one migration invocation.""" + + source: str + output: str + kind: str + runs_migrated: int = 0 + transitions_migrated: int = 0 + resumable_runs: int = 0 + skipped: list[str] = field(default_factory=list) + + def to_dict(self) -> dict[str, Any]: + return { + "source": self.source, + "output": self.output, + "kind": self.kind, + "runs_migrated": self.runs_migrated, + "transitions_migrated": self.transitions_migrated, + "resumable_runs": self.resumable_runs, + "skipped": self.skipped, + } + + +# ---- shared helpers -------------------------------------------------------- + + +def _safe_json_load(value: Any, default: Any = None) -> Any: + if value is None or value == "": + return default + if isinstance(value, (dict, list)): + return value + try: + return json.loads(value) + except (json.JSONDecodeError, TypeError): + return default + + +def _safe_int(value: Any) -> int | None: + try: + if value is None or isinstance(value, bool): + return None + return int(value) + except (TypeError, ValueError): + return None + + +def _choices_from_list(choices: Any) -> list[dict[str, str]]: + """Legacy SQLite stored choices as [{id, text}]; ids are engine jump ids.""" + if not isinstance(choices, list): + return [] + return [{"id": str(c.get("id", "")), "text": str(c.get("text", ""))} for c in choices if isinstance(c, dict)] + + +def _choices_from_map(choices: Any) -> list[dict[str, str]]: + """Compact JSON stored choices as {index: text}; engine jump ids are lost.""" + if not isinstance(choices, dict): + return [] + ordered = sorted(choices.items(), key=lambda item: _safe_int(item[0]) or 0) + return [{"id": "", "text": str(text)} for _, text in ordered] + + +def _legacy_snapshot( + location_id: Any, + observation: Any, + choices: list[dict[str, str]], + params_state: list[str] | None, + game_state: str = "running", + done: bool = False, + reward: float = 0.0, +) -> QuestSnapshot: + """Build a before/after snapshot from legacy fields, marking what is lost.""" + missing = ["saving"] + if params_state is None: + missing.append("params_state") + if choices and not any(choice["id"] for choice in choices): + missing.append("choice_ids") + return QuestSnapshot( + location_id=str(location_id if location_id is not None else ""), + observation=str(observation or ""), + choices=choices, + params_state=list(params_state or []), + reward=reward, + done=done, + game_state=game_state, + saving=None, + unavailable_fields=missing, + ) + + +def _legacy_treatment(agent_config: Any) -> HarnessTreatment: + """Derive the treatment from a legacy agent config and the canonical registry.""" + config = agent_config if isinstance(agent_config, dict) else {} + harness = str(config.get("harness") or "") + model = str(config.get("model") or UNAVAILABLE) + temperature = config.get("temperature") + try: + temperature = float(temperature) if temperature is not None else 0.0 + except (TypeError, ValueError): + temperature = 0.0 + + if harness in HARNESS_SPECS: + return build_treatment( + harness=harness, + model=model, + temperature=temperature, + system_template=str(config.get("system_template") or UNAVAILABLE), + knob_values={"compaction_interval": config.get("compaction_interval")}, + ) + # No compatibility alias: an unresolvable configuration stays explicitly + # unknown. The recorded agent id is preserved on the run, not folded into + # a guessed treatment. + return HarnessTreatment.unknown(harness or UNKNOWN, model, temperature) + + +def _legacy_response(payload: Any, action_index: int | None) -> LLMResponse | None: + if not isinstance(payload, dict): + return None + resolved_action = _safe_int(payload.get("action")) or action_index or 1 + return LLMResponse( + action=resolved_action, + analysis=payload.get("analysis"), + reasoning=payload.get("reasoning"), + memo=payload.get("memo"), + tool_calls=payload.get("tool_calls"), + tool_results=payload.get("tool_results"), + is_default=bool(payload.get("is_default", False)), + parse_mode=payload.get("parse_mode"), + prompt_tokens=_safe_int(payload.get("prompt_tokens")), + completion_tokens=_safe_int(payload.get("completion_tokens")), + total_tokens=_safe_int(payload.get("total_tokens")), + estimated_cost_usd=payload.get("estimated_cost_usd"), + ) + + +def _usage_from_response(response: LLMResponse | None) -> dict[str, Any]: + return response.usage_payload() if response is not None else {} + + +def _legacy_action(choice_index: int | None, choices: list[dict[str, str]]) -> QuestAction: + """Map a legacy action index; ids and timestamps are never invented.""" + choice_id = None + if choice_index is not None and 1 <= choice_index <= len(choices): + choice_id = choices[choice_index - 1]["id"] or None + return QuestAction( + kind="choose", + choice_index=choice_index, + choice_id=choice_id, + performed_at_ms=None, + ) + + +def _build_transitions( + rows: list[dict[str, Any]], + final_state: dict[str, Any] | None, +) -> list[QuestTransition]: + """Map ordered legacy decision rows into v2 transitions. + + Each legacy row records the state *before* its action, so row ``i + 1`` is + the observed post-state of transition ``i`` when their step numbers are + consecutive. A trailing logger-only terminal row is treated as terminal + state, not as an executed action. + """ + terminal_row: dict[str, Any] | None = None + if rows and not rows[-1]["choices"]: + terminal_row = rows[-1] + rows = rows[:-1] + + transitions: list[QuestTransition] = [] + for i, row in enumerate(rows): + before = _legacy_snapshot( + location_id=row["location_id"], + observation=row["observation"], + choices=row["choices"], + params_state=row.get("params_state"), + ) + + next_row = rows[i + 1] if i + 1 < len(rows) else terminal_row + after = QuestSnapshot.unavailable() + structurally_next = ( + next_row is not None + and _safe_int(next_row.get("step")) is not None + and _safe_int(row.get("step")) is not None + and _safe_int(next_row["step"]) == _safe_int(row["step"]) + 1 + ) + if structurally_next: + after = _legacy_snapshot( + location_id=next_row["location_id"], + observation=next_row["observation"], + choices=next_row["choices"], + params_state=next_row.get("params_state"), + done=next_row is terminal_row, + game_state="running" if next_row is not terminal_row else UNAVAILABLE, + ) + if i == len(rows) - 1: + final_snapshot = _final_state_snapshot(final_state, game_state_when_running="running") + if final_snapshot is not None: + after = final_snapshot + + transitions.append( + QuestTransition( + index=i + 1, + before=before, + action=row["action"], + after=after, + response=row.get("response"), + usage=_usage_from_response(row.get("response")), + progress=ProgressState(current=0.0, maximum=100.0, manifest=None), + provenance=PROVENANCE_LEGACY_MAPPED, + replay_status=REPLAY_UNAVAILABLE, + reasoning_mode=None, + ) + ) + return transitions + + +def _final_state_snapshot(final_state: dict[str, Any] | None, *, game_state_when_running: str) -> QuestSnapshot | None: + """Build the post-state from a run_summary ``final_state`` blob. + + ``None`` means the payload carried nothing usable; callers keep their own + prior interpretation instead of inventing an unavailable post-state. + """ + if isinstance(final_state, dict) and final_state: + return _legacy_snapshot( + location_id=final_state.get("location_id"), + observation=final_state.get("text") or final_state.get("observation"), + choices=_choices_from_list(final_state.get("choices")), + params_state=final_state.get("params_state"), + done=bool(final_state.get("done", False)), + reward=float(final_state.get("reward") or 0.0), + game_state=UNAVAILABLE if final_state.get("done") else game_state_when_running, + ) + return None + + +def _terminal_snapshot(rows: list[dict[str, Any]], final_state: dict[str, Any] | None) -> QuestSnapshot: + forced_terminal = _final_state_snapshot(final_state, game_state_when_running=UNAVAILABLE) + if forced_terminal is not None: + return forced_terminal + if rows and not rows[-1]["choices"]: + row = rows[-1] + return _legacy_snapshot( + location_id=row["location_id"], + observation=row["observation"], + choices=[], + params_state=row.get("params_state"), + done=True, + game_state=UNAVAILABLE, + ) + return QuestSnapshot.unavailable() + + +def _build_record( + *, + run_id: Any, + quest_file: str, + quest_name: str, + start_time: Any, + end_time: Any, + run_duration: Any, + agent_id: str, + agent_config: Any, + outcome: str | None, + reward: Any, + benchmark_id: str | None, + rows: list[dict[str, Any]], + final_state: dict[str, Any] | None, +) -> RunRecord: + treatment = _legacy_treatment(agent_config) + transitions = _build_transitions(rows, final_state) + + checksum = UNAVAILABLE + if quest_file and Path(quest_file).exists(): + checksum = quest_checksum(quest_file) + + record = RunRecord( + run_id=run_id, + quest_file=quest_file or UNAVAILABLE, + quest_name=quest_name or (Path(quest_file).stem if quest_file else UNAVAILABLE), + quest_checksum=checksum, + quest_language=UNAVAILABLE, + engine_revision=UNAVAILABLE, + agent_id=agent_id or UNAVAILABLE, + treatment=treatment.to_dict(), + started_at=str(start_time) if start_time else None, + ended_at=str(end_time) if end_time else None, + run_duration=float(run_duration) if run_duration not in (None, "") else None, + benchmark_id=benchmark_id, + outcome=outcome, + reward=float(reward or 0.0), + transitions=transitions, + terminal_snapshot=_terminal_snapshot(rows, final_state), + schema_version=SCHEMA_VERSION, + ) + record.usage = QuestLogger.aggregate_usage(transitions) + record.transcript_diagnostics = QuestLogger.calculate_metrics(transitions, outcome) + record.transcript_diagnostics["migrated_from"] = "legacy" + return record + + +# ---- legacy JSON ----------------------------------------------------------- + + +def _rows_from_legacy_json(steps: list[Any]) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + for step in steps: + if not isinstance(step, dict): + continue + choices = _choices_from_map(step.get("choices")) + decision = step.get("llm_decision") if isinstance(step.get("llm_decision"), dict) else {} + choice_map = decision.get("choice") if isinstance(decision.get("choice"), dict) else {} + choice_index = _safe_int(next(iter(choice_map), None)) if choice_map else None + rows.append( + { + "step": _safe_int(step.get("step")), + "location_id": step.get("location_id"), + "observation": step.get("observation"), + "choices": choices, + "params_state": step.get("params") if isinstance(step.get("params"), list) else None, + "action": _legacy_action(choice_index, choices), + "response": _legacy_response(decision, choice_index), + } + ) + return rows + + +def migrate_legacy_json(payload: dict[str, Any]) -> RunRecord: + """Map one legacy ``run_summary.json`` document into a v2 record.""" + if payload.get("schema_version") == SCHEMA_VERSION: + raise ValueError("Record is already schema v2; migration is a one-time legacy conversion.") + + rows = _rows_from_legacy_json(payload.get("steps") or []) + return _build_record( + run_id=payload.get("run_id"), + quest_file=str(payload.get("quest_file") or ""), + quest_name=str(payload.get("quest_name") or ""), + start_time=payload.get("start_time"), + end_time=payload.get("end_time"), + run_duration=payload.get("run_duration"), + agent_id=str(payload.get("agent_id") or ""), + agent_config=payload.get("agent_config"), + outcome=payload.get("outcome"), + reward=payload.get("reward"), + benchmark_id=payload.get("benchmark_id"), + rows=rows, + final_state=payload.get("final_state") if isinstance(payload.get("final_state"), dict) else None, + ) + + +# ---- legacy SQLite --------------------------------------------------------- + + +def _rows_from_legacy_sqlite(conn: sqlite3.Connection, run_id: Any) -> list[dict[str, Any]]: + cursor = conn.execute( + """ + SELECT step, location_id, observation, choices, action, llm_response + FROM steps + WHERE run_id = ? + ORDER BY step + """, + (run_id,), + ) + rows: list[dict[str, Any]] = [] + for step, location_id, observation, choices_json, action, llm_response in cursor.fetchall(): + choices = _choices_from_list(_safe_json_load(choices_json, [])) + # SQLite stores the runner-clamped action, which compact JSON could not + # distinguish from a model proposal, so it wins whenever it is present. + choice_index = _safe_int(action) + response_payload = _safe_json_load(llm_response) + rows.append( + { + "step": _safe_int(step), + "location_id": location_id, + "observation": observation, + "choices": choices, + "params_state": None, + "action": _legacy_action(choice_index, choices), + "response": _legacy_response(response_payload, choice_index), + } + ) + return rows + + +def _legacy_run_rows(conn: sqlite3.Connection) -> list[dict[str, Any]]: + cursor = conn.execute("PRAGMA table_info(runs)") + columns = {row[1] for row in cursor.fetchall()} + if not columns: + raise ValueError("Source database has no 'runs' table") + if "schema_version" in columns: + raise ValueError("Source database is already schema v2; migration is a one-time legacy conversion.") + + selected = [name for name in LEGACY_RUN_COLUMNS if name in columns] + rows = conn.execute(f"SELECT {', '.join(selected)} FROM runs ORDER BY id").fetchall() + return [dict(zip(selected, row, strict=True)) for row in rows] + + +def migrate_legacy_sqlite(source: Path, output: Path) -> MigrationReport: + """Convert a legacy metrics database into a fresh v2 database.""" + report = MigrationReport(source=str(source), output=str(output), kind="sqlite") + if output.exists(): + raise ValueError(f"Migration output already exists: {output}") + output.parent.mkdir(parents=True, exist_ok=True) + + src = sqlite3.connect(source) + dst = sqlite3.connect(output) + try: + ensure_v2_schema(dst) + for run in _legacy_run_rows(src): + rows = _rows_from_legacy_sqlite(src, run.get("id")) + record = _build_record( + run_id=run.get("id"), + quest_file=str(run.get("quest_file") or ""), + quest_name=str(run.get("quest_name") or ""), + start_time=run.get("start_time"), + end_time=run.get("end_time"), + run_duration=run.get("run_duration"), + agent_id=str(run.get("agent_id") or ""), + agent_config=_safe_json_load(run.get("agent_config")), + outcome=run.get("outcome"), + reward=run.get("reward"), + benchmark_id=run.get("benchmark_id"), + rows=rows, + final_state=None, + ) + _insert_record(dst, record) + report.runs_migrated += 1 + report.transitions_migrated += len(record.transitions) + if record.is_resumable: + report.resumable_runs += 1 + dst.commit() + finally: + src.close() + dst.close() + return report + + +def _insert_record(conn: sqlite3.Connection, record: RunRecord) -> None: + cursor = conn.execute( + """ + INSERT INTO runs ( + schema_version, quest_file, quest_name, quest_checksum, quest_language, + engine_revision, agent_id, treatment, treatment_signature, benchmark_id, + lineage, start_time, end_time, run_duration, outcome, reward, + usage, transcript_diagnostics, progress, terminal_snapshot + ) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + record.schema_version, + record.quest_file, + record.quest_name, + record.quest_checksum, + record.quest_language, + record.engine_revision, + record.agent_id, + json.dumps(record.treatment, ensure_ascii=False), + record.treatment_signature, + record.benchmark_id, + None, + record.started_at, + record.ended_at, + record.run_duration, + record.outcome, + record.reward, + json.dumps(record.usage, ensure_ascii=False), + json.dumps(record.transcript_diagnostics, ensure_ascii=False), + json.dumps(record.progress.to_dict(), ensure_ascii=False), + json.dumps(record.terminal_snapshot.to_dict(), ensure_ascii=False) if record.terminal_snapshot else None, + ), + ) + new_run_id = cursor.lastrowid + for transition in record.transitions: + insert_transition_row(conn, new_run_id, transition) + + +# ---- entry point ----------------------------------------------------------- + + +def migrate_json_tree(source: Path, output: Path) -> MigrationReport: + """Convert one legacy run summary, or a tree of them, into v2 JSON.""" + report = MigrationReport(source=str(source), output=str(output), kind="json") + + if source.is_file(): + payloads = [(source, source.name)] + output_root = output if output.suffix == ".json" else output / source.name + single_file = True + else: + payloads = [(path, str(path.relative_to(source))) for path in sorted(source.rglob("run_summary.json"))] + output_root = output + single_file = False + + if not payloads: + raise ValueError(f"No legacy run_summary.json files found under {source}") + + for path, relative in payloads: + with open(path, encoding="utf-8") as f: + payload = json.load(f) + try: + record = migrate_legacy_json(payload) + except ValueError as exc: + report.skipped.append(f"{path}: {exc}") + continue + + destination = output_root if single_file else output_root / relative + destination.parent.mkdir(parents=True, exist_ok=True) + with open(destination, "w", encoding="utf-8") as f: + json.dump(record.to_dict(), f, indent=2, ensure_ascii=False) + + report.runs_migrated += 1 + report.transitions_migrated += len(record.transitions) + if record.is_resumable: + report.resumable_runs += 1 + + return report + + +def migrate_records(source: str | Path, output: str | Path) -> MigrationReport: + """Migrate a legacy JSON tree or SQLite database into a v2 destination.""" + source_path = Path(source) + output_path = Path(output) + if not source_path.exists(): + raise FileNotFoundError(f"Migration source not found: {source_path}") + + if source_path.is_dir() or source_path.suffix == ".json": + return migrate_json_tree(source_path, output_path) + return migrate_legacy_sqlite(source_path, output_path) diff --git a/llm_quest_benchmark/core/progress.py b/llm_quest_benchmark/core/progress.py new file mode 100644 index 0000000..054ab0a --- /dev/null +++ b/llm_quest_benchmark/core/progress.py @@ -0,0 +1,283 @@ +"""State-based progress manifests and the runtime progress tracker. + +Progress is a diagnostic that is deliberately separate from the authoritative +terminal outcome. Without a curated manifest a run reports terminal-only +progress rather than guessing story advancement. +""" + +from __future__ import annotations + +import functools +import hashlib +import re +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +import yaml + +from llm_quest_benchmark.schemas.records import ProgressState, QuestSnapshot, canonical_json + +MANIFEST_VERSION = 1 +_PREDICATE_KEYS = { + "location_id", + "game_state", + "done", + "params_contains", + "observation_contains", + "params_pattern", +} + + +@dataclass(frozen=True) +class Milestone: + """One state predicate and the progress percentage it awards.""" + + id: str + percent: float + match: dict[str, Any] + description: str = "" + + @staticmethod + @functools.cache + def _compiled_pattern(pattern: str): + return re.compile(pattern) + + def matches(self, snapshot: QuestSnapshot) -> bool: + """All declared predicates must hold for the milestone to be reached.""" + params_text = "\n".join(snapshot.params_state) + + location_ids = self.match.get("location_id") + if location_ids is not None and str(snapshot.location_id) not in location_ids: + return False + + game_states = self.match.get("game_state") + if game_states is not None and snapshot.game_state not in game_states: + return False + + done = self.match.get("done") + if done is not None and bool(snapshot.done) is not bool(done): + return False + + for needle in self.match.get("params_contains") or []: + if needle not in params_text: + return False + + for needle in self.match.get("observation_contains") or []: + if needle not in (snapshot.observation or ""): + return False + + pattern_rule = self.match.get("params_pattern") + if pattern_rule is not None: + regex = Milestone._compiled_pattern(pattern_rule["pattern"]) + hits = sum(1 for line in snapshot.params_state if regex.search(line)) + if hits < int(pattern_rule.get("min_count", 1)): + return False + + return True + + +@dataclass +class ProgressManifest: + """Validated set of quest milestones loaded from YAML.""" + + quest: str + milestones: list[Milestone] + source: str = "" + version: int = MANIFEST_VERSION + _hash_cache: str | None = field(default=None, init=False, repr=False, compare=False) + + @property + def maximum(self) -> float: + """Progress scale ceiling; terminal success always reports 100.""" + return 100.0 + + @property + def hash(self) -> str: + """Digest of the milestone definitions that award progress. + + Recorded with every progress state so a run scored under a since-edited + manifest is detectable instead of silently comparable. + """ + if self._hash_cache is None: + payload = [{"id": m.id, "percent": m.percent, "match": m.match} for m in self.milestones] + self._hash_cache = hashlib.sha256( + canonical_json({"quest": self.quest, "milestones": payload}).encode("utf-8") + ).hexdigest()[:16] + return self._hash_cache + + @classmethod + def from_file(cls, path: str | Path) -> ProgressManifest: + manifest_path = Path(path) + if not manifest_path.exists(): + raise FileNotFoundError(f"Progress manifest not found: {manifest_path}") + with open(manifest_path, encoding="utf-8") as f: + payload = yaml.safe_load(f) + return cls.from_dict(payload, source=str(manifest_path)) + + @classmethod + def from_dict(cls, payload: Any, source: str = "") -> ProgressManifest: + if not isinstance(payload, dict): + raise ValueError(f"Progress manifest must be a mapping: {source or ''}") + + version = payload.get("version", MANIFEST_VERSION) + if version != MANIFEST_VERSION: + raise ValueError(f"Unsupported progress manifest version {version!r} in {source or ''}") + + quest = str(payload.get("quest") or "").strip() + if not quest: + raise ValueError(f"Progress manifest is missing 'quest': {source or ''}") + + raw_milestones = payload.get("milestones") + if not isinstance(raw_milestones, list) or not raw_milestones: + raise ValueError(f"Progress manifest needs a non-empty 'milestones' list: {source or ''}") + + milestones: list[Milestone] = [] + seen_ids: set[str] = set() + previous_percent = -1.0 + for entry in raw_milestones: + if not isinstance(entry, dict): + raise ValueError(f"Milestone entries must be mappings: {source or ''}") + milestone_id = str(entry.get("id") or "").strip() + if not milestone_id: + raise ValueError(f"Milestone is missing 'id': {source or ''}") + if milestone_id in seen_ids: + raise ValueError(f"Duplicate milestone id '{milestone_id}' in {source or ''}") + seen_ids.add(milestone_id) + + if "percent" not in entry: + raise ValueError(f"Milestone '{milestone_id}' is missing 'percent'") + percent = float(entry["percent"]) + if not (0.0 <= percent <= 100.0): + raise ValueError(f"Milestone '{milestone_id}' percent must be within 0..100, got {percent}") + if percent < previous_percent: + raise ValueError(f"Milestone '{milestone_id}' percent decreases; list milestones in ascending order") + previous_percent = percent + + match = entry.get("match") + if not isinstance(match, dict) or not match: + raise ValueError(f"Milestone '{milestone_id}' needs a non-empty 'match' mapping") + unknown = set(match) - _PREDICATE_KEYS + if unknown: + raise ValueError( + f"Milestone '{milestone_id}' has unknown predicates: {sorted(unknown)}. " + f"Supported: {sorted(_PREDICATE_KEYS)}" + ) + _validate_predicates(milestone_id, match) + + milestones.append( + Milestone( + id=milestone_id, + percent=percent, + match=match, + description=str(entry.get("description") or ""), + ) + ) + + return cls(quest=quest, milestones=milestones, source=source, version=MANIFEST_VERSION) + + +def _validate_predicates(milestone_id: str, match: dict[str, Any]) -> None: + for key in ("location_id", "game_state", "params_contains", "observation_contains"): + value = match.get(key) + if value is None: + continue + if not isinstance(value, list) or not all(isinstance(item, str) for item in value): + raise ValueError(f"Milestone '{milestone_id}' predicate '{key}' must be a list of strings") + + if "done" in match and not isinstance(match["done"], bool): + raise ValueError(f"Milestone '{milestone_id}' predicate 'done' must be a boolean") + + pattern_rule = match.get("params_pattern") + if pattern_rule is not None: + if not isinstance(pattern_rule, dict) or "pattern" not in pattern_rule: + raise ValueError(f"Milestone '{milestone_id}' predicate 'params_pattern' needs a 'pattern' key") + try: + re.compile(str(pattern_rule["pattern"])) + except re.error as exc: + raise ValueError(f"Milestone '{milestone_id}' has an invalid params_pattern regex: {exc}") from exc + min_count = pattern_rule.get("min_count", 1) + if not isinstance(min_count, int) or min_count < 1: + raise ValueError(f"Milestone '{milestone_id}' params_pattern min_count must be a positive integer") + + +@dataclass +class ProgressTracker: + """Monotonic progress accumulator over observed snapshots. + + Reached milestones only accumulate, so ``current`` never decreases even when + the backtracking harness restores an earlier checkpoint. + """ + + manifest: ProgressManifest | None = None + _reached: list[str] = field(default_factory=list, init=False) + _current: float = field(default=0.0, init=False) + _stalled: int = field(default=0, init=False) + + @property + def manifest_id(self) -> str | None: + return self.manifest.source if self.manifest else None + + @property + def manifest_hash(self) -> str | None: + return self.manifest.hash if self.manifest else None + + @property + def maximum(self) -> float: + return self.manifest.maximum if self.manifest else 100.0 + + def state(self, newly_reached: list[str] | None = None) -> ProgressState: + return ProgressState( + current=self._current, + maximum=self.maximum, + # Without a manifest the run is unscored: only terminal success moves + # the number, so 0.0 means "not scored", not "no advancement". + scored=self.manifest is not None, + reached=list(self._reached), + newly_reached=list(newly_reached or []), + stalled_transitions=self._stalled, + manifest=self.manifest_id, + manifest_hash=self.manifest_hash, + ) + + def observe(self, snapshot: QuestSnapshot) -> ProgressState: + """Fold one snapshot into progress and return the resulting state.""" + newly_reached: list[str] = [] + if self.manifest: + for milestone in self.manifest.milestones: + if milestone.id in self._reached: + continue + if milestone.matches(snapshot): + self._reached.append(milestone.id) + newly_reached.append(milestone.id) + self._current = max(self._current, milestone.percent) + + # Terminal success is always full progress, with or without a manifest. + if snapshot.game_state == "win": + self._current = self.maximum + + if newly_reached: + self._stalled = 0 + else: + self._stalled += 1 + + return self.state(newly_reached) + + def seed(self, snapshot: QuestSnapshot) -> ProgressState: + """Fold the starting snapshot in without counting it as a stalled step.""" + state = self.observe(snapshot) + self._stalled = 0 + return self.state(state.newly_reached) + + def restore_from(self, state: ProgressState) -> None: + """Seed the tracker from a persisted state when resuming a run.""" + self._reached = list(state.reached) + self._current = float(state.current) + self._stalled = int(state.stalled_transitions) + + +def load_progress_manifest(path: str | None) -> ProgressManifest | None: + """Load an optional manifest path from configuration or a resume record.""" + if not path: + return None + return ProgressManifest.from_file(path) diff --git a/llm_quest_benchmark/core/provenance.py b/llm_quest_benchmark/core/provenance.py new file mode 100644 index 0000000..5f0b0d8 --- /dev/null +++ b/llm_quest_benchmark/core/provenance.py @@ -0,0 +1,51 @@ +"""Quest and engine provenance used to gate replay and resume.""" + +from __future__ import annotations + +import hashlib +import subprocess +from functools import lru_cache +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[2] +ENGINE_ROOT = REPO_ROOT / "space-rangers-quest" +ENGINE_SOURCES = ( + ENGINE_ROOT / "src/lib/qmreader.ts", + ENGINE_ROOT / "src/lib/qmplayer/index.ts", + ENGINE_ROOT / "src/lib/qmplayer/funcs.ts", +) + + +def quest_checksum(quest_file: str | Path) -> str: + """SHA-256 of the quest file bytes, used to detect quest mutation.""" + path = Path(quest_file) + digest = hashlib.sha256(path.read_bytes()).hexdigest() + return f"sha256:{digest}" + + +@lru_cache(maxsize=1) +def engine_revision() -> str: + """Identify the TypeScript engine build backing this run. + + Prefers the submodule commit; falls back to a digest of the engine sources + so a dirty or detached checkout still produces a stable identifier. + """ + try: + proc = subprocess.run( + ["git", "-C", str(ENGINE_ROOT), "rev-parse", "HEAD"], + capture_output=True, + text=True, + check=True, + timeout=10, + ) + revision = proc.stdout.strip() + if revision: + return f"git:{revision}" + except (subprocess.SubprocessError, OSError): + pass + + hasher = hashlib.sha256() + for source in ENGINE_SOURCES: + if source.exists(): + hasher.update(source.read_bytes()) + return f"src:{hasher.hexdigest()[:16]}" diff --git a/llm_quest_benchmark/core/replay.py b/llm_quest_benchmark/core/replay.py new file mode 100644 index 0000000..734a609 --- /dev/null +++ b/llm_quest_benchmark/core/replay.py @@ -0,0 +1,173 @@ +"""Deterministic replay and resume verification for schema-v2 run records. + +Replay re-executes recorded choose timestamps and restore actions against the +real engine and compares every resulting snapshot digest. Resume runs this +verification before any new model inference. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path + +from llm_quest_benchmark.core.progress import ProgressTracker, load_progress_manifest +from llm_quest_benchmark.core.provenance import engine_revision, quest_checksum +from llm_quest_benchmark.environments.qm import QMPlayerEnv +from llm_quest_benchmark.schemas.records import QuestSnapshot, RunRecord + + +class ReplayError(RuntimeError): + """Raised when a record cannot be reproduced against the live engine.""" + + +@dataclass +class ReplayResult: + """Outcome of replaying a full record.""" + + snapshot: QuestSnapshot + checkpoints: list[QuestSnapshot] = field(default_factory=list) + verified_transitions: int = 0 + + +def verify_environment(record: RunRecord, quest_file: str) -> None: + """Verify quest identity and engine revision before replaying.""" + actual_checksum = quest_checksum(quest_file) + if record.quest_checksum and record.quest_checksum != actual_checksum: + raise ReplayError( + f"Quest checksum mismatch for {quest_file}: recorded {record.quest_checksum}, found {actual_checksum}" + ) + + actual_engine = engine_revision() + if record.engine_revision and record.engine_revision != actual_engine: + raise ReplayError(f"Engine revision mismatch: recorded {record.engine_revision}, found {actual_engine}") + + +def _seed_from_record(env: QMPlayerEnv, record: RunRecord) -> QuestSnapshot: + """Put the engine into the exact state the record starts from. + + The engine seeds its PRNG per process (``initGame`` takes a random seed), so + a fresh ``reset`` produces a different ``aleaSeed``/``aleaState`` than the + recorded run even though the quest text is identical. Replay therefore boots + the process and then loads the recorded opening saving, which is the + authoritative engine state: ``restore`` re-derives the observation, choices, + and parameter state from it and fails if they do not reproduce the recorded + digest. + """ + snapshot = env.reset() + if not record.transitions: + return snapshot + + first_transition = record.transitions[0] + first = first_transition.before + if not first.is_resumable: + raise ReplayError( + f"Transition {first_transition.index} is not replayable: its before-state carries no " + f"engine saving (provenance={first_transition.provenance})" + ) + try: + return env.restore(first) + except Exception as exc: # engine refused the saving, or the state diverged + raise ReplayError(f"Recorded opening state could not be reproduced: {exc}") from exc + + +def replay_record(env: QMPlayerEnv, record: RunRecord) -> ReplayResult: + """Replay every recorded transition, verifying each resulting digest. + + The environment must not have been reset yet; this function drives it from + the recorded opening state through the whole recorded trajectory. + """ + snapshot = _seed_from_record(env, record) + checkpoints: list[QuestSnapshot] = [snapshot] + + for transition in record.transitions: + if not transition.is_replayable: + raise ReplayError( + f"Transition {transition.index} is not replayable " + f"(provenance={transition.provenance}, replay_status={transition.replay_status})" + ) + if transition.before.digest != snapshot.digest: + raise ReplayError( + f"Transition {transition.index} before-state diverged ({snapshot.digest} != {transition.before.digest})" + ) + + action = transition.action + if action.is_choose: + index = int(action.choice_index or 0) + if not (1 <= index <= len(snapshot.choices)): + raise ReplayError(f"Transition {transition.index} choice index {index} is out of range") + recorded_id = str(action.choice_id) + live_id = str(snapshot.choices[index - 1]["id"]) + if recorded_id != live_id: + raise ReplayError( + f"Transition {transition.index} choice id diverged (recorded {recorded_id}, found {live_id})" + ) + snapshot = env.step(index, int(action.performed_at_ms)) + checkpoints.append(snapshot) + else: + checkpoint_index = int(action.checkpoint_index or 0) + if not (1 <= checkpoint_index <= len(checkpoints)): + raise ReplayError( + f"Transition {transition.index} restores checkpoint {checkpoint_index}, " + f"but only {len(checkpoints)} are on the active branch" + ) + snapshot = env.restore(checkpoints[checkpoint_index - 1]) + checkpoints = checkpoints[:checkpoint_index] + + if snapshot.digest != transition.after.digest: + raise ReplayError( + f"Transition {transition.index} after-state diverged ({snapshot.digest} != {transition.after.digest})" + ) + + return ReplayResult(snapshot=snapshot, checkpoints=checkpoints, verified_transitions=len(record.transitions)) + + +def restore_progress_tracker(record: RunRecord) -> ProgressTracker: + """Rebuild the progress tracker, including its manifest, from a record. + + A manifest that no longer hashes to what the run was scored under would make + the resumed progress incomparable with the recorded part, so it is rejected + rather than silently re-scored. + """ + manifest_path = record.progress.manifest + manifest = None + if manifest_path and Path(manifest_path).exists(): + manifest = load_progress_manifest(manifest_path) + + recorded_hash = record.progress.manifest_hash + if manifest is not None and recorded_hash and manifest.hash != recorded_hash: + raise ReplayError( + f"Progress manifest {manifest_path} changed since the run was recorded " + f"(recorded {recorded_hash}, found {manifest.hash})" + ) + + if manifest is None and record.progress.scored and record.progress.maximum: + raise ReplayError( + f"Run was scored under progress manifest {manifest_path}, but the file is missing; " + "resuming would silently downgrade progress to unscored." + ) + + tracker = ProgressTracker(manifest=manifest) + tracker.restore_from(record.progress) + return tracker + + +def harness_config_from_record(record: RunRecord): + """Rebuild the harness configuration recorded in a run's treatment.""" + from llm_quest_benchmark.schemas.config import HarnessConfig + + treatment = record.treatment or {} + knobs = treatment.get("knobs") or {} + harness = str(treatment.get("harness") or "") + if not harness or harness == "unknown": + raise ReplayError("Run record has no resolvable harness treatment; it cannot be resumed.") + + return HarnessConfig( + model=str(treatment.get("model") or ""), + system_template=str(treatment.get("system_prompt") or "system_role.jinja"), + harness=harness, + temperature=float(treatment.get("temperature") or 0.0), + benchmark_id=record.benchmark_id, + compaction_interval=int(knobs.get("compaction_interval", 50)), + restore_limit=knobs.get("restore_limit"), + adaptive_stall_steps=knobs.get("adaptive_stall_steps"), + ) diff --git a/llm_quest_benchmark/core/runner.py b/llm_quest_benchmark/core/runner.py index 67f56e8..425b436 100644 --- a/llm_quest_benchmark/core/runner.py +++ b/llm_quest_benchmark/core/runner.py @@ -1,27 +1,66 @@ -"""Quest runner implementation with improved logging and error handling""" +"""Quest runner: canonical transitions, checkpoints, progress, replay and resume.""" -import json import logging -import sqlite3 import threading from collections.abc import Callable from concurrent.futures import ThreadPoolExecutor from concurrent.futures import TimeoutError as FuturesTimeoutError -from copy import deepcopy +from pathlib import Path from typing import Any from llm_quest_benchmark.constants import DEFAULT_QUEST_TIMEOUT from llm_quest_benchmark.core.logging import LogManager, QuestLogger +from llm_quest_benchmark.core.progress import ProgressTracker, load_progress_manifest +from llm_quest_benchmark.core.provenance import engine_revision, quest_checksum +from llm_quest_benchmark.core.replay import ReplayError, replay_record, restore_progress_tracker, verify_environment from llm_quest_benchmark.environments.qm import QMPlayerEnv as QuestEnvironment from llm_quest_benchmark.environments.state import QuestOutcome -from llm_quest_benchmark.players.base import QuestPlayer +from llm_quest_benchmark.harnesses.specs import build_treatment, get_spec +from llm_quest_benchmark.players.base import DecisionContext, QuestPlayer from llm_quest_benchmark.schemas.config import HarnessConfig -from llm_quest_benchmark.schemas.state import AgentState +from llm_quest_benchmark.schemas.records import ( + PROVENANCE_RUNTIME, + REPLAY_PENDING, + ProgressState, + QuestAction, + QuestSnapshot, + QuestTransition, + ResumeLineage, + RunRecord, +) # Configure logging logging.getLogger("quest").setLevel(logging.WARNING) +def _agent_identity(agent: QuestPlayer, agent_config: Any | None) -> str: + """Resolve the run identity before execution starts.""" + if agent_config is not None and getattr(agent_config, "agent_id", None): + return str(agent_config.agent_id) + return str(getattr(agent, "agent_id", None) or str(agent)) + + +def _treatment_payload(agent: QuestPlayer, agent_config: Any | None) -> dict[str, Any]: + """Build the canonical treatment recorded before the first transition.""" + if agent_config is not None: + return agent_config.treatment().to_dict() + + # No config supplied: resolve the treatment from the player itself. + harness = getattr(agent, "harness_name", "") or "human" + spec = get_spec(harness) + return build_treatment( + harness=harness, + model=str(getattr(agent, "model_name", "") or spec.name), + temperature=float(getattr(agent, "temperature", 0.0) or 0.0), + system_template=str(getattr(agent, "system_template", "none")), + knob_values={ + "compaction_interval": getattr(agent, "_compaction_interval", None), + "restore_limit": getattr(agent, "restore_limit", None), + "adaptive_stall_steps": getattr(agent, "adaptive_stall_steps", None), + }, + ).to_dict() + + def run_quest_with_timeout( quest_path: str, agent: QuestPlayer, @@ -30,26 +69,22 @@ def run_quest_with_timeout( debug: bool = False, callbacks: list[Callable[[str, Any], None]] = None, max_steps: int | None = None, + progress_manifest: str | None = None, + resume_record: RunRecord | None = None, + resume_path: str | None = None, ) -> QuestOutcome | None: """Run quest with timeout. max_steps: optional shared cap on agent steps per quest. None (default) - preserves the prior unbounded-loop behavior. + preserves the prior unbounded-loop behavior; reaching the cap yields the + resumable TRUNCATED outcome. """ logger: QuestLogger | None = None executor: ThreadPoolExecutor | None = None - benchmark_id = getattr(agent_config, "benchmark_id", None) if agent_config else None + runner: QuestRunner | None = None try: - # Get agent_id from agent itself if available, or from config - agent_id = getattr(agent, "agent_id", None) - if agent_id is None and agent_config: - agent_id = agent_config.agent_id - - # Initialize logger with agent_id - logger = QuestLogger(debug=debug, agent=agent_id) + logger = QuestLogger(debug=debug, agent=_agent_identity(agent, agent_config)) - # Create quest environment and runner - QuestEnvironment(quest_path) # validates quest path runner = QuestRunner( agent=agent, debug=debug, @@ -57,64 +92,28 @@ def run_quest_with_timeout( quest_logger=logger, agent_config=agent_config, max_steps=max_steps, + progress_manifest=progress_manifest, + resume_record=resume_record, + resume_path=resume_path, ) # Run quest with timeout executor = ThreadPoolExecutor(max_workers=1) future = executor.submit(runner.run, quest_path) try: - outcome = future.result(timeout=max(timeout, 1)) - - # Update run with agent_id and config if provided - if agent_config and agent_id: - try: - # Ensure we have a connection for this thread - logger._init_connection() - - # Store agent config as JSON for better storage/retrieval - agent_config_json = json.dumps(agent_config.__dict__) - - # Also store benchmark_id if provided - if benchmark_id: - # Include benchmark_id in the update if available - logger._local.cursor.execute( - """ - UPDATE runs - SET agent_id = ?, agent_config = ?, benchmark_id = ? - WHERE id = ? - """, - (agent_id, agent_config_json, benchmark_id, logger.current_run_id), - ) - logger.logger.info(f"Updated run {logger.current_run_id} with benchmark_id {benchmark_id}") - else: - logger._local.cursor.execute( - """ - UPDATE runs - SET agent_id = ?, agent_config = ? - WHERE id = ? - """, - (agent_id, agent_config_json, logger.current_run_id), - ) - logger._local.conn.commit() - except sqlite3.OperationalError as e: - if "no such column: agent_id" in str(e): - logger.logger.warning("agent_id column not found in database, skipping update") - else: - raise - - # The outcome is already recorded in the QuestRunner - return outcome + return future.result(timeout=max(timeout, 1)) except FuturesTimeoutError: future.cancel() runner.request_stop("timeout") logger.logger.warning(f"Quest timed out after {timeout} seconds") # Persist timeout as authoritative outcome. - logger.set_quest_outcome( + logger.finish_run( QuestOutcome.TIMEOUT.name, 0.0, - final_state=runner.snapshot_state(), - benchmark_id=benchmark_id, + terminal_snapshot=runner.current_snapshot(), + progress=runner.progress_state(), + diagnostics=runner.runtime_metrics(), ) # Notify callbacks about the timeout @@ -130,8 +129,13 @@ def run_quest_with_timeout( except Exception as e: if logger: logger.logger.error(f"Error running quest: {e}") - if logger: - logger.set_quest_outcome("ERROR", 0.0, benchmark_id=benchmark_id) + logger.finish_run( + QuestOutcome.ERROR.name, + 0.0, + terminal_snapshot=runner.current_snapshot() if runner else None, + progress=runner.progress_state() if runner else None, + diagnostics=runner.runtime_metrics() if runner else None, + ) raise finally: if executor: @@ -139,7 +143,11 @@ def run_quest_with_timeout( class QuestRunner: - """Manages quest execution with logging and metrics""" + """Executes a quest as a sequence of canonical transitions. + + The runner owns the active-branch checkpoint stack, progress tracking, + replay verification on resume, and transition construction. + """ def __init__( self, @@ -149,22 +157,40 @@ def __init__( quest_logger: QuestLogger = None, agent_config=None, max_steps: int | None = None, + progress_manifest: str | None = None, + resume_record: RunRecord | None = None, + resume_path: str | None = None, ): - """Initialize components needed for quest execution. - - max_steps: optional shared cap on agent steps per quest. None (default) - preserves the prior unbounded-loop behavior. - """ self.agent = agent self.debug = debug self.callbacks = callbacks or [] self.step_count = 0 - self.env = None + self.env: QuestEnvironment | None = None self.agent_config = agent_config self.max_steps = max_steps + self.resume_record = resume_record + self.resume_path = resume_path self._stop_requested = threading.Event() self._stop_reason = "" + self._transition_index = 0 + self._checkpoints: list[QuestSnapshot] = [] + self._snapshot: QuestSnapshot | None = None + self._quest_checksum = "" + self._restore_attempts = 0 + self._restores_accepted = 0 + self._restored_distance = 0 + self._progress_at_last_restore: float | None = None + + if resume_record is not None: + # A resumed run keeps the manifest and reached milestones it recorded, + # so progress stays monotonic across the interruption. + self.progress = restore_progress_tracker(resume_record) + else: + self.progress = ProgressTracker(manifest=load_progress_manifest(progress_manifest)) + + self.restore_limit = getattr(agent_config, "restore_limit", None) if agent_config else None + # Set up central logging log_manager = LogManager() log_manager.setup(debug=debug) @@ -173,24 +199,40 @@ def __init__( # Use provided quest logger or create a new one self.quest_logger = quest_logger if self.quest_logger is None: - self.quest_logger = QuestLogger(debug=self.debug, agent=str(self.agent)) + self.quest_logger = QuestLogger(debug=self.debug, agent=_agent_identity(agent, agent_config)) if debug: self.logger.debug(f"QuestRunner initialized with agent: {str(agent)}") + # ---- runtime accessors ------------------------------------------------- + def request_stop(self, reason: str = "requested") -> None: """Signal runner loop to stop as soon as possible.""" self._stop_reason = reason self._stop_requested.set() - def snapshot_state(self) -> dict[str, Any] | None: - """Best-effort snapshot of current environment state.""" - if not self.env: - return None - state = self.env.state if getattr(self.env, "state", None) else None - if isinstance(state, dict): - return deepcopy(state) - return None + def current_snapshot(self) -> QuestSnapshot | None: + """Latest canonical snapshot, usable as a final state on interruption.""" + return self._snapshot + + def progress_state(self) -> ProgressState: + return self.progress.state() + + def runtime_metrics(self) -> dict[str, Any]: + """Runner-owned metrics merged into the persisted run record.""" + progress_recovered = 0.0 + if self._progress_at_last_restore is not None: + progress_recovered = max(0.0, self.progress.state().current - self._progress_at_last_restore) + return { + "restore": { + "attempts": self._restore_attempts, + "accepted": self._restores_accepted, + "restored_distance": self._restored_distance, + # Progress gained after the last accepted restore. + "progress_recovered": round(progress_recovered, 4), + }, + "forced_stop_reason": getattr(self.env, "forced_stop_reason", None) if self.env else None, + } def _notify_callbacks(self, event: str, data: Any = None) -> None: """Notify all callbacks of an event""" @@ -200,26 +242,200 @@ def _notify_callbacks(self, event: str, data: Any = None) -> None: except Exception as e: self.logger.error(f"Error in callback: {e}") + # ---- setup ------------------------------------------------------------- + def initialize(self, quest: str) -> None: - """Initialize environment and logger for a new quest""" + """Initialize environment and write complete run metadata.""" try: if self.debug: self.logger.debug("Initializing environment for quest: %s", quest) - self.env = QuestEnvironment(quest, debug=self.debug) - self.quest_logger.set_quest_file(quest) + language = str(self.resume_record.quest_language) if self.resume_record else "rus" + self.env = QuestEnvironment(quest, language=language, debug=self.debug) + + lineage = None + if self.resume_record is not None: + lineage = ResumeLineage( + source_run_id=self.resume_record.run_id, + source_path=str(self.resume_path or ""), + resumed_from_index=len(self.resume_record.transitions), + ) + + self._quest_checksum = quest_checksum(self.env.quest_file) + self.quest_logger.start_run( + quest_file=self.env.quest_file, + quest_name=Path(self.env.quest_file).stem, + quest_checksum=self._quest_checksum, + quest_language=self.env.language, + engine_revision=engine_revision(), + agent_id=_agent_identity(self.agent, self.agent_config), + treatment=_treatment_payload(self.agent, self.agent_config), + benchmark_id=getattr(self.agent_config, "benchmark_id", None) if self.agent_config else None, + lineage=lineage, + ) self.logger.info(f"Running quest {quest} with agent: {str(self.agent)}") except Exception as e: self.logger.error("Failed to initialize environment: %s", str(e), exc_info=True) raise + def _start_environment(self) -> QuestSnapshot: + """Reset or resume the environment and seed checkpoints and progress.""" + if self.resume_record is None: + snapshot = self.env.reset() + self._checkpoints = [snapshot] + self.progress.seed(snapshot) + return snapshot + + record = self.resume_record + verify_environment(record, self.env.quest_file) + self._verify_resume_treatment(record) + + result = replay_record(self.env, record) + self.logger.info("Verified %s recorded transitions before resuming", result.verified_transitions) + + self._checkpoints = result.checkpoints + self._transition_index = max((t.index for t in record.transitions), default=0) + self.step_count = sum(1 for t in record.transitions if t.action.is_choose) + restore_metrics = (record.transcript_diagnostics or {}).get("restore") or {} + self._restore_attempts = int(restore_metrics.get("attempts") or 0) + self._restores_accepted = int(restore_metrics.get("accepted") or 0) + self._restored_distance = int(restore_metrics.get("restored_distance") or 0) + + # Prior transitions stay part of the resumed run's record. + self.quest_logger.adopt_transitions(record.transitions) + self.agent.rebuild_from_transitions(record.transitions) + return result.snapshot + + def _verify_resume_treatment(self, record: RunRecord) -> None: + """Refuse to resume under a different treatment than was recorded.""" + current = _treatment_payload(self.agent, self.agent_config) + recorded_signature = record.treatment_signature + if recorded_signature and current.get("signature") != recorded_signature: + raise ReplayError( + f"Treatment signature mismatch: recorded {recorded_signature}, current {current.get('signature')}" + ) + + # ---- decision execution ------------------------------------------------ + + def _decision_context(self) -> DecisionContext: + remaining = None + if self.restore_limit is not None: + remaining = max(0, int(self.restore_limit) - self._restores_accepted) + restore_allowed = bool( + getattr(self.agent, "supports_restore", False) + and len(self._checkpoints) > 1 + and (remaining is None or remaining > 0) + ) + return DecisionContext( + step=self.step_count + 1, + checkpoints=list(self._checkpoints), + restore_allowed=restore_allowed, + restores_remaining=remaining, + progress=self.progress.state(), + ) + + def _resolve_action(self, action: QuestAction, context: DecisionContext, snapshot: QuestSnapshot) -> QuestAction: + """Validate the agent's action and clamp it into an executable one.""" + if action.is_restore: + self._restore_attempts += 1 + index = int(action.checkpoint_index or 0) + valid = ( + getattr(self.agent, "supports_restore", False) + and context.restore_allowed + and 1 <= index < len(self._checkpoints) + ) + if valid: + return QuestAction.restore(index) + self.logger.warning( + "Rejecting restore to checkpoint %s (allowed=%s, checkpoints=%s); choosing instead", + index, + context.restore_allowed, + len(self._checkpoints), + ) + response = self.agent.get_last_response() + fallback = int(getattr(response, "action", 1) or 1) + action = QuestAction(kind="choose", choice_index=fallback) + + index = int(action.choice_index or 1) + num_choices = len(snapshot.choices) + if index < 1 or index > num_choices: + self.logger.error("RUNNER ERROR - Action %s out of range 1-%s; defaulting to 1", index, num_choices) + index = 1 + checksum_hex = self._quest_checksum.removeprefix("sha256:") + timestamp_base = 1_700_000_000_000 + (int(checksum_hex[:10], 16) % 100_000_000_000) + return QuestAction.choose( + choice_index=index, + choice_id=str(snapshot.choices[index - 1]["id"]), + performed_at_ms=timestamp_base + self._transition_index + 1, + ) + + def _execute(self, action: QuestAction) -> QuestSnapshot: + if action.is_restore: + # A restore truncates the active branch to the target checkpoint, so + # each one strictly shortens the stack. Restores therefore cannot + # loop: once the stack is one deep the runner offers no restore + # until a choose extends the branch again. + checkpoint_index = int(action.checkpoint_index) + target = self._checkpoints[checkpoint_index - 1] + after = self.env.restore(target) + self._restored_distance += max(0, len(self._checkpoints) - checkpoint_index) + self._checkpoints = self._checkpoints[:checkpoint_index] + self._restores_accepted += 1 + self._progress_at_last_restore = self.progress.state().current + return after + + after = self.env.step(int(action.choice_index), int(action.performed_at_ms)) + self._checkpoints.append(after) + self.step_count += 1 + return after + + def _record_transition( + self, + before: QuestSnapshot, + action: QuestAction, + after: QuestSnapshot, + ) -> QuestTransition: + response = self.agent.get_last_response() + usage = response.usage_payload() if response is not None else {} + + self._transition_index += 1 + transition = QuestTransition( + index=self._transition_index, + before=before, + action=action, + after=after, + response=response, + usage=usage, + progress=self.progress.observe(after), + provenance=PROVENANCE_RUNTIME, + replay_status=REPLAY_PENDING, + reasoning_mode=getattr(self.agent, "reasoning_mode", None), + ) + + self.agent.on_transition(transition) + self._notify_callbacks("game_state", transition) + self.quest_logger.log_transition(transition) + return transition + + def _finish(self, outcome: QuestOutcome, snapshot: QuestSnapshot | None) -> QuestOutcome: + self.agent.on_game_end(snapshot) + self.quest_logger.finish_run( + outcome.name, + reward=snapshot.reward if snapshot else 0.0, + terminal_snapshot=snapshot, + progress=self.progress.state(), + diagnostics=self.runtime_metrics(), + ) + return outcome + + # ---- main loop --------------------------------------------------------- + def run(self, quest: str) -> QuestOutcome: - """Run the quest until completion or error""" + """Run the quest until completion, truncation, or error.""" if not self.agent: self.logger.error("No agent initialized!") return QuestOutcome.ERROR try: - # Initialize environment and notify callbacks self.initialize(quest) self._notify_callbacks( "run_record", @@ -233,188 +449,81 @@ def run(self, quest: str) -> QuestOutcome: self._notify_callbacks("title") self._notify_callbacks("progress", {"step": 0, "message": "Starting quest..."}) - # Get initial state - observation = self.env.reset() + snapshot = self._start_environment() + self._snapshot = snapshot while True: if self._stop_requested.is_set(): self.logger.warning("Quest runner stop requested: %s", self._stop_reason or "unknown") return QuestOutcome.TIMEOUT - if self.max_steps is not None and self.step_count >= self.max_steps: - self.logger.warning( - "Quest exceeded max steps (%s); ending as FAILURE to break a non-terminating loop", - self.max_steps, - ) - if self.env and self.env.state: - self.agent.on_game_end(self.env.state) - - outcome = QuestOutcome.FAILURE - reward = self.env.state.get("reward", 0.0) if self.env and self.env.state else 0.0 - if self.quest_logger: - self.quest_logger.set_quest_outcome( - outcome.name, - reward, - final_state=self.env.state if self.env else None, - ) - - return outcome - - self.step_count += 1 - self._notify_callbacks( - "progress", {"step": self.step_count, "message": f"Processing step {self.step_count}..."} - ) - - # Check if there are any choices available - if not self.env.state["choices"]: - if self.env and self.env.state: - self.agent.on_game_end(self.env.state) + if snapshot.done: + # Terminal state is the after-snapshot of the last executed + # transition, never a synthetic decision row. + outcome = QuestOutcome.SUCCESS if snapshot.game_state == "win" else QuestOutcome.FAILURE + return self._finish(outcome, snapshot) - # Log quest outcome - outcome = QuestOutcome.FAILURE - reward = self.env.state.get("reward", 0.0) if self.env and self.env.state else 0.0 - if self.quest_logger: - self.quest_logger.set_quest_outcome( - outcome.name, - reward, - final_state=self.env.state if self.env else None, - ) + if not snapshot.choices: + self.logger.warning("No choices available at location %s", snapshot.location_id) + return self._finish(QuestOutcome.FAILURE, snapshot) - return outcome + if self.max_steps is not None and self.step_count >= self.max_steps: + self.logger.warning("Quest reached max steps (%s); recording TRUNCATED", self.max_steps) + return self._finish(QuestOutcome.TRUNCATED, snapshot) - current_location_id = self.env.state["location_id"] - current_observation = observation - current_choices = deepcopy(self.env.state["choices"]) + self._notify_callbacks( + "progress", {"step": self.step_count + 1, "message": f"Processing step {self.step_count + 1}..."} + ) - # Get agent's action and take step - action = self.agent.get_action(current_observation, current_choices) + context = self._decision_context() + proposed = self.agent.get_quest_action(snapshot.agent_observation(), snapshot.choices, context) + action = self._resolve_action(proposed, context, snapshot) if self.debug: - self.logger.debug(f"Agent selected action: {action}") - choices_debug = [] - for i, c in enumerate(current_choices): - choices_debug.append(f"{i + 1}: {c['text']}") - self.logger.debug(f"Available choices: {choices_debug}") - - # Validate action is within range (extra safety check) - num_choices = len(current_choices) - if action < 1 or action > num_choices: - self.logger.error(f"RUNNER ERROR - Action {action} out of range 1-{num_choices}") - self.logger.error("Defaulting to action 1") - action = 1 + self.logger.debug("Agent action: %s", action.to_dict()) if self._stop_requested.is_set(): self.logger.info("Quest runner stopped before step after %s", self._stop_reason or "request") return QuestOutcome.TIMEOUT try: - self.logger.debug(f"Taking step with final action: {action}") - observation, done, success, info = self.env.step(action) - - # One canonical executed decision step: consumers receive - # this same object before it is serialized by QuestLogger. - agent_state = AgentState( - step=self.step_count, - location_id=current_location_id, - observation=current_observation, - choices=current_choices, - action=str(action), - llm_response=self.agent.get_last_response(), - ) - self.agent.on_step(agent_state) - self._notify_callbacks("game_state", agent_state) - - if self.quest_logger: - self.quest_logger.log_step(agent_state) - - if done: - self.agent.on_game_end(self.env.state) - - # Log terminal step: the final observation after the last action - outcome = QuestOutcome.SUCCESS if success else QuestOutcome.FAILURE - if self.quest_logger: - terminal_state = AgentState( - step=self.step_count + 1, - location_id=self.env.state.get("location_id", "unknown") - if self.env.state - else "unknown", - observation=observation or f"[Quest ended: {outcome.name}]", - choices=[], - action=outcome.name, - llm_response=None, - ) - self.quest_logger.log_step(terminal_state) - - reward = self.env.state.get("reward", 0.0) if self.env and self.env.state else 0.0 - if self.quest_logger: - # Get benchmark_id from the agent_config parameter if available - benchmark_id = None - if ( - hasattr(self, "agent_config") - and self.agent_config - and hasattr(self.agent_config, "benchmark_id") - ): - benchmark_id = self.agent_config.benchmark_id - - self.quest_logger.set_quest_outcome( - outcome.name, - reward, - benchmark_id, - final_state=self.env.state if self.env else None, - ) - - return outcome - + after = self._execute(action) except Exception as e: if self._stop_requested.is_set(): self.logger.info("Quest runner stopped during step after %s", self._stop_reason or "request") return QuestOutcome.TIMEOUT self.logger.error("Error during step: %s", str(e), exc_info=True) - self._notify_callbacks("error", str(e)) - - # Log error outcome - if self.quest_logger: - # Get benchmark_id from the agent_config parameter if available - benchmark_id = None - if ( - hasattr(self, "agent_config") - and self.agent_config - and hasattr(self.agent_config, "benchmark_id") - ): - benchmark_id = self.agent_config.benchmark_id - - self.quest_logger.set_quest_outcome( - QuestOutcome.ERROR.name, - 0.0, - benchmark_id, - final_state=self.env.state if self.env else None, - ) - raise + self._record_transition(snapshot, action, after) + snapshot = after + self._snapshot = snapshot + + except ReplayError as e: + self.logger.error("Resume verification failed: %s", str(e)) + self._notify_callbacks("error", str(e)) + self.quest_logger.finish_run( + QuestOutcome.ERROR.name, + 0.0, + terminal_snapshot=self._snapshot, + progress=self.progress.state(), + diagnostics=self.runtime_metrics(), + ) + raise except Exception as e: if self._stop_requested.is_set(): self.logger.info("Quest runner stopped after %s", self._stop_reason or "request") return QuestOutcome.TIMEOUT self.logger.error("Error running quest: %s", str(e), exc_info=True) self._notify_callbacks("error", str(e)) - if self.env and self.env.state: - self.agent.on_game_end(self.env.state) - - # Log error outcome - if self.quest_logger: - # Get benchmark_id from the agent_config parameter if available - benchmark_id = None - if hasattr(self, "agent_config") and self.agent_config and hasattr(self.agent_config, "benchmark_id"): - benchmark_id = self.agent_config.benchmark_id - - self.quest_logger.set_quest_outcome( - QuestOutcome.ERROR.name, - 0.0, - benchmark_id, - final_state=self.env.state if self.env else None, - ) - + self.agent.on_game_end(self._snapshot) + self.quest_logger.finish_run( + QuestOutcome.ERROR.name, + 0.0, + terminal_snapshot=self._snapshot, + progress=self.progress.state(), + diagnostics=self.runtime_metrics(), + ) return QuestOutcome.ERROR finally: self._notify_callbacks("close") diff --git a/llm_quest_benchmark/environments/qm.py b/llm_quest_benchmark/environments/qm.py index f39b720..c971421 100644 --- a/llm_quest_benchmark/environments/qm.py +++ b/llm_quest_benchmark/environments/qm.py @@ -1,11 +1,15 @@ """QM environment for Space Rangers quests""" import logging -from typing import Any +from collections import Counter from llm_quest_benchmark.executors.ts_bridge.bridge import QMBridge -from llm_quest_benchmark.schemas.state import QMState -from llm_quest_benchmark.utils.choice_mapper import ChoiceMapper +from llm_quest_benchmark.schemas.records import QuestSnapshot + +LOOP_GUARD_MIN_STATES = 30 +LOOP_GUARD_WINDOW = 10 +LOOP_GUARD_REPEATS = 5 +FORCED_STOP_TEXT = "[Forced stop: repetitive state loop detected before terminal quest outcome]" def find_quest_file(quest_path: str) -> str: @@ -43,14 +47,11 @@ def find_quest_file(quest_path: str) -> str: class QMPlayerEnv: - """Environment for playing QM files using TypeScript bridge - - This environment provides a clean interface to Space Rangers quests. - All game state and logic is handled by the TypeScript bridge, while this class: - 1. Manages the bridge lifecycle - 2. Handles choice mapping between sequential numbers and jump IDs - 3. Formats observations and state - 4. Tracks game history + """Environment for playing QM files using the TypeScript bridge. + + The environment owns canonical snapshots: every reset, step, and restore + returns a ``QuestSnapshot`` carrying the full engine saving and a + deterministic digest over the canonical state fields. """ def __init__(self, quest_file: str, language: str = "rus", debug: bool = False): @@ -63,6 +64,7 @@ def __init__(self, quest_file: str, language: str = "rus", debug: bool = False): """ self.debug = debug self.language = language + self.forced_stop_reason: str | None = None # Initialize logger self.logger = logging.getLogger(self.__class__.__name__) @@ -77,212 +79,123 @@ def __init__(self, quest_file: str, language: str = "rus", debug: bool = False): # Initialize bridge self.bridge = QMBridge(self.quest_file, language=self.language, debug=debug) - self.state_history: list[QMState] = [] - self.choice_mapper: ChoiceMapper | None = None - self._current_state: dict[str, Any] = {} # Internal state storage + self._snapshot: QuestSnapshot | None = None except Exception as e: self.logger.error(f"Failed to initialize QMPlayerEnv: {e}") raise RuntimeError(f"Failed to initialize QMPlayerEnv: {e}") - @staticmethod - def _format_params_state(params_state: Any) -> str: - if not params_state: - return "" - if isinstance(params_state, list): - lines = [str(x).strip() for x in params_state if str(x).strip()] - else: - lines = [str(params_state).strip()] - if not lines: - return "" - return "Status:\n" + "\n".join(lines) - - def _compose_observation_text(self, text: str, params_state: Any) -> str: - base = (text or "").strip() - params_block = self._format_params_state(params_state) - if not params_block: - return base - if not base: - return params_block - return f"{base}\n\n{params_block}" - - def _format_observation(self, state) -> str: - """Format observation text from game state""" - if not state: - return "No state available" - - # Support both QMState-like objects and internal dict state. - if isinstance(state, dict): - text = state.get("text") or "" - params_state = state.get("params_state") or [] - choices = state.get("choices") or [] - else: - text = getattr(state, "text", "") or "" - params_state = getattr(state, "params_state", []) or [] - choices = getattr(state, "choices", []) or [] - - text = self._compose_observation_text(text, params_state) - - # Add choices if available - if choices: - text += "\n\nAvailable actions:\n" - for i, choice in enumerate(choices, 1): - text += f"{i}. {choice['text']}\n" - - return text - - def reset(self) -> str: - """Reset environment to initial state""" + @property + def snapshot(self) -> QuestSnapshot | None: + """Current canonical snapshot, or None before reset.""" + return self._snapshot + + def _require_snapshot(self) -> QuestSnapshot: + if self._snapshot is None: + raise RuntimeError("Environment not initialized - call reset() first") + return self._snapshot + + def reset(self) -> QuestSnapshot: + """Start the quest and return the initial snapshot.""" try: + self.forced_stop_reason = None initial_bridge_state = self.bridge.start_game() if not initial_bridge_state: raise RuntimeError("Failed to get initial state from bridge") - self._current_state = { - "location_id": initial_bridge_state.location_id, - "text": initial_bridge_state.text, - "params_state": initial_bridge_state.params_state, - "choices": initial_bridge_state.choices, - "reward": initial_bridge_state.reward, - "done": initial_bridge_state.game_ended, - "info": {}, - } - - if not self._current_state["choices"]: + self._snapshot = initial_bridge_state.to_snapshot() + if not self._snapshot.choices and not self._snapshot.done: raise RuntimeError("No valid choices in initial state") - - return self._compose_observation_text(self._current_state["text"], self._current_state.get("params_state")) + return self._snapshot except Exception as e: self.logger.error(f"Failed to reset environment: {e}") self.bridge.close() # Clean up on error raise RuntimeError(f"Failed to reset environment: {e}") - def step(self, action: str) -> tuple[str, bool, bool, dict[str, Any]]: - """Take action in environment and return new state - - Args: - action: Action to take (choice number or text) + def _detect_state_loop(self) -> bool: + """Detect a repeating non-terminal state loop across recent engine states. - Returns: - Tuple of (observation, done, success, info) + This is a general guard for quests (like Prison.qm) whose daily-routine + branches can cycle forever without reaching a terminal outcome. """ - if not self._current_state: - raise RuntimeError("Environment not initialized - call reset() first") - - # Check for patterns that might indicate an infinite loop - # This is a general solution that works for any quest that might get stuck - if len(self.bridge.state_history) > 30: # Only check after a reasonable number of steps - # Get the current state text - current_text = self._current_state.get("text", "") - - # Check for repeating daily routine patterns (like in Prison.qm) - repeat_day_pattern_count = 0 - _ = 0 # reserved for future repeat text detection - - # Check last 10 states for repetition - text_fragments = [] - for state in self.bridge.state_history[-10:]: - if state.text: - # Add first 20 chars of each state text for comparison - text_fragment = state.text[:20].strip() - text_fragments.append(text_fragment) - - # Specific check for daily pattern (appears in Prison.qm) - if "Наступил новый день" in state.text: - repeat_day_pattern_count += 1 - - # Count how many times each fragment appears - from collections import Counter - - fragment_counts = Counter(text_fragments) - - # If any single text fragment appears 5+ times in the last 10 states - # or if we see 5+ daily routine messages - if any(count >= 5 for count in fragment_counts.values()) or repeat_day_pattern_count >= 5: - self.logger.warning(f"Detected potential infinite loop after {len(self.bridge.state_history)} steps") + history = self.bridge.state_history + if len(history) <= LOOP_GUARD_MIN_STATES: + return False + + fragments = [state.text[:20].strip() for state in history[-LOOP_GUARD_WINDOW:] if state.text] + counts = Counter(fragments) + return any(count >= LOOP_GUARD_REPEATS for count in counts.values()) + + def _forced_stop_snapshot(self) -> QuestSnapshot: + """Terminal snapshot for the loop guard; never a quest success.""" + current = self._require_snapshot() + return QuestSnapshot( + location_id=current.location_id, + observation=f"{current.observation}\n\n{FORCED_STOP_TEXT}", + choices=[], + params_state=list(current.params_state), + reward=current.reward, + done=True, + game_state="fail", + saving=current.saving, + ) + + def step(self, choice_index: int, performed_at_ms: int) -> QuestSnapshot: + """Execute a one-based choice and return the resulting snapshot. + + ``performed_at_ms`` is recorded with the transition and handed to the + engine, so replaying the same timestamp reproduces dynamic branches. + """ + self._require_snapshot() - synthetic_text = ( - current_text + "\n\n[Forced stop: repetitive state loop detected before terminal quest outcome]" - ) - - forced_info = {"forced_completion": True, "reason": "infinite_loop_detected"} - self._current_state = { - "location_id": self._current_state.get("location_id", "unknown"), - "text": synthetic_text, - "params_state": self._current_state.get("params_state", []), - "choices": [], - "reward": self._current_state.get("reward", 0.0), - "done": True, - "info": forced_info, - } - - # Loop detection is a non-terminal fallback signal, not a quest success. - return ( - self._compose_observation_text( - self._current_state["text"], self._current_state.get("params_state") - ), - True, - False, - forced_info, - ) + if self._detect_state_loop(): + self.logger.warning("Detected potential infinite loop after %s states", len(self.bridge.state_history)) + self.forced_stop_reason = "infinite_loop_detected" + self._snapshot = self._forced_stop_snapshot() + return self._snapshot try: - # Take action in bridge - new_bridge_state = self.bridge.step(action) + new_bridge_state = self.bridge.step(choice_index, performed_at_ms) if not new_bridge_state: raise RuntimeError("Failed to get new state from bridge") - # Update internal state - self._current_state = { - "location_id": new_bridge_state.location_id, - "text": new_bridge_state.text, - "params_state": new_bridge_state.params_state, - "choices": new_bridge_state.choices, - "reward": new_bridge_state.reward, - "done": new_bridge_state.game_ended, - "info": {}, - } - - # Determine success from the TS engine's authoritative gameState. - # "win" = success, "fail"/"dead" = failure, "running" = not ended yet. - success = new_bridge_state.game_state == "win" - - if new_bridge_state.game_ended: + self._snapshot = new_bridge_state.to_snapshot() + + if self._snapshot.done: self.logger.info( - f"Game ended: game_state={new_bridge_state.game_state}, " - f"location={new_bridge_state.location_id}, success={success}" + f"Game ended: game_state={self._snapshot.game_state}, " + f"location={self._snapshot.location_id}, success={self._snapshot.game_state == 'win'}" ) - - return ( - self._compose_observation_text(self._current_state["text"], self._current_state.get("params_state")), - self._current_state["done"], - success, - self._current_state["info"], - ) + return self._snapshot except Exception as e: self.logger.error(f"Failed to take step: {e}") self.bridge.close() # Clean up on error raise RuntimeError(f"Failed to take step: {e}") - def get_state(self) -> dict[str, Any]: - """Get current environment state""" - if not self._current_state: - raise RuntimeError("Environment not initialized - call reset() first") - return self._current_state.copy() + def restore(self, snapshot: QuestSnapshot) -> QuestSnapshot: + """Restore the engine to an exact recorded snapshot. + + The restored snapshot must reproduce the recorded digest; a mismatch + means the engine or the record diverged and the run cannot continue. + """ + if not snapshot.is_resumable: + raise ValueError("Cannot restore a snapshot without a full engine saving") + + try: + restored_state = self.bridge.load_saving(snapshot.saving) + except Exception as e: + self.logger.error(f"Failed to restore snapshot: {e}") + raise RuntimeError(f"Failed to restore snapshot: {e}") + + restored = restored_state.to_snapshot() + if restored.digest != snapshot.digest: + raise RuntimeError( + f"Restored state digest does not match the recorded snapshot ({restored.digest} != {snapshot.digest})" + ) + self.forced_stop_reason = None + self._snapshot = restored + return restored def close(self): """Clean up resources""" if hasattr(self, "bridge"): self.bridge.close() - - @property - def state(self) -> dict[str, Any]: - """Get current state for renderer compatibility""" - if not self._current_state: - return {} - return self._current_state.copy() - - def current_observation(self) -> str: - """Get current observation for renderer compatibility""" - if not self._current_state: - return "" - return self._format_observation(self._current_state) diff --git a/llm_quest_benchmark/environments/state.py b/llm_quest_benchmark/environments/state.py index 1483c89..5a72a74 100644 --- a/llm_quest_benchmark/environments/state.py +++ b/llm_quest_benchmark/environments/state.py @@ -9,17 +9,24 @@ class QuestOutcome(Enum): SUCCESS = 1 FAILURE = 0 ERROR = -1 - TIMEOUT = -2 # Add timeout state + TIMEOUT = -2 # Wall-clock timeout + TRUNCATED = -3 # Explicit step limit reached; the run stays resumable @property def is_error(self) -> bool: """Whether this outcome represents an error state""" return self in (QuestOutcome.ERROR, QuestOutcome.TIMEOUT) + @property + def is_resumable(self) -> bool: + """Whether a run ending in this outcome can be continued from its record""" + return self is QuestOutcome.TRUNCATED + @property def exit_code(self) -> int: """Get the appropriate exit code for this outcome. Normal outcomes (SUCCESS, FAILURE) return 0, + TRUNCATED returns 0 (a deliberate, resumable stop), TIMEOUT returns 124 (standard Unix timeout code), ERROR returns 2 (standard Unix error code)""" if self == QuestOutcome.TIMEOUT: diff --git a/llm_quest_benchmark/executors/benchmark.py b/llm_quest_benchmark/executors/benchmark.py index 6aa188d..d21116a 100644 --- a/llm_quest_benchmark/executors/benchmark.py +++ b/llm_quest_benchmark/executors/benchmark.py @@ -12,12 +12,14 @@ from pathlib import Path from typing import Any -from llm_quest_benchmark.core.logging import DEFAULT_DB_PATH +from llm_quest_benchmark.core.logging import default_db_path, ensure_v2_schema, verify_v2_schema +from llm_quest_benchmark.core.provenance import engine_revision, quest_checksum from llm_quest_benchmark.core.runner import run_quest_with_timeout from llm_quest_benchmark.environments.state import QuestOutcome from llm_quest_benchmark.harnesses.factory import create_harness from llm_quest_benchmark.llm import tracing from llm_quest_benchmark.schemas.config import BenchmarkConfig +from llm_quest_benchmark.schemas.records import SCHEMA_VERSION # Configure logging logging.basicConfig( @@ -50,51 +52,8 @@ def _agent_model(agent_config) -> str: def _agent_id(agent_config) -> str: - """Return the stable result identifier for legacy and harness configs.""" - return getattr(agent_config, "harness_id", None) or agent_config.agent_id - - -def _agent_template(agent_config) -> str: - """Return legacy template name for result artifacts.""" - if hasattr(agent_config, "action_template"): - return agent_config.action_template - - harness_templates = { - "minimal": "stub.jinja", - "reasoning_recent": "reasoning.jinja", - "reasoning_full": "reasoning.jinja", - "memo_compact": "stateful_compact.jinja", - "hinted_compact": "stateful_compact_hints.jinja", - "tool_compact": "tool_augmented.jinja", - "tool_hinted": "tool_augmented_hints.jinja", - "programmatic_memory": "programmatic_memory.jinja", - "planner": "planner.jinja", - "compaction_no_memo": "reasoning.jinja", - "memo_cot": "memo_cot.jinja", - "memo_extended": "memo_extended.jinja", - "memo_structured": "memo_structured.jinja", - } - return harness_templates.get(_agent_harness(agent_config), "reasoning.jinja") - - -def _agent_memory_mode(agent_config) -> str: - """Return legacy memory mode for result artifacts.""" - if hasattr(agent_config, "memory_mode"): - return agent_config.memory_mode - - harness_memory_modes = { - "reasoning_full": "full_transcript", - "memo_compact": "compaction", - "hinted_compact": "compaction", - "tool_compact": "compaction", - "tool_hinted": "compaction", - "planner": "compaction", - "compaction_no_memo": "compaction", - "memo_cot": "compaction", - "memo_extended": "compaction", - "memo_structured": "compaction", - } - return harness_memory_modes.get(_agent_harness(agent_config), "default") + """Return the stable result identifier derived from the treatment.""" + return agent_config.agent_id def _harnesses_by_model(results: list[dict[str, Any]]) -> dict[str, set[str]]: @@ -133,13 +92,14 @@ def _result_entry( reward: float = 0.0, error: str | None = None, ) -> dict[str, Any]: + treatment = agent_config.treatment().to_dict() return { "quest": quest, "model": _agent_model(agent_config), "temperature": agent_config.temperature, "harness": _agent_harness(agent_config), - "template": _agent_template(agent_config), - "memory_mode": _agent_memory_mode(agent_config), + "treatment": treatment, + "treatment_signature": treatment["signature"], "agent_id": _agent_id(agent_config), "attempt": attempt, "outcome": outcome, @@ -150,11 +110,14 @@ def _result_entry( def _mark_run_timeout(run_id: int | None, quest: str, agent_config, benchmark_id: str, timeout: int) -> None: """Record a parent-enforced timeout for a killed child process.""" - agent_config_json = json.dumps(agent_config.__dict__) end_time = datetime.utcnow() - conn = sqlite3.connect(DEFAULT_DB_PATH) + treatment = agent_config.treatment().to_dict() + conn = sqlite3.connect(default_db_path()) try: + ensure_v2_schema(conn) if run_id is not None: + # The child already wrote complete run metadata before executing, + # so the parent only records the terminal outcome. row = conn.execute("SELECT start_time FROM runs WHERE id = ?", (run_id,)).fetchone() run_duration = None if row and row[0]: @@ -166,40 +129,38 @@ def _mark_run_timeout(run_id: int | None, quest: str, agent_config, benchmark_id conn.execute( """ UPDATE runs - SET agent_id = ?, agent_config = ?, benchmark_id = ?, outcome = ?, - reward = ?, end_time = ?, run_duration = ? + SET outcome = ?, reward = ?, end_time = ?, run_duration = ? WHERE id = ? """, - ( - _agent_id(agent_config), - agent_config_json, - benchmark_id, - QuestOutcome.TIMEOUT.name, - 0.0, - end_time, - run_duration, - run_id, - ), + (QuestOutcome.TIMEOUT.name, 0.0, end_time, run_duration, run_id), ) else: + # The child died before writing its run row; record the attempt with + # the metadata the parent can prove. conn.execute( """ INSERT INTO runs - (quest_file, quest_name, start_time, end_time, agent_id, agent_config, - outcome, reward, run_duration, benchmark_id) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + (schema_version, quest_file, quest_name, quest_checksum, quest_language, + engine_revision, agent_id, treatment, treatment_signature, benchmark_id, + start_time, end_time, run_duration, outcome, reward) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( + SCHEMA_VERSION, quest, Path(quest).stem, + quest_checksum(quest) if Path(quest).exists() else "unavailable", + "rus", + engine_revision(), + _agent_id(agent_config), + json.dumps(treatment, ensure_ascii=False), + treatment["signature"], + benchmark_id, end_time, end_time, - _agent_id(agent_config), - agent_config_json, - QuestOutcome.TIMEOUT.name, 0.0, + QuestOutcome.TIMEOUT.name, 0.0, - benchmark_id, ), ) conn.commit() @@ -214,6 +175,7 @@ def _run_benchmark_task(task: dict[str, Any], result_queue) -> None: quest = task["quest"] attempt = task["attempt"] max_steps = task.get("max_steps") + progress_manifest = task.get("progress_manifest") def callback(event: str, data: Any = None) -> None: if event == "run_record" and isinstance(data, dict): @@ -234,6 +196,8 @@ def callback(event: str, data: Any = None) -> None: debug=agent_config.debug, compaction_interval=agent_config.compaction_interval, system_template=agent_config.system_template, + restore_limit=agent_config.restore_limit, + adaptive_stall_steps=agent_config.adaptive_stall_steps, ) outcome = run_quest_with_timeout( quest, @@ -243,6 +207,7 @@ def callback(event: str, data: Any = None) -> None: debug=agent_config.debug, callbacks=[callback], max_steps=max_steps, + progress_manifest=progress_manifest, ) outcome_name = outcome.name if outcome else QuestOutcome.TIMEOUT.name result_queue.put( @@ -304,25 +269,45 @@ def get_quest_files(quest_paths: list[str], max_quests: int | None = None) -> li return quest_files -def _load_benchmark_runs_from_db(benchmark_id: str, db_path: str = DEFAULT_DB_PATH) -> list[dict[str, Any]]: - """Load DB runs associated with a benchmark id.""" - if not Path(db_path).exists(): +def _load_benchmark_runs_from_db(benchmark_id: str, db_path: str | None = None) -> list[dict[str, Any]]: + """Load schema-v2 DB runs associated with a benchmark id. + + The database is resolved at call time, so workers and tests read the same + database they wrote. A pre-v2 database raises with migration guidance rather + than being probed column by column. + """ + resolved = db_path or default_db_path() + if not Path(resolved).exists(): return [] - conn = sqlite3.connect(db_path) + conn = sqlite3.connect(resolved) conn.row_factory = sqlite3.Row try: + if not verify_v2_schema(conn): + return [] rows = conn.execute( """ - SELECT id, quest_file, quest_name, start_time, end_time, agent_id, - agent_config, outcome, reward, run_duration, benchmark_id + SELECT id, schema_version, quest_file, quest_name, quest_checksum, quest_language, + engine_revision, agent_id, treatment, treatment_signature, benchmark_id, + start_time, end_time, run_duration, outcome, reward, usage, + transcript_diagnostics, progress FROM runs WHERE benchmark_id = ? ORDER BY id """, (benchmark_id,), ).fetchall() - return [dict(row) for row in rows] + runs = [] + for row in rows: + run = dict(row) + for key in ("treatment", "usage", "transcript_diagnostics", "progress"): + if isinstance(run.get(key), str) and run[key]: + try: + run[key] = json.loads(run[key]) + except json.JSONDecodeError: + run[key] = {} + runs.append(run) + return runs finally: conn.close() @@ -349,6 +334,7 @@ def _write_benchmark_artifacts(config: BenchmarkConfig, results: list[dict[str, "runs": agent.runs, "system_template": agent.system_template, "harness": _agent_harness(agent), + "treatment": agent.treatment().to_dict(), } for agent in config.agents ], @@ -372,6 +358,7 @@ def _write_benchmark_artifacts(config: BenchmarkConfig, results: list[dict[str, "benchmark_id": config.benchmark_id, "max_quests": config.max_quests, "max_workers": config.max_workers, + "progress_manifest": config.progress_manifest, "agents": [ { "model": agent.model, @@ -381,6 +368,9 @@ def _write_benchmark_artifacts(config: BenchmarkConfig, results: list[dict[str, "runs": agent.runs, "skip_single": agent.skip_single, "debug": agent.debug, + "compaction_interval": agent.compaction_interval, + "restore_limit": agent.restore_limit, + "adaptive_stall_steps": agent.adaptive_stall_steps, } for agent in config.agents ], @@ -437,6 +427,7 @@ def run_benchmark(config: BenchmarkConfig, progress_callback=None) -> list[dict[ "attempt": attempt, "benchmark_id": config.benchmark_id, "max_steps": config.max_steps, + "progress_manifest": config.progress_manifest, } ) @@ -671,13 +662,6 @@ def print_summary(results: list[dict[str, Any]]) -> None: print("\nResults Summary:") print("=" * 80) - # Calculate total steps (if available) - steps_info_available = any("steps" in r for r in results) - - if steps_info_available: - total_steps = sum(len(r.get("steps", [])) for r in results) - steps_by_model = {} - # Group by agent variant (model, or "model [harness]" when a model is run # under more than one harness in these results). harnesses_by_model = _harnesses_by_model(results) @@ -690,12 +674,6 @@ def print_summary(results: list[dict[str, Any]]) -> None: timeout = len([r for r in model_results if r["outcome"] == QuestOutcome.TIMEOUT.name]) total = len(model_results) - # Calculate steps for this group (if available) - if steps_info_available: - model_steps = sum(len(r.get("steps", [])) for r in model_results) - avg_steps = model_steps / total if total > 0 else 0 - steps_by_model[group] = (model_steps, avg_steps) - label = "Agent" if "[" in group else "Model" print(f"\n{label}: {group}") print(f"Total quests: {total}") @@ -704,17 +682,6 @@ def print_summary(results: list[dict[str, Any]]) -> None: print(f"Error: {error} ({error / total * 100:.1f}%)") print(f"Timeout: {timeout} ({timeout / total * 100:.1f}%)") - if steps_info_available: - print(f"Total steps: {model_steps}") - print(f"Average steps per quest: {avg_steps:.1f}") - - # Print overall steps summary (if available) - if steps_info_available: - print("\nOverall Steps Summary:") - print("=" * 80) - print(f"Total steps across all models: {total_steps}") - print(f"Average steps per quest: {total_steps / len(results):.1f}") - # List errors if any errors = [r for r in results if r.get("error")] if errors: diff --git a/llm_quest_benchmark/executors/cli/commands.py b/llm_quest_benchmark/executors/cli/commands.py index d554f70..46bb8c7 100644 --- a/llm_quest_benchmark/executors/cli/commands.py +++ b/llm_quest_benchmark/executors/cli/commands.py @@ -26,10 +26,11 @@ MODEL_CHOICES, SYSTEM_ROLE_TEMPLATE, ) -from llm_quest_benchmark.core.analyzer import analyze_benchmark, analyze_quest_run +from llm_quest_benchmark.core.analyzer import analyze_benchmark, analyze_quest_run, load_transitions from llm_quest_benchmark.core.benchmark_report import render_benchmark_report from llm_quest_benchmark.core.leaderboard import generate_leaderboard -from llm_quest_benchmark.core.logging import LogManager +from llm_quest_benchmark.core.logging import LogManager, default_db_path +from llm_quest_benchmark.core.replay import harness_config_from_record from llm_quest_benchmark.core.runner import run_quest_with_timeout from llm_quest_benchmark.environments.state import QuestOutcome from llm_quest_benchmark.executors.benchmark import ( @@ -37,10 +38,12 @@ print_summary, run_benchmark, ) -from llm_quest_benchmark.harnesses.factory import HARNESS_REGISTRY, create_harness +from llm_quest_benchmark.harnesses.factory import create_harness +from llm_quest_benchmark.harnesses.specs import LLM_HARNESS_NAMES from llm_quest_benchmark.llm import tracing from llm_quest_benchmark.renderers.terminal import RichRenderer from llm_quest_benchmark.schemas.config import BenchmarkConfig, HarnessConfig +from llm_quest_benchmark.schemas.records import RunRecord, load_run_record # Initialize logging log_manager = LogManager() @@ -51,7 +54,7 @@ rich_markup_mode="rich", ) -HARNESS_CHOICES = list(HARNESS_REGISTRY.keys()) +HARNESS_CHOICES = list(LLM_HARNESS_NAMES) def version_callback(value: bool): @@ -68,9 +71,9 @@ def _parse_run_dir_id(path: Path) -> int: return -1 -def _load_run_summary(path: Path) -> dict[str, Any]: - with open(path, encoding="utf-8") as f: - return json.load(f) +def _load_run_summary(path: Path) -> RunRecord: + """Load a schema-v2 run record; legacy files are rejected with guidance.""" + return load_run_record(str(path)) def _count_quest_collections(quests_root: Path) -> list[dict[str, Any]]: @@ -106,46 +109,20 @@ def _summarize_quest_collections(collections: list[dict[str, Any]]) -> dict[str, } -def _coerce_choices(step: dict[str, Any]) -> dict[str, str]: - """Normalize old/new step choice formats to {index: text} map.""" - choices = step.get("choices") - if isinstance(choices, dict): - return {str(k): str(v) for k, v in choices.items()} - if isinstance(choices, list): - return {str(i): c.get("text", "") for i, c in enumerate(choices, start=1) if isinstance(c, dict)} - indexed = step.get("choices_indexed") - if isinstance(indexed, list): - return { - str(c.get("index")): c.get("text", "") - for c in indexed - if isinstance(c, dict) and c.get("index") is not None - } - return {} - - -def _coerce_selected_choice(step: dict[str, Any], choices_map: dict[str, str]) -> dict[str, str] | None: - """Extract selected choice from old/new step schema.""" - llm_decision = step.get("llm_decision") or {} - if isinstance(llm_decision, dict): - choice_map = llm_decision.get("choice") - if isinstance(choice_map, dict) and choice_map: - return {str(k): str(v) for k, v in choice_map.items()} - - selected_choice = step.get("selected_choice") - if isinstance(selected_choice, dict): - idx = selected_choice.get("index") - text = selected_choice.get("text") - if idx is not None and text is not None: - return {str(idx): str(text)} - - action_index = step.get("action_index") or step.get("action") - try: - idx = str(int(action_index)) - except (TypeError, ValueError): - return None - if idx not in choices_map: - return None - return {idx: choices_map[idx]} +def _choices_map(choices: list[dict[str, str]]) -> dict[str, str]: + """Index the choices of a snapshot for display.""" + return {str(index): choice.get("text", "") for index, choice in enumerate(choices, start=1)} + + +def _selected_label(transition: Any, choices_map: dict[str, str]) -> str: + """Render the executed action of a transition for display.""" + action = transition.action + if action.is_restore: + return f"restore checkpoint {action.checkpoint_index}" + index = str(action.choice_index) if action.choice_index is not None else "" + if index and index in choices_map: + return f"{index}:{choices_map[index]}" + return "none" def _handle_quest_outcome(outcome: QuestOutcome, log_prefix: str) -> None: @@ -201,35 +178,36 @@ def analyze_run( typer.echo(f"run_summary not found: {summary_path}", err=True) raise typer.Exit(code=1) - data = _load_run_summary(summary_path) - steps = data.get("steps") or [] - outcome = data.get("outcome", "UNKNOWN") - quest_name = data.get("quest_name", "unknown") - agent_id = data.get("agent_id", "unknown") + record = _load_run_summary(summary_path) + outcome = record.outcome or "UNKNOWN" typer.echo(f"Run: {summary_path}") - typer.echo(f"Quest: {quest_name}") - typer.echo(f"Agent: {agent_id}") + typer.echo(f"Quest: {record.quest_name}") + typer.echo(f"Agent: {record.agent_id}") + typer.echo(f"Treatment: {record.treatment_signature}") typer.echo(f"Outcome: {outcome}") - typer.echo(f"Total Steps: {len(steps)}") + typer.echo(f"Progress: {record.progress.current:.1f}% of {record.progress.maximum:.1f}%") + typer.echo(f"Total Steps: {len(record.transitions)}") + if record.lineage: + typer.echo(f"Resumed from: {record.lineage.source_path} (run {record.lineage.source_run_id})") decision_rows = [] - for step in steps: - if not isinstance(step, dict): + for transition in record.transitions: + choices_map = _choices_map(transition.before.choices) + if len(choices_map) <= 1 and not transition.action.is_restore: continue - choices_map = _coerce_choices(step) - if len(choices_map) <= 1: - continue - llm_decision = step.get("llm_decision") if isinstance(step.get("llm_decision"), dict) else {} + response = transition.response decision_rows.append( { - "step": step.get("step"), - "observation": step.get("observation", ""), + "step": transition.index, + "observation": transition.before.agent_observation(), "choices": choices_map, - "selected": _coerce_selected_choice(step, choices_map), - "analysis": llm_decision.get("analysis"), - "reasoning": llm_decision.get("reasoning"), - "is_default": bool(llm_decision.get("is_default", False)), + "selected": _selected_label(transition, choices_map), + "analysis": response.analysis if response else None, + "reasoning": response.reasoning if response else None, + "is_default": bool(response.is_default) if response else False, + "reasoning_mode": transition.reasoning_mode, + "progress": transition.progress.current, } ) @@ -239,9 +217,11 @@ def analyze_run( typer.echo("\nDecision Trace:") for row in decision_rows[:max_steps]: - selected = row["selected"] or {} - selected_str = ", ".join(f"{k}:{v}" for k, v in selected.items()) if selected else "none" - typer.echo(f"- step {row['step']}: selected [{selected_str}] default={row['is_default']}") + mode = f" mode={row['reasoning_mode']}" if row["reasoning_mode"] else "" + typer.echo( + f"- step {row['step']}: selected [{row['selected']}] " + f"default={row['is_default']} progress={row['progress']:.1f}%{mode}" + ) if row["reasoning"]: typer.echo(f" reasoning: {row['reasoning']}") if row["analysis"]: @@ -255,10 +235,7 @@ def analyze_run( typer.echo("- available choices:") for idx, text in last["choices"].items(): typer.echo(f" {idx}: {text}") - selected = last["selected"] or {} - if selected: - chosen_idx, chosen_text = next(iter(selected.items())) - typer.echo(f"- selected: {chosen_idx}: {chosen_text}") + typer.echo(f"- selected: {last['selected']}") except typer.Exit: raise @@ -354,6 +331,15 @@ def run( help="Harness to use for quest decisions.", ), compaction_interval: int = typer.Option(50, help="Advanced override for compaction interval."), + restore_limit: int | None = typer.Option(None, help="Restore budget; valid only for the backtracking harness."), + adaptive_stall_steps: int | None = typer.Option( + None, help="Stall trigger; valid only for the adaptive_reasoning harness." + ), + max_steps: int | None = typer.Option(None, help="Stop after this many steps and record a resumable TRUNCATED run."), + progress_manifest: Path | None = typer.Option(None, help="Curated YAML progress manifest for this quest."), + resume_from: Path | None = typer.Option( + None, help="Resume a schema-v2 run_summary.json; quest and treatment come from the record." + ), timeout: int = typer.Option(60, help="Timeout in seconds for run (0 for no timeout)."), skip: bool = typer.Option(True, help="Auto-select single choices without asking agent."), debug: bool = typer.Option(False, help="Enable debug logging and output, remove terminal UI."), @@ -365,30 +351,52 @@ def run( Example: llm-quest run --quest quests/boat.qm --model sonnet --debug + llm-quest run --quest quests/Boat.qm --max-steps 5 + llm-quest run --resume-from results//Boat/run_12/run_summary.json """ try: log_manager.setup(debug) - # Create agent config - agent_config = HarnessConfig( - model=model, - system_template=system_template, - harness=harness, - temperature=temperature, - skip_single=skip, - debug=debug, - compaction_interval=compaction_interval, - ) + resume_record = None + if resume_from is not None: + resume_record = load_run_record(str(resume_from)) + if not resume_record.is_resumable: + typer.echo( + f"Run {resume_from} is not resumable " + f"(outcome={resume_record.outcome}); only verified TRUNCATED runs can continue.", + err=True, + ) + raise typer.Exit(code=1) + # Quest and treatment come from the record, not from the CLI flags. + agent_config = harness_config_from_record(resume_record) + quest = Path(resume_record.quest_file) + agent_config.skip_single = skip + agent_config.debug = debug + log.warning(f"Resuming run {resume_record.run_id} from {resume_from}") + else: + agent_config = HarnessConfig( + model=model, + system_template=system_template, + harness=harness, + temperature=temperature, + skip_single=skip, + debug=debug, + compaction_interval=compaction_interval, + restore_limit=restore_limit, + adaptive_stall_steps=adaptive_stall_steps, + ) # Create agent agent = create_harness( - harness=harness, - model=model, - system_template=system_template, - temperature=temperature, - skip_single=skip, - debug=debug, - compaction_interval=compaction_interval, + harness=agent_config.harness, + model=agent_config.model, + system_template=agent_config.system_template, + temperature=agent_config.temperature, + skip_single=agent_config.skip_single, + debug=agent_config.debug, + compaction_interval=agent_config.compaction_interval, + restore_limit=agent_config.restore_limit, + adaptive_stall_steps=agent_config.adaptive_stall_steps, ) log.warning(f"Starting quest run with agent {str(agent)}") @@ -433,6 +441,10 @@ def close_callback(event, data): timeout=timeout, agent_config=agent_config, callbacks=callbacks, + max_steps=max_steps, + progress_manifest=str(progress_manifest) if progress_manifest else None, + resume_record=resume_record, + resume_path=str(resume_from) if resume_from else None, ) _handle_quest_outcome(result, "Quest run") @@ -523,10 +535,12 @@ def download_quests() -> None: @app.command() def analyze( quest: str | None = typer.Option(None, help="Name of the quest to analyze (e.g. 'boat.qm')."), - benchmark: str | None = typer.Option(None, help="Name of the benchmark to analyze (e.g. 'baseline')."), + benchmark: str | None = typer.Option(None, help="Benchmark ID to analyze (e.g. 'CLI_benchmark_20260101_...')."), run_id: int | None = typer.Option(None, help="Specific run ID to analyze in detail."), last: bool = typer.Option(False, help="Analyze the most recent quest run."), - db: Path = typer.Option("metrics.db", help="Path to SQLite database."), + db: Path | None = typer.Option( + None, help="Path to SQLite database (defaults to $LLM_QUEST_DB_PATH, else metrics.db)." + ), export: Path | None = typer.Option(None, help="Export results to JSON file."), format: str = typer.Option("summary", help="Output format (summary, detail, or compact)."), debug: bool = typer.Option(False, help="Enable debug logging and output."), @@ -557,6 +571,7 @@ def analyze( raise typer.Exit(code=1) # Validate database exists + db = db or Path(default_db_path()) if not db.exists(): typer.echo(f"Database not found: {db}", err=True) raise typer.Exit(code=1) @@ -577,100 +592,69 @@ def analyze( # Analyze specific run by ID if run_id: - # First check schema to handle older database versions - cursor.execute("PRAGMA table_info(runs)") - columns = [column[1] for column in cursor.fetchall()] - - # Construct query based on available columns - select_fields = ["r.id", "r.quest_name", "r.start_time", "r.end_time", "r.agent_id", "r.agent_config"] - if "outcome" in columns: - select_fields.append("r.outcome") - else: - select_fields.append("'UNKNOWN' as outcome") - - if "reward" in columns: - select_fields.append("r.reward") - else: - select_fields.append("0.0 as reward") - - if "run_duration" in columns: - select_fields.append("r.run_duration") - else: - select_fields.append("NULL as run_duration") - - query = f""" - SELECT {", ".join(select_fields)} - FROM runs r - WHERE r.id = ? - """ - cursor.execute(query, (run_id,)) - - run = cursor.fetchone() - if not run: - typer.echo(f"Run ID {run_id} not found", err=True) - raise typer.Exit(code=1) - - run_id, quest_name, start_time, end_time, agent_id, agent_config, outcome, reward, run_duration = run - - # Get steps for this run cursor.execute( """ - SELECT step, location_id, observation, choices, action, llm_response - FROM steps - WHERE run_id = ? - ORDER BY step - """, + SELECT id, quest_name, start_time, end_time, agent_id, treatment, treatment_signature, + outcome, reward, run_duration, usage, transcript_diagnostics, progress + FROM runs + WHERE id = ? + """, (run_id,), ) - steps = [] - step_count = 0 - success_choices = 0 - total_choices = 0 - - for step_data in cursor.fetchall(): - step_count += 1 - step_num, location_id, obs, choices_json, action, llm_response = step_data - choices = json.loads(choices_json) if choices_json else [] - total_choices += len(choices) - if len(choices) == 1: - success_choices += 1 - - step = { - "step": step_num, - "location_id": location_id, - "observation": obs, - "choices": choices, - "action": action, - "llm_response": json.loads(llm_response) if llm_response else None, - } - steps.append(step) + run = cursor.fetchone() + if not run: + typer.echo(f"Run ID {run_id} not found", err=True) + raise typer.Exit(code=1) + ( + run_id, + quest_name, + start_time, + end_time, + agent_id, + treatment_json, + treatment_signature, + outcome, + reward, + run_duration, + usage_json, + diagnostics_json, + progress_json, + ) = run + + treatment = json.loads(treatment_json) if treatment_json else {} + transitions = load_transitions(conn, run_id) + decision_points = sum(1 for t in transitions if len(t.before.choices) > 1) + + # Exported in canonical domains, matching run_summary.json. run_data = { - "run_id": run_id, - "quest_name": quest_name, - "start_time": start_time, - "end_time": end_time, - "agent_id": agent_id, - "agent_config": json.loads(agent_config) if agent_config else None, - "outcome": outcome, - "reward": reward, - "run_duration": run_duration, - "steps": steps, - "stats": { - "total_steps": step_count, - "total_choices": total_choices, - "auto_choices": success_choices, - "decision_points": total_choices - success_choices, + "schema_version": 2, + "run": { + "id": run_id, + "agent_id": agent_id, + "started_at": start_time, + "ended_at": end_time, + "duration": run_duration, }, + "quest": {"name": quest_name}, + "treatment": treatment, + "terminal": {"outcome": outcome, "reward": reward}, + "usage": json.loads(usage_json) if usage_json else {}, + "progress": json.loads(progress_json) if progress_json else {}, + "transcript_diagnostics": json.loads(diagnostics_json) if diagnostics_json else {}, + "transitions": [t.to_dict() for t in transitions], } # Export if requested if export: with open(export, "w") as f: - json.dump(run_data, f, indent=2) + json.dump(run_data, f, indent=2, ensure_ascii=False) typer.echo(f"Results exported to {export}") + duration_text = f"{run_duration:.2f}" if run_duration is not None else "n/a" + progress_text = f"{run_data['progress'].get('current', 0.0):.1f}%" + # Print human-readable summary based on format if format == "summary": typer.echo("\n📊 Run Summary") @@ -678,12 +662,14 @@ def analyze( typer.echo(f"Run ID: {run_id}") typer.echo(f"Quest: {quest_name}") typer.echo(f"Agent: {agent_id}") + typer.echo(f"Treatment: {treatment_signature}") typer.echo(f"Start Time: {start_time}") - typer.echo(f"Duration: {run_duration:.2f} seconds") + typer.echo(f"Duration: {duration_text} seconds") typer.echo(f"Outcome: {outcome}") typer.echo(f"Reward: {reward}") - typer.echo(f"Total Steps: {step_count}") - typer.echo(f"Decision Points: {total_choices - success_choices}") + typer.echo(f"Progress: {progress_text}") + typer.echo(f"Total Transitions: {len(transitions)}") + typer.echo(f"Decision Points: {decision_points}") elif format == "detail": typer.echo("\n📊 Run Details") @@ -693,48 +679,42 @@ def analyze( typer.echo(f"Agent: {agent_id}") typer.echo(f"Start Time: {start_time}") typer.echo(f"End Time: {end_time}") - typer.echo(f"Duration: {run_duration:.2f} seconds") + typer.echo(f"Duration: {duration_text} seconds") typer.echo(f"Outcome: {outcome}") typer.echo(f"Reward: {reward}") + typer.echo(f"Progress: {progress_text}") - # Agent config details if available - if run_data.get("agent_config"): - typer.echo("\nAgent Configuration:") - for key, value in run_data["agent_config"].items(): - typer.echo(f" {key}: {value}") + typer.echo("\nTreatment:") + for key, value in sorted(treatment.items()): + typer.echo(f" {key}: {value}") - # Step details - typer.echo(f"\nSteps ({len(steps)} total):") - for i, step in enumerate(steps, 1): - typer.echo(f"\n🔹 Step {i}:") - typer.echo(f" Location: {step['location_id']}") + typer.echo(f"\nTransitions ({len(transitions)} total):") + for transition in transitions: + typer.echo(f"\n🔹 Transition {transition.index}:") + typer.echo(f" Location: {transition.before.location_id}") - # Truncate observation for readability - obs = step["observation"] + obs = transition.before.observation if len(obs) > 100: obs = obs[:97] + "..." typer.echo(f" Observation: {obs}") - # Show choices - if step["choices"]: - typer.echo(f" Choices ({len(step['choices'])}):") - for j, choice in enumerate(step["choices"], 1): + if transition.before.choices: + typer.echo(f" Choices ({len(transition.before.choices)}):") + for j, choice in enumerate(transition.before.choices, 1): choice_text = choice["text"] if len(choice_text) > 50: choice_text = choice_text[:47] + "..." typer.echo(f" {j}. {choice_text}") - typer.echo(f" Action: {step['action']}") + typer.echo(f" Action: {_selected_label(transition, _choices_map(transition.before.choices))}") - # Show LLM reasoning if available and debug is enabled - if debug and step.get("llm_response"): - llm_resp = step["llm_response"] - if isinstance(llm_resp, dict) and llm_resp.get("reasoning"): - typer.echo(f" Reasoning: {llm_resp['reasoning']}") + if debug and transition.response and transition.response.reasoning: + typer.echo(f" Reasoning: {transition.response.reasoning}") elif format == "compact": typer.echo( - f"Run {run_id}: {quest_name} - {outcome} (Reward: {reward}) - Steps: {step_count} - Agent: {agent_id}" + f"Run {run_id}: {quest_name} - {outcome} (Reward: {reward}) - " + f"Transitions: {len(transitions)} - Agent: {agent_id}" ) # Analyze quest runs @@ -790,14 +770,21 @@ def analyze( typer.echo(f"\n🔸 Run {i} (ID: {run.get('id', 'unknown')}):") typer.echo(f" Start Time: {run['start_time']}") typer.echo(f" Agent: {run.get('model', 'unknown')}") + typer.echo(f" Harness: {run.get('harness', 'unknown')}") typer.echo(f" Outcome: {run['outcome']}") typer.echo(f" Reward: {run['reward']}") - # Only show steps in debug mode to avoid output overload - if debug and run.get("steps"): - typer.echo(f"\n Steps ({len(run['steps'])} total):") - for step in run["steps"]: - typer.echo(f" Step {step['step']}: Action {step['action']}") + # Only show transitions in debug mode to avoid output overload + if debug and run.get("transitions"): + typer.echo(f"\n Transitions ({len(run['transitions'])} total):") + for transition in run["transitions"]: + action = transition["action"] + label = ( + f"restore {action['checkpoint_index']}" + if action["kind"] == "restore" + else f"choose {action['choice_index']}" + ) + typer.echo(f" Transition {transition['index']}: {label}") # Analyze benchmark results else: @@ -823,7 +810,9 @@ def analyze( @app.command() def cleanup( - db_path: Path = typer.Option("metrics.db", help="Path to SQLite database file."), + db_path: Path | None = typer.Option( + None, help="Path to SQLite database file (defaults to $LLM_QUEST_DB_PATH, else metrics.db)." + ), older_than: str | None = typer.Option(None, help="ISO date (YYYY-MM-DD) to delete records older than this date."), all: bool = typer.Option(False, help="Delete all records from the database."), truncate_json: bool = typer.Option(False, help="Also remove JSON result files from results/ directory."), @@ -841,6 +830,7 @@ def cleanup( """ try: # Check if database exists + db_path = db_path or Path(default_db_path()) if not db_path.exists(): typer.echo(f"Database not found: {db_path}", err=True) raise typer.Exit(code=1) @@ -858,11 +848,11 @@ def cleanup( # Get initial counts cursor.execute("SELECT count(*) FROM runs") initial_runs = cursor.fetchone()[0] - cursor.execute("SELECT count(*) FROM steps") - initial_steps = cursor.fetchone()[0] + cursor.execute("SELECT count(*) FROM transitions") + initial_transitions = cursor.fetchone()[0] deleted_runs = 0 - deleted_steps = 0 + deleted_transitions = 0 # Delete by date if older_than: @@ -875,18 +865,18 @@ def cleanup( cursor.execute("SELECT id FROM runs WHERE start_time < ?", (cutoff_date.isoformat(),)) run_ids = [row[0] for row in cursor.fetchall()] - # Delete steps first + # Delete transitions first if run_ids: placeholders = ",".join("?" for _ in run_ids) - cursor.execute(f"DELETE FROM steps WHERE run_id IN ({placeholders})", run_ids) - deleted_steps = cursor.rowcount + cursor.execute(f"DELETE FROM transitions WHERE run_id IN ({placeholders})", run_ids) + deleted_transitions = cursor.rowcount # Then delete runs cursor.execute(f"DELETE FROM runs WHERE id IN ({placeholders})", run_ids) deleted_runs = cursor.rowcount conn.commit() - typer.echo(f"Deleted {deleted_runs} runs and {deleted_steps} steps") + typer.echo(f"Deleted {deleted_runs} runs and {deleted_transitions} transitions") except ValueError: typer.echo(f"Invalid date format: {older_than}. Use YYYY-MM-DD format.", err=True) @@ -896,16 +886,16 @@ def cleanup( elif all: typer.echo("Deleting all records from database") - # Delete steps first (due to foreign key constraints) - cursor.execute("DELETE FROM steps") - deleted_steps = cursor.rowcount + # Delete transitions first (due to foreign key constraints) + cursor.execute("DELETE FROM transitions") + deleted_transitions = cursor.rowcount # Then delete runs cursor.execute("DELETE FROM runs") deleted_runs = cursor.rowcount conn.commit() - typer.echo(f"Deleted {deleted_runs} runs and {deleted_steps} steps") + typer.echo(f"Deleted {deleted_runs} runs and {deleted_transitions} transitions") else: typer.echo("No action specified. Use --older-than or --all to specify what to delete.") @@ -935,12 +925,15 @@ def cleanup( # Print summary of changes cursor.execute("SELECT count(*) FROM runs") final_runs = cursor.fetchone()[0] - cursor.execute("SELECT count(*) FROM steps") - final_steps = cursor.fetchone()[0] + cursor.execute("SELECT count(*) FROM transitions") + final_transitions = cursor.fetchone()[0] typer.echo("\nSummary:") typer.echo(f"Runs: {initial_runs} -> {final_runs} ({initial_runs - final_runs} removed)") - typer.echo(f"Steps: {initial_steps} -> {final_steps} ({initial_steps - final_steps} removed)") + typer.echo( + f"Transitions: {initial_transitions} -> {final_transitions} " + f"({initial_transitions - final_transitions} removed)" + ) except Exception as e: typer.echo(f"Error during cleanup: {str(e)}", err=True) diff --git a/llm_quest_benchmark/executors/ts_bridge/bridge.py b/llm_quest_benchmark/executors/ts_bridge/bridge.py index d20794b..9388dd6 100644 --- a/llm_quest_benchmark/executors/ts_bridge/bridge.py +++ b/llm_quest_benchmark/executors/ts_bridge/bridge.py @@ -1,5 +1,6 @@ """TypeScript bridge for QM file parsing and execution""" +import hashlib import json import logging import os @@ -23,18 +24,21 @@ class QMBridge: """Bridge to TypeScript QM parser and executor""" - def __init__(self, quest_file: str, language: str = "rus", debug: bool = False): - """Initialize bridge with quest file path""" + def __init__(self, quest_file: str, language: str = "rus", debug: bool = False, seed: str | None = None): + """Initialize bridge with quest file path and deterministic engine seed.""" self.quest_file = Path(quest_file).resolve() self.language = language self.debug = debug self.process = None self.parser_script = Path(__file__).parent / "consoleplayer.ts" self.state_history: list[QMBridgeState] = [] + self.seed = seed # Validate quest file exists if not self.quest_file.exists(): raise FileNotFoundError(f"Quest file not found: {quest_file}") + if self.seed is None: + self.seed = hashlib.sha256(self.quest_file.read_bytes()).hexdigest() # Validate parser script exists if not self.parser_script.exists(): @@ -72,6 +76,7 @@ def _build_node_env(self) -> dict[str, str]: env["NODE_OPTIONS"] = f"{node_options} {legacy_flag}".strip() if self.language: env["QM_LANG"] = self.language + env["QM_SEED"] = str(self.seed) return env def _try_parse_json_object(self, line: str) -> dict[str, Any] | None: @@ -281,6 +286,43 @@ def _read_response_json(self, timeout: int = 10, require_state: bool = True) -> raise TimeoutError(f"Timed out waiting for valid bridge state packet. Last JSON keys: [{keys}].\n{details}") raise TimeoutError(f"Timeout waiting for JSON response from TypeScript bridge.\n{details}") + def _send_command(self, command: dict[str, Any]) -> None: + """Write one structured protocol command to the engine process.""" + if not self.process or not self.process.stdin: + raise RuntimeError("Game process not started") + self.process.stdin.write(json.dumps(command, ensure_ascii=False) + "\n") + self.process.stdin.flush() + + @staticmethod + def _state_from_response(response: dict[str, Any]) -> QMBridgeState: + """Build a bridge state from one protocol packet, validating required fields.""" + state = response.get("state") + saving = response.get("saving") + if not isinstance(state, dict) or not isinstance(saving, dict): + raise RuntimeError("Invalid response format: missing 'state' or 'saving' field") + if "text" not in state or "choices" not in state or "gameState" not in state: + raise RuntimeError("Invalid response format: missing required state fields") + if "locationId" not in saving: + raise RuntimeError("Invalid response format: missing saving.locationId") + + params_state_raw = state.get("paramsState") or [] + params_state = [clean_qm_text(p) for p in params_state_raw if isinstance(p, str) and clean_qm_text(p)] + game_state_raw = state.get("gameState", "running") + return QMBridgeState( + location_id=str(saving["locationId"]), + text=clean_qm_text(state.get("text", "")), + params_state=params_state, + choices=[ + {"id": str(c["jumpId"]), "text": clean_qm_text(c["text"])} + for c in (state.get("choices") or []) + if isinstance(c, dict) and c.get("active", False) + ], + reward=0.0, + game_ended=game_state_raw != "running", + game_state=game_state_raw, + saving=saving, + ) + def parse_quest_locations(self) -> dict[str, Any]: """Parse quest file and return metadata including locations and start location""" cmd = ["node", "-r", "ts-node/register", str(self.parser_script), str(self.quest_file), "--parse"] @@ -367,31 +409,7 @@ def start_game(self) -> QMBridgeState: if noise and self.debug: logger.debug("Skipped %d non-protocol stdout lines on start", len(noise)) - state = response.get("state") - saving = response.get("saving") - if not isinstance(state, dict) or not isinstance(saving, dict): - raise RuntimeError("Invalid response format: missing 'state' or 'saving' field") - if "text" not in state or "choices" not in state or "gameState" not in state: - raise RuntimeError("Invalid response format: missing required state fields") - if "locationId" not in saving: - raise RuntimeError("Invalid response format: missing saving.locationId") - - params_state_raw = state.get("paramsState") or [] - params_state = [clean_qm_text(p) for p in params_state_raw if isinstance(p, str) and clean_qm_text(p)] - game_state_raw = state.get("gameState", "running") - initial_state = QMBridgeState( - location_id=str(saving["locationId"]), - text=clean_qm_text(state.get("text", "")), - params_state=params_state, - choices=[ - {"id": str(c["jumpId"]), "text": clean_qm_text(c["text"])} - for c in (state.get("choices") or []) - if isinstance(c, dict) and c.get("active", False) - ], - reward=0.0, - game_ended=game_state_raw != "running", - game_state=game_state_raw, - ) + initial_state = self._state_from_response(response) if not initial_state.choices and not initial_state.game_ended: raise RuntimeError("No valid choices in initial state") @@ -409,38 +427,13 @@ def get_current_state(self) -> QMBridgeState: raise RuntimeError("Game not started") try: - self.process.stdin.write("get_state\n") - self.process.stdin.flush() + self._send_command({"cmd": "state"}) response_data, noise = self._read_protocol_message(timeout=10.0) if noise and self.debug: logger.debug("Skipped %d non-protocol stdout lines on get_state", len(noise)) - state = response_data.get("state") - saving = response_data.get("saving") - if not isinstance(state, dict) or not isinstance(saving, dict): - raise RuntimeError("Invalid response format: missing 'state' or 'saving' field") - if "text" not in state or "choices" not in state or "gameState" not in state: - raise RuntimeError("Invalid response format: missing required state fields") - if "locationId" not in saving: - raise RuntimeError("Invalid response format: missing saving.locationId") - - params_state_raw = state.get("paramsState") or [] - params_state = [clean_qm_text(p) for p in params_state_raw if isinstance(p, str) and clean_qm_text(p)] - game_state_raw = state.get("gameState", "running") - current_state = QMBridgeState( - location_id=str(saving["locationId"]), - text=clean_qm_text(state.get("text", "")), - params_state=params_state, - choices=[ - {"id": str(c["jumpId"]), "text": clean_qm_text(c["text"])} - for c in (state.get("choices") or []) - if isinstance(c, dict) and c.get("active", False) - ], - reward=0.0, - game_ended=game_state_raw != "running", - game_state=game_state_raw, - ) + current_state = self._state_from_response(response_data) if not current_state.choices and not current_state.game_ended: raise RuntimeError("No valid choices in current state") @@ -452,6 +445,27 @@ def get_current_state(self) -> QMBridgeState: self.close() # Clean up on error raise RuntimeError(f"Failed to get current state: {str(e)}") + def load_saving(self, saving: dict[str, Any]) -> QMBridgeState: + """Restore the engine to an exact recorded saving.""" + if not self.process: + raise RuntimeError("Game not started") + if not isinstance(saving, dict) or not saving: + raise ValueError("load_saving requires a non-empty engine saving") + + try: + self._send_command({"cmd": "load", "saving": saving}) + response_data, noise = self._read_protocol_message(timeout=10.0) + if noise and self.debug: + logger.debug("Skipped %d non-protocol stdout lines on load", len(noise)) + + restored = self._state_from_response(response_data) + self.state_history.append(restored) + return restored + except Exception as e: + logger.error(f"Failed to load saving: {str(e)}") + self.close() + raise RuntimeError(f"Failed to load saving: {str(e)}") + def validate_choice(self, choice_num: int) -> str | None: """Validate choice number and return corresponding jump ID""" current_state = self.state_history[-1] if self.state_history else self.get_current_state() @@ -472,8 +486,13 @@ def validate_choice(self, choice_num: int) -> str | None: return current_state.choices[choice_num - 1]["id"] - def step(self, choice_num: int) -> QMBridgeState: - """Take a step in the game with choice number (1-based)""" + def step(self, choice_num: int, performed_at_ms: int) -> QMBridgeState: + """Take a step in the game with choice number (1-based). + + ``performed_at_ms`` is the exact transition timestamp handed to the + engine's ``performJump``. It is recorded with the transition so replay + and resume reproduce dynamic quest behaviour exactly. + """ if not self.process: raise RuntimeError("Game not started") @@ -490,49 +509,13 @@ def step(self, choice_num: int) -> QMBridgeState: logger.debug(f"Current choices: {choices_debug}") logger.debug(f"Current choices raw: {current_state.choices}") - # Send jump ID to process - self.process.stdin.write(f"{jump_id}\n") - self.process.stdin.flush() - - # Read until we get a protocol JSON state. - try: - response_data, noise = self._read_protocol_message(timeout=10.0) - except TimeoutError: - logger.warning("No protocol response received from TypeScript bridge, trying get_state fallback") - return self.get_current_state() + self._send_command({"cmd": "jump", "jumpId": int(jump_id), "performedAtMs": int(performed_at_ms)}) + response_data, noise = self._read_protocol_message(timeout=10.0) if noise and self.debug: logger.debug("Skipped %d non-protocol stdout lines on step", len(noise)) - state = response_data.get("state") - saving = response_data.get("saving") - if not isinstance(state, dict) or not isinstance(saving, dict): - raise RuntimeError("Invalid response format: missing 'state' or 'saving' field") - if "text" not in state or "choices" not in state or "gameState" not in state: - raise RuntimeError("Invalid response format: missing required state fields") - if "locationId" not in saving: - raise RuntimeError("Invalid response format: missing saving.locationId") - - params_state_raw = state.get("paramsState") or [] - params_state = [clean_qm_text(p) for p in params_state_raw if isinstance(p, str) and clean_qm_text(p)] - - choices = [ - {"id": str(c["jumpId"]), "text": clean_qm_text(c["text"])} - for c in (state.get("choices") or []) - if isinstance(c, dict) and c.get("active", False) - ] - - game_state_raw = state.get("gameState", "running") - new_state = QMBridgeState( - location_id=str(saving["locationId"]), - text=clean_qm_text(state.get("text", "")), - params_state=params_state, - choices=choices, - reward=0.0, - game_ended=game_state_raw != "running", - game_state=game_state_raw, - ) - + new_state = self._state_from_response(response_data) self.state_history.append(new_state) return new_state diff --git a/llm_quest_benchmark/executors/ts_bridge/consoleplayer.ts b/llm_quest_benchmark/executors/ts_bridge/consoleplayer.ts index aed0828..df41a31 100644 --- a/llm_quest_benchmark/executors/ts_bridge/consoleplayer.ts +++ b/llm_quest_benchmark/executors/ts_bridge/consoleplayer.ts @@ -5,7 +5,7 @@ import { parse } from "../../../space-rangers-quest/src/lib/qmreader"; import * as fs from "fs"; import * as process from "process"; import { QMPlayer } from "../../../space-rangers-quest/src/lib/qmplayer"; -import { performJump } from "../../../space-rangers-quest/src/lib/qmplayer/funcs"; +import { initGame, performJump } from "../../../space-rangers-quest/src/lib/qmplayer/funcs"; // Get the quest file path and language from command line arguments if (process.argv.length < 3) { @@ -34,9 +34,24 @@ try { const qm = parse(data); -// Initialize player +// Initialize from an explicit stable seed. The bridge records timestamps for +// every jump; a stable initial PRNG state makes independent runs comparable. const player = new QMPlayer(qm, language as "rus" | "eng"); -player.start(); +const seed = process.env.QM_SEED || "llm-quest-default"; +player.loadSaving(initGame(qm, seed)); + +function emitState() { + console.log(JSON.stringify({ + state: player.getState(), + saving: player.getSaving() + })); +} + +function emitError(message: string) { + // Protocol errors go to stdout so the Python bridge fails fast instead of + // waiting for a state packet that will never arrive. + console.log(JSON.stringify({ error: message })); +} // If parse mode, output raw QM structure and exit if (parseMode) { @@ -55,12 +70,12 @@ if (parseMode) { } // Output initial raw state -console.log(JSON.stringify({ - state: player.getState(), - saving: player.getSaving() -})); +emitState(); -// Read commands and return raw state +// Structured command protocol: one JSON object per stdin line. +// {"cmd":"state"} +// {"cmd":"jump","jumpId":,"performedAtMs":} +// {"cmd":"load","saving":{...}} const rl = readline.createInterface({ input: process.stdin, output: process.stdout, @@ -68,34 +83,64 @@ const rl = readline.createInterface({ rl.on('line', (input) => { try { - const command = input.trim(); + const raw = input.trim(); + if (!raw) { + return; + } - // Handle special commands - if (command === "get_state") { - console.log(JSON.stringify({ - state: player.getState(), - saving: player.getSaving() - })); + let command: any; + try { + command = JSON.parse(raw); + } catch (parseError) { + emitError(`Malformed command JSON: ${raw}`); + return; + } + if (!command || typeof command !== "object") { + emitError("Command must be a JSON object"); return; } - // Try to perform jump - const jumpId = parseInt(command, 10); - if (!isNaN(jumpId)) { - // IMPORTANT: keep stdout protocol clean. - // space-rangers-quest performJump defaults showDebug=true, which can emit console.info lines - // (e.g. autojump logs) to stdout and break the Python bridge parser. - const currentSaving = player.getSaving(); - const nextSaving = performJump(jumpId, qm, currentSaving, Date.now(), false); - player.loadSaving(nextSaving); - console.log(JSON.stringify({ - state: player.getState(), - saving: player.getSaving() - })); - } else { - console.error(JSON.stringify({ error: "Invalid jump ID" })); + switch (command.cmd) { + case "state": { + emitState(); + return; + } + case "jump": { + const jumpId = Number(command.jumpId); + const performedAtMs = Number(command.performedAtMs); + if (!Number.isFinite(jumpId)) { + emitError("jump requires a numeric jumpId"); + return; + } + if (!Number.isFinite(performedAtMs)) { + // Deterministic replay depends on the caller-supplied timestamp, + // so the bridge never invents one. + emitError("jump requires a numeric performedAtMs"); + return; + } + // IMPORTANT: keep stdout protocol clean. + // space-rangers-quest performJump defaults showDebug=true, which can emit console.info lines + // (e.g. autojump logs) to stdout and break the Python bridge parser. + const nextSaving = performJump(jumpId, qm, player.getSaving(), performedAtMs, false); + player.loadSaving(nextSaving); + emitState(); + return; + } + case "load": { + if (!command.saving || typeof command.saving !== "object") { + emitError("load requires a saving object"); + return; + } + player.loadSaving(command.saving); + emitState(); + return; + } + default: { + emitError(`Unknown command: ${String(command.cmd)}`); + return; + } } } catch (error) { - console.error(JSON.stringify({ error: String(error) })); + emitError(String(error)); } }); diff --git a/llm_quest_benchmark/harnesses/adaptive.py b/llm_quest_benchmark/harnesses/adaptive.py new file mode 100644 index 0000000..b0f5de5 --- /dev/null +++ b/llm_quest_benchmark/harnesses/adaptive.py @@ -0,0 +1,109 @@ +"""Experimental adaptive-reasoning harness. + +Reasoning depth is a prompt-level policy: concise by default, deep once the +state repeats or curated progress stalls. It does not depend on any +provider-specific reasoning control. +""" + +from llm_quest_benchmark.constants import DEFAULT_MODEL, DEFAULT_TEMPERATURE, SYSTEM_ROLE_TEMPLATE +from llm_quest_benchmark.harnesses.base import BaseHarness +from llm_quest_benchmark.harnesses.memory import DefaultMemory +from llm_quest_benchmark.players.base import DecisionContext +from llm_quest_benchmark.schemas.records import QuestAction + +DEFAULT_ADAPTIVE_STALL_STEPS = 3 +MODE_CONCISE = "concise" +MODE_DEEP = "deep" + + +class AdaptiveReasoningHarness(BaseHarness): + """Recent-context harness that deepens its prompt only when triggered.""" + + harness_name = "adaptive_reasoning" + + def __init__( + self, + model_name: str = DEFAULT_MODEL, + system_template: str = SYSTEM_ROLE_TEMPLATE, + action_template: str = "adaptive_reasoning.jinja", + temperature: float = DEFAULT_TEMPERATURE, + skip_single: bool = False, + debug: bool = False, + adaptive_stall_steps: int | None = None, + memory_module=None, + **_, + ): + super().__init__( + model_name=model_name, + system_template=system_template, + action_template=action_template, + temperature=temperature, + skip_single=skip_single, + debug=debug, + memory_module=memory_module or DefaultMemory(), + ) + self.adaptive_stall_steps = ( + int(adaptive_stall_steps) if adaptive_stall_steps is not None else DEFAULT_ADAPTIVE_STALL_STEPS + ) + self.reasoning_mode = MODE_CONCISE + self._context = DecisionContext() + + def reset(self) -> None: + super().reset() + self.reasoning_mode = MODE_CONCISE + self._context = DecisionContext() + + def get_quest_action( + self, + observation: str, + choices: list[dict[str, str]], + context: DecisionContext, + ) -> QuestAction: + self._context = context + return super().get_quest_action(observation, choices, context) + + def _select_mode(self, state_signature: str) -> tuple[str, str | None]: + """Pick the reasoning mode and the trigger that caused it. + + Recovery is implicit: once neither trigger holds the next decision goes + back to concise mode while all decision history is retained. + """ + if self._state_action_counts.get(state_signature): + return MODE_DEEP, "repeated_state" + stalled = int(self._context.progress.stalled_transitions) + if stalled >= self.adaptive_stall_steps: + return MODE_DEEP, f"progress_stalled_{stalled}" + return MODE_CONCISE, None + + def _build_prompt(self, observation: str, choices: list[dict[str, str]], mode: str, trigger: str | None) -> str: + template = self.prompt_renderer.get_template(self.action_template) + return template.render( + observation=observation, + choices=[{"text": choice.get("text", "")} for choice in choices], + mode=mode, + trigger=trigger, + progress=self._context.progress.current, + ).strip() + + def _get_action_impl(self, observation: str, choices: list[dict[str, str]]) -> int: + try: + state_signature = self._state_signature(observation, choices) + mode, trigger = self._select_mode(state_signature) + self.reasoning_mode = mode + + contextual_state = self._build_contextual_state(observation) + prompt = self._build_prompt(contextual_state, choices, mode, trigger) + parsed_response = self._parse_with_retries(prompt, observation, choices) + if parsed_response.action < 1 or parsed_response.action > len(choices): + parsed_response.action = 1 + + self.history.append(parsed_response) + self._last_response = parsed_response + self._remember_decision(observation, choices, state_signature, parsed_response) + return parsed_response.action + except Exception as exc: + self.logger.error("Adaptive reasoning harness error during LLM call: %s", exc) + default_response = self._error_default_response(exc) + self.history.append(default_response) + self._last_response = default_response + return 1 diff --git a/llm_quest_benchmark/harnesses/backtracking.py b/llm_quest_benchmark/harnesses/backtracking.py new file mode 100644 index 0000000..6597428 --- /dev/null +++ b/llm_quest_benchmark/harnesses/backtracking.py @@ -0,0 +1,181 @@ +"""Experimental backtracking harness. + +This is the only harness whose specification declares the choose-or-restore +loop. Restores are budgeted by ``restore_limit`` and remain visible as +transitions in the persisted record. +""" + +from typing import Any + +from llm_quest_benchmark.constants import DEFAULT_MODEL, DEFAULT_TEMPERATURE, SYSTEM_ROLE_TEMPLATE +from llm_quest_benchmark.harnesses.base import BaseHarness, _parse_json_response +from llm_quest_benchmark.harnesses.memory import CompactionMemory +from llm_quest_benchmark.players.base import DecisionContext +from llm_quest_benchmark.schemas.records import QuestAction, QuestSnapshot + +DEFAULT_RESTORE_LIMIT = 3 +CHECKPOINT_SUMMARY_CHARS = 140 + + +def parse_restore_request(response: str, max_checkpoint: int) -> int | None: + """Extract a valid restore checkpoint index from a model response. + + Returns None when the model did not ask for a restore or asked for an index + outside the active branch. + """ + if max_checkpoint < 1: + return None + + payload, _ = _parse_json_response(response) + if not isinstance(payload, dict): + return None + + requested = payload.get("checkpoint") + if str(payload.get("action") or "").strip().lower() != "restore": + # A bare {"restore": N} is also accepted. + requested = payload.get("restore", None) if "restore" in payload else None + if requested is None: + return None + + try: + index = int(requested) + except (TypeError, ValueError): + return None + if 1 <= index <= max_checkpoint: + return index + return None + + +def summarize_checkpoint(index: int, snapshot: QuestSnapshot) -> str: + """One-line checkpoint description shown to the model.""" + text = " ".join((snapshot.observation or "").split()) + if len(text) > CHECKPOINT_SUMMARY_CHARS: + text = text[:CHECKPOINT_SUMMARY_CHARS] + "..." + params = "; ".join(snapshot.params_state[:4]) + suffix = f" | {params}" if params else "" + return f"[{index}] location {snapshot.location_id}: {text}{suffix}" + + +class BacktrackingHarness(BaseHarness): + """Compacted-memory harness that may choose an option or restore a checkpoint.""" + + harness_name = "backtracking" + supports_restore = True + + def __init__( + self, + model_name: str = DEFAULT_MODEL, + system_template: str = SYSTEM_ROLE_TEMPLATE, + action_template: str = "backtracking.jinja", + temperature: float = DEFAULT_TEMPERATURE, + skip_single: bool = False, + debug: bool = False, + compaction_interval: int = 50, + restore_limit: int | None = None, + memory_module=None, + **_, + ): + super().__init__( + model_name=model_name, + system_template=system_template, + action_template=action_template, + temperature=temperature, + skip_single=skip_single, + debug=debug, + memory_module=memory_module or CompactionMemory(compaction_interval=compaction_interval), + ) + self.restore_limit = int(restore_limit) if restore_limit is not None else DEFAULT_RESTORE_LIMIT + self._compaction_interval = compaction_interval + self._context = DecisionContext() + self._pending_restore: int | None = None + + def reset(self) -> None: + super().reset() + self._context = DecisionContext() + self._pending_restore = None + + def get_quest_action( + self, + observation: str, + choices: list[dict[str, str]], + context: DecisionContext, + ) -> QuestAction: + """Choose a current option or restore a recorded checkpoint.""" + self._context = context + self._pending_restore = None + action_index = self.get_action(observation, choices) + if self._pending_restore is not None: + checkpoint = self._pending_restore + self._pending_restore = None + return QuestAction.restore(checkpoint) + return QuestAction(kind="choose", choice_index=action_index) + + def _restorable_checkpoints(self) -> list[str]: + # The last checkpoint is the current state, so it is never restorable. + return [ + summarize_checkpoint(index, snapshot) + for index, snapshot in enumerate(self._context.checkpoints[:-1], start=1) + ] + + def _build_prompt(self, observation: str, choices: list[dict[str, str]]) -> str: + template = self.prompt_renderer.get_template(self.action_template) + return template.render( + observation=observation, + choices=[{"text": choice.get("text", "")} for choice in choices], + checkpoints=self._restorable_checkpoints(), + restore_allowed=self._context.restore_allowed, + restores_remaining=self._context.restores_remaining, + ).strip() + + def _get_action_impl(self, observation: str, choices: list[dict[str, str]]) -> int: + try: + state_signature = self._state_signature(observation, choices) + contextual_state = self._build_contextual_state(observation) + prompt = self._build_prompt(contextual_state, choices) + + raw_response = self._call_llm(prompt) + usage: dict[str, Any] = self.llm.get_last_usage() + + max_checkpoint = max(0, len(self._context.checkpoints) - 1) + checkpoint = None + if self._context.restore_allowed: + checkpoint = parse_restore_request(raw_response, max_checkpoint) + + parsed_response = self._parse_llm_response(raw_response, len(choices)) + if checkpoint is None and parsed_response.is_default: + retry_raw = self._call_llm(self._format_retry_prompt(observation, choices)) + usage = self._merge_usage(usage, self.llm.get_last_usage()) + retry_parsed = self._parse_llm_response(retry_raw, len(choices)) + if not retry_parsed.is_default: + retry_parsed.parse_mode = f"retry_{retry_parsed.parse_mode or 'parsed'}" + parsed_response = retry_parsed + + if checkpoint is None: + action_before_policy = parsed_response.action + parsed_response.action = self._apply_safety_filter(choices, parsed_response.action) + if parsed_response.action != action_before_policy and not parsed_response.reasoning: + parsed_response.reasoning = "policy_safety_override" + else: + self._pending_restore = checkpoint + parsed_response.parse_mode = "restore" + parsed_response.reasoning = parsed_response.reasoning or f"restore_checkpoint_{checkpoint}" + + if parsed_response.action < 1 or parsed_response.action > len(choices): + parsed_response.action = 1 + + usage_payload = self._normalize_usage(usage) + parsed_response.prompt_tokens = usage_payload["prompt_tokens"] + parsed_response.completion_tokens = usage_payload["completion_tokens"] + parsed_response.total_tokens = usage_payload["total_tokens"] + parsed_response.estimated_cost_usd = usage_payload["estimated_cost_usd"] + + self.history.append(parsed_response) + self._last_response = parsed_response + self._remember_decision(observation, choices, state_signature, parsed_response) + return parsed_response.action + except Exception as exc: + self.logger.error("Backtracking harness error during LLM call: %s", exc) + default_response = self._error_default_response(exc) + self.history.append(default_response) + self._last_response = default_response + return 1 diff --git a/llm_quest_benchmark/harnesses/base.py b/llm_quest_benchmark/harnesses/base.py index fd8864b..b6dd348 100644 --- a/llm_quest_benchmark/harnesses/base.py +++ b/llm_quest_benchmark/harnesses/base.py @@ -13,6 +13,7 @@ from llm_quest_benchmark.llm.client import get_llm_client, parse_model_name from llm_quest_benchmark.llm.prompt import PromptRenderer from llm_quest_benchmark.players.base import QuestPlayer +from llm_quest_benchmark.schemas.records import QuestSnapshot, QuestTransition from llm_quest_benchmark.schemas.response import LLMResponse RISKY_CHOICE_KEYWORDS = ( @@ -194,22 +195,28 @@ def parse_llm_response( memo_raw = response_json.get("memo") memo = str(memo_raw) if memo_raw is not None else None - action_value = response_json.get("action") or response_json.get("result") or response_json.get("choice") - if action_value is not None: + # "action" may name the action kind rather than the choice (the + # backtracking harness answers {"action":"restore","result":2}), so each + # candidate key is tried in turn and the first valid choice number wins. + for key in ("action", "result", "choice"): + action_value = response_json.get(key) + if action_value is None: + continue try: action = int(action_value) - if _validate_action_number(action, num_choices, debug, logger): - return LLMResponse( - action=action, - reasoning=reasoning, - analysis=analysis, - memo=memo, - is_default=False, - parse_mode=json_parse_mode or "json", - ) except (ValueError, TypeError): if debug and logger: - logger.error("Invalid action value in JSON: %s", action_value) + logger.debug("Non-numeric '%s' value in JSON: %s", key, action_value) + continue + if _validate_action_number(action, num_choices, debug, logger): + return LLMResponse( + action=action, + reasoning=reasoning, + analysis=analysis, + memo=memo, + is_default=False, + parse_mode=json_parse_mode or "json", + ) try: action = int(response.strip()) @@ -249,6 +256,8 @@ def parse_llm_response( class BaseHarness(QuestPlayer): """Abstract LLM harness base class.""" + OBSERVATION_HISTORY_LIMIT = 20 + def __init__( self, model_name, @@ -323,25 +332,59 @@ def reset(self) -> None: if self.memory_module is not None: self.memory_module.reset() - def get_action(self, observation: str, choices: list[dict[str, str]]) -> int: + def _error_default_response(self, exc: Exception) -> LLMResponse: + """Fallback response recorded on provider failure; harness names the marker.""" + return LLMResponse( + action=1, + is_default=True, + parse_mode="error_default", + reasoning=f"{self.harness_name}_error: {exc}", + ) + + def _remember_observation(self, observation: str) -> None: clean = (observation or "").strip() if clean: self._observation_history.append(clean) - if len(self._observation_history) > 20: - self._observation_history = self._observation_history[-20:] + del self._observation_history[: -self.OBSERVATION_HISTORY_LIMIT] + + def get_action(self, observation: str, choices: list[dict[str, str]]) -> int: + self._remember_observation(observation) return super().get_action(observation, choices) def on_game_start(self) -> None: super().on_game_start() self.reset() - def on_game_end(self, final_state: dict[str, Any]) -> None: + def on_game_end(self, final_snapshot: QuestSnapshot | None) -> None: if self.debug: - self.logger.debug("Game ended with state: %s", final_state) + self.logger.debug("Game ended with state: %s", final_snapshot) def get_last_response(self) -> LLMResponse | None: return self._last_response + def rebuild_from_transitions(self, transitions: list[QuestTransition]) -> None: + """Replay recorded decisions into harness memory without model calls. + + Resume must continue with the same memory the interrupted run had, so + each recorded choose transition is folded back through the same + bookkeeping path a live decision uses. + """ + for transition in transitions: + self.on_transition(transition) + if not transition.action.is_choose or transition.response is None: + continue + observation = transition.before.agent_observation() + choices = transition.before.choices + self._remember_observation(observation) + self.history.append(transition.response) + self._last_response = transition.response + self._remember_decision( + observation, + choices, + self._state_signature(observation, choices), + transition.response, + ) + def _build_contextual_state(self, state: str) -> str: if self.memory_module is None: return state diff --git a/llm_quest_benchmark/harnesses/factory.py b/llm_quest_benchmark/harnesses/factory.py index 561f657..ab518ad 100644 --- a/llm_quest_benchmark/harnesses/factory.py +++ b/llm_quest_benchmark/harnesses/factory.py @@ -1,6 +1,8 @@ """Factory for creating harness-based quest players.""" from llm_quest_benchmark.constants import DEFAULT_MODEL +from llm_quest_benchmark.harnesses.adaptive import AdaptiveReasoningHarness +from llm_quest_benchmark.harnesses.backtracking import BacktrackingHarness from llm_quest_benchmark.harnesses.memo import ( CompactionNoMemoHarness, HintedCompactHarness, @@ -12,12 +14,28 @@ from llm_quest_benchmark.harnesses.minimal import MinimalHarness from llm_quest_benchmark.harnesses.planner import PlannerHarness from llm_quest_benchmark.harnesses.reasoning import ReasoningFullTranscriptHarness, ReasoningRecentHarness +from llm_quest_benchmark.harnesses.specs import ( + HARNESS_SPECS, + SPECIAL_HARNESSES, + is_random_choice_harness, + parse_random_choice_seed, + valid_harness_names, +) from llm_quest_benchmark.harnesses.tool_harness import ProgrammaticMemoryHarness, ToolCompactHarness, ToolHintedHarness from llm_quest_benchmark.players.base import QuestPlayer from llm_quest_benchmark.players.human import HumanPlayer from llm_quest_benchmark.players.random import RandomPlayer -HARNESS_REGISTRY = { +__all__ = [ + "HARNESS_CLASSES", + "HARNESS_SPECS", + "SPECIAL_HARNESSES", + "create_harness", + "is_random_choice_harness", +] + +# Implementation for each model-driven harness specification. +HARNESS_CLASSES = { "minimal": MinimalHarness, "reasoning_recent": ReasoningRecentHarness, "reasoning_full": ReasoningFullTranscriptHarness, @@ -31,24 +49,10 @@ "memo_cot": MemoCotHarness, "memo_extended": MemoExtendedHarness, "memo_structured": MemoStructuredHarness, + "backtracking": BacktrackingHarness, + "adaptive_reasoning": AdaptiveReasoningHarness, } -SPECIAL_HARNESSES = ("human", "random_choice", "random_choice_") - - -def _parse_random_choice_seed(identifier: str) -> tuple[bool, int | None]: - if identifier == "random_choice": - return True, None - prefix = "random_choice_" - if identifier.startswith(prefix) and identifier[len(prefix) :].isdigit(): - return True, int(identifier[len(prefix) :]) - return False, None - - -def is_random_choice_harness(identifier: str) -> bool: - is_random, _ = _parse_random_choice_seed(identifier) - return is_random - def create_harness( harness: str, @@ -58,10 +62,12 @@ def create_harness( debug: bool = False, compaction_interval: int = 50, system_template: str = "system_role.jinja", + restore_limit: int | None = None, + adaptive_stall_steps: int | None = None, ) -> QuestPlayer: - valid = [*sorted(HARNESS_REGISTRY), *SPECIAL_HARNESSES] - is_random_harness, seed = _parse_random_choice_seed(harness) - is_random_model, _ = _parse_random_choice_seed(model) + valid = valid_harness_names() + is_random_harness, seed = parse_random_choice_seed(harness) + is_random_model, _ = parse_random_choice_seed(model) if is_random_harness: if is_random_model and model != "random_choice": raise ValueError("Encode random seeds in harness, for example harness='random_choice_123'") @@ -72,7 +78,7 @@ def create_harness( raise ValueError(f"Unknown harness '{harness}'. Valid: {valid}") if harness == "human": return HumanPlayer(skip_single=skip_single) - if harness not in HARNESS_REGISTRY: + if harness not in HARNESS_CLASSES: raise ValueError(f"Unknown harness '{harness}'. Valid: {valid}") if is_random_model: raise ValueError( @@ -82,12 +88,18 @@ def create_harness( raise ValueError(f"Unknown random_choice model '{model}'. Valid: {valid}") if model == "human": raise ValueError("Use harness='human' for human runs instead of pairing human model with an LLM harness") - cls = HARNESS_REGISTRY[harness] - return cls( - model_name=model, - temperature=temperature, - skip_single=skip_single, - debug=debug, - compaction_interval=compaction_interval, - system_template=system_template, - ) + + spec = HARNESS_SPECS[harness] + kwargs = { + "model_name": model, + "temperature": temperature, + "skip_single": skip_single, + "debug": debug, + "compaction_interval": compaction_interval, + "system_template": system_template, + } + if "restore_limit" in spec.knobs and restore_limit is not None: + kwargs["restore_limit"] = restore_limit + if "adaptive_stall_steps" in spec.knobs and adaptive_stall_steps is not None: + kwargs["adaptive_stall_steps"] = adaptive_stall_steps + return HARNESS_CLASSES[harness](**kwargs) diff --git a/llm_quest_benchmark/harnesses/planner.py b/llm_quest_benchmark/harnesses/planner.py index 810440c..c96d390 100644 --- a/llm_quest_benchmark/harnesses/planner.py +++ b/llm_quest_benchmark/harnesses/planner.py @@ -192,7 +192,7 @@ def on_game_start(self) -> None: self.current_plan = None self._plan_history = [] - def on_game_end(self, final_state: dict[str, Any]) -> None: + def on_game_end(self, final_snapshot) -> None: if self.debug: logging.getLogger(self.__class__.__name__).debug("Planner finished with plan: %s", self.current_plan) - super().on_game_end(final_state) + super().on_game_end(final_snapshot) diff --git a/llm_quest_benchmark/harnesses/specs.py b/llm_quest_benchmark/harnesses/specs.py new file mode 100644 index 0000000..163e1e7 --- /dev/null +++ b/llm_quest_benchmark/harnesses/specs.py @@ -0,0 +1,357 @@ +"""Canonical harness specifications and treatment signatures. + +A harness specification declares the material components of a treatment: +prompt, memory, tools, loop, and reasoning policy. Reports read those +components directly instead of inferring them from harness names. +""" + +from __future__ import annotations + +import hashlib +from dataclasses import dataclass, field +from typing import Any + +from llm_quest_benchmark.schemas.records import canonical_json + +# Memory policies. +MEMORY_NONE = "none" +MEMORY_RECENT_WINDOW = "recent_window" +MEMORY_FULL_TRANSCRIPT = "full_transcript" +MEMORY_COMPACTION = "compaction" + +# Decision loop policies. +LOOP_SINGLE_CALL = "single_call" +LOOP_TOOL_SELECT_ACT = "tool_select_act" +LOOP_PLAN_ACT = "plan_act" +LOOP_CHOOSE_OR_RESTORE = "choose_or_restore" +LOOP_ADAPTIVE_DEPTH = "adaptive_depth" +LOOP_INTERACTIVE = "interactive" +LOOP_RANDOM = "random" + +# Reasoning policies. +REASONING_NONE = "none" +REASONING_CONCISE = "concise" +REASONING_CHAIN_OF_THOUGHT = "chain_of_thought" +REASONING_STRUCTURED = "structured" +REASONING_PLAN = "plan" +REASONING_ADAPTIVE = "adaptive" + +COMPACTION_TOOLS = ("calculator", "scratchpad", "quest_history") +PROGRAMMATIC_TOOLS = ("calculator", "scratchpad", "history_read", "history_search") + + +@dataclass(frozen=True) +class HarnessSpec: + """Material description of one harness treatment.""" + + name: str + prompt: str + memory: str + loop: str + reasoning: str + tools: tuple[str, ...] = () + knobs: tuple[str, ...] = () + experimental: bool = False + requires_model: bool = True + + def to_dict(self) -> dict[str, Any]: + return { + "harness": self.name, + "prompt": self.prompt, + "memory": self.memory, + "tools": list(self.tools), + "loop": self.loop, + "reasoning": self.reasoning, + } + + +HARNESS_SPECS: dict[str, HarnessSpec] = { + "minimal": HarnessSpec( + name="minimal", + prompt="stub.jinja", + memory=MEMORY_RECENT_WINDOW, + loop=LOOP_SINGLE_CALL, + reasoning=REASONING_NONE, + ), + "reasoning_recent": HarnessSpec( + name="reasoning_recent", + prompt="reasoning.jinja", + memory=MEMORY_RECENT_WINDOW, + loop=LOOP_SINGLE_CALL, + reasoning=REASONING_CONCISE, + ), + "reasoning_full": HarnessSpec( + name="reasoning_full", + prompt="reasoning.jinja", + memory=MEMORY_FULL_TRANSCRIPT, + loop=LOOP_SINGLE_CALL, + reasoning=REASONING_CONCISE, + ), + "memo_compact": HarnessSpec( + name="memo_compact", + prompt="stateful_compact.jinja", + memory=MEMORY_COMPACTION, + loop=LOOP_SINGLE_CALL, + reasoning=REASONING_CONCISE, + knobs=("compaction_interval",), + ), + "hinted_compact": HarnessSpec( + name="hinted_compact", + prompt="stateful_compact_hints.jinja", + memory=MEMORY_COMPACTION, + loop=LOOP_SINGLE_CALL, + reasoning=REASONING_CONCISE, + knobs=("compaction_interval",), + ), + "tool_compact": HarnessSpec( + name="tool_compact", + prompt="tool_augmented.jinja", + memory=MEMORY_COMPACTION, + loop=LOOP_TOOL_SELECT_ACT, + reasoning=REASONING_CONCISE, + tools=COMPACTION_TOOLS, + knobs=("compaction_interval",), + ), + "tool_hinted": HarnessSpec( + name="tool_hinted", + prompt="tool_augmented_hints.jinja", + memory=MEMORY_COMPACTION, + loop=LOOP_TOOL_SELECT_ACT, + reasoning=REASONING_CONCISE, + tools=COMPACTION_TOOLS, + knobs=("compaction_interval",), + ), + "programmatic_memory": HarnessSpec( + name="programmatic_memory", + prompt="programmatic_memory.jinja", + memory=MEMORY_RECENT_WINDOW, + loop=LOOP_TOOL_SELECT_ACT, + reasoning=REASONING_CONCISE, + tools=PROGRAMMATIC_TOOLS, + ), + "planner": HarnessSpec( + name="planner", + prompt="planner.jinja", + memory=MEMORY_COMPACTION, + loop=LOOP_PLAN_ACT, + reasoning=REASONING_PLAN, + knobs=("compaction_interval",), + ), + "compaction_no_memo": HarnessSpec( + name="compaction_no_memo", + prompt="reasoning.jinja", + memory=MEMORY_COMPACTION, + loop=LOOP_SINGLE_CALL, + reasoning=REASONING_CONCISE, + knobs=("compaction_interval",), + ), + "memo_cot": HarnessSpec( + name="memo_cot", + prompt="memo_cot.jinja", + memory=MEMORY_COMPACTION, + loop=LOOP_SINGLE_CALL, + reasoning=REASONING_CHAIN_OF_THOUGHT, + knobs=("compaction_interval",), + ), + "memo_extended": HarnessSpec( + name="memo_extended", + prompt="memo_extended.jinja", + memory=MEMORY_COMPACTION, + loop=LOOP_SINGLE_CALL, + reasoning=REASONING_CONCISE, + knobs=("compaction_interval",), + ), + "memo_structured": HarnessSpec( + name="memo_structured", + prompt="memo_structured.jinja", + memory=MEMORY_COMPACTION, + loop=LOOP_SINGLE_CALL, + reasoning=REASONING_STRUCTURED, + knobs=("compaction_interval",), + ), + "backtracking": HarnessSpec( + name="backtracking", + prompt="backtracking.jinja", + memory=MEMORY_COMPACTION, + loop=LOOP_CHOOSE_OR_RESTORE, + reasoning=REASONING_CONCISE, + knobs=("compaction_interval", "restore_limit"), + experimental=True, + ), + "adaptive_reasoning": HarnessSpec( + name="adaptive_reasoning", + prompt="adaptive_reasoning.jinja", + memory=MEMORY_RECENT_WINDOW, + loop=LOOP_ADAPTIVE_DEPTH, + reasoning=REASONING_ADAPTIVE, + knobs=("adaptive_stall_steps",), + experimental=True, + ), + "human": HarnessSpec( + name="human", + prompt="none", + memory=MEMORY_NONE, + loop=LOOP_INTERACTIVE, + reasoning=REASONING_NONE, + requires_model=False, + ), + "random_choice": HarnessSpec( + name="random_choice", + prompt="none", + memory=MEMORY_NONE, + loop=LOOP_RANDOM, + reasoning=REASONING_NONE, + knobs=("seed",), + requires_model=False, + ), +} + +# Harnesses that support restoring a recorded checkpoint. +RESTORE_HARNESSES = frozenset(name for name, spec in HARNESS_SPECS.items() if spec.loop == LOOP_CHOOSE_OR_RESTORE) + +# Knobs that are only meaningful for a specific harness. +EXCLUSIVE_KNOBS = { + "restore_limit": "backtracking", + "adaptive_stall_steps": "adaptive_reasoning", +} + +LLM_HARNESS_NAMES = tuple(sorted(name for name, spec in HARNESS_SPECS.items() if spec.requires_model)) +SPECIAL_HARNESSES = ("human", "random_choice", "random_choice_") + + +def parse_random_choice_seed(identifier: str) -> tuple[bool, int | None]: + """Return ``(is_random_choice, seed)`` for a harness or model identifier.""" + if identifier == "random_choice": + return True, None + prefix = "random_choice_" + if identifier.startswith(prefix) and identifier[len(prefix) :].isdigit(): + return True, int(identifier[len(prefix) :]) + return False, None + + +def is_random_choice_harness(identifier: str) -> bool: + is_random, _ = parse_random_choice_seed(identifier) + return is_random + + +def get_spec(harness: str) -> HarnessSpec: + """Resolve a harness identifier to its canonical specification.""" + is_random, _ = parse_random_choice_seed(harness) + if is_random: + return HARNESS_SPECS["random_choice"] + spec = HARNESS_SPECS.get(harness) + if spec is None: + valid = [*sorted(HARNESS_SPECS), "random_choice_"] + raise ValueError(f"Unknown harness '{harness}'. Valid: {valid}") + return spec + + +def valid_harness_names() -> list[str]: + return [*sorted(HARNESS_SPECS), "random_choice_"] + + +@dataclass +class HarnessTreatment: + """Canonical, hashable description of one executed treatment.""" + + harness: str + prompt: str + memory: str + tools: list[str] + loop: str + reasoning: str + model: str + temperature: float + system_prompt: str + knobs: dict[str, Any] = field(default_factory=dict) + + def canonical_payload(self) -> dict[str, Any]: + return { + "harness": self.harness, + "prompt": self.prompt, + "memory": self.memory, + "tools": sorted(self.tools), + "loop": self.loop, + "reasoning": self.reasoning, + "model": self.model, + "temperature": round(float(self.temperature), 6), + "system_prompt": self.system_prompt, + "knobs": {k: self.knobs[k] for k in sorted(self.knobs)}, + } + + @property + def signature(self) -> str: + digest = hashlib.sha256(canonical_json(self.canonical_payload()).encode("utf-8")).hexdigest() + return f"t2_{digest[:16]}" + + def to_dict(self) -> dict[str, Any]: + payload = self.canonical_payload() + payload["signature"] = self.signature + return payload + + @classmethod + def unknown(cls, harness: str, model: str, temperature: float) -> HarnessTreatment: + """Treatment for a configuration whose components cannot be resolved. + + Migration uses this for legacy records; it records an explicit + ``unknown`` component rather than guessing a compatibility alias. + """ + return cls( + harness=harness or "unknown", + prompt="unknown", + memory="unknown", + tools=[], + loop="unknown", + reasoning="unknown", + model=model or "unknown", + temperature=float(temperature or 0.0), + system_prompt="unknown", + knobs={}, + ) + + +def build_treatment( + harness: str, + model: str, + temperature: float, + system_template: str, + knob_values: dict[str, Any] | None = None, +) -> HarnessTreatment: + """Build the canonical treatment for a harness configuration. + + Only knobs declared material by the harness specification enter the + signature, so an irrelevant knob can never split otherwise-equal runs. + """ + spec = get_spec(harness) + knob_values = knob_values or {} + knobs: dict[str, Any] = {} + for knob in spec.knobs: + value = knob_values.get(knob) + if value is not None: + knobs[knob] = value + + is_random, seed = parse_random_choice_seed(harness) + if is_random and seed is not None: + knobs["seed"] = seed + + return HarnessTreatment( + harness=harness, + prompt=spec.prompt, + memory=spec.memory, + tools=list(spec.tools), + loop=spec.loop, + reasoning=spec.reasoning, + model=model, + temperature=float(temperature), + system_prompt=system_template if spec.requires_model else "none", + knobs=knobs, + ) + + +def validate_exclusive_knobs(harness: str, knob_values: dict[str, Any]) -> None: + """Reject harness-exclusive knobs supplied for a different harness.""" + for knob, owner in EXCLUSIVE_KNOBS.items(): + if knob_values.get(knob) is None: + continue + if harness != owner: + raise ValueError(f"{knob} is only valid for harness: {owner}, got harness: {harness}") diff --git a/llm_quest_benchmark/harnesses/tool_harness.py b/llm_quest_benchmark/harnesses/tool_harness.py index 0e45ffd..cd3942f 100644 --- a/llm_quest_benchmark/harnesses/tool_harness.py +++ b/llm_quest_benchmark/harnesses/tool_harness.py @@ -7,8 +7,8 @@ from llm_quest_benchmark.harnesses.memory import CompactionMemory, DefaultMemory from llm_quest_benchmark.harnesses.tools import QuestHistoryTool, Scratchpad, calculator from llm_quest_benchmark.harnesses.trajectory import MAX_READ_COUNT, MAX_SEARCH_RESULTS, Trajectory +from llm_quest_benchmark.schemas.records import QuestTransition from llm_quest_benchmark.schemas.response import LLMResponse -from llm_quest_benchmark.schemas.state import AgentState class ToolCompactHarness(BaseHarness): @@ -252,8 +252,8 @@ class ProgrammaticMemoryHarness(ToolCompactHarness): duplicated invariant. Memory, tool set, and step bookkeeping are overridden: this harness carries no compaction and no clipped step log. `DefaultMemory` is the single bounded recent-context source in the prompt; the trajectory - is an on-demand retrieval view of the canonical `AgentState` objects emitted - by the runner after each executed decision. + is an on-demand retrieval view of the canonical `QuestTransition` objects + emitted by the runner after each executed decision. """ harness_name = "programmatic_memory" @@ -285,9 +285,9 @@ def __init__( tools=[calculator, self._scratchpad_tool, self._trajectory], ) - def on_step(self, agent_state: AgentState) -> None: - """Index the runner's canonical executed decision for retrieval.""" - self._trajectory.append(agent_state) + def on_transition(self, transition: QuestTransition) -> None: + """Index the runner's canonical executed transition for retrieval.""" + self._trajectory.append(transition) def _tool_descriptions(self) -> list[str]: return [ @@ -385,8 +385,8 @@ def _execute_tool_calls(self, tool_calls: list[dict[str, Any]]) -> list[str]: return results def _log_step(self, observation: str, choices: list[dict[str, str]], response: LLMResponse) -> None: - # The runner emits the canonical AgentState to on_step after env.step; - # this harness has no separate clipped step log to populate here. + # The runner emits the canonical QuestTransition to on_transition after + # env.step; this harness has no separate clipped step log to populate here. pass def reset(self) -> None: diff --git a/llm_quest_benchmark/harnesses/trajectory.py b/llm_quest_benchmark/harnesses/trajectory.py index 0f91149..5a86217 100644 --- a/llm_quest_benchmark/harnesses/trajectory.py +++ b/llm_quest_benchmark/harnesses/trajectory.py @@ -1,16 +1,16 @@ -"""Bounded retrieval over canonical executed quest steps. +"""Bounded retrieval over canonical executed quest transitions. This is an in-memory retrieval/index view for the ``programmatic_memory`` -harness only. The runner creates each ``AgentState`` once, delivers that exact -object to the harness and callbacks, then ``QuestLogger`` serializes it to the -persisted ``run_summary.json`` trajectory. This component stores references to -those canonical executed-decision objects for one episode; it creates no -parallel step representation and writes nothing to disk. +harness only. The runner creates each ``QuestTransition`` once, delivers that +exact object to the harness and callbacks, then ``QuestLogger`` serializes it to +the persisted schema-v2 record. This component stores references to those +canonical transitions for one episode; it creates no parallel step +representation and writes nothing to disk. """ import re -from llm_quest_benchmark.schemas.state import AgentState +from llm_quest_benchmark.schemas.records import QuestTransition MAX_READ_COUNT = 6 MAX_SEARCH_RESULTS = 5 @@ -43,29 +43,37 @@ def _coerce_positive_int(value) -> int | None: return None -def _selected_choice(entry: AgentState) -> str: - """Return the selected choice text from a canonical decision state.""" - try: - action = int(entry.action) - except (TypeError, ValueError): +def _action_label(entry: QuestTransition) -> str: + """Render the executed action for retrieval output.""" + if entry.action.is_restore: + return f"restore checkpoint {entry.action.checkpoint_index}" + return str(entry.action.choice_index) + + +def _selected_choice(entry: QuestTransition) -> str: + """Return the selected choice text from a canonical transition.""" + if not entry.action.is_choose: return "" - if 1 <= action <= len(entry.choices): - return entry.choices[action - 1].get("text", "") + index = entry.action.choice_index + if index is None: + return "" + if 1 <= index <= len(entry.before.choices): + return entry.before.choices[index - 1].get("text", "") return "" class Trajectory: - """Append-only references to canonical executed decision steps. + """Append-only references to canonical executed transitions. - The runner owns step construction and ordering. This view stores each - ``AgentState`` reference once after the action has executed, resets between - episodes, and never mutates the state object. Read/search bound their + The runner owns transition construction and ordering. This view stores each + ``QuestTransition`` reference once after the action has executed, resets + between episodes, and never mutates the transition. Read/search bound their OUTPUT by entry count and character budget; they never truncate stored state. """ def __init__(self): - self._steps: list[AgentState] = [] + self._steps: list[QuestTransition] = [] def __len__(self) -> int: return len(self._steps) @@ -73,13 +81,13 @@ def __len__(self) -> int: def reset(self) -> None: self._steps = [] - def append(self, agent_state: AgentState) -> AgentState: - """Store the canonical executed-decision state by reference.""" - self._steps.append(agent_state) - return agent_state + def append(self, transition: QuestTransition) -> QuestTransition: + """Store the canonical executed transition by reference.""" + self._steps.append(transition) + return transition - def recent(self, window: int) -> list[AgentState]: - """Return canonical references for the last ``window`` steps.""" + def recent(self, window: int) -> list[QuestTransition]: + """Return canonical references for the last ``window`` transitions.""" if window <= 0: return [] return list(self._steps[-window:]) @@ -124,8 +132,8 @@ def search(self, query: str, limit) -> str: scored = [] for entry in self._steps: - choices_text = " ".join(choice.get("text", "") for choice in entry.choices) - haystack = " ".join([entry.observation, choices_text, _selected_choice(entry)]).lower() + choices_text = " ".join(choice.get("text", "") for choice in entry.before.choices) + haystack = " ".join([entry.before.agent_observation(), choices_text, _selected_choice(entry)]).lower() entry_tokens = set(_SEARCH_TOKEN_PATTERN.findall(haystack)) score = sum(1 for token in tokens if token in entry_tokens) if score > 0: @@ -134,15 +142,15 @@ def search(self, query: str, limit) -> str: if not scored: return f"no matches for query in {len(self._steps)} recorded steps" - scored.sort(key=lambda item: (item[0], item[1].step), reverse=True) + scored.sort(key=lambda item: (item[0], item[1].index), reverse=True) entries = [entry for _, entry in scored[:bounded_limit]] return self._format_entries(entries) @staticmethod - def _format_entries(entries: list[AgentState]) -> str: + def _format_entries(entries: list[QuestTransition]) -> str: """Join formatted entries into a result hard-bounded by MAX_OUTPUT_CHARS. - Only the formatted output is truncated; canonical ``AgentState`` + Only the formatted output is truncated; canonical ``QuestTransition`` objects remain untouched. Room for the trailing omission marker ("N of M steps shown...") is @@ -181,11 +189,11 @@ def _format_entries(entries: list[AgentState]) -> str: lines = [] total_len = 0 for entry in entries: - choices_text = "; ".join(choice.get("text", "") for choice in entry.choices) or "(none)" + choices_text = "; ".join(choice.get("text", "") for choice in entry.before.choices) or "(none)" selected_choice = _selected_choice(entry) line = ( - f"Step {entry.step}: observation={entry.observation} | " - f"choices={choices_text} | selected={entry.action}: {selected_choice}" + f"Step {entry.index}: observation={entry.before.agent_observation()} | " + f"choices={choices_text} | selected={_action_label(entry)}: {selected_choice}" ) projected_len = total_len + len(line) + (1 if lines else 0) if lines and projected_len > entry_budget: @@ -198,7 +206,6 @@ def _format_entries(entries: list[AgentState]) -> str: break lines.append(line) total_len = projected_len - text = "\n".join(lines) included = len(lines) if included < total_entries: diff --git a/llm_quest_benchmark/llm/prompt.py b/llm_quest_benchmark/llm/prompt.py index 61132a2..c0b7498 100644 --- a/llm_quest_benchmark/llm/prompt.py +++ b/llm_quest_benchmark/llm/prompt.py @@ -1,16 +1,14 @@ """Prompt renderer and history tracker for LLM agents""" from pathlib import Path -from typing import Any from jinja2 import Environment, FileSystemLoader, Template from llm_quest_benchmark.constants import DEFAULT_TEMPLATE, PROMPT_TEMPLATES_DIR, SYSTEM_ROLE_TEMPLATE -from llm_quest_benchmark.schemas.state import QMState class PromptRenderer: - """Handles prompt rendering and history tracking for LLM agents""" + """Handles prompt rendering for LLM agents""" def __init__( self, @@ -28,7 +26,6 @@ def __init__( action_template (str, optional): Action template name to use. Defaults to DEFAULT_TEMPLATE. """ self.env = env - self.history: list[dict[str, Any]] = [] self.templates_dir = templates_dir or PROMPT_TEMPLATES_DIR self.system_template_name = system_template self.action_template_name = action_template @@ -69,27 +66,6 @@ def render_system_prompt(self, **kwargs) -> str: """ return self.system_template.render(**kwargs) - def add_to_history(self, state: dict[str, Any] | QMState) -> None: - """Add state to history - - Args: - state (Union[Dict[str, Any], QMState]): State to add to history - """ - # Convert QMState to dict for history tracking - if isinstance(state, QMState): - self.history.append( - { - "action": "", # Will be updated by step - "text": state.text, - "choices": state.choices, - "reward": state.reward, - "done": state.done, - "info": state.info, - } - ) - else: - self.history.append(state) - def get_template(self, template_name: str) -> Template: """Get a specific template by name @@ -101,19 +77,6 @@ def get_template(self, template_name: str) -> Template: """ return self.jinja_env.get_template(template_name) - def get_history(self, last_n: int | None = None) -> list[dict[str, Any]]: - """Get history, optionally limited to last N entries - - Args: - last_n (Optional[int], optional): Number of entries to return. Defaults to None. - - Returns: - List[Dict[str, Any]]: History entries - """ - if last_n is not None: - return self.history[-last_n:] - return self.history - def get_system_template_content(self) -> str: """Get raw system template content""" return self.system_template_content diff --git a/llm_quest_benchmark/players/base.py b/llm_quest_benchmark/players/base.py index c0edd02..5d2def8 100644 --- a/llm_quest_benchmark/players/base.py +++ b/llm_quest_benchmark/players/base.py @@ -1,15 +1,38 @@ """Base class for quest players and harnesses.""" from abc import ABC, abstractmethod +from dataclasses import dataclass, field from typing import Any +from llm_quest_benchmark.schemas.records import ProgressState, QuestAction, QuestSnapshot, QuestTransition from llm_quest_benchmark.schemas.response import LLMResponse -from llm_quest_benchmark.schemas.state import AgentState + + +@dataclass +class DecisionContext: + """Runner-supplied context for one decision. + + ``checkpoints`` is the active-branch checkpoint stack, one-based for the + harness: index 1 is the first recorded state on the active branch. + """ + + step: int = 0 + checkpoints: list[QuestSnapshot] = field(default_factory=list) + restore_allowed: bool = False + restores_remaining: int | None = None + progress: ProgressState = field(default_factory=ProgressState) class QuestPlayer(ABC): """Abstract base class for quest players""" + # Only harnesses whose specification declares the choose-or-restore loop may + # emit restore actions. Everything else can only choose a current option. + supports_restore: bool = False + + # Canonical harness identifier used to resolve this player's treatment. + harness_name: str = "" + def __init__(self, skip_single: bool = False): """Initialize player with skip_single option""" self.skip_single = skip_single @@ -41,15 +64,31 @@ def get_action(self, observation: str, choices: list) -> int: ) return 1 - # Get action from implementation + # Implementations that produce richer response metadata replace + # `_last_response` themselves. Simple players get a fresh response for + # every decision; never leak a prior auto/default marker forward. + previous_response = self._last_response action = self._get_action_impl(observation, choices) - - # Store basic response if implementation didn't set it - if not self._last_response: + if self._last_response is previous_response: self._last_response = LLMResponse(action=action) return action + def get_quest_action( + self, + observation: str, + choices: list[dict[str, str]], + context: DecisionContext, + ) -> QuestAction: + """Return the quest action for this decision. + + The default implementation can only choose a current option. The choice + id and transition timestamp are filled in by the runner, which owns the + engine mapping. + """ + index = self.get_action(observation, choices) + return QuestAction(kind="choose", choice_index=index) + @abstractmethod def _get_action_impl(self, observation: str, choices: list) -> int: """Implementation of action selection logic""" @@ -59,8 +98,8 @@ def get_last_response(self) -> LLMResponse: """Get the last response from the player or harness.""" return self._last_response - def on_step(self, agent_state: AgentState) -> None: - """Receive one canonical executed decision step from the runner.""" + def on_transition(self, transition: QuestTransition) -> None: + """Receive one canonical executed transition from the runner.""" pass @abstractmethod @@ -72,10 +111,19 @@ def on_game_start(self) -> None: """Called when game starts""" self._last_response = None - def on_game_end(self, final_state: dict[str, Any]) -> None: + def on_game_end(self, final_snapshot: QuestSnapshot | None) -> None: """Called when game ends""" pass + def rebuild_from_transitions(self, transitions: list[QuestTransition]) -> None: + """Rebuild harness memory from a resumed run's recorded transitions.""" + for transition in transitions: + self.on_transition(transition) + def __str__(self) -> str: """String representation of the player""" return self.__class__.__name__ + + def describe_state(self) -> dict[str, Any]: + """Optional harness-specific diagnostics recorded with the run.""" + return {} diff --git a/llm_quest_benchmark/players/human.py b/llm_quest_benchmark/players/human.py index b5d74f4..6114430 100644 --- a/llm_quest_benchmark/players/human.py +++ b/llm_quest_benchmark/players/human.py @@ -1,14 +1,16 @@ """Interactive console player for Space Rangers quests""" import logging -from typing import Any from llm_quest_benchmark.players.base import QuestPlayer +from llm_quest_benchmark.schemas.records import QuestSnapshot class HumanPlayer(QuestPlayer): """Interactive console player that takes input from user""" + harness_name = "human" + def __init__(self, skip_single: bool = False, debug: bool = False): super().__init__(skip_single=skip_single) self.debug = debug @@ -44,6 +46,6 @@ def on_game_start(self) -> None: if self.debug: self.logger.debug("Starting new game") - def on_game_end(self, final_state: dict[str, Any]) -> None: + def on_game_end(self, final_snapshot: QuestSnapshot | None) -> None: """Called when game ends""" pass diff --git a/llm_quest_benchmark/players/random.py b/llm_quest_benchmark/players/random.py index a8fea29..89a1dc1 100644 --- a/llm_quest_benchmark/players/random.py +++ b/llm_quest_benchmark/players/random.py @@ -12,6 +12,8 @@ class RandomPlayer(QuestPlayer): Used for testing quests and finding edge cases. """ + harness_name = "random_choice" + def __init__(self, seed: int = None, debug: bool = False, skip_single: bool = False): """Initialize random player. @@ -26,7 +28,8 @@ def __init__(self, seed: int = None, debug: bool = False, skip_single: bool = Fa if debug: self.logger.setLevel(logging.DEBUG) self.rng = random.Random(seed) - # Keep the persisted identifier stable for existing result artifacts. + # The seed is a material knob, so it belongs in the harness identifier. + self.harness_name = f"random_choice_{seed}" if seed is not None else "random_choice" self.agent_id = f"random_{seed}" if seed is not None else "random" def _get_action_impl(self, observation: str, choices: list[dict[str, str]]) -> int: @@ -47,3 +50,18 @@ def _get_action_impl(self, observation: str, choices: list[dict[str, str]]) -> i def reset(self) -> None: """Reset player state; nothing to reset for random choice.""" pass + + def rebuild_from_transitions(self, transitions) -> None: + """Advance the RNG past the decisions a resumed run already made. + + A seeded random policy is only deterministic if its stream position is + restored, so each recorded decision consumes exactly the draw it + originally consumed. Auto-selected single choices never consumed one. + """ + for transition in transitions: + if not transition.action.is_choose: + continue + choices = transition.before.choices + if self.skip_single and len(choices) == 1: + continue + self.rng.randint(1, len(choices)) diff --git a/llm_quest_benchmark/prompt_templates/adaptive_reasoning.jinja b/llm_quest_benchmark/prompt_templates/adaptive_reasoning.jinja new file mode 100644 index 0000000..d5dd384 --- /dev/null +++ b/llm_quest_benchmark/prompt_templates/adaptive_reasoning.jinja @@ -0,0 +1,25 @@ +Current story state: +{{ observation }} + +Available actions: +{% for choice in choices %} +{{ loop.index }}. {{ choice.text }} +{% endfor %} + +Goal: maximize chances to complete the mission successfully. +{% if mode == "deep" %} +Deep planning mode{% if trigger %} (trigger: {{ trigger }}){% endif %}. Progress so far: {{ progress }}%. +Before answering, work through this explicitly: +1. What has changed since the last time you saw a similar state? +2. Which earlier actions failed to advance the quest, and why? +3. What is the concrete next objective, and which action serves it? +4. Which options are dead ends or repeat a known failure? + +Return ONLY valid JSON (no markdown/code fences), exactly: +{"analysis":"","reasoning":"","result":} +{% else %} +Concise mode. Pick the action that keeps progress alive and gathers useful information. + +Return ONLY valid JSON (no markdown/code fences), exactly: +{"analysis":"","reasoning":"","result":} +{% endif %} diff --git a/llm_quest_benchmark/prompt_templates/backtracking.jinja b/llm_quest_benchmark/prompt_templates/backtracking.jinja new file mode 100644 index 0000000..b1a0185 --- /dev/null +++ b/llm_quest_benchmark/prompt_templates/backtracking.jinja @@ -0,0 +1,26 @@ +Current story state: +{{ observation }} + +Available actions: +{% for choice in choices %} +{{ loop.index }}. {{ choice.text }} +{% endfor %} + +{% if restore_allowed %} +Recorded checkpoints you may return to instead of acting: +{% for checkpoint in checkpoints %} +{{ checkpoint }} +{% endfor %} +Restores remaining: {{ restores_remaining if restores_remaining is not none else "unlimited" }} +Restore only when the current branch is clearly lost or blocked; otherwise act. +{% else %} +No checkpoint restore is available for this decision. You must choose an action. +{% endif %} + +Goal: maximize chances to complete the mission successfully. + +Return ONLY valid JSON (no markdown/code fences), exactly one of: +{"action":"choose","analysis":"","reasoning":"","result":} +{% if restore_allowed %} +{"action":"restore","analysis":"","reasoning":"","checkpoint":,"result":} +{% endif %} diff --git a/llm_quest_benchmark/renderers/base.py b/llm_quest_benchmark/renderers/base.py index 10ffb5f..dae75e1 100644 --- a/llm_quest_benchmark/renderers/base.py +++ b/llm_quest_benchmark/renderers/base.py @@ -3,7 +3,7 @@ import time from abc import ABC, abstractmethod -from llm_quest_benchmark.schemas.state import AgentState +from llm_quest_benchmark.schemas.records import QuestTransition class BaseRenderer(ABC): @@ -18,11 +18,11 @@ def _sleep_for_readability(self, seconds: float = 1.0) -> None: time.sleep(seconds) @abstractmethod - def render_game_state(self, state: AgentState) -> None: - """Render the current game state + def render_game_state(self, transition: QuestTransition) -> None: + """Render the game state produced by one executed transition Args: - state (AgentState): Current game state including observation, choices, action, etc. + transition (QuestTransition): Executed transition with before/after snapshots """ pass diff --git a/llm_quest_benchmark/renderers/benchmark_result.py b/llm_quest_benchmark/renderers/benchmark_result.py index 872a531..c9f1af9 100644 --- a/llm_quest_benchmark/renderers/benchmark_result.py +++ b/llm_quest_benchmark/renderers/benchmark_result.py @@ -6,7 +6,6 @@ from rich.console import Console from rich.panel import Panel from rich.table import Table -from rich.text import Text from llm_quest_benchmark.renderers.base import BaseRenderer @@ -23,143 +22,15 @@ def __init__(self, debug: bool = False): self.debug = debug self.console = Console() - def render_game_state(self, state: dict[str, Any] | None = None) -> None: + def render_game_state(self, transition: Any = None) -> None: """Required implementation of abstract method from BaseRenderer. Not used in benchmark analysis but required by the interface. Args: - state: Game state dictionary (unused) + transition: Executed transition (unused) """ pass # Not used for benchmark analysis - def render_config(self, config: dict[str, Any]) -> None: - """Render benchmark configuration - - Args: - config: Benchmark configuration dictionary - """ - self.console.print("\n[bold cyan]Benchmark Configuration[/]") - self.console.print("=" * 80) - table = Table(show_header=False, box=None) - table.add_column("Setting", style="cyan") - table.add_column("Value", style="green") - - table.add_row("Quest Timeout", f"{config['quest_timeout']}s") - table.add_row("Benchmark Timeout", f"{config['benchmark_timeout']}s") - table.add_row("Debug Mode", str(config["debug"])) - table.add_row("Max Workers", str(config["max_workers"])) - - self.console.print(table) - - def render_agents(self, agents: list) -> None: - """Render agent configurations - - Args: - agents: List of agent configuration dictionaries - """ - self.console.print("\n[bold cyan]Agent Configurations[/]") - self.console.print("=" * 80) - - for agent in agents: - table = Table(show_header=False, box=None) - table.add_column("Setting", style="cyan") - table.add_column("Value", style="green") - - table.add_row("Model", agent["model"]) - table.add_row("Template", agent["template"]) - table.add_row("Temperature", str(agent["temperature"])) - table.add_row("Skip Single", str(agent.get("skip_single", False))) - - self.console.print(table) - self.console.print() - - def render_summary(self, summary: dict[str, Any], quests: list) -> None: - """Render overall benchmark summary""" - self.console.print("\n[bold cyan]Overall Results[/]") - self.console.print("=" * 80) - - # Outcome table - outcome_table = Table(title="Outcome Summary", box=box.ROUNDED) - outcome_table.add_column("Outcome", style="cyan") - outcome_table.add_column("Count", style="magenta") - outcome_table.add_column("Percentage", style="green") - - total = summary["total_runs"] - for outcome, count in summary["outcomes"].items(): - outcome_table.add_row(outcome, str(count), f"{count / total * 100:.1f}%" if total else "0%") - - # Step table - step_table = Table(title="Step Statistics", box=box.ROUNDED) - step_table.add_column("Model", style="cyan") - step_table.add_column("Total Steps", style="magenta") - step_table.add_column("Avg Steps/Run", style="green") - - # Global stats - if "steps" in summary: - step_table.add_row( - "[bold]All Models[/]", str(summary["steps"]["total"]), f"{summary['steps']['average']:.1f}" - ) - - # Per-model stats - for model, stats in summary["steps"]["by_model"].items(): - step_table.add_row(model, str(stats["total"]), f"{stats['average']:.1f}") - - # Render both panels - self.console.print(Panel(outcome_table, title="Benchmark Outcomes", expand=False)) - self.console.print(Panel(step_table, title="Step Analysis", expand=False)) - - def render_quest_details(self, quest: dict[str, Any], debug: bool = False) -> None: - """Render detailed results for a single quest - - Args: - quest: Quest results dictionary - debug: Whether to show debug information - """ - self.console.print(f"\n[bold]{quest['name']}[/]") - self.console.print("-" * 40) - - # Quest summary table - table = Table(show_header=True, box=None) - table.add_column("Outcome", style="cyan") - table.add_column("Count", justify="right", style="green") - table.add_column("Percentage", justify="right", style="blue") - - for outcome, count in quest["outcomes"].items(): - percentage = (count / quest["total_runs"]) * 100 - table.add_row(outcome, str(count), f"{percentage:.1f}%") - - self.console.print(table) - - if debug: - # Detailed run information - for result in quest["results"]: - self.console.print(f"\n[dim]Run with {result['model']} (temp={result['temperature']})[/]") - self.console.print(f"Outcome: {result['outcome']}") - - if result.get("error"): - self.console.print(Text(f"Error: {result['error']}", style="red")) - - # Show steps if available - for step in result.get("steps", []): - self.console.print(f"\n[bold]Step {step.get('step', '?')}:[/]") - if step.get("state"): - self.console.print( - Text( - step["state"][:200] + "..." if len(step["state"]) > 200 else step["state"], style="blue" - ) - ) - - if step.get("choices"): - self.console.print("\nChoices:") - for choice in step["choices"]: - self.console.print(f" {choice['id']}: {choice['text']}") - - if step.get("response"): - self.console.print(f"\nSelected: {step['response']}") - - if step.get("reasoning"): - self.console.print(Text(f"\nReasoning: {step['reasoning']}", style="yellow")) - def render_benchmark_results(self, data: dict[str, Any], debug: bool = False) -> None: """Render complete benchmark results @@ -171,9 +42,9 @@ def render_benchmark_results(self, data: dict[str, Any], debug: bool = False) -> self.console.print("\n[bold cyan]Benchmark Results[/]") self.console.print("=" * 80) - # Print benchmark name if available - if "benchmark_name" in data: - self.console.print(f"\nBenchmark: {data['benchmark_name']}") + # Print benchmark id if available + if "benchmark_id" in data: + self.console.print(f"\nBenchmark: {data['benchmark_id']}") # Overall statistics summary = data["summary"] @@ -217,3 +88,17 @@ def render_benchmark_results(self, data: dict[str, Any], debug: bool = False) -> quest_table.add_row(quest["name"], str(quest["runs"]), f"{quest['success_rate']:.1f}%") self.console.print(Panel(quest_table, title="Quest Statistics", expand=False)) + + # Treatment statistics, grouped by canonical signature and components. + # Printed as one short line per component instead of a wide table: a + # seven-column table is narrower than its content on an 80-column + # terminal, so rich would silently truncate names such as the harness. + if data.get("treatments"): + self.console.print("\n[bold cyan]Treatment Statistics[/]") + self.console.print("=" * 80) + for treatment in data["treatments"]: + self.console.print(f"\nSignature: {treatment['signature']}") + for component in ("harness", "memory", "loop", "reasoning"): + self.console.print(f" {component}: {treatment[component]}") + self.console.print(f" runs: {treatment['runs']}") + self.console.print(f" success rate: {treatment['success_rate']:.1f}%") diff --git a/llm_quest_benchmark/renderers/null.py b/llm_quest_benchmark/renderers/null.py index 76342ce..784e13b 100644 --- a/llm_quest_benchmark/renderers/null.py +++ b/llm_quest_benchmark/renderers/null.py @@ -1,14 +1,13 @@ """Null renderer that does nothing - used for debug mode and when no rendering is needed""" -from typing import Any - from llm_quest_benchmark.renderers.base import BaseRenderer +from llm_quest_benchmark.schemas.records import QuestTransition class NoRenderer(BaseRenderer): """Null renderer implementation that does nothing""" - def render_game_state(self, state: dict[str, Any]) -> None: + def render_game_state(self, transition: QuestTransition) -> None: """Do nothing implementation of game state rendering""" pass diff --git a/llm_quest_benchmark/renderers/progress.py b/llm_quest_benchmark/renderers/progress.py index a5097d2..6157b64 100644 --- a/llm_quest_benchmark/renderers/progress.py +++ b/llm_quest_benchmark/renderers/progress.py @@ -44,7 +44,7 @@ def __init__(self, total_quests: int, total_runs: int): # Print initial header self.console.print("\n[bold cyan]Benchmark Progress[/]") - def render_game_state(self, state: dict[str, Any]) -> None: + def render_game_state(self, transition: Any) -> None: """No game state rendering needed for automated players""" pass diff --git a/llm_quest_benchmark/renderers/terminal.py b/llm_quest_benchmark/renderers/terminal.py index 0209c47..ebc2ed7 100644 --- a/llm_quest_benchmark/renderers/terminal.py +++ b/llm_quest_benchmark/renderers/terminal.py @@ -4,13 +4,13 @@ from llm_quest_benchmark.constants import READABILITY_DELAY from llm_quest_benchmark.renderers.base import BaseRenderer +from llm_quest_benchmark.schemas.records import QuestTransition from llm_quest_benchmark.schemas.response import LLMResponse -from llm_quest_benchmark.schemas.state import AgentState from llm_quest_benchmark.utils import choice_mapper, text_processor class NoRenderer: - def render_game_state(self, state: AgentState): + def render_game_state(self, transition: QuestTransition): """Render complete game state with RPG elements""" pass @@ -60,10 +60,10 @@ def render_error(self, message: str): """Render error message""" self.console.print(f"\nError: {message}", style="red bold") - def render_game_state(self, state: AgentState): - """Render game state""" + def render_game_state(self, transition: QuestTransition): + """Render one executed transition""" self.console.clear() - self.step_number = state.step + self.step_number = transition.index # Print step separator self.console.print(f"\n{'=' * 80}", style="blue") @@ -71,16 +71,24 @@ def render_game_state(self, state: AgentState): self.console.print(f"{'=' * 80}\n", style="blue") # Show LLM response first - self.render_llm_response(state.llm_response) + self.render_llm_response(transition.response) # Add separator after LLM response self.console.print(f"\n{'-' * 40}\n", style="dim") - self.render_quest_text(state.observation) - self.render_choices(state.choices) + if transition.action.is_restore: + self.console.print( + f"[magenta]Restored checkpoint {transition.action.checkpoint_index}[/]", + ) - def render_llm_response(self, response: LLMResponse): + self.render_quest_text(transition.after.observation) + self.render_parameters(transition.after.params_state) + self.render_choices(transition.after.choices) + + def render_llm_response(self, response: LLMResponse | None): """Render LLM's response""" self.console.print("\n[yellow bold]Agent's Thoughts:[/]") + if response is None: + return if response.analysis: self.console.print(f"[yellow]{response.analysis.strip()}[/]") if response.reasoning: diff --git a/llm_quest_benchmark/schemas/__init__.py b/llm_quest_benchmark/schemas/__init__.py index cb0338f..4e49dc6 100644 --- a/llm_quest_benchmark/schemas/__init__.py +++ b/llm_quest_benchmark/schemas/__init__.py @@ -1,16 +1,20 @@ """Schema exports for LLM Quest Benchmark""" __all__ = [ - "QMState", - "AgentState", "LLMResponse", "QMBridgeState", "BenchmarkConfig", "HarnessConfig", + "ProgressState", + "QuestAction", + "QuestSnapshot", + "QuestTransition", + "RunRecord", + "SCHEMA_VERSION", ] # Import directly from the schema modules using relative imports from .bridge import QMBridgeState from .config import BenchmarkConfig, HarnessConfig +from .records import SCHEMA_VERSION, ProgressState, QuestAction, QuestSnapshot, QuestTransition, RunRecord from .response import LLMResponse -from .state import AgentState, QMState diff --git a/llm_quest_benchmark/schemas/bridge.py b/llm_quest_benchmark/schemas/bridge.py index 27ad071..9d71f31 100644 --- a/llm_quest_benchmark/schemas/bridge.py +++ b/llm_quest_benchmark/schemas/bridge.py @@ -1,11 +1,18 @@ """Bridge dataclasses for TypeScript integration""" from dataclasses import dataclass, field +from typing import Any + +from llm_quest_benchmark.schemas.records import QuestSnapshot @dataclass class QMBridgeState: - """State object returned by TypeScript bridge""" + """State object returned by TypeScript bridge. + + ``saving`` is the full engine ``GameState`` (including its PRNG state), which + is what makes exact restore and deterministic replay possible. + """ location_id: str text: str @@ -14,3 +21,17 @@ class QMBridgeState: game_ended: bool game_state: str = "running" # "running" | "win" | "fail" | "dead" from TS engine params_state: list[str] = field(default_factory=list) + saving: dict[str, Any] = field(default_factory=dict) + + def to_snapshot(self) -> QuestSnapshot: + """Convert to the canonical snapshot used by records and replay.""" + return QuestSnapshot( + location_id=str(self.location_id), + observation=self.text, + choices=[{"id": str(c["id"]), "text": c["text"]} for c in self.choices], + params_state=list(self.params_state), + reward=self.reward, + done=self.game_ended, + game_state=self.game_state, + saving=self.saving or None, + ) diff --git a/llm_quest_benchmark/schemas/config.py b/llm_quest_benchmark/schemas/config.py index 93d6af8..0b1e673 100644 --- a/llm_quest_benchmark/schemas/config.py +++ b/llm_quest_benchmark/schemas/config.py @@ -26,18 +26,6 @@ "name": "Default Benchmark", } -COMPACTION_HARNESSES = { - "memo_compact", - "hinted_compact", - "tool_compact", - "tool_hinted", - "planner", - "compaction_no_memo", - "memo_cot", - "memo_extended", - "memo_structured", -} - def get_default_benchmark_yaml() -> str: """Get the default benchmark configuration from default.yaml file""" @@ -79,6 +67,8 @@ class HarnessConfig: debug: bool = False benchmark_id: str | None = None compaction_interval: int = 50 + restore_limit: int | None = None + adaptive_stall_steps: int | None = None def __init__( self, @@ -91,14 +81,12 @@ def __init__( debug: bool = False, benchmark_id: str | None = None, compaction_interval: int = 50, - **legacy_keys, + restore_limit: int | None = None, + adaptive_stall_steps: int | None = None, + **unexpected_keys, ): - if "template" in legacy_keys or "action_template" in legacy_keys: - raise ValueError("Use harness: key instead of template:") - if "memory_mode" in legacy_keys: - raise ValueError("Use harness: key instead of memory_mode:") - if legacy_keys: - unexpected = ", ".join(sorted(legacy_keys)) + if unexpected_keys: + unexpected = ", ".join(sorted(unexpected_keys)) raise TypeError(f"Unexpected HarnessConfig key(s): {unexpected}") self.model = model @@ -110,19 +98,21 @@ def __init__( self.debug = debug self.benchmark_id = benchmark_id self.compaction_interval = compaction_interval + self.restore_limit = restore_limit + self.adaptive_stall_steps = adaptive_stall_steps self.__post_init__() def __post_init__(self): self.system_template = normalize_template_name(self.system_template) - from llm_quest_benchmark.harnesses.factory import HARNESS_REGISTRY, SPECIAL_HARNESSES, is_random_choice_harness - - if ( - self.harness not in HARNESS_REGISTRY - and self.harness != "human" - and not is_random_choice_harness(self.harness) - ): - valid = [*sorted(HARNESS_REGISTRY), *SPECIAL_HARNESSES] - raise ValueError(f"Invalid harness: {self.harness}. Supported harnesses: {valid}") + from llm_quest_benchmark.harnesses.specs import ( + HARNESS_SPECS, + is_random_choice_harness, + valid_harness_names, + validate_exclusive_knobs, + ) + + if self.harness not in HARNESS_SPECS and not is_random_choice_harness(self.harness): + raise ValueError(f"Invalid harness: {self.harness}. Supported harnesses: {valid_harness_names()}") if self.harness == "human" and self.model != "human": raise ValueError("Use model: human with harness: human") if self.model == "human" and self.harness != "human": @@ -143,19 +133,41 @@ def __post_init__(self): if self.compaction_interval < 1: raise ValueError(f"compaction_interval must be >= 1, got {self.compaction_interval}") + validate_exclusive_knobs(self.harness, self.knob_values()) + if self.restore_limit is not None and self.restore_limit < 1: + raise ValueError(f"restore_limit must be >= 1, got {self.restore_limit}") + if self.adaptive_stall_steps is not None and self.adaptive_stall_steps < 1: + raise ValueError(f"adaptive_stall_steps must be >= 1, got {self.adaptive_stall_steps}") + + def knob_values(self) -> dict[str, int | None]: + """Material knob values considered by the harness specification.""" + return { + "compaction_interval": self.compaction_interval, + "restore_limit": self.restore_limit, + "adaptive_stall_steps": self.adaptive_stall_steps, + } + + def treatment(self): + """Canonical treatment for this configuration.""" + from llm_quest_benchmark.harnesses.specs import build_treatment + + return build_treatment( + harness=self.harness, + model=self.model, + temperature=self.temperature, + system_template=self.system_template, + knob_values=self.knob_values(), + ) + @property def harness_id(self) -> str: - """Generate a stable harness ID based on configuration values""" - import hashlib - - interval_tag = f"_ci{self.compaction_interval}" if self.harness in COMPACTION_HARNESSES else "" - config_str = f"{self.model}_{self.temperature}_{self.harness}_{self.system_template}{interval_tag}" - hash_val = hashlib.md5(config_str.encode()).hexdigest()[:8] - return f"{self.model}_t{self.temperature}_{self.harness}_{hash_val}" + """Stable harness ID derived from the canonical treatment signature.""" + signature = self.treatment().signature + return f"{self.model}_t{self.temperature}_{self.harness}_{signature.split('_', 1)[1][:8]}" @property def agent_id(self) -> str: - """DB-compatible alias for harness_id""" + """Run identity used for the results directory and DB rows.""" return self.harness_id @@ -175,6 +187,7 @@ class BenchmarkConfig: benchmark_id: str | None = None # Unique ID for the benchmark run max_quests: int | None = None # Maximum number of quests to run (useful for testing) max_workers: int | None = None # Optional parallel workers for future benchmark scheduling + progress_manifest: str | None = None # Optional curated milestone manifest (YAML) def __post_init__(self): # Validate quest paths @@ -192,6 +205,12 @@ def __post_init__(self): if self.max_steps is not None and self.max_steps < 1: raise ValueError(f"max_steps must be >= 1, got {self.max_steps}") + if self.progress_manifest: + # Fail fast on an invalid manifest rather than mid-benchmark. + from llm_quest_benchmark.core.progress import ProgressManifest + + ProgressManifest.from_file(self.progress_manifest) + @classmethod def from_yaml(cls, yaml_path: str) -> "BenchmarkConfig": """Create config from YAML file""" @@ -200,13 +219,6 @@ def from_yaml(cls, yaml_path: str) -> "BenchmarkConfig": # Convert agent configs if "agents" in data: - agents = [] - for agent in data["agents"]: - if "template" in agent: - raise ValueError("Use harness: key instead of template:") - if "memory_mode" in agent: - raise ValueError("Use harness: key instead of memory_mode:") - agents.append(HarnessConfig(**agent)) - data["agents"] = agents + data["agents"] = [HarnessConfig(**agent) for agent in data["agents"]] return cls(**data) diff --git a/llm_quest_benchmark/schemas/records.py b/llm_quest_benchmark/schemas/records.py new file mode 100644 index 0000000..e4bf3a2 --- /dev/null +++ b/llm_quest_benchmark/schemas/records.py @@ -0,0 +1,484 @@ +"""Canonical schema-v2 run record types. + +These types are the only transition/run representation accepted at runtime. +Legacy shapes are read exclusively by ``llm_quest_benchmark.core.migration``. +""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import asdict, dataclass, field +from pathlib import Path +from typing import Any + +from llm_quest_benchmark.schemas.response import LLMResponse + +SCHEMA_VERSION = 2 + +# Provenance of a persisted transition. +PROVENANCE_RUNTIME = "runtime" +PROVENANCE_LEGACY_MAPPED = "legacy_mapped" + +# Replay status of a persisted transition. +REPLAY_PENDING = "pending" +REPLAY_VERIFIED = "verified" +REPLAY_UNAVAILABLE = "unavailable" + +ACTION_CHOOSE = "choose" +ACTION_RESTORE = "restore" +ACTION_KINDS = (ACTION_CHOOSE, ACTION_RESTORE) + +# Marker for state that a legacy record cannot prove. +GAME_STATE_UNAVAILABLE = "unavailable" + +# Nested domains every schema-v2 run record must carry. +RUN_RECORD_DOMAINS = ( + "run", + "quest", + "treatment", + "lineage", + "terminal", + "usage", + "progress", + "transcript_diagnostics", + "transitions", +) + +MIGRATION_HINT = "run `scripts/migrate_records.py --source PATH --output PATH` to convert legacy records." + + +def canonical_json(payload: Any) -> str: + """Stable JSON encoding used for digests and signatures.""" + return json.dumps(payload, sort_keys=True, ensure_ascii=False, separators=(",", ":"), default=str) + + +def _digest(payload: Any) -> str: + return hashlib.sha256(canonical_json(payload).encode("utf-8")).hexdigest() + + +def _require_mapping(value: Any, domain: str) -> dict[str, Any]: + """Return a required record domain, rejecting anything that is not a mapping.""" + if not isinstance(value, dict): + raise ValueError( + f"Run record domain '{domain}' must be an object, got {type(value).__name__}; {MIGRATION_HINT}" + ) + return value + + +@dataclass +class QuestAction: + """One executed environment action. + + ``choose`` carries the one-based choice index, the engine jump id, and the + exact ``performed_at_ms`` handed to ``performJump``. ``restore`` carries the + one-based checkpoint index selected by the backtracking harness. + """ + + kind: str + choice_index: int | None = None + choice_id: str | None = None + performed_at_ms: int | None = None + checkpoint_index: int | None = None + + @classmethod + def choose(cls, choice_index: int, choice_id: str, performed_at_ms: int) -> QuestAction: + return cls( + kind=ACTION_CHOOSE, + choice_index=int(choice_index), + choice_id=str(choice_id), + performed_at_ms=int(performed_at_ms), + ) + + @classmethod + def restore(cls, checkpoint_index: int) -> QuestAction: + return cls(kind=ACTION_RESTORE, checkpoint_index=int(checkpoint_index)) + + @property + def is_choose(self) -> bool: + return self.kind == ACTION_CHOOSE + + @property + def is_restore(self) -> bool: + return self.kind == ACTION_RESTORE + + def to_dict(self) -> dict[str, Any]: + return asdict(self) + + @classmethod + def from_dict(cls, payload: dict[str, Any]) -> QuestAction: + """Rebuild an action, rejecting any kind outside the canonical set.""" + if not isinstance(payload, dict): + raise ValueError(f"Quest action must be an object, got {type(payload).__name__}") + kind = str(payload.get("kind") or "") + if kind not in ACTION_KINDS: + raise ValueError(f"Unsupported quest action kind {kind!r}; expected one of {list(ACTION_KINDS)}") + return cls( + kind=kind, + choice_index=payload.get("choice_index"), + choice_id=payload.get("choice_id"), + performed_at_ms=payload.get("performed_at_ms"), + checkpoint_index=payload.get("checkpoint_index"), + ) + + +@dataclass +class QuestSnapshot: + """Complete environment state at one point in a run.""" + + location_id: str + observation: str + choices: list[dict[str, str]] = field(default_factory=list) + params_state: list[str] = field(default_factory=list) + reward: float = 0.0 + done: bool = False + game_state: str = "running" + saving: dict[str, Any] | None = None + digest: str = "" + # Fields a legacy record could not prove. Never part of the digest. + unavailable_fields: list[str] = field(default_factory=list) + + def __post_init__(self) -> None: + if not self.digest: + self.digest = self.compute_digest() + + @classmethod + def unavailable(cls, missing: list[str] | None = None) -> QuestSnapshot: + """Snapshot placeholder for state a legacy record cannot reconstruct.""" + return cls( + location_id="", + observation="", + game_state=GAME_STATE_UNAVAILABLE, + unavailable_fields=missing or ["location_id", "observation", "choices", "params_state", "saving"], + ) + + @property + def is_unavailable(self) -> bool: + return self.game_state == GAME_STATE_UNAVAILABLE + + def digest_payload(self) -> dict[str, Any]: + """Canonical fields covered by the snapshot digest.""" + return { + "location_id": str(self.location_id), + "observation": self.observation, + "choices": [{"id": str(c.get("id", "")), "text": c.get("text", "")} for c in self.choices], + "params_state": list(self.params_state), + "done": bool(self.done), + "game_state": self.game_state, + "saving": self.saving, + } + + def compute_digest(self) -> str: + return _digest(self.digest_payload()) + + def agent_observation(self) -> str: + """Observation text as presented to an agent (quest text plus params).""" + base = (self.observation or "").strip() + lines = [str(p).strip() for p in self.params_state if str(p).strip()] + if not lines: + return base + params_block = "Status:\n" + "\n".join(lines) + return f"{base}\n\n{params_block}" if base else params_block + + @property + def is_resumable(self) -> bool: + """A snapshot can seed a resumed run only with a full engine saving.""" + return isinstance(self.saving, dict) and bool(self.saving) and not self.is_unavailable + + def to_dict(self) -> dict[str, Any]: + return asdict(self) + + @classmethod + def from_dict(cls, payload: dict[str, Any]) -> QuestSnapshot: + choices_raw = payload.get("choices") or [] + choices = [ + {"id": str(c.get("id", "")), "text": str(c.get("text", ""))} for c in choices_raw if isinstance(c, dict) + ] + return cls( + location_id=str(payload.get("location_id", "")), + observation=str(payload.get("observation") or ""), + choices=choices, + params_state=[str(p) for p in (payload.get("params_state") or [])], + reward=float(payload.get("reward") or 0.0), + done=bool(payload.get("done", False)), + game_state=str(payload.get("game_state") or "running"), + saving=payload.get("saving") if isinstance(payload.get("saving"), dict) else None, + digest=str(payload.get("digest") or ""), + unavailable_fields=[str(f) for f in (payload.get("unavailable_fields") or [])], + ) + + +@dataclass +class ProgressState: + """Monotonic milestone progress derived from a curated manifest. + + ``scored`` says whether a curated manifest was in effect. An unscored state + is terminal-only progress: it never guesses story advancement, so a 0.0 + ``current`` means "no manifest", not "no progress made". + + ``current`` never decreases within a run, including after a restore. + ``maximum`` is the highest percentage the active manifest can award, so + ``current / maximum`` is a well-defined completion ratio. ``manifest_hash`` + pins the exact milestone definitions that produced these numbers, so + progress recorded under a since-edited manifest is detectable. + """ + + current: float = 0.0 + maximum: float = 100.0 + scored: bool = False + reached: list[str] = field(default_factory=list) + newly_reached: list[str] = field(default_factory=list) + stalled_transitions: int = 0 + manifest: str | None = None + manifest_hash: str | None = None + + def to_dict(self) -> dict[str, Any]: + return asdict(self) + + @classmethod + def from_dict(cls, payload: dict[str, Any] | None) -> ProgressState: + payload = payload or {} + return cls( + current=float(payload.get("current") or 0.0), + maximum=float(payload.get("maximum") or 100.0), + scored=bool(payload.get("scored", False)), + reached=[str(m) for m in (payload.get("reached") or [])], + newly_reached=[str(m) for m in (payload.get("newly_reached") or [])], + stalled_transitions=int(payload.get("stalled_transitions") or 0), + manifest=payload.get("manifest"), + manifest_hash=payload.get("manifest_hash"), + ) + + +@dataclass +class QuestTransition: + """One executed environment transition plus its agent-side provenance.""" + + index: int + before: QuestSnapshot + action: QuestAction + after: QuestSnapshot + response: LLMResponse | None = None + usage: dict[str, Any] = field(default_factory=dict) + progress: ProgressState = field(default_factory=ProgressState) + provenance: str = PROVENANCE_RUNTIME + replay_status: str = REPLAY_PENDING + reasoning_mode: str | None = None + + @property + def is_replayable(self) -> bool: + """Whether this transition carries every deterministic replay input.""" + if self.replay_status == REPLAY_UNAVAILABLE: + return False + if not self.before.is_resumable or not self.after.is_resumable: + return False + if self.action.is_choose: + return self.action.choice_id is not None and self.action.performed_at_ms is not None + if self.action.is_restore: + return self.action.checkpoint_index is not None + return False + + def to_dict(self) -> dict[str, Any]: + return { + "index": self.index, + "before": self.before.to_dict(), + "action": self.action.to_dict(), + "after": self.after.to_dict(), + "response": self.response.to_dict() if self.response else None, + "usage": dict(self.usage), + "progress": self.progress.to_dict(), + "provenance": self.provenance, + "replay_status": self.replay_status, + "reasoning_mode": self.reasoning_mode, + } + + @classmethod + def from_dict(cls, payload: dict[str, Any]) -> QuestTransition: + if not isinstance(payload, dict): + raise ValueError(f"Transition must be an object, got {type(payload).__name__}; {MIGRATION_HINT}") + for domain in ("before", "action", "after"): + if domain not in payload: + raise ValueError(f"Transition is missing '{domain}'; {MIGRATION_HINT}") + response_payload = payload.get("response") + response = None + if isinstance(response_payload, dict) and response_payload: + response = LLMResponse(**{k: v for k, v in response_payload.items() if k in LLMResponse.__annotations__}) + return cls( + index=int(payload.get("index") or 0), + before=QuestSnapshot.from_dict(payload.get("before") or {}), + action=QuestAction.from_dict(payload.get("action") or {}), + after=QuestSnapshot.from_dict(payload.get("after") or {}), + response=response, + usage=dict(payload.get("usage") or {}), + progress=ProgressState.from_dict(payload.get("progress")), + provenance=str(payload.get("provenance") or PROVENANCE_RUNTIME), + replay_status=str(payload.get("replay_status") or REPLAY_PENDING), + reasoning_mode=payload.get("reasoning_mode"), + ) + + +@dataclass +class ResumeLineage: + """Link from a resumed run back to the record it continued.""" + + source_run_id: Any + source_path: str + resumed_from_index: int + + def to_dict(self) -> dict[str, Any]: + return asdict(self) + + @classmethod + def from_dict(cls, payload: dict[str, Any] | None) -> ResumeLineage | None: + if not payload: + return None + return cls( + source_run_id=payload.get("source_run_id"), + source_path=str(payload.get("source_path") or ""), + resumed_from_index=int(payload.get("resumed_from_index") or 0), + ) + + +@dataclass +class RunRecord: + """Schema-v2 run record shared by ``run_summary.json`` and SQLite. + + The serialized form groups these fields into the canonical domains + ``run``, ``quest``, ``treatment``, ``lineage``, ``terminal``, ``usage``, + ``progress``, ``transcript_diagnostics``, and ``transitions``. Nothing is + written or read at the top level except ``schema_version``. + """ + + run_id: Any + quest_file: str + quest_name: str + quest_checksum: str + quest_language: str + engine_revision: str + agent_id: str + treatment: dict[str, Any] + started_at: str | None = None + ended_at: str | None = None + run_duration: float | None = None + benchmark_id: str | None = None + lineage: ResumeLineage | None = None + outcome: str | None = None + reward: float = 0.0 + usage: dict[str, Any] = field(default_factory=dict) + transcript_diagnostics: dict[str, Any] = field(default_factory=dict) + progress: ProgressState = field(default_factory=ProgressState) + terminal_snapshot: QuestSnapshot | None = None + transitions: list[QuestTransition] = field(default_factory=list) + schema_version: int = SCHEMA_VERSION + + @property + def treatment_signature(self) -> str: + return str(self.treatment.get("signature") or "") + + @property + def is_resumable(self) -> bool: + """Resumable runs carry a complete, deterministic engine lineage.""" + if self.outcome not in ("TRUNCATED",): + return False + if not self.transitions: + return False + if any(t.provenance != PROVENANCE_RUNTIME or not t.is_replayable for t in self.transitions): + return False + return self.transitions[-1].after.is_resumable + + def to_dict(self) -> dict[str, Any]: + return { + "schema_version": self.schema_version, + "run": { + "id": self.run_id, + "agent_id": self.agent_id, + "started_at": self.started_at, + "ended_at": self.ended_at, + "duration": self.run_duration, + "benchmark_id": self.benchmark_id, + }, + "quest": { + "file": self.quest_file, + "name": self.quest_name, + "checksum": self.quest_checksum, + "language": self.quest_language, + "engine_revision": self.engine_revision, + }, + "treatment": self.treatment, + "lineage": self.lineage.to_dict() if self.lineage else None, + "terminal": { + "outcome": self.outcome, + "reward": self.reward, + "snapshot": self.terminal_snapshot.to_dict() if self.terminal_snapshot else None, + }, + "usage": self.usage, + "progress": self.progress.to_dict(), + "transcript_diagnostics": self.transcript_diagnostics, + "transitions": [t.to_dict() for t in self.transitions], + } + + @classmethod + def from_dict(cls, payload: dict[str, Any]) -> RunRecord: + """Rebuild a record, rejecting anything that is not a complete v2 document.""" + if not isinstance(payload, dict): + raise ValueError(f"Run record must be an object, got {type(payload).__name__}; {MIGRATION_HINT}") + + version = payload.get("schema_version") + if version != SCHEMA_VERSION: + raise ValueError(f"Unsupported run record schema_version {version!r}; {MIGRATION_HINT}") + + missing = [domain for domain in RUN_RECORD_DOMAINS if domain not in payload] + if missing: + raise ValueError(f"Run record is missing canonical domain(s): {', '.join(missing)}; {MIGRATION_HINT}") + + run = _require_mapping(payload["run"], "run") + quest = _require_mapping(payload["quest"], "quest") + treatment = _require_mapping(payload["treatment"], "treatment") + terminal = _require_mapping(payload["terminal"], "terminal") + usage = _require_mapping(payload["usage"], "usage") + progress = _require_mapping(payload["progress"], "progress") + diagnostics = _require_mapping(payload["transcript_diagnostics"], "transcript_diagnostics") + + lineage_payload = payload["lineage"] + if lineage_payload is not None and not isinstance(lineage_payload, dict): + raise ValueError(f"Run record domain 'lineage' must be an object or null; {MIGRATION_HINT}") + + transitions_payload = payload["transitions"] + if not isinstance(transitions_payload, list): + raise ValueError(f"Run record domain 'transitions' must be a list; {MIGRATION_HINT}") + + terminal_snapshot = terminal.get("snapshot") + if terminal_snapshot is not None and not isinstance(terminal_snapshot, dict): + raise ValueError(f"Run record field 'terminal.snapshot' must be an object or null; {MIGRATION_HINT}") + + return cls( + run_id=run.get("id"), + quest_file=str(quest.get("file") or ""), + quest_name=str(quest.get("name") or ""), + quest_checksum=str(quest.get("checksum") or ""), + quest_language=str(quest.get("language") or ""), + engine_revision=str(quest.get("engine_revision") or ""), + agent_id=str(run.get("agent_id") or ""), + treatment=dict(treatment), + started_at=run.get("started_at"), + ended_at=run.get("ended_at"), + run_duration=run.get("duration"), + benchmark_id=run.get("benchmark_id"), + lineage=ResumeLineage.from_dict(lineage_payload), + outcome=terminal.get("outcome"), + reward=float(terminal.get("reward") or 0.0), + usage=dict(usage), + transcript_diagnostics=dict(diagnostics), + progress=ProgressState.from_dict(progress), + terminal_snapshot=QuestSnapshot.from_dict(terminal_snapshot) if terminal_snapshot else None, + transitions=[QuestTransition.from_dict(t) for t in transitions_payload], + ) + + +def load_run_record(path: str) -> RunRecord: + """Load a schema-v2 ``run_summary.json`` from disk.""" + with open(Path(path), encoding="utf-8") as f: + payload = json.load(f) + if not isinstance(payload, dict): + raise ValueError(f"Run record is not a JSON object: {path}") + return RunRecord.from_dict(payload) diff --git a/llm_quest_benchmark/schemas/response.py b/llm_quest_benchmark/schemas/response.py index 0644e4e..b217d08 100644 --- a/llm_quest_benchmark/schemas/response.py +++ b/llm_quest_benchmark/schemas/response.py @@ -20,6 +20,15 @@ class LLMResponse: total_tokens: int | None = None estimated_cost_usd: float | None = None + def usage_payload(self) -> dict: + """Canonical usage domain persisted on transitions and run rows.""" + return { + "prompt_tokens": self.prompt_tokens or 0, + "completion_tokens": self.completion_tokens or 0, + "total_tokens": self.total_tokens or 0, + "estimated_cost_usd": self.estimated_cost_usd, + } + def to_choice_string(self) -> str: """Convert to choice string (1-based action number)""" return str(self.action) diff --git a/llm_quest_benchmark/schemas/state.py b/llm_quest_benchmark/schemas/state.py deleted file mode 100644 index 71c0e9c..0000000 --- a/llm_quest_benchmark/schemas/state.py +++ /dev/null @@ -1,59 +0,0 @@ -"""State dataclasses for quest environments""" - -from dataclasses import dataclass -from typing import Any - -from .response import LLMResponse - - -@dataclass -class QMState: - """A QM game state""" - - location_id: str - text: str - choices: list[dict[str, str]] # [{id: str, text: str}] - reward: float - done: bool - info: dict[str, Any] - - -@dataclass -class AgentState: - """State for tracking agent interactions with quest""" - - step: int # Current step number - location_id: str # Current location in quest - observation: str # Current game text/observation - choices: list[dict[str, str]] # Available choices - action: str # Agent's chosen action (e.g. "1" or "Go north") - llm_response: LLMResponse # Agent's response (LLM or not) - - @classmethod - def from_qm_state(cls, qm_state: QMState, step: int, action: str, llm_response: LLMResponse) -> "AgentState": - """Create AgentState from QMState and agent response""" - return cls( - step=step, - location_id=qm_state.location_id, - observation=qm_state.text, - choices=qm_state.choices, - action=action, - llm_response=llm_response, - ) - - def __str__(self) -> str: - """String representation of AgentState""" - choices_str = "\n".join([f"{i + 1}. {choice['text']}" for i, choice in enumerate(self.choices)]) - return f"Step {self.step}.\nObservation: {self.observation}.\nChoices:\n{choices_str}\nLLM Response:\n{str(self.llm_response)}" - - def to_dict(self) -> dict[str, Any]: - """Convert AgentState to dictionary""" - return { - "step": self.step, - "location_id": self.location_id, - "observation": self.observation, - "choices": self.choices, - "action": self.action, - "llm_response": self.llm_response.to_dict() if self.llm_response else None, - "game_ended": False, # Default value, can be updated by caller - } diff --git a/llm_quest_benchmark/tests/conftest.py b/llm_quest_benchmark/tests/conftest.py index 52c2e29..f4c43aa 100644 --- a/llm_quest_benchmark/tests/conftest.py +++ b/llm_quest_benchmark/tests/conftest.py @@ -2,10 +2,22 @@ import pytest +import llm_quest_benchmark.core.logging as logging_module from llm_quest_benchmark.constants import DEFAULT_QUEST from llm_quest_benchmark.core.logging import LogManager +@pytest.fixture(autouse=True) +def isolated_runtime_artifacts(tmp_path, monkeypatch): + """Keep every test and spawned benchmark worker off developer artifacts.""" + db_path = tmp_path / "metrics.db" + results_path = tmp_path / "results" + monkeypatch.setenv(logging_module.DB_PATH_ENV_VAR, str(db_path)) + monkeypatch.setenv(logging_module.RESULTS_DIR_ENV_VAR, str(results_path)) + monkeypatch.setattr(logging_module, "DEFAULT_DB_PATH", str(db_path)) + monkeypatch.setattr(logging_module, "RESULTS_DIR", results_path) + + @pytest.fixture def test_logger(): """Get a test logger""" diff --git a/llm_quest_benchmark/tests/core/test_analyzer.py b/llm_quest_benchmark/tests/core/test_analyzer.py index 57e623c..670c6c3 100644 --- a/llm_quest_benchmark/tests/core/test_analyzer.py +++ b/llm_quest_benchmark/tests/core/test_analyzer.py @@ -1,4 +1,4 @@ -"""Tests for quest run analyzer""" +"""Tests for the schema-v2 quest run analyzer""" import json import os @@ -11,7 +11,15 @@ from typer.testing import CliRunner from llm_quest_benchmark.core.analyzer import analyze_quest_run +from llm_quest_benchmark.core.logging import ensure_v2_schema from llm_quest_benchmark.executors.cli.commands import app +from llm_quest_benchmark.harnesses.specs import build_treatment +from llm_quest_benchmark.schemas.records import ( + SCHEMA_VERSION, + QuestAction, + QuestSnapshot, + QuestTransition, +) @contextmanager @@ -25,132 +33,118 @@ def isolated_filesystem(): os.chdir(previous_cwd) -def setup_test_db(db_path: Path): - """Set up test database with sample data""" - conn = sqlite3.connect(db_path) - cursor = conn.cursor() - - # Create tables - cursor.execute(""" - CREATE TABLE IF NOT EXISTS runs ( - id INTEGER PRIMARY KEY, - quest_name TEXT, - start_time TIMESTAMP, - end_time TIMESTAMP, - model TEXT, - template TEXT, - outcome TEXT, - reward REAL, - benchmark_name TEXT - )""") - - cursor.execute(""" - CREATE TABLE IF NOT EXISTS steps ( - run_id INTEGER, - step INTEGER, - observation TEXT, - choices TEXT, - action INTEGER, - reward REAL, - llm_response TEXT, - FOREIGN KEY(run_id) REFERENCES runs(id) - )""") - - # Insert test data - now = datetime.now() - cursor.execute( - """ - INSERT INTO runs (quest_name, start_time, end_time, model, template, outcome, reward, benchmark_name) - VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ("test1.qm", now - timedelta(hours=1), now, "test-model", "test-template", "SUCCESS", 1.0, "baseline"), - ) - run1_id = cursor.lastrowid +def _treatment(model: str = "gpt-5-mini", harness: str = "reasoning_recent") -> dict: + return build_treatment( + harness=harness, + model=model, + temperature=0.4, + system_template="system_role.jinja", + ).to_dict() - cursor.execute( - """ - INSERT INTO runs (quest_name, start_time, end_time, model, template, outcome, reward, benchmark_name) - VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ("test1.qm", now - timedelta(minutes=30), now, "test-model", "test-template", "FAILURE", 0.0, "baseline"), - ) - run2_id = cursor.lastrowid - # Insert another run with different benchmark - cursor.execute( +def _insert_run(conn, quest_name, start_time, end_time, outcome, reward, benchmark_id, treatment): + cursor = conn.execute( """ - INSERT INTO runs (quest_name, start_time, end_time, model, template, outcome, reward, benchmark_name) - VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - ("test2.qm", now, now + timedelta(minutes=30), "test-model", "test-template", "SUCCESS", 0.8, "experimental"), - ) - _ = cursor.lastrowid - - # Insert steps for first run - steps1 = [ - ( - run1_id, - 1, - "You are at a trading station.", - json.dumps([{"id": "1", "text": "Talk to merchant"}, {"id": "2", "text": "Leave station"}]), - 1, - 0.0, - '{"action": 1, "reasoning": "Should talk to merchant"}', - ), + INSERT INTO runs ( + schema_version, quest_file, quest_name, quest_checksum, quest_language, + engine_revision, agent_id, treatment, treatment_signature, benchmark_id, + start_time, end_time, run_duration, outcome, reward, usage, transcript_diagnostics, progress + ) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, ( - run1_id, - 2, - "The merchant greets you.", - json.dumps([{"id": "1", "text": "Buy"}, {"id": "2", "text": "Sell"}]), - 2, - 1.0, - '{"action": 2, "reasoning": "Better to sell"}', + SCHEMA_VERSION, + f"quests/{quest_name}", + quest_name, + "sha256:test", + "rus", + "git:test", + "llm_test-agent", + json.dumps(treatment), + treatment["signature"], + benchmark_id, + start_time, + end_time, + 5.0, + outcome, + reward, + json.dumps({"total_tokens": 10}), + json.dumps({"total_transitions": 2}), + json.dumps({"current": 50.0, "maximum": 100.0, "scored": True}), ), - ] - cursor.executemany( - """ - INSERT INTO steps (run_id, step, observation, choices, action, reward, llm_response) - VALUES (?, ?, ?, ?, ?, ?, ?) - """, - steps1, ) + return cursor.lastrowid - # Insert steps for second run - steps2 = [ - ( - run2_id, - 1, - "You are at a trading station.", - json.dumps([{"id": "1", "text": "Talk to merchant"}, {"id": "2", "text": "Leave station"}]), - 2, - 0.0, - '{"action": 2, "reasoning": "Should leave"}', - ) - ] - cursor.executemany( + +def _insert_transition(conn, run_id, index, choice_index): + before = QuestSnapshot( + location_id=str(index), + observation="You are at a trading station.", + choices=[{"id": "11", "text": "Talk to merchant"}, {"id": "12", "text": "Leave station"}], + saving={"locationId": index}, + ) + after = QuestSnapshot(location_id=str(index + 1), observation="The merchant greets you.", saving={}) + transition = QuestTransition( + index=index, + before=before, + action=QuestAction.choose(choice_index, before.choices[choice_index - 1]["id"], 1735689600000 + index), + after=after, + ) + payload = transition.to_dict() + conn.execute( """ - INSERT INTO steps (run_id, step, observation, choices, action, reward, llm_response) - VALUES (?, ?, ?, ?, ?, ?, ?) - """, - steps2, + INSERT INTO transitions ( + run_id, transition_index, before_state, action, after_state, + response, usage, progress, provenance, replay_status, reasoning_mode + ) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + run_id, + index, + json.dumps(payload["before"]), + json.dumps(payload["action"]), + json.dumps(payload["after"]), + None, + json.dumps({}), + json.dumps(payload["progress"]), + "runtime", + "pending", + None, + ), ) + +def setup_test_db(db_path: Path): + """Set up a schema-v2 test database with sample data""" + conn = sqlite3.connect(db_path) + ensure_v2_schema(conn) + + now = datetime.now() + treatment = _treatment(model="test-model") + + run1 = _insert_run(conn, "test1.qm", now - timedelta(hours=1), now, "SUCCESS", 1.0, "baseline", treatment) + run2 = _insert_run(conn, "test1.qm", now - timedelta(minutes=30), now, "FAILURE", 0.0, "baseline", treatment) + _insert_run(conn, "test2.qm", now, now + timedelta(minutes=30), "SUCCESS", 0.8, "experimental", treatment) + + _insert_transition(conn, run1, 1, 1) + _insert_transition(conn, run1, 2, 2) + _insert_transition(conn, run2, 1, 2) + conn.commit() conn.close() def test_analyze_quest_run(tmp_path): - """Test analyzing a specific quest run from database""" + """Analyze a specific quest run from the v2 database""" db_path = tmp_path / "metrics.db" setup_test_db(db_path) - # Test CLI command runner = CliRunner() with isolated_filesystem(): result = runner.invoke(app, ["analyze", "--quest", "test1.qm", "--db", str(db_path), "--debug"]) assert result.exit_code == 0 - # Check that summary info is present assert "Quest Run Summary" in result.stdout assert "test1.qm" in result.stdout assert "test-model" in result.stdout @@ -158,73 +152,66 @@ def test_analyze_quest_run(tmp_path): assert "FAILURE" in result.stdout assert "Total Runs: 2" in result.stdout - # Step info checking is no longer part of the output format - pass - def test_analyze_benchmark(tmp_path): - """Test analyzing benchmark results from database""" + """Analyze benchmark results from the v2 database""" db_path = tmp_path / "metrics.db" setup_test_db(db_path) - # Test CLI command for all benchmarks runner = CliRunner() with isolated_filesystem(): result = runner.invoke(app, ["analyze", "--benchmark", "baseline", "--db", str(db_path)]) assert result.exit_code == 0 - # Check that summary info is present assert "Benchmark Results" in result.stdout assert "Benchmark: baseline" in result.stdout assert "Total Runs: 2" in result.stdout assert "Success Rate: 50.0%" in result.stdout assert "Average Success Reward: 1.00" in result.stdout - # Check that model stats are present assert "Model Performance" in result.stdout assert "test-model" in result.stdout assert "50.0%" in result.stdout - # Check that quest stats are present assert "Quest Results" in result.stdout assert "test1.qm" in result.stdout +def test_analyze_benchmark_groups_by_treatment_signature(tmp_path): + """Benchmark analysis groups by canonical treatment, not by harness name""" + db_path = tmp_path / "metrics.db" + setup_test_db(db_path) + + runner = CliRunner() + with isolated_filesystem(): + result = runner.invoke(app, ["analyze", "--benchmark", "baseline", "--db", str(db_path)]) + assert result.exit_code == 0 + assert "Treatment Statistics" in result.stdout + assert "reasoning_recent" in result.stdout + + def test_analyze_specific_benchmark(tmp_path): - """Test analyzing a specific benchmark from database""" + """Analyze a specific benchmark id from the v2 database""" db_path = tmp_path / "metrics.db" setup_test_db(db_path) - # Test CLI command for specific benchmark runner = CliRunner() with isolated_filesystem(): result = runner.invoke(app, ["analyze", "--benchmark", "experimental", "--db", str(db_path)]) assert result.exit_code == 0 - # Check that summary info is present - assert "Benchmark Results" in result.stdout assert "Benchmark: experimental" in result.stdout assert "Total Runs: 1" in result.stdout assert "Success Rate: 100.0%" in result.stdout assert "Average Success Reward: 0.80" in result.stdout - - # Check that model stats are present - assert "Model Performance" in result.stdout - assert "test-model" in result.stdout - assert "100.0%" in result.stdout - - # Check that quest stats are present - assert "Quest Results" in result.stdout assert "test2.qm" in result.stdout -def test_analyze_metrics(tmp_path): - """Test analyze command with a valid metrics file""" - # Create test database +def test_analyze_metrics_returns_transitions(tmp_path): + """analyze_quest_run returns canonical transitions for each run""" db_path = tmp_path / "metrics.db" setup_test_db(db_path) - # Test analyze_quest_run function results = analyze_quest_run("test1.qm", db_path) assert results["quest_name"] == "test1.qm" assert results["total_runs"] == 2 @@ -232,20 +219,15 @@ def test_analyze_metrics(tmp_path): assert results["outcomes"]["FAILURE"] == 1 assert len(results["runs"]) == 2 - # Test CLI command - runner = CliRunner() - with isolated_filesystem(): - result = runner.invoke(app, ["analyze", "--quest", "test1.qm", "--db", str(db_path)]) - assert result.exit_code == 0 - assert "Quest Run Summary" in result.stdout - assert "test1.qm" in result.stdout - assert "test-model" in result.stdout - assert "SUCCESS" in result.stdout - assert "FAILURE" in result.stdout + run_with_two = next(run for run in results["runs"] if len(run["transitions"]) == 2) + assert run_with_two["model"] == "test-model" + assert run_with_two["harness"] == "reasoning_recent" + assert run_with_two["transitions"][0]["action"]["kind"] == "choose" + assert run_with_two["transitions"][0]["action"]["performed_at_ms"] == 1735689600001 -def test_analyze_no_metrics_dir(tmp_path): - """Test analyze command with non-existent metrics directory""" +def test_analyze_no_metrics_dir(): + """analyze with a non-existent database""" runner = CliRunner() with isolated_filesystem(): result = runner.invoke(app, ["analyze", "--quest", "test1.qm"]) @@ -254,24 +236,10 @@ def test_analyze_no_metrics_dir(tmp_path): def test_analyze_empty_metrics_dir(tmp_path): - """Test analyze command with empty metrics directory""" - # Create empty database + """analyze with an empty v2 database""" db_path = tmp_path / "metrics.db" conn = sqlite3.connect(db_path) - cursor = conn.cursor() - cursor.execute(""" - CREATE TABLE IF NOT EXISTS runs ( - id INTEGER PRIMARY KEY, - quest_name TEXT, - start_time TIMESTAMP, - end_time TIMESTAMP, - model TEXT, - template TEXT, - outcome TEXT, - reward REAL, - benchmark_name TEXT - )""") - conn.commit() + ensure_v2_schema(conn) conn.close() runner = CliRunner() @@ -282,8 +250,7 @@ def test_analyze_empty_metrics_dir(tmp_path): def test_analyze_invalid_file(tmp_path): - """Test analyze command with invalid database file""" - # Create invalid database file + """analyze with an invalid database file""" db_path = tmp_path / "metrics.db" with open(db_path, "w") as f: f.write("invalid data") @@ -295,9 +262,8 @@ def test_analyze_invalid_file(tmp_path): assert "Error analyzing quest run" in result.output -def test_analyze_invalid_benchmark_file(tmp_path): - """Test analyze command with invalid benchmark name""" - # Create test database +def test_analyze_invalid_benchmark_id(tmp_path): + """analyze with an unknown benchmark id""" db_path = tmp_path / "metrics.db" setup_test_db(db_path) @@ -308,66 +274,15 @@ def test_analyze_invalid_benchmark_file(tmp_path): assert "No benchmark data found for nonexistent" in result.output -def test_analyze_benchmark_directory(tmp_path): - """Test analyze command with benchmark directory""" - # Create test database with multiple benchmarks +def test_analyze_run_by_id_prints_transitions(tmp_path): + """analyze --run-id renders canonical transitions""" db_path = tmp_path / "metrics.db" - conn = sqlite3.connect(db_path) - cursor = conn.cursor() - - # Create tables - cursor.execute(""" - CREATE TABLE IF NOT EXISTS runs ( - id INTEGER PRIMARY KEY, - quest_name TEXT, - start_time TIMESTAMP, - end_time TIMESTAMP, - model TEXT, - template TEXT, - outcome TEXT, - reward REAL, - benchmark_name TEXT - )""") - - # Insert test data for multiple benchmarks - now = datetime.now() - test_data = [ - ("test1.qm", now, "test-model", "test-template", "SUCCESS", 1.0, "benchmark1"), - ("test2.qm", now, "test-model", "test-template", "FAILURE", 0.0, "benchmark1"), - ("test3.qm", now, "test-model", "test-template", "SUCCESS", 0.5, "benchmark2"), - ] - - for quest, time, model, template, outcome, reward, benchmark in test_data: - cursor.execute( - """ - INSERT INTO runs (quest_name, start_time, end_time, model, template, outcome, reward, benchmark_name) - VALUES (?, ?, ?, ?, ?, ?, ?, ?) - """, - (quest, time, time, model, template, outcome, reward, benchmark), - ) - - conn.commit() - conn.close() + setup_test_db(db_path) - # Test CLI command runner = CliRunner() with isolated_filesystem(): - result = runner.invoke(app, ["analyze", "--benchmark", "benchmark1", "--db", str(db_path)]) + result = runner.invoke(app, ["analyze", "--run-id", "1", "--db", str(db_path), "--format", "detail"]) assert result.exit_code == 0 - - # Check that summary info is present - assert "Benchmark Results" in result.stdout - assert "Benchmark: benchmark1" in result.stdout - assert "Total Runs: 2" in result.stdout - assert "Success Rate: 50.0%" in result.stdout - assert "Average Success Reward: 1.00" in result.stdout - - # Check that model stats are present - assert "Model Performance" in result.stdout - assert "test-model" in result.stdout - assert "50.0%" in result.stdout - - # Check that quest stats are present - assert "Quest Results" in result.stdout - assert "test1.qm" in result.stdout - assert "test2.qm" in result.stdout + assert "Transitions (2 total)" in result.stdout + assert "Treatment:" in result.stdout + assert "Talk to merchant" in result.stdout diff --git a/llm_quest_benchmark/tests/core/test_migration.py b/llm_quest_benchmark/tests/core/test_migration.py new file mode 100644 index 0000000..e54c76e --- /dev/null +++ b/llm_quest_benchmark/tests/core/test_migration.py @@ -0,0 +1,333 @@ +"""Tests for the one-time legacy -> schema-v2 migration.""" + +import json +import sqlite3 + +import pytest + +from llm_quest_benchmark.core.migration import migrate_legacy_json, migrate_records +from llm_quest_benchmark.schemas.records import SCHEMA_VERSION, RunRecord + +LEGACY_AGENT_CONFIG = { + "model": "gpt-5-mini", + "harness": "reasoning_recent", + "temperature": 0.4, + "system_template": "system_role.jinja", + "compaction_interval": 50, +} + + +def _legacy_json(agent_config=LEGACY_AGENT_CONFIG, final_state=True) -> dict: + return { + "run_id": 12, + "quest_file": "quests/Boat.qm", + "quest_name": "Boat", + "start_time": "2026-02-15T00:00:00", + "end_time": "2026-02-15T00:00:20", + "agent_id": "llm_gpt-5-mini", + "agent_config": agent_config, + "outcome": "FAILURE", + "reward": 0.0, + "run_duration": 20.0, + "benchmark_id": "bench_legacy", + "final_state": ( + { + "location_id": "9", + "text": "The end", + "choices": [], + "reward": 0.0, + "done": True, + "info": {}, + } + if final_state + else None + ), + "usage": {"prompt_tokens": 20, "completion_tokens": 10, "total_tokens": 30}, + "metrics": {"total_steps": 2}, + "steps": [ + { + "step": 1, + "location_id": "1", + "observation": "start", + "choices": {"1": "go", "2": "stay"}, + "llm_decision": { + "analysis": "a", + "reasoning": "r", + "is_default": False, + "parse_mode": "json_direct", + "choice": {"1": "go"}, + "prompt_tokens": 10, + "completion_tokens": 5, + "total_tokens": 15, + "estimated_cost_usd": 0.001, + }, + }, + { + "step": 2, + "location_id": "2", + "observation": "middle", + "choices": {"1": "north", "2": "south"}, + "llm_decision": { + "analysis": "b", + "reasoning": "r2", + "is_default": False, + "choice": {"2": "south"}, + "prompt_tokens": 10, + "completion_tokens": 5, + "total_tokens": 15, + "estimated_cost_usd": 0.001, + }, + }, + { + # Logger-only terminal pseudo-step: state, not an executed action. + "step": 3, + "location_id": "9", + "observation": "The end", + "choices": {}, + "llm_decision": {"is_default": True, "choice": None}, + }, + ], + } + + +def test_legacy_json_maps_deterministically_to_expected_v2_record(): + record = migrate_legacy_json(_legacy_json()) + + assert record.schema_version == SCHEMA_VERSION + assert record.run_id == 12 + assert record.quest_name == "Boat" + assert record.agent_id == "llm_gpt-5-mini" + assert record.benchmark_id == "bench_legacy" + assert record.outcome == "FAILURE" + + # Three legacy rows, but only two executed actions. + assert len(record.transitions) == 2 + first, second = record.transitions + + assert first.before.location_id == "1" + assert [c["text"] for c in first.before.choices] == ["go", "stay"] + assert first.action.choice_index == 1 + assert first.after.location_id == "2" + assert first.after.observation == "middle" + + assert second.action.choice_index == 2 + # The last after-state comes from final_state when it exists. + assert second.after.location_id == "9" + assert second.after.observation == "The end" + assert second.after.done is True + + # Usage and transcript diagnostics are recomputed from mapped transitions. + assert record.usage["total_tokens"] == 30 + assert record.usage["prompt_tokens"] == 20 + assert record.transcript_diagnostics["total_steps"] == 2 + assert record.transcript_diagnostics["migrated_from"] == "legacy" + + +def test_migrated_records_mark_unknowable_fields_and_block_resume(): + record = migrate_legacy_json(_legacy_json()) + + for transition in record.transitions: + assert transition.provenance == "legacy_mapped" + assert transition.replay_status == "unavailable" + assert transition.action.performed_at_ms is None + assert transition.action.choice_id is None + assert transition.before.saving is None + assert "saving" in transition.before.unavailable_fields + assert "choice_ids" in transition.before.unavailable_fields + assert "params_state" in transition.before.unavailable_fields + assert not transition.is_replayable + + assert record.quest_language == "unavailable" + assert record.engine_revision == "unavailable" + assert not record.is_resumable + + +def test_missing_post_state_is_marked_unavailable_not_invented(): + payload = _legacy_json(final_state=False) + # Drop the terminal pseudo-step so the last action has no observed post-state. + payload["steps"] = payload["steps"][:2] + + record = migrate_legacy_json(payload) + + assert len(record.transitions) == 2 + assert record.transitions[0].after.location_id == "2" # observed next row + assert record.transitions[1].after.is_unavailable + assert record.transitions[1].after.location_id == "" + + +def test_non_consecutive_rows_do_not_become_a_post_state(): + payload = _legacy_json(final_state=False) + payload["steps"] = payload["steps"][:2] + payload["steps"][1]["step"] = 7 # a gap: row 2 is not the observed next state + + record = migrate_legacy_json(payload) + + assert record.transitions[0].after.is_unavailable + + +def test_unknown_configuration_gets_an_explicit_unknown_treatment(): + record = migrate_legacy_json(_legacy_json(agent_config=None)) + + treatment = record.treatment + assert treatment["harness"] == "unknown" + assert treatment["prompt"] == "unknown" + assert treatment["memory"] == "unknown" + assert treatment["loop"] == "unknown" + assert treatment["reasoning"] == "unknown" + assert treatment["signature"].startswith("t2_") + + +def test_known_configuration_is_resolved_from_the_canonical_registry(): + record = migrate_legacy_json(_legacy_json()) + + treatment = record.treatment + assert treatment["harness"] == "reasoning_recent" + assert treatment["memory"] == "recent_window" + assert treatment["loop"] == "single_call" + assert treatment["prompt"] == "reasoning.jinja" + + +def test_migrating_a_v2_record_is_rejected(): + with pytest.raises(ValueError, match="already schema v2"): + migrate_legacy_json({"schema_version": SCHEMA_VERSION, "transitions": []}) + + +def test_migrate_json_tree_writes_v2_records(tmp_path): + source = tmp_path / "results" / "llm_gpt-5-mini" / "Boat" / "run_12" + source.mkdir(parents=True) + (source / "run_summary.json").write_text(json.dumps(_legacy_json()), encoding="utf-8") + + output = tmp_path / "results_v2" + report = migrate_records(tmp_path / "results", output) + + assert report.kind == "json" + assert report.runs_migrated == 1 + assert report.transitions_migrated == 2 + assert report.resumable_runs == 0 + + migrated_path = output / "llm_gpt-5-mini" / "Boat" / "run_12" / "run_summary.json" + assert migrated_path.exists() + record = RunRecord.from_dict(json.loads(migrated_path.read_text(encoding="utf-8"))) + assert len(record.transitions) == 2 + + +def _legacy_sqlite(path) -> None: + conn = sqlite3.connect(path) + conn.execute( + """ + CREATE TABLE runs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + quest_file TEXT, quest_name TEXT, start_time TIMESTAMP, end_time TIMESTAMP, + agent_id TEXT, agent_config TEXT, outcome TEXT, reward REAL, + run_duration REAL, benchmark_id TEXT + ) + """ + ) + conn.execute( + """ + CREATE TABLE steps ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + run_id INTEGER, step INTEGER, location_id TEXT, observation TEXT, + choices TEXT, action TEXT, llm_response TEXT + ) + """ + ) + cursor = conn.execute( + """ + INSERT INTO runs (quest_file, quest_name, start_time, end_time, agent_id, agent_config, + outcome, reward, run_duration, benchmark_id) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + "quests/Boat.qm", + "Boat", + "2026-02-15T00:00:00", + "2026-02-15T00:00:20", + "llm_gpt-5-mini", + json.dumps(LEGACY_AGENT_CONFIG), + "FAILURE", + 0.0, + 20.0, + "bench_legacy", + ), + ) + run_id = cursor.lastrowid + choices = json.dumps([{"id": "11", "text": "go"}, {"id": "12", "text": "stay"}]) + conn.executemany( + "INSERT INTO steps (run_id, step, location_id, observation, choices, action, llm_response) " + "VALUES (?, ?, ?, ?, ?, ?, ?)", + [ + # The model proposed 2, but the runner clamped the executed action to 1. + (run_id, 1, "1", "start", choices, "1", json.dumps({"action": 2, "reasoning": "r"})), + (run_id, 2, "2", "middle", choices, "2", json.dumps({"action": 2, "reasoning": "r2"})), + (run_id, 3, "9", "The end", json.dumps([]), "FAILURE", None), + ], + ) + conn.commit() + conn.close() + + +def test_migrate_legacy_sqlite_creates_a_v2_database(tmp_path): + source = tmp_path / "legacy.db" + _legacy_sqlite(source) + output = tmp_path / "metrics_v2.db" + + report = migrate_records(source, output) + + assert report.kind == "sqlite" + assert report.runs_migrated == 1 + assert report.transitions_migrated == 2 + assert report.resumable_runs == 0 + + conn = sqlite3.connect(output) + try: + run = conn.execute( + "SELECT schema_version, quest_name, agent_id, treatment_signature, outcome FROM runs" + ).fetchone() + assert run[0] == SCHEMA_VERSION + assert run[1] == "Boat" + assert run[2] == "llm_gpt-5-mini" + assert run[3].startswith("t2_") + assert run[4] == "FAILURE" + + transitions = conn.execute( + "SELECT transition_index, action, provenance, replay_status FROM transitions ORDER BY transition_index" + ).fetchall() + assert len(transitions) == 2 + first_action = json.loads(transitions[0][1]) + # SQLite's steps.action wins: it is the runner-executed action, while + # the model proposal in llm_response could disagree. + assert first_action["choice_index"] == 1 + assert first_action["choice_id"] == "11" # engine jump ids survive in SQLite + assert first_action["performed_at_ms"] is None + assert transitions[0][2] == "legacy_mapped" + assert transitions[0][3] == "unavailable" + finally: + conn.close() + + +def test_migrate_rejects_an_existing_output_database(tmp_path): + source = tmp_path / "legacy.db" + _legacy_sqlite(source) + output = tmp_path / "metrics_v2.db" + output.write_text("", encoding="utf-8") + + with pytest.raises(ValueError, match="output already exists"): + migrate_records(source, output) + + +def test_migrate_rejects_a_v2_source_database(tmp_path): + from llm_quest_benchmark.core.logging import ensure_v2_schema + + source = tmp_path / "already_v2.db" + conn = sqlite3.connect(source) + ensure_v2_schema(conn) + conn.close() + + with pytest.raises(ValueError, match="already schema v2"): + migrate_records(source, tmp_path / "out.db") + + +def test_migrate_missing_source_raises(tmp_path): + with pytest.raises(FileNotFoundError): + migrate_records(tmp_path / "nope", tmp_path / "out") diff --git a/llm_quest_benchmark/tests/core/test_progress.py b/llm_quest_benchmark/tests/core/test_progress.py new file mode 100644 index 0000000..2c2556b --- /dev/null +++ b/llm_quest_benchmark/tests/core/test_progress.py @@ -0,0 +1,201 @@ +"""Tests for state-based progress manifests and the monotonic tracker.""" + +from pathlib import Path + +import pytest + +from llm_quest_benchmark.core.progress import ProgressManifest, ProgressTracker, load_progress_manifest +from llm_quest_benchmark.schemas.records import QuestSnapshot + +BOAT_MANIFEST = Path(__file__).resolve().parents[3] / "configs" / "progress" / "Boat.yaml" + + +def _snapshot(location_id="1", params=None, observation="text", game_state="running", done=False) -> QuestSnapshot: + return QuestSnapshot( + location_id=location_id, + observation=observation, + choices=[{"id": "11", "text": "Continue"}], + params_state=params or [], + game_state=game_state, + done=done, + saving={"locationId": location_id}, + ) + + +def _manifest(tmp_path: Path, body: str) -> Path: + path = tmp_path / "manifest.yaml" + path.write_text(body, encoding="utf-8") + return path + + +VALID_BODY = """ +quest: Demo +version: 1 +milestones: + - id: entered + percent: 25 + match: + location_id: ["2"] + - id: armed + percent: 60 + match: + params_contains: ["Sword"] + - id: two_across + percent: 80 + match: + params_pattern: + pattern: "^xxx - " + min_count: 2 +""" + + +def test_manifest_validates_and_loads(tmp_path): + manifest = ProgressManifest.from_file(_manifest(tmp_path, VALID_BODY)) + + assert manifest.quest == "Demo" + assert [m.id for m in manifest.milestones] == ["entered", "armed", "two_across"] + assert manifest.maximum == 100.0 + + +def test_manifest_rejects_duplicate_ids(tmp_path): + body = VALID_BODY.replace("id: armed", "id: entered") + with pytest.raises(ValueError, match="Duplicate milestone id"): + ProgressManifest.from_file(_manifest(tmp_path, body)) + + +def test_manifest_rejects_descending_percentages(tmp_path): + body = VALID_BODY.replace("percent: 60", "percent: 10") + with pytest.raises(ValueError, match="percent decreases"): + ProgressManifest.from_file(_manifest(tmp_path, body)) + + +def test_manifest_rejects_out_of_range_percent(tmp_path): + body = VALID_BODY.replace("percent: 60", "percent: 160") + with pytest.raises(ValueError, match="within 0..100"): + ProgressManifest.from_file(_manifest(tmp_path, body)) + + +def test_manifest_rejects_unknown_predicates(tmp_path): + body = VALID_BODY.replace(' location_id: ["2"]', " vibes: high") + with pytest.raises(ValueError, match="unknown predicates"): + ProgressManifest.from_file(_manifest(tmp_path, body)) + + +def test_manifest_rejects_empty_milestones(tmp_path): + with pytest.raises(ValueError, match="non-empty 'milestones' list"): + ProgressManifest.from_file(_manifest(tmp_path, "quest: Demo\nversion: 1\nmilestones: []\n")) + + +def test_manifest_rejects_unsupported_version(tmp_path): + with pytest.raises(ValueError, match="Unsupported progress manifest version"): + ProgressManifest.from_file(_manifest(tmp_path, VALID_BODY.replace("version: 1", "version: 7"))) + + +def test_predicates_match_location_params_and_pattern(tmp_path): + manifest = ProgressManifest.from_file(_manifest(tmp_path, VALID_BODY)) + entered, armed, two_across = manifest.milestones + + assert entered.matches(_snapshot(location_id="2")) + assert not entered.matches(_snapshot(location_id="3")) + assert armed.matches(_snapshot(params=["Sword of dawn"])) + assert not armed.matches(_snapshot(params=["Shield"])) + assert two_across.matches(_snapshot(params=["xxx - Ах(2)", "xxx - Вау(1)"])) + assert not two_across.matches(_snapshot(params=["xxx - Ах(2)", "Вау (1) - xxx"])) + + +def test_progress_is_monotonic_across_a_restore(tmp_path): + tracker = ProgressTracker(manifest=ProgressManifest.from_file(_manifest(tmp_path, VALID_BODY))) + + tracker.seed(_snapshot(location_id="1")) + advanced = tracker.observe(_snapshot(location_id="2")) + assert advanced.current == pytest.approx(25.0) + assert advanced.newly_reached == ["entered"] + + # Restoring to the starting state must not lower recorded progress. + after_restore = tracker.observe(_snapshot(location_id="1")) + assert after_restore.current == pytest.approx(25.0) + assert after_restore.newly_reached == [] + assert after_restore.reached == ["entered"] + + +def test_stall_counter_resets_on_a_new_milestone(tmp_path): + tracker = ProgressTracker(manifest=ProgressManifest.from_file(_manifest(tmp_path, VALID_BODY))) + tracker.seed(_snapshot(location_id="1")) + + assert tracker.observe(_snapshot(location_id="1")).stalled_transitions == 1 + assert tracker.observe(_snapshot(location_id="1")).stalled_transitions == 2 + assert tracker.observe(_snapshot(location_id="2")).stalled_transitions == 0 + assert tracker.observe(_snapshot(location_id="2")).stalled_transitions == 1 + + +def test_terminal_success_is_always_full_progress(tmp_path): + tracker = ProgressTracker(manifest=ProgressManifest.from_file(_manifest(tmp_path, VALID_BODY))) + tracker.seed(_snapshot(location_id="1")) + + state = tracker.observe(_snapshot(location_id="9", game_state="win", done=True)) + + assert state.current == pytest.approx(100.0) + assert state.maximum == pytest.approx(100.0) + + +def test_without_a_manifest_progress_is_terminal_only(): + """No manifest means no guessing: only terminal success moves the needle.""" + tracker = ProgressTracker(manifest=None) + tracker.seed(_snapshot(location_id="1")) + + assert tracker.observe(_snapshot(location_id="5")).current == 0.0 + assert tracker.observe(_snapshot(location_id="9", game_state="fail", done=True)).current == 0.0 + + winning = ProgressTracker(manifest=None) + assert winning.observe(_snapshot(game_state="win", done=True)).current == pytest.approx(100.0) + + +def test_restore_from_seeds_a_resumed_tracker(tmp_path): + tracker = ProgressTracker(manifest=ProgressManifest.from_file(_manifest(tmp_path, VALID_BODY))) + tracker.seed(_snapshot(location_id="1")) + tracker.observe(_snapshot(location_id="2")) + state = tracker.state() + + resumed = ProgressTracker(manifest=tracker.manifest) + resumed.restore_from(state) + + assert resumed.state().current == pytest.approx(25.0) + assert resumed.state().reached == ["entered"] + # Already-reached milestones are not re-awarded on resume. + assert resumed.observe(_snapshot(location_id="2")).newly_reached == [] + + +def test_bundled_boat_manifest_is_valid_and_state_based(): + manifest = load_progress_manifest(str(BOAT_MANIFEST)) + + assert manifest is not None + assert manifest.quest == "Boat" + ids = [m.id for m in manifest.milestones] + assert "crossing_started" in ids + assert "all_gods_across" in ids + + tracker = ProgressTracker(manifest=manifest) + tracker.seed(_snapshot(location_id="1")) + state = tracker.observe( + _snapshot( + location_id="6", + params=[ + "Осталось 11 часов", + "Положение Богов:", + "xxx - Ах(2)", + "xxx - Бах(5)", + "Вау (1) - xxx", + "Гэ (10) - xxx", + ], + ) + ) + + assert "crossing_started" in state.reached + assert "two_gods_across" in state.reached + assert "three_gods_across" not in state.reached + assert state.current == pytest.approx(70.0) + + +def test_load_progress_manifest_returns_none_without_a_path(): + assert load_progress_manifest(None) is None + assert load_progress_manifest("") is None diff --git a/llm_quest_benchmark/tests/core/test_replay.py b/llm_quest_benchmark/tests/core/test_replay.py new file mode 100644 index 0000000..6d1e707 --- /dev/null +++ b/llm_quest_benchmark/tests/core/test_replay.py @@ -0,0 +1,290 @@ +"""Tests for deterministic replay verification and resume assembly.""" + +import copy +import json +from pathlib import Path + +import pytest + +from llm_quest_benchmark.core.replay import ( + ReplayError, + harness_config_from_record, + replay_record, + restore_progress_tracker, + verify_environment, +) +from llm_quest_benchmark.harnesses.specs import build_treatment +from llm_quest_benchmark.schemas.records import ( + ProgressState, + QuestAction, + QuestSnapshot, + QuestTransition, + RunRecord, +) + +REPO_ROOT = Path(__file__).resolve().parents[3] +BOAT = str(REPO_ROOT / "quests" / "Boat.qm") + + +def _snapshot(location_id: str, timestamp: int, choices=None) -> QuestSnapshot: + """Snapshot whose content depends on the transition timestamp. + + Real quests can branch on the timestamp handed to performJump, so the fake + engine mirrors that: a replayed timestamp change must change the state. + """ + return QuestSnapshot( + location_id=location_id, + observation=f"At {location_id} (t={timestamp})", + choices=choices if choices is not None else [{"id": "11", "text": "A"}, {"id": "12", "text": "B"}], + saving={"locationId": location_id, "t": timestamp}, + ) + + +class _FakeEngine: + """Deterministic fake engine: state is a function of action and timestamp.""" + + def __init__(self): + self.quest_file = BOAT + self.language = "rus" + self.forced_stop_reason = None + self._snapshot = None + self.calls = [] + + def reset(self): + self._snapshot = _snapshot("start", 0) + return self._snapshot + + def step(self, choice_index, performed_at_ms): + self.calls.append((choice_index, performed_at_ms)) + location = f"{self._snapshot.location_id}>{choice_index}" + self._snapshot = _snapshot(location, performed_at_ms) + return self._snapshot + + def restore(self, snapshot): + if snapshot.saving is None: + raise ValueError("Cannot restore a snapshot without a full engine saving") + self._snapshot = snapshot + return snapshot + + def close(self): + return None + + +def _recorded_run() -> tuple[RunRecord, _FakeEngine]: + """Execute a short run against the fake engine and record it.""" + engine = _FakeEngine() + snapshot = engine.reset() + transitions = [] + for index, (choice, timestamp) in enumerate([(1, 1000), (2, 2000), (1, 3000)], start=1): + before = snapshot + after = engine.step(choice, timestamp) + transitions.append( + QuestTransition( + index=index, + before=before, + action=QuestAction.choose(choice, before.choices[choice - 1]["id"], timestamp), + after=after, + ) + ) + snapshot = after + + record = RunRecord( + run_id=1, + quest_file=BOAT, + quest_name="Boat", + quest_checksum="", + quest_language="rus", + engine_revision="", + agent_id="agent", + treatment=build_treatment("reasoning_recent", "gpt-5-mini", 0.4, "system_role.jinja").to_dict(), + outcome="TRUNCATED", + transitions=transitions, + progress=ProgressState(current=30.0, reached=["a"]), + ) + return record, engine + + +def test_replay_of_an_unchanged_record_verifies_every_transition(): + record, _ = _recorded_run() + + result = replay_record(_FakeEngine(), record) + + assert result.verified_transitions == 3 + assert result.snapshot.digest == record.transitions[-1].after.digest + # Checkpoints are the active branch: the start plus one per executed choose. + assert len(result.checkpoints) == 4 + + +def test_replay_reexecutes_the_recorded_timestamps(): + record, _ = _recorded_run() + engine = _FakeEngine() + + replay_record(engine, record) + + assert engine.calls == [(1, 1000), (2, 2000), (1, 3000)] + + +def test_replay_detects_a_changed_action(): + """Even a self-consistent action edit is caught by the resulting state.""" + record, _ = _recorded_run() + record.transitions[1].action.choice_index = 1 + record.transitions[1].action.choice_id = "11" + + with pytest.raises(ReplayError, match="after-state diverged"): + replay_record(_FakeEngine(), record) + + +def test_replay_detects_a_changed_timestamp(): + record, _ = _recorded_run() + record.transitions[1].action.performed_at_ms = 2999 + + with pytest.raises(ReplayError, match="after-state diverged"): + replay_record(_FakeEngine(), record) + + +def test_replay_detects_a_changed_choice_id(): + record, _ = _recorded_run() + record.transitions[0].action.choice_id = "999" + + with pytest.raises(ReplayError, match="choice id diverged"): + replay_record(_FakeEngine(), record) + + +def test_replay_detects_a_mutated_resulting_state(): + record, _ = _recorded_run() + record.transitions[2].after = QuestSnapshot( + location_id="tampered", observation="tampered", saving={"locationId": "tampered"} + ) + + with pytest.raises(ReplayError, match="after-state diverged"): + replay_record(_FakeEngine(), record) + + +def test_replay_detects_a_mutated_before_state(): + record, _ = _recorded_run() + record.transitions[1].before = QuestSnapshot( + location_id="tampered", observation="tampered", saving={"locationId": "tampered"} + ) + + with pytest.raises(ReplayError, match="before-state diverged"): + replay_record(_FakeEngine(), record) + + +def test_replay_rejects_non_replayable_transitions(): + record, _ = _recorded_run() + record.transitions[0].provenance = "legacy_mapped" + record.transitions[0].replay_status = "unavailable" + + with pytest.raises(ReplayError, match="not replayable"): + replay_record(_FakeEngine(), record) + + +def test_replay_reproduces_restores_and_truncates_the_active_branch(): + record, engine = _recorded_run() + # Append a restore back to the first checkpoint, then one more choose. + current = record.transitions[-1].after + target = record.transitions[0].before + engine.restore(target) + record.transitions.append( + QuestTransition( + index=4, + before=current, + action=QuestAction.restore(1), + after=target, + ) + ) + after = engine.step(2, 4000) + record.transitions.append( + QuestTransition( + index=5, + before=target, + action=QuestAction.choose(2, "12", 4000), + after=after, + ) + ) + + result = replay_record(_FakeEngine(), record) + + assert result.verified_transitions == 5 + assert result.snapshot.digest == after.digest + # After restoring to checkpoint 1 and taking one step, the branch is 2 deep. + assert len(result.checkpoints) == 2 + + +def test_verify_environment_detects_a_changed_quest(): + record, _ = _recorded_run() + record.quest_checksum = "sha256:not-this-quest" + + with pytest.raises(ReplayError, match="Quest checksum mismatch"): + verify_environment(record, BOAT) + + +def test_verify_environment_detects_a_changed_engine(): + record, _ = _recorded_run() + record.engine_revision = "git:0000000000000000000000000000000000000000" + + with pytest.raises(ReplayError, match="Engine revision mismatch"): + verify_environment(record, BOAT) + + +def test_harness_config_is_rebuilt_from_the_recorded_treatment(): + record, _ = _recorded_run() + + config = harness_config_from_record(record) + + assert config.harness == "reasoning_recent" + assert config.model == "gpt-5-mini" + assert config.temperature == pytest.approx(0.4) + assert config.system_template == "system_role.jinja" + # Rebuilding must reproduce the exact recorded signature. + assert config.treatment().signature == record.treatment_signature + + +def test_harness_config_rejects_an_unknown_migrated_treatment(): + record, _ = _recorded_run() + record.treatment = {"harness": "unknown", "model": "unavailable"} + + with pytest.raises(ReplayError, match="no resolvable harness treatment"): + harness_config_from_record(record) + + +def test_backtracking_knobs_survive_the_record_round_trip(): + record, _ = _recorded_run() + record.treatment = build_treatment( + "backtracking", "gpt-5-mini", 0.4, "system_role.jinja", {"restore_limit": 2, "compaction_interval": 10} + ).to_dict() + + config = harness_config_from_record(record) + + assert config.harness == "backtracking" + assert config.restore_limit == 2 + assert config.compaction_interval == 10 + + +def test_restore_progress_tracker_recovers_the_recorded_state(tmp_path): + manifest = tmp_path / "manifest.yaml" + manifest.write_text( + 'quest: Demo\nversion: 1\nmilestones:\n - id: a\n percent: 30\n match:\n location_id: ["x"]\n', + encoding="utf-8", + ) + record, _ = _recorded_run() + record.progress = ProgressState(current=30.0, reached=["a"], stalled_transitions=2, manifest=str(manifest)) + + tracker = restore_progress_tracker(record) + + assert tracker.manifest is not None + assert tracker.state().current == pytest.approx(30.0) + assert tracker.state().reached == ["a"] + assert tracker.state().stalled_transitions == 2 + + +def test_record_json_round_trip_survives_replay(): + """A record read back from disk must replay exactly like the in-memory one.""" + record, _ = _recorded_run() + reloaded = RunRecord.from_dict(json.loads(json.dumps(record.to_dict()))) + + result = replay_record(_FakeEngine(), reloaded) + + assert result.verified_transitions == len(record.transitions) + assert result.snapshot.digest == record.transitions[-1].after.digest + assert copy.deepcopy(reloaded.to_dict()) == record.to_dict() diff --git a/llm_quest_benchmark/tests/core/test_runner.py b/llm_quest_benchmark/tests/core/test_runner.py index 2f5b563..ea5a190 100644 --- a/llm_quest_benchmark/tests/core/test_runner.py +++ b/llm_quest_benchmark/tests/core/test_runner.py @@ -1,251 +1,396 @@ -"""Tests for runner timeout and max-steps handling.""" +"""Tests for runner transitions, truncation, checkpoints, and timeout handling.""" from concurrent.futures import TimeoutError as FuturesTimeoutError from types import SimpleNamespace +import pytest + +from llm_quest_benchmark.constants import DEFAULT_QUEST from llm_quest_benchmark.core.runner import QuestRunner, run_quest_with_timeout from llm_quest_benchmark.environments.state import QuestOutcome +from llm_quest_benchmark.harnesses.specs import build_treatment +from llm_quest_benchmark.players.base import DecisionContext, QuestPlayer +from llm_quest_benchmark.schemas.records import QuestAction, QuestSnapshot +from llm_quest_benchmark.schemas.response import LLMResponse + +QUEST = str(DEFAULT_QUEST) + + +def _snapshot(location_id: str, done: bool = False, game_state: str = "running", choices=None) -> QuestSnapshot: + return QuestSnapshot( + location_id=location_id, + observation=f"Observation at {location_id}", + choices=[] if done else (choices if choices is not None else [{"id": "11", "text": "Continue"}]), + params_state=[f"Loc: {location_id}"], + done=done, + game_state=game_state, + saving={"locationId": location_id}, + ) -def test_timeout_records_benchmark_id(monkeypatch): - recorded = {} - - class DummyFuture: - def result(self, timeout): # noqa: ARG002 - raise FuturesTimeoutError() - - def cancel(self): - return None +class _FakeEnv: + """Snapshot-returning environment stub with checkpoint restore.""" - class DummyExecutor: - def __init__(self, max_workers): # noqa: ARG002 - self.future = DummyFuture() - - def submit(self, fn, quest): # noqa: ARG002 - return self.future - - def shutdown(self, wait=False, cancel_futures=True): # noqa: ARG002 - return None + def __init__(self, terminate_at: int | None = None, success: bool = True): + self.quest_file = QUEST + self.language = "rus" + self.forced_stop_reason = None + self.terminate_at = terminate_at + self.success = success + self.taken = 0 + self.restored = [] + self._snapshot = None - class DummyLogger: - def __init__(self, debug=False, agent=None): # noqa: ARG002 - self.current_run_id = 1 - self.logger = SimpleNamespace( - warning=lambda *a, **k: None, - error=lambda *a, **k: None, - info=lambda *a, **k: None, - ) + @property + def snapshot(self): + return self._snapshot - def set_quest_file(self, quest_path): # noqa: ARG002 - return None + def reset(self): + self.taken = 0 + self._snapshot = _snapshot("loc0") + return self._snapshot + + def step(self, choice_index, performed_at_ms): + assert isinstance(performed_at_ms, int) and performed_at_ms > 0 + self.taken += 1 + done = self.terminate_at is not None and self.taken >= self.terminate_at + game_state = "running" if not done else ("win" if self.success else "fail") + self._snapshot = _snapshot(f"loc{self.taken}", done=done, game_state=game_state) + return self._snapshot + + def restore(self, snapshot): + self.restored.append(snapshot.location_id) + self._snapshot = snapshot + return snapshot + + def close(self): + return None - def _init_connection(self): - return None - def set_quest_outcome(self, outcome, reward, final_state=None, benchmark_id=None): - recorded["outcome"] = outcome - recorded["reward"] = reward - recorded["final_state"] = final_state - recorded["benchmark_id"] = benchmark_id +class _FakeAgent(QuestPlayer): + def __init__(self): + super().__init__() + self.action_calls = 0 + self.transitions = [] + self.agent_id = "fake_agent" + self.harness_name = "human" - class DummyRunner: - def __init__(self, **kwargs): # noqa: ARG002 - return None + def reset(self): + return None - def run(self, quest): # noqa: ARG002 - return QuestOutcome.SUCCESS + def on_game_start(self): + return None - def request_stop(self, reason): # noqa: ARG002 - recorded["stop_reason"] = reason + def on_game_end(self, final_snapshot): + return None - def snapshot_state(self): - return {"location_id": "X", "done": False} + def on_transition(self, transition): + self.transitions.append(transition) - monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestLogger", DummyLogger) - monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestEnvironment", lambda *a, **k: object()) - monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestRunner", DummyRunner) - monkeypatch.setattr("llm_quest_benchmark.core.runner.ThreadPoolExecutor", DummyExecutor) - - agent = SimpleNamespace(agent_id="llm_test") - cfg = SimpleNamespace(agent_id="llm_test", benchmark_id="bench_timeout_1") + def _get_action_impl(self, observation, choices): + self.action_calls += 1 + self._last_response = LLMResponse(action=1) + return 1 - outcome = run_quest_with_timeout("quests/mock.qm", agent, timeout=1, agent_config=cfg) + def __str__(self): + return "FakeAgent" - assert outcome == QuestOutcome.TIMEOUT - assert recorded["stop_reason"] == "timeout" - assert recorded["outcome"] == QuestOutcome.TIMEOUT.name - assert recorded["benchmark_id"] == "bench_timeout_1" - assert recorded["final_state"] == {"location_id": "X", "done": False} +class _RestoringAgent(_FakeAgent): + """Emits one restore to checkpoint 1 on its third decision.""" -class _NonTerminatingEnv: - """Fake quest environment with a single choice that never ends the quest.""" + supports_restore = True def __init__(self): - self.state = {"choices": [{"text": "Continue"}], "location_id": "loc", "reward": 0.0} + super().__init__() + self.contexts = [] - def reset(self): - return "Same observation forever." + def get_quest_action(self, observation, choices, context: DecisionContext) -> QuestAction: + self.contexts.append(context) + self.action_calls += 1 + self._last_response = LLMResponse(action=1) + if self.action_calls == 3 and context.restore_allowed: + return QuestAction.restore(1) + return QuestAction(kind="choose", choice_index=1) + + +class _FakeConfig: + """Minimal agent config exposing the canonical treatment the runner records.""" + + def __init__(self, agent_id="fake_agent_id", benchmark_id=None, restore_limit=None): + self.agent_id = agent_id + self.benchmark_id = benchmark_id + self.restore_limit = restore_limit + + def treatment(self): + return build_treatment( + harness="reasoning_recent", + model="gpt-5-mini", + temperature=0.4, + system_template="system_role.jinja", + ) - def step(self, action): # noqa: ARG002 - return "Same observation forever.", False, False, {} +class _DummyQuestLogger: + def __init__(self): + self.current_run_id = 1 + self.started = None + self.transitions = [] + self.outcomes = [] -class _TerminatingEnv: - """Fake quest environment that ends successfully after `terminate_at` steps.""" + def start_run(self, **kwargs): + self.started = kwargs + return 1 - def __init__(self, terminate_at: int): - self.terminate_at = terminate_at - self._taken = 0 - self.state = {"choices": [{"text": "Continue"}], "location_id": "loc", "reward": 0.0} + def log_transition(self, transition): + self.transitions.append(transition) - def reset(self): - return "Initial observation." + def adopt_transitions(self, transitions): + for transition in transitions: + self.log_transition(transition) - def step(self, action): # noqa: ARG002 - self._taken += 1 - done = self._taken >= self.terminate_at - if done: - self.state = {"choices": [], "location_id": "loc", "reward": 1.0} - return "Observation.", done, done, {} + def finish_run(self, outcome, reward=0.0, terminal_snapshot=None, progress=None, diagnostics=None): + self.outcomes.append( + { + "outcome": outcome, + "reward": reward, + "terminal_snapshot": terminal_snapshot, + "progress": progress, + "diagnostics": diagnostics, + } + ) -class _FakeAgent: - def __init__(self): - self.action_calls = 0 - self.step_states = [] +def _runner(monkeypatch, env, agent, **kwargs): + monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestEnvironment", lambda *a, **k: env) + logger = _DummyQuestLogger() + runner = QuestRunner(agent=agent, quest_logger=logger, **kwargs) + return runner, logger - def reset(self): - return None - def on_game_start(self): - return None +def test_run_records_one_transition_per_executed_action(monkeypatch): + env = _FakeEnv(terminate_at=3) + agent = _FakeAgent() + runner, logger = _runner(monkeypatch, env, agent) - def on_game_end(self, final_state): # noqa: ARG002 - return None + outcome = runner.run(QUEST) - def on_step(self, agent_state): - self.step_states.append(agent_state) + assert outcome == QuestOutcome.SUCCESS + assert agent.action_calls == 3 + assert len(logger.transitions) == 3 + # No synthetic terminal decision row: terminal state is the last after-snapshot. + assert logger.transitions[-1].after.done is True + assert logger.outcomes[-1]["terminal_snapshot"] is logger.transitions[-1].after + assert [t.index for t in logger.transitions] == [1, 2, 3] + assert agent.transitions == logger.transitions - def get_action(self, observation, choices): # noqa: ARG002 - self.action_calls += 1 - return 1 - def get_last_response(self): - return None +def test_transition_carries_before_action_and_after(monkeypatch): + env = _FakeEnv(terminate_at=1) + agent = _FakeAgent() + runner, logger = _runner(monkeypatch, env, agent) - def __str__(self): - return "FakeAgent" + runner.run(QUEST) + transition = logger.transitions[0] + assert transition.before.location_id == "loc0" + assert transition.after.location_id == "loc1" + assert transition.action.is_choose + assert transition.action.choice_index == 1 + assert transition.action.choice_id == "11" + assert transition.action.performed_at_ms is not None + assert transition.provenance == "runtime" -class _DummyQuestLogger: - def __init__(self): - self.current_run_id = 1 - self.steps_logged = 0 - self.step_states = [] - self.outcomes = [] - def set_quest_file(self, quest_path): # noqa: ARG002 - return None +def test_run_metadata_is_initialized_before_execution(monkeypatch): + env = _FakeEnv(terminate_at=1) + agent = _FakeAgent() + runner, logger = _runner(monkeypatch, env, agent) - def log_step(self, agent_state): - self.steps_logged += 1 - self.step_states.append(agent_state) + runner.run(QUEST) - def set_quest_outcome(self, outcome, reward, benchmark_id=None, final_state=None): - self.outcomes.append( - {"outcome": outcome, "reward": reward, "benchmark_id": benchmark_id, "final_state": final_state} - ) + assert logger.started["quest_file"] == QUEST + assert logger.started["quest_checksum"].startswith("sha256:") + assert logger.started["engine_revision"] + assert logger.started["treatment"]["signature"].startswith("t2_") def test_max_steps_none_preserves_unbounded_loop(monkeypatch): - """Backward compatibility: omitting max_steps (None) must not cut a quest - short before its natural terminal state, regardless of step count.""" - env = _TerminatingEnv(terminate_at=5) - monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestEnvironment", lambda *a, **k: env) - + """Omitting max_steps must not cut a quest short before its terminal state.""" + env = _FakeEnv(terminate_at=5) agent = _FakeAgent() - quest_logger = _DummyQuestLogger() - runner = QuestRunner(agent=agent, quest_logger=quest_logger, max_steps=None) + runner, logger = _runner(monkeypatch, env, agent, max_steps=None) - outcome = runner.run("quests/mock.qm") + outcome = runner.run(QUEST) assert outcome == QuestOutcome.SUCCESS assert agent.action_calls == 5 assert runner.step_count == 5 - assert quest_logger.outcomes[-1]["outcome"] == QuestOutcome.SUCCESS.name - + assert logger.outcomes[-1]["outcome"] == QuestOutcome.SUCCESS.name -def test_max_steps_caps_a_non_terminating_quest_as_failure(monkeypatch): - """A quest that never reaches a terminal state must stop at max_steps and - be recorded as FAILURE, not loop forever.""" - env = _NonTerminatingEnv() - monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestEnvironment", lambda *a, **k: env) +def test_max_steps_records_resumable_truncated_outcome(monkeypatch): + """An explicit step limit is a deliberate stop, not a quest failure.""" + env = _FakeEnv(terminate_at=None) agent = _FakeAgent() - quest_logger = _DummyQuestLogger() - runner = QuestRunner(agent=agent, quest_logger=quest_logger, max_steps=3) + runner, logger = _runner(monkeypatch, env, agent, max_steps=3) - outcome = runner.run("quests/mock.qm") + outcome = runner.run(QUEST) - assert outcome == QuestOutcome.FAILURE - assert agent.action_calls == 3 # exactly max_steps actions taken, no more - assert runner.step_count == 3 - assert quest_logger.outcomes # the cap path must record an outcome, not just return - assert quest_logger.outcomes[-1]["outcome"] == QuestOutcome.FAILURE.name + assert outcome == QuestOutcome.TRUNCATED + assert outcome.is_resumable + assert agent.action_calls == 3 + assert len(logger.transitions) == 3 + assert logger.outcomes[-1]["outcome"] == QuestOutcome.TRUNCATED.name def test_max_steps_larger_than_natural_termination_does_not_interfere(monkeypatch): - """A generous max_steps must not change the outcome of a quest that - terminates naturally well before the cap.""" - env = _TerminatingEnv(terminate_at=2) - monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestEnvironment", lambda *a, **k: env) - + env = _FakeEnv(terminate_at=2) agent = _FakeAgent() - quest_logger = _DummyQuestLogger() - runner = QuestRunner(agent=agent, quest_logger=quest_logger, max_steps=60) + runner, _ = _runner(monkeypatch, env, agent, max_steps=60) - outcome = runner.run("quests/mock.qm") - - assert outcome == QuestOutcome.SUCCESS + assert runner.run(QUEST) == QuestOutcome.SUCCESS assert agent.action_calls == 2 - assert runner.step_count == 2 -def test_runner_delivers_one_canonical_state_to_agent_callback_and_logger(monkeypatch): - env = _TerminatingEnv(terminate_at=1) - monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestEnvironment", lambda *a, **k: env) - agent = _FakeAgent() - quest_logger = _DummyQuestLogger() - callback_states = [] +def test_backtracking_restore_truncates_only_the_active_branch(monkeypatch): + env = _FakeEnv(terminate_at=5) + agent = _RestoringAgent() + runner, logger = _runner(monkeypatch, env, agent, max_steps=6) + + runner.run(QUEST) + + restores = [t for t in logger.transitions if t.action.is_restore] + assert len(restores) == 1 + assert restores[0].action.checkpoint_index == 1 + # The restore is visible in the chronological log, before and after it. + assert len(logger.transitions) > 1 + assert env.restored == ["loc0"] + assert runner._restores_accepted == 1 + assert logger.outcomes[-1]["diagnostics"]["restore"]["accepted"] == 1 + + +def test_restore_is_rejected_for_harnesses_without_the_capability(monkeypatch): + class _SneakyAgent(_FakeAgent): + def get_quest_action(self, observation, choices, context): + self.action_calls += 1 + self._last_response = LLMResponse(action=1) + return QuestAction.restore(1) + + env = _FakeEnv(terminate_at=2) + agent = _SneakyAgent() + runner, logger = _runner(monkeypatch, env, agent, max_steps=4) - runner = QuestRunner( - agent=agent, - quest_logger=quest_logger, - callbacks=[lambda event, data: callback_states.append(data) if event == "game_state" else None], + runner.run(QUEST) + + assert not any(t.action.is_restore for t in logger.transitions) + assert env.restored == [] + assert runner._restore_attempts > 0 + assert runner._restores_accepted == 0 + + +def test_restore_limit_closes_the_budget(monkeypatch): + env = _FakeEnv(terminate_at=None) + agent = _RestoringAgent() + runner, _ = _runner( + monkeypatch, + env, + agent, + max_steps=6, + agent_config=_FakeConfig(restore_limit=1), ) - assert runner.run("quests/mock.qm") == QuestOutcome.SUCCESS - assert len(agent.step_states) == 1 - assert len(callback_states) == 1 - assert quest_logger.steps_logged == 2 # executed decision plus logger-only terminal state - assert agent.step_states[0] is callback_states[0] is quest_logger.step_states[0] + runner.run(QUEST) + assert runner._restores_accepted <= 1 + # Once the budget is spent the runner stops offering restore. + assert agent.contexts[-1].restores_remaining == 0 + assert agent.contexts[-1].restore_allowed is False -def test_run_quest_with_timeout_forwards_max_steps_to_runner(monkeypatch): - """run_quest_with_timeout must thread max_steps through to QuestRunner.""" - captured = {} + +def test_timeout_records_outcome_and_final_snapshot(monkeypatch): + recorded = {} + + class DummyFuture: + def result(self, timeout): # noqa: ARG002 + raise FuturesTimeoutError() + + def cancel(self): + return None + + class DummyExecutor: + def __init__(self, max_workers): # noqa: ARG002 + self.future = DummyFuture() + + def submit(self, fn, quest): # noqa: ARG002 + return self.future + + def shutdown(self, wait=False, cancel_futures=True): # noqa: ARG002 + return None class DummyLogger: def __init__(self, debug=False, agent=None): # noqa: ARG002 self.current_run_id = 1 self.logger = SimpleNamespace( - warning=lambda *a, **k: None, error=lambda *a, **k: None, info=lambda *a, **k: None + warning=lambda *a, **k: None, + error=lambda *a, **k: None, + info=lambda *a, **k: None, ) - def set_quest_file(self, quest_path): # noqa: ARG002 + def finish_run(self, outcome, reward=0.0, terminal_snapshot=None, progress=None, diagnostics=None): + recorded.update( + { + "outcome": outcome, + "reward": reward, + "terminal_snapshot": terminal_snapshot, + "diagnostics": diagnostics, + } + ) + + class DummyRunner: + def __init__(self, **kwargs): # noqa: ARG002 + return None + + def run(self, quest): # noqa: ARG002 + return QuestOutcome.SUCCESS + + def request_stop(self, reason): + recorded["stop_reason"] = reason + + def current_snapshot(self): + return _snapshot("loc7") + + def progress_state(self): return None + def runtime_metrics(self): + return {"restore": {"attempts": 0}} + + monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestLogger", DummyLogger) + monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestRunner", DummyRunner) + monkeypatch.setattr("llm_quest_benchmark.core.runner.ThreadPoolExecutor", DummyExecutor) + + agent = SimpleNamespace(agent_id="llm_test") + cfg = SimpleNamespace(agent_id="llm_test", benchmark_id="bench_timeout_1") + + outcome = run_quest_with_timeout(QUEST, agent, timeout=1, agent_config=cfg) + + assert outcome == QuestOutcome.TIMEOUT + assert recorded["stop_reason"] == "timeout" + assert recorded["outcome"] == QuestOutcome.TIMEOUT.name + assert recorded["terminal_snapshot"].location_id == "loc7" + + +def test_run_quest_with_timeout_forwards_options_to_runner(monkeypatch): + captured = {} + + class DummyLogger: + def __init__(self, debug=False, agent=None): # noqa: ARG002 + self.current_run_id = 1 + self.logger = SimpleNamespace( + warning=lambda *a, **k: None, error=lambda *a, **k: None, info=lambda *a, **k: None + ) + class DummyExecutorForResult: def __init__(self, max_workers): # noqa: ARG002 pass @@ -270,12 +415,56 @@ def run(self, quest): # noqa: ARG002 return _Runner() monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestLogger", DummyLogger) - monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestEnvironment", lambda *a, **k: object()) monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestRunner", fake_quest_runner) monkeypatch.setattr("llm_quest_benchmark.core.runner.ThreadPoolExecutor", DummyExecutorForResult) agent = SimpleNamespace(agent_id="llm_test") - outcome = run_quest_with_timeout("quests/mock.qm", agent, timeout=1, max_steps=42) + outcome = run_quest_with_timeout( + QUEST, agent, timeout=1, max_steps=42, progress_manifest="configs/progress/Boat.yaml" + ) assert outcome == QuestOutcome.SUCCESS assert captured["max_steps"] == 42 + assert captured["progress_manifest"] == "configs/progress/Boat.yaml" + + +def test_run_uses_agent_config_identity_for_the_run(monkeypatch): + """Identity comes from the treatment config, not from a post-export patch.""" + env = _FakeEnv(terminate_at=1) + agent = _FakeAgent() + config = _FakeConfig(agent_id="gpt-5-mini_t0.4_reasoning_recent_deadbeef", benchmark_id="bench_1") + monkeypatch.setattr("llm_quest_benchmark.core.runner.QuestEnvironment", lambda *a, **k: env) + logger = _DummyQuestLogger() + runner = QuestRunner(agent=agent, quest_logger=logger, agent_config=config) + + runner.run(QUEST) + + assert logger.started["agent_id"] == "gpt-5-mini_t0.4_reasoning_recent_deadbeef" + assert logger.started["benchmark_id"] == "bench_1" + + +def test_progress_is_monotonic_and_terminal_success_is_full(monkeypatch, tmp_path): + manifest = tmp_path / "manifest.yaml" + manifest.write_text( + """ +quest: Fake +version: 1 +milestones: + - id: first + percent: 40 + match: + location_id: ["loc1"] +""", + encoding="utf-8", + ) + + env = _FakeEnv(terminate_at=3) + agent = _FakeAgent() + runner, logger = _runner(monkeypatch, env, agent, progress_manifest=str(manifest)) + + runner.run(QUEST) + + progress_values = [t.progress.current for t in logger.transitions] + assert progress_values == sorted(progress_values) + assert progress_values[0] == pytest.approx(40.0) + assert progress_values[-1] == pytest.approx(100.0) diff --git a/llm_quest_benchmark/tests/environments/test_outcome_detection.py b/llm_quest_benchmark/tests/environments/test_outcome_detection.py index 46fc797..5e00213 100644 --- a/llm_quest_benchmark/tests/environments/test_outcome_detection.py +++ b/llm_quest_benchmark/tests/environments/test_outcome_detection.py @@ -7,6 +7,9 @@ import logging from llm_quest_benchmark.schemas.bridge import QMBridgeState +from llm_quest_benchmark.schemas.records import QuestSnapshot + +PERFORMED_AT_MS = 1735689600000 def _make_bridge_state(game_state: str = "running", text: str = "", location_id: str = "1") -> QMBridgeState: @@ -17,6 +20,7 @@ def _make_bridge_state(game_state: str = "running", text: str = "", location_id: reward=0.0, game_ended=game_state != "running", game_state=game_state, + saving={"locationId": location_id, "state": game_state}, ) @@ -44,10 +48,20 @@ def test_running_state(self): def test_default_game_state(self): state = QMBridgeState(location_id="1", text="", choices=[], reward=0.0, game_ended=False) assert state.game_state == "running" + assert state.saving == {} + + def test_snapshot_conversion_carries_game_state_and_saving(self): + snapshot = _make_bridge_state("win", text="You won").to_snapshot() + + assert isinstance(snapshot, QuestSnapshot) + assert snapshot.game_state == "win" + assert snapshot.done is True + assert snapshot.saving == {"locationId": "1", "state": "win"} + assert snapshot.digest == snapshot.compute_digest() class TestEnvironmentOutcomeFromGameState: - """Verify the environment determines success from game_state, not text.""" + """Verify the environment carries the engine's authoritative game_state.""" def _make_env(self): from llm_quest_benchmark.environments.qm import QMPlayerEnv @@ -55,16 +69,14 @@ def _make_env(self): env = QMPlayerEnv.__new__(QMPlayerEnv) env.debug = False env.language = "eng" + env.forced_stop_reason = None env.logger = logging.getLogger("test_outcome") - env._current_state = { - "location_id": "1", - "text": "start", - "params_state": [], - "choices": [{"id": "1", "text": "Go"}], - "reward": 0.0, - "done": False, - "info": {}, - } + env._snapshot = QuestSnapshot( + location_id="1", + observation="start", + choices=[{"id": "1", "text": "Go"}], + saving={"locationId": "1"}, + ) return env def _patch_bridge_step(self, env, game_state: str, text: str = ""): @@ -73,7 +85,7 @@ def _patch_bridge_step(self, env, game_state: str, text: str = ""): class FakeBridge: state_history = [_make_bridge_state("running")] - def step(self, _action): + def step(self, _choice_index, _performed_at_ms): return _make_bridge_state(game_state, text=text) def close(self): @@ -84,43 +96,43 @@ def close(self): def test_win_is_success(self): env = self._make_env() self._patch_bridge_step(env, "win") - _, done, success, _ = env.step("1") - assert done is True - assert success is True + snapshot = env.step(1, PERFORMED_AT_MS) + assert snapshot.done is True + assert snapshot.game_state == "win" def test_fail_is_failure(self): env = self._make_env() self._patch_bridge_step(env, "fail") - _, done, success, _ = env.step("1") - assert done is True - assert success is False + snapshot = env.step(1, PERFORMED_AT_MS) + assert snapshot.done is True + assert snapshot.game_state == "fail" def test_dead_is_failure(self): env = self._make_env() self._patch_bridge_step(env, "dead") - _, done, success, _ = env.step("1") - assert done is True - assert success is False + snapshot = env.step(1, PERFORMED_AT_MS) + assert snapshot.done is True + assert snapshot.game_state == "dead" def test_win_with_misleading_failure_text(self): """gameState=win must override misleading failure text.""" env = self._make_env() self._patch_bridge_step(env, "win", text="mission failed completely, you died") - _, done, success, _ = env.step("1") - assert done is True - assert success is True + snapshot = env.step(1, PERFORMED_AT_MS) + assert snapshot.done is True + assert snapshot.game_state == "win" def test_fail_with_misleading_success_text(self): """gameState=fail must override misleading success text like 'congratulations'.""" env = self._make_env() self._patch_bridge_step(env, "fail", text="congratulations on your 10000 credits reward") - _, done, success, _ = env.step("1") - assert done is True - assert success is False + snapshot = env.step(1, PERFORMED_AT_MS) + assert snapshot.done is True + assert snapshot.game_state == "fail" def test_running_is_not_done(self): env = self._make_env() self._patch_bridge_step(env, "running") - _, done, success, _ = env.step("1") - assert done is False - assert success is False + snapshot = env.step(1, PERFORMED_AT_MS) + assert snapshot.done is False + assert snapshot.game_state == "running" diff --git a/llm_quest_benchmark/tests/environments/test_qm.py b/llm_quest_benchmark/tests/environments/test_qm.py index 7415260..c6d28ef 100644 --- a/llm_quest_benchmark/tests/environments/test_qm.py +++ b/llm_quest_benchmark/tests/environments/test_qm.py @@ -1,51 +1,86 @@ """Tests for QM environment""" import logging +import time import pytest from llm_quest_benchmark.constants import DEFAULT_QUEST from llm_quest_benchmark.environments.qm import QMPlayerEnv +from llm_quest_benchmark.schemas.records import QuestSnapshot + + +def _now_ms() -> int: + return int(time.time() * 1000) def test_qm_env_lifecycle(): - """Test QM environment lifecycle - initialization, reset, step, close""" + """QM environment lifecycle - initialization, reset, step, close""" env = QMPlayerEnv(str(DEFAULT_QUEST)) try: - # Test initialization assert env.quest_file == str(DEFAULT_QUEST) - assert env._current_state == {} - - # Test reset - observation = env.reset() - assert isinstance(observation, str) - assert len(observation) > 0 - state = env.get_state() - assert "choices" in state - assert len(state["choices"]) > 0 - - # Test step - observation, done, success, info = env.step("1") - assert isinstance(observation, str) - assert isinstance(done, bool) - assert isinstance(success, bool) - assert isinstance(info, dict) + assert env.snapshot is None + + snapshot = env.reset() + assert isinstance(snapshot, QuestSnapshot) + assert snapshot.observation + assert snapshot.choices + assert snapshot.saving, "reset must carry the full engine saving" + assert snapshot.digest == snapshot.compute_digest() + + after = env.step(1, _now_ms()) + assert isinstance(after, QuestSnapshot) + assert after.saving + assert after.digest != snapshot.digest + finally: + env.close() + + +def test_qm_env_restore_reproduces_the_recorded_digest(): + """Exact engine restore must reproduce the recorded snapshot digest.""" + env = QMPlayerEnv(str(DEFAULT_QUEST)) + try: + start = env.reset() + env.step(1, _now_ms()) + restored = env.restore(start) + + assert restored.digest == start.digest + assert restored.location_id == start.location_id + assert env.snapshot.digest == start.digest + finally: + env.close() + + +def test_qm_env_restore_rejects_snapshot_without_saving(): + env = QMPlayerEnv(str(DEFAULT_QUEST)) + try: + env.reset() + with pytest.raises(ValueError, match="full engine saving"): + env.restore(QuestSnapshot(location_id="1", observation="x", saving=None)) finally: env.close() def test_qm_env_error_handling(): - """Test QM environment error handling""" - # Test invalid quest file + """QM environment error handling""" with pytest.raises(RuntimeError): QMPlayerEnv("nonexistent.qm") - # Test step without reset env = QMPlayerEnv(str(DEFAULT_QUEST)) try: with pytest.raises(RuntimeError): - env.step("1") + env.step(1, _now_ms()) + finally: + env.close() + + +def test_qm_env_rejects_invalid_choice(): + env = QMPlayerEnv(str(DEFAULT_QUEST)) + try: + env.reset() + with pytest.raises(RuntimeError): + env.step(999, _now_ms()) finally: env.close() @@ -59,41 +94,30 @@ class _FakeBridge: def __init__(self, states): self.state_history = states - def step(self, _action): + def step(self, _choice_index, _performed_at_ms): raise AssertionError("step() should not be called when loop detection triggers") def test_infinite_loop_detection_sets_terminal_failure_state(): - """Loop guard must produce a terminal env state, not a dangling non-final snapshot.""" + """Loop guard must produce a terminal snapshot, not a dangling non-final one.""" env = QMPlayerEnv.__new__(QMPlayerEnv) env.debug = False env.language = "rus" + env.forced_stop_reason = None env.logger = logging.getLogger("test_qm_loop_guard") env.bridge = _FakeBridge([_FakeBridgeState("Наступил новый день") for _ in range(31)]) - env._current_state = { - "location_id": "1", - "text": "Наступил новый день", - "params_state": ["День: 10"], - "choices": [{"id": "1", "text": "Ждать"}], - "reward": 0.0, - "done": False, - "info": {}, - } - - observation, done, success, info = env.step("1") - - assert done is True - assert success is False - assert info["forced_completion"] is True - assert env.state["done"] is True - assert env.state["choices"] == [] - assert "Forced stop" in observation - - # Test invalid choice - env = QMPlayerEnv(str(DEFAULT_QUEST)) - try: - env.reset() - with pytest.raises(RuntimeError): - env.step("999") # Invalid choice number - finally: - env.close() + env._snapshot = QuestSnapshot( + location_id="1", + observation="Наступил новый день", + choices=[{"id": "1", "text": "Ждать"}], + params_state=["День: 10"], + saving={"locationId": 1}, + ) + + snapshot = env.step(1, _now_ms()) + + assert snapshot.done is True + assert snapshot.game_state == "fail" + assert snapshot.choices == [] + assert "Forced stop" in snapshot.observation + assert env.forced_stop_reason == "infinite_loop_detected" diff --git a/llm_quest_benchmark/tests/executors/cli/test_commands.py b/llm_quest_benchmark/tests/executors/cli/test_commands.py index 1bd972e..8205fc2 100644 --- a/llm_quest_benchmark/tests/executors/cli/test_commands.py +++ b/llm_quest_benchmark/tests/executors/cli/test_commands.py @@ -1,5 +1,8 @@ """Tests for CLI commands""" +import json +import sqlite3 +from pathlib import Path from unittest.mock import Mock from typer.testing import CliRunner @@ -7,10 +10,64 @@ from llm_quest_benchmark.constants import DEFAULT_QUEST from llm_quest_benchmark.executors.cli import commands from llm_quest_benchmark.executors.cli.commands import app +from llm_quest_benchmark.harnesses.specs import build_treatment +from llm_quest_benchmark.schemas.records import ( + ProgressState, + QuestAction, + QuestSnapshot, + QuestTransition, + RunRecord, +) +from llm_quest_benchmark.schemas.response import LLMResponse runner = CliRunner() +def _record(outcome: str = "FAILURE", resumable: bool = False) -> RunRecord: + before = QuestSnapshot( + location_id="1", + observation="State one", + choices=[{"id": "11", "text": "Go left"}, {"id": "12", "text": "Go right"}], + saving={"locationId": 1} if resumable else None, + ) + after = QuestSnapshot( + location_id="2", + observation="State two", + choices=[], + done=True, + game_state="fail", + saving={"locationId": 2} if resumable else None, + ) + return RunRecord( + run_id=1, + quest_file="quests/Boat.qm", + quest_name="TestQuest", + quest_checksum="sha256:test", + quest_language="rus", + engine_revision="git:test", + agent_id="llm_test", + treatment=build_treatment("reasoning_recent", "gpt-5-mini", 0.4, "system_role.jinja").to_dict(), + outcome=outcome, + progress=ProgressState(current=40.0), + terminal_snapshot=after, + transitions=[ + QuestTransition( + index=1, + before=before, + action=QuestAction.choose(2, "12", 1735689600000), + after=after, + response=LLMResponse(action=2, analysis="Need progress", reasoning="Right seems safer"), + progress=ProgressState(current=40.0), + ) + ], + ) + + +def _write_record(path: Path, record: RunRecord) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(record.to_dict(), ensure_ascii=False), encoding="utf-8") + + def test_version(): """Test version command""" result = runner.invoke(app, ["--version"]) @@ -29,17 +86,43 @@ def test_run_quest(): def test_run_quest_invalid_args(): """Test run command with invalid arguments""" - # Test invalid model result = runner.invoke(app, ["run", "--quest", str(DEFAULT_QUEST), "--model", "invalid-model"]) assert result.exit_code == 2 - # Test missing quest file result = runner.invoke( app, ["run", "--quest", "nonexistent.qm", "--model", "random_choice", "--harness", "random_choice"] ) assert result.exit_code == 2 +def test_run_rejects_restore_limit_for_other_harnesses(): + result = runner.invoke( + app, + [ + "run", + "--quest", + str(DEFAULT_QUEST), + "--model", + "gpt-5-mini", + "--harness", + "reasoning_recent", + "--restore-limit", + "2", + ], + ) + assert result.exit_code == 2 + + +def test_run_rejects_resuming_a_non_resumable_record(tmp_path): + path = tmp_path / "run_summary.json" + _write_record(path, _record(outcome="FAILURE")) + + result = runner.invoke(app, ["run", "--resume-from", str(path)]) + + assert result.exit_code == 1 + assert "not resumable" in result.output + + def test_analyze_invalid_input(): """Test analyze command with invalid input""" result = runner.invoke(app, ["analyze"]) @@ -55,54 +138,35 @@ def test_benchmark_missing_config(): def test_analyze_run_with_run_summary_path(tmp_path): - """Test analyze-run against explicit run_summary path.""" + """analyze-run against an explicit schema-v2 run_summary path.""" summary_path = tmp_path / "run_summary.json" - summary_path.write_text( - """ -{ - "quest_name": "TestQuest", - "agent_id": "llm_test", - "outcome": "FAILURE", - "steps": [ - { - "step": 1, - "observation": "State one", - "choices": {"1": "Go left", "2": "Go right"}, - "llm_decision": { - "analysis": "Need progress", - "reasoning": "Right seems safer", - "is_default": false, - "choice": {"2": "Go right"} - } - } - ] -} -""".strip(), - encoding="utf-8", - ) + _write_record(summary_path, _record()) result = runner.invoke(app, ["analyze-run", "--run-summary", str(summary_path)]) assert result.exit_code == 0 assert "Decision Steps: 1" in result.stdout assert "selected [2:Go right]" in result.stdout + assert "Treatment: t2_" in result.stdout + assert "Progress: 40.0%" in result.stdout + + +def test_analyze_run_rejects_legacy_records(tmp_path): + """A pre-v2 run summary must point at the migration command, not be parsed.""" + summary_path = tmp_path / "run_summary.json" + summary_path.write_text(json.dumps({"run_id": 1, "steps": []}), encoding="utf-8") + + result = runner.invoke(app, ["analyze-run", "--run-summary", str(summary_path)]) + + assert result.exit_code == 2 + assert "scripts/migrate_records.py" in result.output def test_analyze_run_autolocates_latest_run(monkeypatch, tmp_path): - """Test analyze-run latest-run discovery with --agent and --quest.""" + """analyze-run latest-run discovery with --agent and --quest.""" monkeypatch.chdir(tmp_path) - run_dir = tmp_path / "results" / "llm_test" / "QuestA" / "run_42" - run_dir.mkdir(parents=True) - summary_path = run_dir / "run_summary.json" - summary_path.write_text( - """ -{ - "quest_name": "QuestA", - "agent_id": "llm_test", - "outcome": "SUCCESS", - "steps": [] -} -""".strip(), - encoding="utf-8", + _write_record( + tmp_path / "results" / "llm_test" / "QuestA" / "run_42" / "run_summary.json", + _record(outcome="SUCCESS"), ) result = runner.invoke(app, ["analyze-run", "--agent", "llm_test", "--quest", "QuestA"]) @@ -110,6 +174,20 @@ def test_analyze_run_autolocates_latest_run(monkeypatch, tmp_path): assert "Outcome: SUCCESS" in result.stdout +def test_cleanup_counts_transitions(tmp_path): + from llm_quest_benchmark.core.logging import ensure_v2_schema + + db_path = tmp_path / "metrics.db" + conn = sqlite3.connect(db_path) + ensure_v2_schema(conn) + conn.close() + + result = runner.invoke(app, ["cleanup", "--db-path", str(db_path), "--all", "--no-backup"]) + + assert result.exit_code == 0 + assert "transitions" in result.stdout + + def test_download_quests_command_prints_summary(monkeypatch): """Test quest downloader wrapper prints collection counts.""" fake_run = Mock() diff --git a/llm_quest_benchmark/tests/harnesses/test_adaptive.py b/llm_quest_benchmark/tests/harnesses/test_adaptive.py new file mode 100644 index 0000000..493ca75 --- /dev/null +++ b/llm_quest_benchmark/tests/harnesses/test_adaptive.py @@ -0,0 +1,115 @@ +"""Tests for the experimental adaptive-reasoning harness.""" + +from unittest.mock import Mock + +from llm_quest_benchmark.harnesses.adaptive import MODE_CONCISE, MODE_DEEP, AdaptiveReasoningHarness +from llm_quest_benchmark.players.base import DecisionContext +from llm_quest_benchmark.schemas.records import ProgressState + +CHOICES = [{"id": "11", "text": "Search the room"}, {"id": "12", "text": "Leave"}] + + +def _context(stalled: int = 0, current: float = 20.0) -> DecisionContext: + return DecisionContext(progress=ProgressState(current=current, stalled_transitions=stalled)) + + +def _mock_llm(*responses): + llm = Mock() + llm.get_completion.side_effect = list(responses) or ['{"analysis":"a","reasoning":"r","result":1}'] + llm.get_last_usage.return_value = { + "prompt_tokens": 10, + "completion_tokens": 5, + "total_tokens": 15, + "estimated_cost_usd": 0.0, + } + return llm + + +def _decide(harness, observation, context): + harness.llm = _mock_llm('{"analysis":"a","reasoning":"r","result":1}') + action = harness.get_quest_action(observation, CHOICES, context) + prompt = harness.llm.get_completion.call_args_list[0].args[0] + return action, prompt + + +def test_routine_states_use_concise_mode(): + harness = AdaptiveReasoningHarness(model_name="gpt-5-mini") + + action, prompt = _decide(harness, "A quiet corridor.", _context()) + + assert action.is_choose + assert harness.reasoning_mode == MODE_CONCISE + assert "Concise mode." in prompt + assert "Deep planning mode" not in prompt + + +def test_progress_stall_switches_the_next_decision_to_deep_mode(): + harness = AdaptiveReasoningHarness(model_name="gpt-5-mini", adaptive_stall_steps=3) + + _, concise_prompt = _decide(harness, "Scene one.", _context(stalled=2)) + assert harness.reasoning_mode == MODE_CONCISE + assert "Concise mode." in concise_prompt + + _, deep_prompt = _decide(harness, "Scene two.", _context(stalled=3)) + assert harness.reasoning_mode == MODE_DEEP + assert "Deep planning mode" in deep_prompt + assert "progress_stalled_3" in deep_prompt + + +def test_repeated_state_switches_to_deep_mode(): + harness = AdaptiveReasoningHarness(model_name="gpt-5-mini") + + _decide(harness, "The same locked door.", _context()) + assert harness.reasoning_mode == MODE_CONCISE + + _, prompt = _decide(harness, "The same locked door.", _context()) + + assert harness.reasoning_mode == MODE_DEEP + assert "repeated_state" in prompt + + +def test_recovery_resets_the_trigger_without_losing_history(): + harness = AdaptiveReasoningHarness(model_name="gpt-5-mini", adaptive_stall_steps=2) + + _decide(harness, "Scene one.", _context(stalled=0)) + _decide(harness, "Scene two.", _context(stalled=2)) + assert harness.reasoning_mode == MODE_DEEP + history_before = len(harness._decision_history) + + # A new milestone resets the stall counter and the state is not a repeat. + _, prompt = _decide(harness, "A brand new hall.", _context(stalled=0)) + + assert harness.reasoning_mode == MODE_CONCISE + assert "Concise mode." in prompt + assert len(harness._decision_history) == history_before + 1 + assert len(harness.history) == 3 + + +def test_adaptive_harness_defaults_to_a_bounded_stall_trigger(): + assert AdaptiveReasoningHarness(model_name="gpt-5-mini").adaptive_stall_steps == 3 + assert AdaptiveReasoningHarness(model_name="gpt-5-mini", adaptive_stall_steps=7).adaptive_stall_steps == 7 + + +def test_reset_returns_the_harness_to_concise_mode(): + harness = AdaptiveReasoningHarness(model_name="gpt-5-mini") + _decide(harness, "Scene.", _context(stalled=9)) + assert harness.reasoning_mode == MODE_DEEP + + harness.reset() + + assert harness.reasoning_mode == MODE_CONCISE + assert harness._context.progress.stalled_transitions == 0 + + +def test_adaptive_harness_recovers_from_provider_errors(): + harness = AdaptiveReasoningHarness(model_name="gpt-5-mini") + llm = Mock() + llm.get_completion.side_effect = RuntimeError("provider unavailable") + llm.get_last_usage.return_value = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0} + harness.llm = llm + + action = harness.get_quest_action("Broken.", CHOICES, _context()) + + assert action.is_choose + assert action.choice_index == 1 + assert harness.get_last_response().is_default is True diff --git a/llm_quest_benchmark/tests/harnesses/test_backtracking.py b/llm_quest_benchmark/tests/harnesses/test_backtracking.py new file mode 100644 index 0000000..590d497 --- /dev/null +++ b/llm_quest_benchmark/tests/harnesses/test_backtracking.py @@ -0,0 +1,176 @@ +"""Tests for the experimental backtracking harness.""" + +from unittest.mock import Mock + +from llm_quest_benchmark.harnesses.backtracking import BacktrackingHarness, parse_restore_request +from llm_quest_benchmark.harnesses.factory import create_harness +from llm_quest_benchmark.harnesses.specs import HARNESS_SPECS, RESTORE_HARNESSES +from llm_quest_benchmark.players.base import DecisionContext +from llm_quest_benchmark.schemas.records import QuestSnapshot + +CHOICES = [{"id": "11", "text": "Press on"}, {"id": "12", "text": "Turn back"}] + + +def _snapshot(location_id: str) -> QuestSnapshot: + return QuestSnapshot( + location_id=location_id, + observation=f"You are at {location_id}", + choices=CHOICES, + params_state=["HP: 10"], + saving={"locationId": location_id}, + ) + + +def _context(checkpoints=3, restore_allowed=True, remaining=2) -> DecisionContext: + return DecisionContext( + step=checkpoints, + checkpoints=[_snapshot(f"loc{i}") for i in range(checkpoints)], + restore_allowed=restore_allowed, + restores_remaining=remaining, + ) + + +def _mock_llm(*responses): + llm = Mock() + llm.get_completion.side_effect = list(responses) + llm.get_last_usage.return_value = { + "prompt_tokens": 10, + "completion_tokens": 5, + "total_tokens": 15, + "estimated_cost_usd": 0.0, + } + return llm + + +def test_backtracking_is_the_only_restore_capable_harness(): + assert {"backtracking"} == RESTORE_HARNESSES + assert BacktrackingHarness.supports_restore is True + + for name, spec in HARNESS_SPECS.items(): + if name == "backtracking": + continue + harness = create_harness(name, model="gpt-5-mini" if spec.requires_model else name) + assert harness.supports_restore is False, f"{name} must not be able to restore" + + +def test_other_harnesses_cannot_emit_restore_actions(): + harness = create_harness("reasoning_recent", model="gpt-5-mini") + harness.llm = _mock_llm('{"action":"restore","checkpoint":1,"result":2}') + + action = harness.get_quest_action("state", CHOICES, _context()) + + assert action.is_choose + assert action.checkpoint_index is None + + +def test_parse_restore_request_accepts_explicit_restore(): + assert parse_restore_request('{"action":"restore","checkpoint":2,"result":1}', 3) == 2 + assert parse_restore_request('{"restore":1}', 3) == 1 + + +def test_parse_restore_request_rejects_out_of_range_and_non_restore(): + assert parse_restore_request('{"action":"restore","checkpoint":9}', 3) is None + assert parse_restore_request('{"action":"restore","checkpoint":0}', 3) is None + assert parse_restore_request('{"action":"choose","result":1}', 3) is None + assert parse_restore_request("not json at all", 3) is None + # No restorable checkpoint means no restore, whatever the model asked for. + assert parse_restore_request('{"action":"restore","checkpoint":1}', 0) is None + + +def test_backtracking_harness_returns_a_restore_action(): + harness = BacktrackingHarness(model_name="gpt-5-mini") + harness.llm = _mock_llm('{"action":"restore","analysis":"dead end","checkpoint":1,"result":2}') + + action = harness.get_quest_action("You are stuck.", CHOICES, _context()) + + assert action.is_restore + assert action.checkpoint_index == 1 + # The response still carries a usable fallback action if the runner rejects it. + assert harness.get_last_response().action == 2 + assert harness.get_last_response().parse_mode == "restore" + + +def test_backtracking_harness_returns_a_choose_action_by_default(): + harness = BacktrackingHarness(model_name="gpt-5-mini") + harness.llm = _mock_llm('{"action":"choose","analysis":"keep going","reasoning":"r","result":1}') + + action = harness.get_quest_action("You are fine.", CHOICES, _context()) + + assert action.is_choose + assert action.choice_index == 1 + + +def test_backtracking_harness_ignores_restore_when_it_is_not_allowed(): + harness = BacktrackingHarness(model_name="gpt-5-mini") + harness.llm = _mock_llm('{"action":"restore","checkpoint":1,"result":2}') + + action = harness.get_quest_action("Budget spent.", CHOICES, _context(restore_allowed=False, remaining=0)) + + assert action.is_choose + assert action.choice_index == 2 + + +def test_backtracking_prompt_lists_restorable_checkpoints_and_budget(): + harness = BacktrackingHarness(model_name="gpt-5-mini") + harness.llm = _mock_llm('{"action":"choose","result":1}') + + harness.get_quest_action("Current scene.", CHOICES, _context(checkpoints=3, remaining=2)) + prompt = harness.llm.get_completion.call_args_list[0].args[0] + + assert "[1] location loc0" in prompt + assert "[2] location loc1" in prompt + # The current state is the last checkpoint and is never offered for restore. + assert "[3] location loc2" not in prompt + assert "Restores remaining: 2" in prompt + + +def test_backtracking_prompt_omits_restore_when_unavailable(): + harness = BacktrackingHarness(model_name="gpt-5-mini") + harness.llm = _mock_llm('{"action":"choose","result":1}') + + harness.get_quest_action("Current scene.", CHOICES, _context(restore_allowed=False)) + prompt = harness.llm.get_completion.call_args_list[0].args[0] + + assert "No checkpoint restore is available" in prompt + assert "must choose an action" in prompt + + +def test_backtracking_harness_defaults_to_a_bounded_restore_limit(): + assert BacktrackingHarness(model_name="gpt-5-mini").restore_limit == 3 + assert BacktrackingHarness(model_name="gpt-5-mini", restore_limit=1).restore_limit == 1 + + +def test_backtracking_harness_skips_single_choice_states_without_restoring(): + harness = BacktrackingHarness(model_name="gpt-5-mini", skip_single=True) + harness.llm = _mock_llm() + + action = harness.get_quest_action("Only one door.", [{"id": "1", "text": "Enter"}], _context()) + + assert action.is_choose + assert action.choice_index == 1 + assert harness.llm.get_completion.call_count == 0 + + +def test_backtracking_harness_recovers_from_provider_errors(): + harness = BacktrackingHarness(model_name="gpt-5-mini") + llm = Mock() + llm.get_completion.side_effect = RuntimeError("provider unavailable") + llm.get_last_usage.return_value = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0} + harness.llm = llm + + action = harness.get_quest_action("Broken.", CHOICES, _context()) + + assert action.is_choose + assert action.choice_index == 1 + assert harness.get_last_response().is_default is True + + +def test_backtracking_reset_clears_pending_restore_state(): + harness = BacktrackingHarness(model_name="gpt-5-mini") + harness.llm = _mock_llm('{"action":"restore","checkpoint":1,"result":1}') + harness.get_quest_action("Stuck.", CHOICES, _context()) + + harness.reset() + + assert harness._pending_restore is None + assert harness._context.checkpoints == [] diff --git a/llm_quest_benchmark/tests/harnesses/test_factory.py b/llm_quest_benchmark/tests/harnesses/test_factory.py index 6d2b1cf..0606972 100644 --- a/llm_quest_benchmark/tests/harnesses/test_factory.py +++ b/llm_quest_benchmark/tests/harnesses/test_factory.py @@ -1,8 +1,9 @@ import pytest -from llm_quest_benchmark.harnesses.factory import HARNESS_REGISTRY, create_harness +from llm_quest_benchmark.harnesses.factory import HARNESS_CLASSES, create_harness from llm_quest_benchmark.harnesses.memo import MemoCompactHarness from llm_quest_benchmark.harnesses.minimal import MinimalHarness +from llm_quest_benchmark.harnesses.specs import HARNESS_SPECS from llm_quest_benchmark.players.human import HumanPlayer from llm_quest_benchmark.players.random import RandomPlayer from llm_quest_benchmark.schemas.config import BenchmarkConfig, HarnessConfig @@ -15,12 +16,18 @@ def test_create_minimal_harness(): def test_all_harness_names_instantiate(): - for harness_name, harness_cls in HARNESS_REGISTRY.items(): + for harness_name, harness_cls in HARNESS_CLASSES.items(): harness = create_harness(harness_name, model="gpt-5-mini") assert isinstance(harness, harness_cls) +def test_every_model_driven_spec_has_an_implementation(): + model_driven = {name for name, spec in HARNESS_SPECS.items() if spec.requires_model} + + assert model_driven == set(HARNESS_CLASSES) + + def test_create_human_harness(): harness = create_harness("human") @@ -70,6 +77,20 @@ def test_human_model_requires_human_harness(): create_harness("minimal", model="human") +def test_backtracking_harness_receives_restore_limit(): + harness = create_harness("backtracking", model="gpt-5-mini", restore_limit=2) + + assert harness.restore_limit == 2 + assert harness.supports_restore is True + + +def test_adaptive_harness_receives_stall_steps(): + harness = create_harness("adaptive_reasoning", model="gpt-5-mini", adaptive_stall_steps=5) + + assert harness.adaptive_stall_steps == 5 + assert harness.supports_restore is False + + def test_harness_config_stable_harness_id(): config = HarnessConfig(harness="memo_compact", model="gpt-5-mini") @@ -124,23 +145,47 @@ def test_harness_config_rejects_human_model_with_llm_harness(): HarnessConfig(harness="minimal", model="human") -def test_harness_config_allows_retired_exp4_aliases(): +def test_harness_config_allows_retired_exp4_harnesses(): for harness_name in ("compaction_no_memo", "memo_cot", "memo_extended", "memo_structured"): config = HarnessConfig(harness=harness_name, model="gpt-5-mini") assert config.harness == harness_name -def test_harness_config_rejects_old_template_key(): - with pytest.raises(ValueError, match="Use harness: key instead of template:"): +def test_harness_config_rejects_removed_template_key(): + with pytest.raises(TypeError, match="template"): HarnessConfig(model="gpt-5-mini", template="reasoning.jinja") -def test_harness_config_rejects_old_memory_mode_key(): - with pytest.raises(ValueError, match="Use harness: key instead of memory_mode:"): +def test_harness_config_rejects_removed_memory_mode_key(): + with pytest.raises(TypeError, match="memory_mode"): HarnessConfig(model="gpt-5-mini", harness="memo_compact", memory_mode="compaction") +def test_restore_limit_is_rejected_for_non_backtracking_harnesses(): + with pytest.raises(ValueError, match="restore_limit is only valid for harness: backtracking"): + HarnessConfig(harness="reasoning_recent", model="gpt-5-mini", restore_limit=3) + + +def test_adaptive_stall_steps_is_rejected_for_other_harnesses(): + with pytest.raises(ValueError, match="adaptive_stall_steps is only valid for harness: adaptive_reasoning"): + HarnessConfig(harness="memo_compact", model="gpt-5-mini", adaptive_stall_steps=3) + + +def test_backtracking_accepts_restore_limit(): + config = HarnessConfig(harness="backtracking", model="gpt-5-mini", restore_limit=3) + + assert config.restore_limit == 3 + assert config.treatment().knobs["restore_limit"] == 3 + + +def test_adaptive_reasoning_accepts_stall_steps(): + config = HarnessConfig(harness="adaptive_reasoning", model="gpt-5-mini", adaptive_stall_steps=4) + + assert config.adaptive_stall_steps == 4 + assert config.treatment().knobs["adaptive_stall_steps"] == 4 + + def test_benchmark_config_from_yaml_parses_harness(tmp_path): quest_path = tmp_path / "quest.qm" quest_path.write_text("", encoding="utf-8") @@ -179,7 +224,7 @@ def test_benchmark_config_from_yaml_rejects_template(tmp_path): encoding="utf-8", ) - with pytest.raises(ValueError, match="Use harness: key instead of template:"): + with pytest.raises(TypeError, match="template"): BenchmarkConfig.from_yaml(str(config_path)) @@ -199,12 +244,12 @@ def test_benchmark_config_from_yaml_rejects_memory_mode(tmp_path): encoding="utf-8", ) - with pytest.raises(ValueError, match="Use harness: key instead of memory_mode:"): + with pytest.raises(TypeError, match="memory_mode"): BenchmarkConfig.from_yaml(str(config_path)) def test_benchmark_config_max_steps_defaults_to_none(tmp_path): - """Backward compatibility: configs without max_steps stay unbounded.""" + """Configs without max_steps stay unbounded.""" quest_path = tmp_path / "quest.qm" quest_path.write_text("", encoding="utf-8") config_path = tmp_path / "benchmark.yaml" @@ -222,11 +267,25 @@ def test_benchmark_config_max_steps_defaults_to_none(tmp_path): config = BenchmarkConfig.from_yaml(str(config_path)) assert config.max_steps is None + assert config.progress_manifest is None -def test_benchmark_config_from_yaml_parses_max_steps(tmp_path): +def test_benchmark_config_from_yaml_parses_max_steps_and_progress_manifest(tmp_path): quest_path = tmp_path / "quest.qm" quest_path.write_text("", encoding="utf-8") + manifest_path = tmp_path / "progress.yaml" + manifest_path.write_text( + """ +quest: Quest +version: 1 +milestones: + - id: start + percent: 10 + match: + location_id: ["1"] +""", + encoding="utf-8", + ) config_path = tmp_path / "benchmark.yaml" config_path.write_text( f""" @@ -236,6 +295,7 @@ def test_benchmark_config_from_yaml_parses_max_steps(tmp_path): - model: gpt-5-mini harness: memo_compact max_steps: 40 +progress_manifest: {manifest_path} """, encoding="utf-8", ) @@ -243,6 +303,17 @@ def test_benchmark_config_from_yaml_parses_max_steps(tmp_path): config = BenchmarkConfig.from_yaml(str(config_path)) assert config.max_steps == 40 + assert config.progress_manifest == str(manifest_path) + + +def test_benchmark_config_rejects_invalid_progress_manifest(tmp_path): + quest_path = tmp_path / "quest.qm" + quest_path.write_text("", encoding="utf-8") + manifest = tmp_path / "bad.yaml" + manifest.write_text("quest: Boat\nversion: 1\nmilestones: []\n", encoding="utf-8") + + with pytest.raises(ValueError, match="non-empty 'milestones' list"): + BenchmarkConfig(quests=[str(quest_path)], agents=[], progress_manifest=str(manifest)) def test_benchmark_config_rejects_non_positive_max_steps(tmp_path): diff --git a/llm_quest_benchmark/tests/harnesses/test_harnesses.py b/llm_quest_benchmark/tests/harnesses/test_harnesses.py index f3b5bab..cc361e3 100644 --- a/llm_quest_benchmark/tests/harnesses/test_harnesses.py +++ b/llm_quest_benchmark/tests/harnesses/test_harnesses.py @@ -2,7 +2,9 @@ from unittest.mock import Mock -from llm_quest_benchmark.harnesses.factory import HARNESS_REGISTRY, create_harness +from llm_quest_benchmark.harnesses.adaptive import AdaptiveReasoningHarness +from llm_quest_benchmark.harnesses.backtracking import BacktrackingHarness +from llm_quest_benchmark.harnesses.factory import HARNESS_CLASSES, create_harness from llm_quest_benchmark.harnesses.memo import ( CompactionNoMemoHarness, HintedCompactHarness, @@ -16,9 +18,9 @@ from llm_quest_benchmark.harnesses.planner import PlannerHarness from llm_quest_benchmark.harnesses.reasoning import ReasoningFullTranscriptHarness, ReasoningRecentHarness from llm_quest_benchmark.harnesses.tool_harness import ProgrammaticMemoryHarness, ToolCompactHarness, ToolHintedHarness -from llm_quest_benchmark.schemas.state import AgentState +from llm_quest_benchmark.schemas.records import QuestAction, QuestSnapshot, QuestTransition -HARNESS_SPECS = { +HARNESS_CONFIGURATIONS = { "minimal": (MinimalHarness, "stub.jinja", DefaultMemory), "reasoning_recent": (ReasoningRecentHarness, "reasoning.jinja", DefaultMemory), "reasoning_full": (ReasoningFullTranscriptHarness, "reasoning.jinja", FullTranscriptMemory), @@ -32,11 +34,13 @@ "memo_cot": (MemoCotHarness, "memo_cot.jinja", CompactionMemory), "memo_extended": (MemoExtendedHarness, "memo_extended.jinja", CompactionMemory), "memo_structured": (MemoStructuredHarness, "memo_structured.jinja", CompactionMemory), + "backtracking": (BacktrackingHarness, "backtracking.jinja", CompactionMemory), + "adaptive_reasoning": (AdaptiveReasoningHarness, "adaptive_reasoning.jinja", DefaultMemory), } def assert_harness_configuration(harness_name: str) -> None: - expected_class, expected_template, expected_memory_class = HARNESS_SPECS[harness_name] + expected_class, expected_template, expected_memory_class = HARNESS_CONFIGURATIONS[harness_name] harness = create_harness(harness_name, model="gpt-5-mini") @@ -90,11 +94,11 @@ def test_exp4_retired_harness_configuration(): def test_all_registry_harnesses_have_configuration_specs(): - assert set(HARNESS_REGISTRY) == set(HARNESS_SPECS) + assert set(HARNESS_CLASSES) == set(HARNESS_CONFIGURATIONS) def test_all_registry_harnesses_instantiate_with_expected_names(): - for harness_name in HARNESS_REGISTRY: + for harness_name in HARNESS_CLASSES: harness = create_harness(harness_name, model="gpt-5-mini") assert harness.harness_name == harness_name @@ -400,18 +404,24 @@ def _record_executed_step( observation: str, choices: list[dict[str, str]], action: int, -) -> AgentState: +) -> QuestTransition: """Deliver the post-env-step lifecycle event a runner would emit.""" - agent_state = AgentState( - step=len(harness._trajectory) + 1, + index = len(harness._trajectory) + 1 + before = QuestSnapshot( location_id="test", observation=observation, - choices=choices, - action=str(action), - llm_response=harness.get_last_response(), + choices=[{"id": str(i + 1), "text": c.get("text", "")} for i, c in enumerate(choices)], + saving={"locationId": index}, ) - harness.on_step(agent_state) - return agent_state + transition = QuestTransition( + index=index, + before=before, + action=QuestAction.choose(action, str(action), 1735689600000 + index), + after=QuestSnapshot(location_id="test", observation="after", saving={"locationId": index + 1}), + response=harness.get_last_response(), + ) + harness.on_transition(transition) + return transition def test_programmatic_memory_harness_can_use_history_search(): @@ -424,7 +434,7 @@ def test_programmatic_memory_harness_can_use_history_search(): first_observation = "Merchant mentions low fuel." first_choices = [{"text": "Buy fuel"}, {"text": "Keep flying"}] first_action = harness.get_action(first_observation, first_choices) - first_state = _record_executed_step(harness, first_observation, first_choices, first_action) + first_transition = _record_executed_step(harness, first_observation, first_choices, first_action) second_observation = "Your fuel gauge is blinking." second_choices = [{"text": "Refuel"}, {"text": "Attack pirates"}] @@ -435,7 +445,7 @@ def test_programmatic_memory_harness_can_use_history_search(): assert response.tool_calls[0]["tool"] == "history_search" assert response.tool_results assert "Merchant mentions low fuel" in response.tool_results[0] - assert harness._trajectory.recent(1)[0] is first_state + assert harness._trajectory.recent(1)[0] is first_transition _record_executed_step(harness, second_observation, second_choices, action) assert len(harness._trajectory) == 2 @@ -553,12 +563,12 @@ def test_programmatic_memory_normal_path_appends_exactly_one_step(): choices = [{"text": "A"}, {"text": "B"}] action = harness.get_action(observation, choices) - state = _record_executed_step(harness, observation, choices, action) + transition = _record_executed_step(harness, observation, choices, action) assert action == 2 assert len(harness._trajectory) == 1 - assert harness._trajectory.recent(1)[0] is state - assert state.action == "2" + assert harness._trajectory.recent(1)[0] is transition + assert transition.action.choice_index == 2 def test_programmatic_memory_retry_path_appends_exactly_one_step(): @@ -586,12 +596,12 @@ def test_programmatic_memory_safety_override_path_appends_exactly_one_step(): ] action = harness.get_action("Risky moment.", choices) - state = _record_executed_step(harness, "Risky moment.", choices, action) + transition = _record_executed_step(harness, "Risky moment.", choices, action) assert action == 2 # safety filter overrides the risky first choice assert len(harness._trajectory) == 1 - assert state.action == "2" - assert state.choices[1]["text"] == "Постараться пройти мимо" + assert transition.action.choice_index == 2 + assert transition.before.choices[1]["text"] == "Постараться пройти мимо" def test_programmatic_memory_error_default_path_appends_exactly_one_step(): @@ -615,7 +625,7 @@ def test_programmatic_memory_error_default_path_appends_exactly_one_step(): assert harness.get_last_response().is_default is True assert harness.get_last_response().parse_mode == "error_default" assert len(harness._trajectory) == 1 - assert harness._trajectory.recent(1)[0].action == "1" + assert harness._trajectory.recent(1)[0].action.choice_index == 1 def test_programmatic_memory_skip_single_path_appends_exactly_one_step(): @@ -624,13 +634,13 @@ def test_programmatic_memory_skip_single_path_appends_exactly_one_step(): choices = [{"text": "Open the only door"}] action = harness.get_action(observation, choices) - state = _record_executed_step(harness, observation, choices, action) + transition = _record_executed_step(harness, observation, choices, action) assert action == 1 assert harness.get_last_response().reasoning == "auto_single_choice" assert len(harness._trajectory) == 1 - assert state.action == "1" - assert state.choices[0]["text"] == "Open the only door" + assert transition.action.choice_index == 1 + assert transition.before.choices[0]["text"] == "Open the only door" def test_programmatic_memory_multi_turn_bookkeeping_stays_exactly_one_per_turn(): @@ -659,4 +669,4 @@ def test_programmatic_memory_multi_turn_bookkeeping_stays_exactly_one_per_turn() action = harness.get_action("Turn 3 single choice, LLM path since skip_single is off.", single_choice) _record_executed_step(harness, "Turn 3 single choice, LLM path since skip_single is off.", single_choice, action) - assert [state.step for state in harness._trajectory.recent(10)] == [1, 2, 3] + assert [t.index for t in harness._trajectory.recent(10)] == [1, 2, 3] diff --git a/llm_quest_benchmark/tests/harnesses/test_specs.py b/llm_quest_benchmark/tests/harnesses/test_specs.py new file mode 100644 index 0000000..b472821 --- /dev/null +++ b/llm_quest_benchmark/tests/harnesses/test_specs.py @@ -0,0 +1,131 @@ +"""Tests for canonical harness specifications and treatment signatures.""" + +import json + +import pytest + +from llm_quest_benchmark.harnesses.specs import ( + HARNESS_SPECS, + HarnessTreatment, + build_treatment, + get_spec, + valid_harness_names, + validate_exclusive_knobs, +) + + +def _treatment(**overrides) -> HarnessTreatment: + kwargs = { + "harness": "reasoning_recent", + "model": "gpt-5-mini", + "temperature": 0.4, + "system_template": "system_role.jinja", + } + kwargs.update(overrides) + return build_treatment(**kwargs) + + +def test_every_spec_declares_all_material_components(): + for name, spec in HARNESS_SPECS.items(): + assert spec.name == name + assert spec.prompt + assert spec.memory + assert spec.loop + assert spec.reasoning + assert isinstance(spec.tools, tuple) + + +def test_treatment_serializes_the_declared_components(): + treatment = _treatment(harness="tool_compact", knob_values={"compaction_interval": 25}) + payload = treatment.to_dict() + + assert payload["harness"] == "tool_compact" + assert payload["prompt"] == "tool_augmented.jinja" + assert payload["memory"] == "compaction" + assert payload["loop"] == "tool_select_act" + assert payload["tools"] == sorted(["calculator", "scratchpad", "quest_history"]) + assert payload["knobs"] == {"compaction_interval": 25} + assert payload["signature"].startswith("t2_") + + +def test_equivalent_configurations_hash_identically(): + first = _treatment() + second = _treatment() + + assert first.signature == second.signature + assert json.dumps(first.to_dict(), sort_keys=True) == json.dumps(second.to_dict(), sort_keys=True) + + +def test_material_differences_produce_different_signatures(): + base = _treatment().signature + + assert _treatment(harness="reasoning_full").signature != base + assert _treatment(model="gpt-5-nano").signature != base + assert _treatment(temperature=0.7).signature != base + assert _treatment(system_template="other.jinja").signature != base + assert ( + _treatment(harness="memo_compact", knob_values={"compaction_interval": 10}).signature + != _treatment(harness="memo_compact", knob_values={"compaction_interval": 50}).signature + ) + + +def test_irrelevant_knobs_do_not_split_otherwise_equal_treatments(): + """A knob a harness does not declare must never enter its signature.""" + with_knob = _treatment(knob_values={"compaction_interval": 10, "restore_limit": 5}) + without_knob = _treatment(knob_values={}) + + assert with_knob.signature == without_knob.signature + assert with_knob.knobs == {} + + +def test_random_choice_seed_is_a_material_knob(): + seeded = build_treatment("random_choice_7", "random_choice", 0.0, "none") + unseeded = build_treatment("random_choice", "random_choice", 0.0, "none") + + assert seeded.knobs["seed"] == 7 + assert seeded.signature != unseeded.signature + + +def test_non_model_harnesses_record_no_system_prompt(): + for harness in ("human", "random_choice"): + payload = build_treatment(harness, harness, 0.0, "system_role.jinja").to_dict() + assert payload["system_prompt"] == "none" + + +def test_unknown_treatment_is_explicit_and_hashable(): + unknown = HarnessTreatment.unknown("legacy_thing", "old-model", 0.3) + payload = unknown.to_dict() + + assert payload["prompt"] == "unknown" + assert payload["memory"] == "unknown" + assert payload["loop"] == "unknown" + assert payload["reasoning"] == "unknown" + assert payload["signature"].startswith("t2_") + + +def test_get_spec_resolves_seeded_random_choice_and_rejects_unknown_names(): + assert get_spec("random_choice_42").name == "random_choice" + assert get_spec("memo_compact").name == "memo_compact" + + with pytest.raises(ValueError, match="Unknown harness"): + get_spec("not_a_harness") + + +def test_valid_harness_names_include_the_seeded_random_form(): + names = valid_harness_names() + + assert "random_choice_" in names + assert "backtracking" in names + assert "adaptive_reasoning" in names + + +def test_exclusive_knobs_are_rejected_for_other_harnesses(): + validate_exclusive_knobs("backtracking", {"restore_limit": 2}) + validate_exclusive_knobs("adaptive_reasoning", {"adaptive_stall_steps": 2}) + validate_exclusive_knobs("memo_compact", {"restore_limit": None}) + + with pytest.raises(ValueError, match="restore_limit is only valid for harness: backtracking"): + validate_exclusive_knobs("memo_compact", {"restore_limit": 2}) + + with pytest.raises(ValueError, match="adaptive_stall_steps is only valid for harness: adaptive_reasoning"): + validate_exclusive_knobs("planner", {"adaptive_stall_steps": 2}) diff --git a/llm_quest_benchmark/tests/harnesses/test_trajectory.py b/llm_quest_benchmark/tests/harnesses/test_trajectory.py index 73674b0..a6edbf0 100644 --- a/llm_quest_benchmark/tests/harnesses/test_trajectory.py +++ b/llm_quest_benchmark/tests/harnesses/test_trajectory.py @@ -7,7 +7,7 @@ Trajectory, _coerce_positive_int, ) -from llm_quest_benchmark.schemas.state import AgentState +from llm_quest_benchmark.schemas.records import QuestAction, QuestSnapshot, QuestTransition def _append_step( @@ -16,17 +16,22 @@ def _append_step( choices: list[str], selected_action: int, selected_choice: str, -) -> AgentState: +) -> QuestTransition: expected_choice = choices[selected_action - 1] if 1 <= selected_action <= len(choices) else "" assert selected_choice == expected_choice + index = len(trajectory) + 1 + before = QuestSnapshot( + location_id="test", + observation=observation, + choices=[{"id": str(i + 1), "text": choice} for i, choice in enumerate(choices)], + saving={"locationId": index}, + ) return trajectory.append( - AgentState( - step=len(trajectory) + 1, - location_id="test", - observation=observation, - choices=[{"text": choice} for choice in choices], - action=str(selected_action), - llm_response=None, + QuestTransition( + index=index, + before=before, + action=QuestAction.choose(selected_action, str(selected_action), 1735689600000 + index), + after=QuestSnapshot(location_id="test", observation="after", saving={"locationId": index + 1}), ) ) @@ -55,11 +60,11 @@ def test_append_retains_full_observation_and_choices_without_clipping(): selected_choice=long_choice, ) - assert step.step == 1 - assert step.observation == long_observation - assert step.choices == [{"text": long_choice}, {"text": "Leave"}] - assert step.action == "1" - assert step.choices[0]["text"] == long_choice + assert step.index == 1 + assert step.before.observation == long_observation + assert [c["text"] for c in step.before.choices] == [long_choice, "Leave"] + assert step.action.choice_index == 1 + assert step.before.choices[0]["text"] == long_choice assert trajectory.recent(1)[0] is step read_output = trajectory.read(1, 1) @@ -73,8 +78,8 @@ def test_append_preserves_insertion_order_and_increments_step(): assert len(trajectory) == 3 recent = trajectory.recent(10) - assert [s.step for s in recent] == [1, 2, 3] - assert [s.observation for s in recent] == [ + assert [s.index for s in recent] == [1, 2, 3] + assert [s.before.observation for s in recent] == [ "Observation 1 with detail.", "Observation 2 with detail.", "Observation 3 with detail.", @@ -332,13 +337,13 @@ def test_search_matches_short_token_only_as_a_whole_word(): def test_single_oversized_entry_is_hard_bounded_not_returned_in_full(): """A single entry larger than MAX_OUTPUT_CHARS must still yield a result <= MAX_OUTPUT_CHARS: only the FORMATTED output is truncated (with a clear - marker), the stored canonical AgentState itself is never touched.""" + marker), the stored canonical QuestTransition itself is never touched.""" trajectory = Trajectory() huge_observation = "x" * (MAX_OUTPUT_CHARS * 2) _append_step(trajectory, huge_observation, ["A"], 1, "A") # Stored, full-fidelity entry is never truncated. - assert trajectory.recent(1)[0].observation == huge_observation + assert trajectory.recent(1)[0].before.observation == huge_observation output = trajectory.read(1, 1) assert len(output) <= MAX_OUTPUT_CHARS # hard bound, no exceptions @@ -369,7 +374,7 @@ def test_first_entry_near_full_budget_plus_second_entry_never_silently_slices(): # full (untruncated) observation text never appears verbatim. assert first_observation not in result # Stored history is untouched regardless of what the formatted output did. - assert trajectory.recent(2)[0].observation == first_observation + assert trajectory.recent(2)[0].before.observation == first_observation def test_output_omits_whole_trailing_entries_once_budget_is_exceeded(): @@ -422,7 +427,7 @@ def test_reset_restarts_step_numbering_from_one(): new_step = _append_step(trajectory, "Fresh episode start.", ["Go"], 1, "Go") - assert new_step.step == 1 + assert new_step.index == 1 def test_never_mutates_earlier_entries_on_append(): @@ -430,6 +435,6 @@ def test_never_mutates_earlier_entries_on_append(): first = _append_step(trajectory, "First.", ["A"], 1, "A") _append_step(trajectory, "Second.", ["B"], 1, "B") - assert first.step == 1 - assert first.observation == "First." - assert trajectory.recent(10)[0] == first + assert first.index == 1 + assert first.before.observation == "First." + assert trajectory.recent(10)[0] is first diff --git a/llm_quest_benchmark/tests/integration/test_benchmark.py b/llm_quest_benchmark/tests/integration/test_benchmark.py index 4eeccec..87db4a2 100644 --- a/llm_quest_benchmark/tests/integration/test_benchmark.py +++ b/llm_quest_benchmark/tests/integration/test_benchmark.py @@ -95,7 +95,9 @@ def test_benchmark_e2e(caplog, tmp_path): assert result["quest"] == str(quest_path) assert result["model"] == "random_policy" assert result["temperature"] == 0.0 - assert result["template"] == "reasoning.jinja" + assert result["harness"] == "random_choice" + assert result["treatment"]["loop"] == "random" + assert result["treatment_signature"].startswith("t2_") assert result["attempt"] == 1 assert "agent_id" in result assert "outcome" in result diff --git a/llm_quest_benchmark/tests/integration/test_mode_agents_e2e.py b/llm_quest_benchmark/tests/integration/test_mode_agents_e2e.py index 274d56e..4e680fa 100644 --- a/llm_quest_benchmark/tests/integration/test_mode_agents_e2e.py +++ b/llm_quest_benchmark/tests/integration/test_mode_agents_e2e.py @@ -133,6 +133,7 @@ def test_programmatic_memory_harness_deterministic_smoke_persists_retrieval_to_r existing LLMResponse -> QuestLogger -> run_summary.json path (contract item 5). """ monkeypatch.setattr(logging_module, "RESULTS_DIR", tmp_path) + monkeypatch.setattr(logging_module, "DEFAULT_DB_PATH", str(tmp_path / "metrics.db")) agent = create_harness("programmatic_memory", model="gpt-5-mini", skip_single=True) agent.llm = FakeLLM("programmatic_memory") @@ -146,17 +147,18 @@ def test_programmatic_memory_harness_deterministic_smoke_persists_retrieval_to_r assert len(summary_paths) == 1 data = json.loads(summary_paths[0].read_text(encoding="utf-8")) - assert data["quest_name"] == "Boat" - steps = data["steps"] - assert len(steps) >= 2 + assert data["schema_version"] == 2 + assert data["quest"]["name"] == "Boat" + transitions = data["transitions"] + assert len(transitions) >= 2 - retrieval_steps = [ - step - for step in steps - if (step.get("llm_decision") or {}).get("tool_calls") - and step["llm_decision"]["tool_calls"][0].get("tool") == "history_search" + retrieval_transitions = [ + transition + for transition in transitions + if (transition.get("response") or {}).get("tool_calls") + and transition["response"]["tool_calls"][0].get("tool") == "history_search" ] - assert retrieval_steps, "expected at least one persisted history_search tool call" - retrieved_decision = retrieval_steps[0]["llm_decision"] - assert retrieved_decision["tool_results"] - assert "history_search(" in retrieved_decision["tool_results"][0] + assert retrieval_transitions, "expected at least one persisted history_search tool call" + retrieved_response = retrieval_transitions[0]["response"] + assert retrieved_response["tool_results"] + assert "history_search(" in retrieved_response["tool_results"][0] diff --git a/llm_quest_benchmark/tests/integration/test_quest_e2e.py b/llm_quest_benchmark/tests/integration/test_quest_e2e.py index 3d02d1a..7e6702e 100644 --- a/llm_quest_benchmark/tests/integration/test_quest_e2e.py +++ b/llm_quest_benchmark/tests/integration/test_quest_e2e.py @@ -35,7 +35,7 @@ def mock_callback(event: str, data: Any) -> None: if event == "progress": caplog.info(f"Progress update - Step {data['step']}: {data['message']}") elif event == "game_state": - caplog.info(f"Game state update - Step {data.step}") + caplog.info(f"Game state update - Transition {data.index}") elif event == "error": caplog.error(f"Error: {data}") @@ -76,7 +76,7 @@ def mock_callback(event: str, data: Any) -> None: if event == "progress": caplog.info(f"Progress update - Step {data['step']}: {data['message']}") elif event == "game_state": - caplog.info(f"Game state update - Step {data.step}") + caplog.info(f"Game state update - Transition {data.index}") elif event == "error": caplog.error(f"Error: {data}") diff --git a/llm_quest_benchmark/tests/integration/test_resume.py b/llm_quest_benchmark/tests/integration/test_resume.py new file mode 100644 index 0000000..a5bd384 --- /dev/null +++ b/llm_quest_benchmark/tests/integration/test_resume.py @@ -0,0 +1,181 @@ +"""End-to-end truncate/resume and restore against the real QM engine.""" + +import json +from pathlib import Path + +import pytest + +from llm_quest_benchmark.core import logging as logging_module +from llm_quest_benchmark.core.replay import harness_config_from_record, replay_record, verify_environment +from llm_quest_benchmark.core.runner import run_quest_with_timeout +from llm_quest_benchmark.environments.qm import QMPlayerEnv +from llm_quest_benchmark.environments.state import QuestOutcome +from llm_quest_benchmark.harnesses.factory import create_harness +from llm_quest_benchmark.schemas.config import HarnessConfig +from llm_quest_benchmark.schemas.records import load_run_record + +REPO_ROOT = Path(__file__).resolve().parents[3] +BOAT = str(REPO_ROOT / "quests" / "Boat.qm") +HARNESS = "random_choice_7" +MAX_STEPS = 6 + + +@pytest.fixture +def isolated_results(tmp_path, monkeypatch): + """Keep run records and the metrics database inside the test directory.""" + monkeypatch.setattr(logging_module, "RESULTS_DIR", tmp_path / "results") + monkeypatch.setattr(logging_module, "DEFAULT_DB_PATH", str(tmp_path / "metrics.db")) + return tmp_path + + +def _config() -> HarnessConfig: + return HarnessConfig(harness=HARNESS, model="random_choice", skip_single=False) + + +def _agent(): + return create_harness(harness=HARNESS, model="random_choice", skip_single=False) + + +def _run(max_steps, resume_record=None, resume_path=None) -> QuestOutcome: + return run_quest_with_timeout( + quest_path=BOAT, + agent=_agent(), + timeout=120, + agent_config=_config(), + max_steps=max_steps, + resume_record=resume_record, + resume_path=resume_path, + ) + + +def _latest_record(results_dir: Path): + summaries = sorted( + results_dir.rglob("run_summary.json"), + key=lambda path: int(path.parent.name.split("_", 1)[1]), + ) + assert summaries, "expected at least one exported run record" + return summaries[-1], load_run_record(str(summaries[-1])) + + +def _actions(record): + return [(t.action.kind, t.action.choice_index, t.action.checkpoint_index) for t in record.transitions] + + +def _quest_state(snapshot): + """Observable quest state, excluding engine bookkeeping. + + Two independently executed runs stamp their own wall-clock time into the + engine saving's performedJumps log, so the raw saving (and therefore the + snapshot digest) cannot match across runs. Everything the quest actually + presents must. + """ + return ( + snapshot.location_id, + snapshot.observation, + tuple(snapshot.params_state), + tuple((c["id"], c["text"]) for c in snapshot.choices), + snapshot.game_state, + snapshot.done, + ) + + +@pytest.mark.integration +@pytest.mark.timeout(180) +def test_truncated_run_resumes_to_the_same_state_as_an_uninterrupted_run(isolated_results): + results_dir = isolated_results / "results" + + # 1. Uninterrupted reference run. + assert _run(MAX_STEPS) == QuestOutcome.TRUNCATED + _, reference = _latest_record(results_dir) + assert len(reference.transitions) == MAX_STEPS + + # 2. Truncate early, then resume from the record. + assert _run(3) == QuestOutcome.TRUNCATED + truncated_path, truncated = _latest_record(results_dir) + assert truncated.outcome == "TRUNCATED" + assert truncated.is_resumable + assert len(truncated.transitions) == 3 + + assert _run(MAX_STEPS, resume_record=truncated, resume_path=str(truncated_path)) == QuestOutcome.TRUNCATED + _, resumed = _latest_record(results_dir) + + # Prior transitions are preserved and the run reaches the same state. + assert len(resumed.transitions) == MAX_STEPS + assert _actions(resumed) == _actions(reference) + assert _quest_state(resumed.terminal_snapshot) == _quest_state(reference.terminal_snapshot) + assert resumed.lineage is not None + assert resumed.lineage.source_run_id == truncated.run_id + assert resumed.lineage.resumed_from_index == 3 + + +@pytest.mark.integration +@pytest.mark.timeout(180) +def test_recorded_run_replays_against_the_real_engine(isolated_results): + results_dir = isolated_results / "results" + assert _run(4) == QuestOutcome.TRUNCATED + _, record = _latest_record(results_dir) + + verify_environment(record, record.quest_file) + env = QMPlayerEnv(record.quest_file, language=record.quest_language) + try: + result = replay_record(env, record) + finally: + env.close() + + assert result.verified_transitions == len(record.transitions) + assert result.snapshot.digest == record.transitions[-1].after.digest + + +@pytest.mark.integration +@pytest.mark.timeout(180) +def test_restore_returns_to_the_exact_recorded_checkpoint(isolated_results): + """A real restore reproduces the recorded snapshot digest exactly.""" + env = QMPlayerEnv(BOAT) + try: + start = env.reset() + checkpoints = [start] + for step in range(3): + checkpoints.append(env.step(1, 1735689600000 + step)) + + restored = env.restore(checkpoints[1]) + + assert restored.digest == checkpoints[1].digest + assert restored.saving == checkpoints[1].saving + # The engine really moved: continuing from here matches the branch again. + assert env.snapshot.digest == checkpoints[1].digest + finally: + env.close() + + +@pytest.mark.integration +@pytest.mark.timeout(180) +def test_resume_is_rejected_when_the_quest_changed(isolated_results, tmp_path): + results_dir = isolated_results / "results" + assert _run(2) == QuestOutcome.TRUNCATED + path, record = _latest_record(results_dir) + + tampered = json.loads(path.read_text(encoding="utf-8")) + tampered["quest"]["checksum"] = "sha256:not-this-quest" + tampered_path = tmp_path / "tampered.json" + tampered_path.write_text(json.dumps(tampered), encoding="utf-8") + tampered_record = load_run_record(str(tampered_path)) + + with pytest.raises(Exception, match="Quest checksum mismatch"): + _run(4, resume_record=tampered_record, resume_path=str(tampered_path)) + + # Nothing about the original record changed. + assert load_run_record(str(path)).quest_checksum == record.quest_checksum + + +@pytest.mark.integration +@pytest.mark.timeout(180) +def test_resume_config_is_rebuilt_from_the_record(isolated_results): + results_dir = isolated_results / "results" + assert _run(2) == QuestOutcome.TRUNCATED + _, record = _latest_record(results_dir) + + config = harness_config_from_record(record) + + assert config.harness == HARNESS + assert config.model == "random_choice" + assert config.treatment().signature == record.treatment_signature diff --git a/llm_quest_benchmark/tests/integration/test_ts_bridge.py b/llm_quest_benchmark/tests/integration/test_ts_bridge.py index e0944f9..ecc55a4 100644 --- a/llm_quest_benchmark/tests/integration/test_ts_bridge.py +++ b/llm_quest_benchmark/tests/integration/test_ts_bridge.py @@ -1,5 +1,6 @@ """Tests for TypeScript bridge""" +import json from pathlib import Path from unittest.mock import Mock @@ -8,13 +9,15 @@ from llm_quest_benchmark.constants import DEFAULT_QUEST from llm_quest_benchmark.executors.ts_bridge.bridge import QMBridge +MOCK_SAVING = {"locationId": 1, "aleaState": [0.5, 0.25, 0.125, 7], "performedJumps": []} MOCK_PROTOCOL_RESPONSE = { "state": { "text": "Test observation", "choices": [{"jumpId": "1", "text": "Choice 1", "active": True}], "gameState": "running", + "paramsState": ["HP: 10"], }, - "saving": {"locationId": 1}, + "saving": MOCK_SAVING, } @@ -46,34 +49,72 @@ def test_bridge_invalid_quest(): QMBridge("nonexistent.qm") -def test_bridge_game_flow(monkeypatch): - """Test complete game flow with mocked protocol layer.""" +def test_bridge_game_flow_carries_saving_and_timestamp(monkeypatch): + """Complete game flow with a mocked protocol layer.""" bridge = QMBridge(str(DEFAULT_QUEST)) + sent = [] + process = Mock() + process.stdin = Mock() + process.stdin.write.side_effect = sent.append + process.poll.return_value = None try: monkeypatch.setattr(bridge, "_read_protocol_message", lambda **kw: (MOCK_PROTOCOL_RESPONSE, [])) - bridge.process = Mock() - bridge.process.stdin = Mock() - bridge.process.poll.return_value = None + monkeypatch.setattr( + "llm_quest_benchmark.executors.ts_bridge.bridge.subprocess.Popen", + lambda *args, **kwargs: process, + ) state = bridge.start_game() assert state.location_id == "1" assert state.text == "Test observation" + assert state.params_state == ["HP: 10"] + assert state.saving == MOCK_SAVING assert len(state.choices) == 1 assert state.choices[0]["id"] == "1" - assert state.choices[0]["text"] == "Choice 1" assert not state.game_ended - state = bridge.step("1") - assert state.location_id == "1" - assert state.text == "Test observation" - assert len(state.choices) == 1 - assert not state.game_ended + state = bridge.step(1, 1735689600123) + assert state.saving == MOCK_SAVING + command = json.loads(sent[-1]) + assert command == {"cmd": "jump", "jumpId": 1, "performedAtMs": 1735689600123} state = bridge.get_current_state() + assert json.loads(sent[-1]) == {"cmd": "state"} assert state.location_id == "1" - assert state.text == "Test observation" - assert len(state.choices) == 1 - assert not state.game_ended + + state = bridge.load_saving(MOCK_SAVING) + assert json.loads(sent[-1]) == {"cmd": "load", "saving": MOCK_SAVING} + assert state.saving == MOCK_SAVING + finally: + bridge.close() + + +def test_bridge_state_converts_to_canonical_snapshot(monkeypatch): + bridge = QMBridge(str(DEFAULT_QUEST)) + try: + monkeypatch.setattr(bridge, "_read_protocol_message", lambda **kw: (MOCK_PROTOCOL_RESPONSE, [])) + bridge.process = Mock() + bridge.process.stdin = Mock() + bridge.process.poll.return_value = None + + snapshot = bridge.start_game().to_snapshot() + + assert snapshot.location_id == "1" + assert snapshot.observation == "Test observation" + assert snapshot.params_state == ["HP: 10"] + assert snapshot.saving == MOCK_SAVING + assert snapshot.digest == snapshot.compute_digest() + assert snapshot.is_resumable + finally: + bridge.close() + + +def test_bridge_load_saving_requires_a_saving(): + bridge = QMBridge(str(DEFAULT_QUEST)) + try: + bridge.process = Mock() + with pytest.raises(ValueError, match="non-empty engine saving"): + bridge.load_saving({}) finally: bridge.close() diff --git a/llm_quest_benchmark/tests/players/test_random_player.py b/llm_quest_benchmark/tests/players/test_random_player.py new file mode 100644 index 0000000..ddd9739 --- /dev/null +++ b/llm_quest_benchmark/tests/players/test_random_player.py @@ -0,0 +1,21 @@ +from llm_quest_benchmark.players.random import RandomPlayer + + +def test_random_player_replaces_prior_auto_single_response(): + player = RandomPlayer(seed=7, skip_single=True) + + assert player.get_action("intro", [{"id": "1", "text": "Continue"}]) == 1 + assert player.get_last_response().is_default is True + + action = player.get_action( + "decision", + [ + {"id": "2", "text": "Left"}, + {"id": "3", "text": "Right"}, + ], + ) + + response = player.get_last_response() + assert response.action == action + assert response.is_default is False + assert response.reasoning is None diff --git a/llm_quest_benchmark/tests/test_benchmark_report.py b/llm_quest_benchmark/tests/test_benchmark_report.py index 3f7e552..d30a3bc 100644 --- a/llm_quest_benchmark/tests/test_benchmark_report.py +++ b/llm_quest_benchmark/tests/test_benchmark_report.py @@ -4,69 +4,103 @@ from pathlib import Path from llm_quest_benchmark.core.benchmark_report import render_benchmark_report +from llm_quest_benchmark.harnesses.specs import build_treatment +from llm_quest_benchmark.schemas.records import ( + ProgressState, + QuestAction, + QuestSnapshot, + QuestTransition, + RunRecord, +) +from llm_quest_benchmark.schemas.response import LLMResponse + + +def _treatment(model: str, harness: str) -> dict: + return build_treatment( + harness=harness, + model=model, + temperature=0.4, + system_template="system_role.jinja", + ).to_dict() + + +def _db_run(run_id: int, model: str, harness: str, outcome: str, benchmark_id: str) -> dict: + treatment = _treatment(model, harness) + return { + "id": run_id, + "quest_file": "quests/Boat.qm", + "quest_name": "Boat", + "start_time": "2026-02-15T00:00:00", + "end_time": "2026-02-15T00:00:10", + "agent_id": f"{model}_t0.4_{harness}_{treatment['signature'][3:11]}", + "treatment": json.dumps(treatment), + "treatment_signature": treatment["signature"], + "outcome": outcome, + "reward": 1.0 if outcome == "SUCCESS" else 0.0, + "run_duration": 10.0, + "benchmark_id": benchmark_id, + } -def test_render_benchmark_report_reads_run_summaries(tmp_path, monkeypatch): - monkeypatch.chdir(tmp_path) - - benchmark_id = "bench_test_1" - benchmark_dir = Path("results/benchmarks") / benchmark_id - benchmark_dir.mkdir(parents=True, exist_ok=True) - - db_runs = [ - { - "id": 1, - "quest_file": "quests/Boat.qm", - "quest_name": "Boat", - "start_time": "2026-02-15T00:00:00", - "end_time": "2026-02-15T00:00:10", - "agent_id": "llm_gpt-5-mini", - "agent_config": json.dumps({"model": "gpt-5-mini"}), - "outcome": "SUCCESS", - "reward": 1.0, - "run_duration": 10.0, - "benchmark_id": benchmark_id, - } - ] - summary = {"benchmark_id": benchmark_id, "db_runs": db_runs, "results": []} - (benchmark_dir / "benchmark_summary.json").write_text( - json.dumps(summary, ensure_ascii=False), - encoding="utf-8", +def _write_run_record(agent_id: str, run_id: int, treatment: dict) -> None: + before = QuestSnapshot( + location_id="1", + observation="state", + choices=[{"id": "11", "text": "go"}, {"id": "12", "text": "stop"}], + saving={"locationId": 1}, ) - - run_summary_dir = Path("results/llm_gpt-5-mini/Boat/run_1") - run_summary_dir.mkdir(parents=True, exist_ok=True) - run_summary = { - "run_id": 1, - "quest_name": "Boat", - "agent_id": "llm_gpt-5-mini", - "outcome": "SUCCESS", - "run_duration": 8.5, - "usage": { + after = QuestSnapshot(location_id="2", observation="next", choices=[], done=True, game_state="win", saving={}) + record = RunRecord( + run_id=run_id, + quest_file="quests/Boat.qm", + quest_name="Boat", + quest_checksum="sha256:test", + quest_language="rus", + engine_revision="git:test", + agent_id=agent_id, + treatment=treatment, + outcome="SUCCESS", + reward=1.0, + run_duration=8.5, + usage={ "prompt_tokens": 100, "completion_tokens": 20, "total_tokens": 120, "estimated_cost_usd": 0.001, "priced_steps": 1, }, - "steps": [ - { - "step": 1, - "observation": "state", - "choices": {"1": "go", "2": "stop"}, - "llm_decision": { - "analysis": "short", - "reasoning": "pick go", - "is_default": False, - "choice": {"1": "go"}, - }, - } + progress=ProgressState(current=100.0), + transitions=[ + QuestTransition( + index=1, + before=before, + action=QuestAction.choose(1, "11", 1735689600000), + after=after, + response=LLMResponse(action=1, analysis="short", reasoning="pick go"), + usage={"prompt_tokens": 100, "completion_tokens": 20, "total_tokens": 120}, + progress=ProgressState(current=100.0), + ) ], - } - (run_summary_dir / "run_summary.json").write_text( - json.dumps(run_summary, ensure_ascii=False), + ) + run_dir = Path("results") / agent_id / "Boat" / f"run_{run_id}" + run_dir.mkdir(parents=True, exist_ok=True) + (run_dir / "run_summary.json").write_text(json.dumps(record.to_dict(), ensure_ascii=False), encoding="utf-8") + + +def test_render_benchmark_report_reads_v2_run_records(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + + benchmark_id = "bench_test_1" + benchmark_dir = Path("results/benchmarks") / benchmark_id + benchmark_dir.mkdir(parents=True, exist_ok=True) + + run = _db_run(1, "gpt-5-mini", "reasoning_recent", "SUCCESS", benchmark_id) + summary = {"benchmark_id": benchmark_id, "db_runs": [run], "results": []} + (benchmark_dir / "benchmark_summary.json").write_text( + json.dumps(summary, ensure_ascii=False), encoding="utf-8", ) + _write_run_record(run["agent_id"], 1, json.loads(run["treatment"])) report, selected = render_benchmark_report( benchmark_ids=[benchmark_id], @@ -77,7 +111,10 @@ def test_render_benchmark_report_reads_run_summaries(tmp_path, monkeypatch): assert "| Total runs | 1 |" in report assert "| Success | 1 |" in report assert "| Total tokens | 120 |" in report - assert "| gpt-5-mini | 1 | 1 | 0 | 0 | 0 | 100.0% | 120 | 0.001000 | 0.0% |" in report + # Model comes from the recorded treatment, not from parsing the agent id. + assert "| gpt-5-mini |" in report + assert run["treatment_signature"] in report + assert "100.0%" in report def test_render_benchmark_report_splits_same_model_different_harness(tmp_path, monkeypatch): @@ -89,24 +126,9 @@ def test_render_benchmark_report_splits_same_model_different_harness(tmp_path, m benchmark_dir = Path("results/benchmarks") / benchmark_id benchmark_dir.mkdir(parents=True, exist_ok=True) - def _db_run(run_id, harness, outcome): - return { - "id": run_id, - "quest_file": "quests/Boat.qm", - "quest_name": "Boat", - "start_time": "2026-02-15T00:00:00", - "end_time": "2026-02-15T00:00:10", - "agent_id": f"llm_gemini-3-flash_{harness}", - "agent_config": json.dumps({"model": "gemini-3-flash", "harness": harness}), - "outcome": outcome, - "reward": 1.0 if outcome == "SUCCESS" else 0.0, - "run_duration": 10.0, - "benchmark_id": benchmark_id, - } - db_runs = [ - _db_run(1, "reasoning_recent", "SUCCESS"), - _db_run(2, "programmatic_memory", "FAILURE"), + _db_run(1, "gpt-5-mini", "reasoning_recent", "SUCCESS", benchmark_id), + _db_run(2, "gpt-5-mini", "programmatic_memory", "FAILURE", benchmark_id), ] summary = {"benchmark_id": benchmark_id, "db_runs": db_runs, "results": []} (benchmark_dir / "benchmark_summary.json").write_text( @@ -120,7 +142,9 @@ def _db_run(run_id, harness, outcome): ) assert selected == [benchmark_id] - assert "gemini-3-flash [reasoning_recent]" in report - assert "gemini-3-flash [programmatic_memory]" in report - # Neither harness-qualified row should collapse into a bare "gemini-3-flash" row. - assert "| gemini-3-flash |" not in report + assert "gpt-5-mini [reasoning_recent]" in report + assert "gpt-5-mini [programmatic_memory]" in report + # Neither harness-qualified row should collapse into a bare "gpt-5-mini" row. + assert "| gpt-5-mini |" not in report + # Materially different treatments must carry different signatures. + assert db_runs[0]["treatment_signature"] != db_runs[1]["treatment_signature"] diff --git a/llm_quest_benchmark/tests/test_benchmark_summary.py b/llm_quest_benchmark/tests/test_benchmark_summary.py index b21654a..153b528 100644 --- a/llm_quest_benchmark/tests/test_benchmark_summary.py +++ b/llm_quest_benchmark/tests/test_benchmark_summary.py @@ -9,8 +9,6 @@ def _result(model, harness, outcome, agent_id=None): "model": model, "temperature": 0.4, "harness": harness, - "template": "reasoning.jinja", - "memory_mode": "default", "agent_id": agent_id or f"llm_{model}_{harness}", "attempt": 1, "outcome": outcome, diff --git a/llm_quest_benchmark/tests/test_database.py b/llm_quest_benchmark/tests/test_database.py index d00c6b1..6c75a0a 100644 --- a/llm_quest_benchmark/tests/test_database.py +++ b/llm_quest_benchmark/tests/test_database.py @@ -1,308 +1,278 @@ import json import os +import sqlite3 import tempfile import pytest from llm_quest_benchmark.core import logging as logging_module from llm_quest_benchmark.core.logging import QuestLogger +from llm_quest_benchmark.schemas.records import ( + SCHEMA_VERSION, + ProgressState, + QuestAction, + QuestSnapshot, + QuestTransition, +) from llm_quest_benchmark.schemas.response import LLMResponse -from llm_quest_benchmark.schemas.state import AgentState + +TREATMENT = { + "harness": "reasoning_recent", + "prompt": "reasoning.jinja", + "memory": "recent_window", + "tools": [], + "loop": "single_call", + "reasoning": "concise", + "model": "gpt-5-mini", + "temperature": 0.4, + "system_prompt": "system_role.jinja", + "knobs": {}, + "signature": "t2_0123456789abcdef", +} @pytest.fixture def quest_logger(): """Create a temporary quest logger for testing""" - # Create a temporary database file fd, db_path = tempfile.mkstemp(suffix=".db") os.close(fd) - # Create logger with the temporary database logger = QuestLogger(db_path=db_path, debug=True) yield logger - # Clean up logger.close() os.unlink(db_path) -def test_quest_logger_initialization(quest_logger): - """Test quest logger initialization""" - # Ensure we have a connection for this thread +def _snapshot(location_id: str = "room1", observation: str = "You are in a room", choices=None) -> QuestSnapshot: + return QuestSnapshot( + location_id=location_id, + observation=observation, + choices=choices + if choices is not None + else [{"id": "11", "text": "Go north"}, {"id": "12", "text": "Go south"}], + params_state=["HP: 10"], + saving={"locationId": int(location_id[-1]) if location_id[-1].isdigit() else 1}, + ) + + +def _transition(index: int, choice_index: int, response: LLMResponse | None = None) -> QuestTransition: + before = _snapshot(location_id=f"room{index}") + after = _snapshot(location_id=f"room{index + 1}") + return QuestTransition( + index=index, + before=before, + action=QuestAction.choose(choice_index, before.choices[choice_index - 1]["id"], 1735689600000 + index), + after=after, + response=response or LLMResponse(action=choice_index), + usage={ + "prompt_tokens": response.prompt_tokens if response else 0, + "completion_tokens": response.completion_tokens if response else 0, + "total_tokens": response.total_tokens if response else 0, + "estimated_cost_usd": response.estimated_cost_usd if response else None, + }, + progress=ProgressState(current=float(index * 10)), + ) + + +def _start_run(logger: QuestLogger, quest_file: str, agent_id: str = "llm_test-agent") -> int: + return logger.start_run( + quest_file=quest_file, + quest_name=quest_file.split("/")[-1].removesuffix(".qm"), + quest_checksum="sha256:test", + quest_language="rus", + engine_revision="git:test", + agent_id=agent_id, + treatment=TREATMENT, + ) + + +def test_quest_logger_creates_v2_tables(quest_logger): quest_logger._init_connection() - # Check that the database was created assert os.path.exists(quest_logger.db_path) - # Check that tables were created quest_logger._local.cursor.execute("SELECT name FROM sqlite_master WHERE type='table'") - tables = quest_logger._local.cursor.fetchall() - table_names = [table[0] for table in tables] + table_names = [table[0] for table in quest_logger._local.cursor.fetchall()] assert "runs" in table_names - assert "steps" in table_names - - -def test_quest_logger_log_step(quest_logger): - """Test logging a single step""" - # Set up quest file - quest_logger.set_quest_file("test_quest.qm") - - # Create agent state - agent_state = AgentState( - step=1, - location_id="room1", - observation="You are in a room", - choices=[{"id": "1", "text": "Go north"}, {"id": "2", "text": "Go south"}], - action="1", - llm_response=LLMResponse(action=1, analysis="I should go north", reasoning="The north path looks safer"), - ) - - # Log the step - quest_logger.log_step(agent_state) + assert "transitions" in table_names + assert "steps" not in table_names - # Ensure we have a connection for this thread - quest_logger._init_connection() - # Check run was created - quest_logger._local.cursor.execute("SELECT * FROM runs") - run = quest_logger._local.cursor.fetchone() - assert run is not None - assert run[1] == "test_quest.qm" # quest_file - - # Check step was logged - get column names first - quest_logger._local.cursor.execute("PRAGMA table_info(steps)") - columns = quest_logger._local.cursor.fetchall() - column_names = [col[1] for col in columns] - print(f"Steps table columns: {column_names}") - - # Now query the step data - quest_logger._local.cursor.execute("SELECT * FROM steps WHERE run_id = ?", (run[0],)) - step = quest_logger._local.cursor.fetchone() - assert step is not None - - # Get the indices for each column - run_id_idx = column_names.index("run_id") - step_idx = column_names.index("step") - location_id_idx = column_names.index("location_id") - observation_idx = column_names.index("observation") - choices_idx = column_names.index("choices") - action_idx = column_names.index("action") - llm_response_idx = column_names.index("llm_response") - - # Check values using column indices - assert step[run_id_idx] == run[0] # run_id - assert step[step_idx] == 1 # step number - assert step[location_id_idx] == "room1" # location_id - assert step[observation_idx] == "You are in a room" # observation - assert "Go north" in step[choices_idx] # choices - assert step[action_idx] == "1" # action - assert "I should go north" in step[llm_response_idx] # llm_response - - -def test_quest_logger_multiple_steps(quest_logger): - """Test logging multiple steps in sequence""" - # Set up quest file once at the beginning - quest_logger.set_quest_file("test_quest.qm") - run_id = quest_logger.current_run_id - - # Create and log multiple steps - steps = [ - AgentState( - step=1, - location_id="room1", - observation="Room 1", - choices=[{"id": "1", "text": "North"}, {"id": "2", "text": "South"}], - action="1", - llm_response=LLMResponse(action=1), - ), - AgentState( - step=2, - location_id="room2", - observation="Room 2", - choices=[{"id": "1", "text": "East"}, {"id": "2", "text": "West"}], - action="2", - llm_response=LLMResponse(action=2), - ), - AgentState( - step=3, - location_id="room3", - observation="Room 3", - choices=[{"id": "1", "text": "Up"}, {"id": "2", "text": "Down"}], - action="1", - llm_response=LLMResponse(action=1), - ), - ] +def test_quest_logger_rejects_legacy_database(tmp_path): + """A pre-v2 database must fail loudly instead of being silently upgraded.""" + db_path = tmp_path / "legacy.db" + conn = sqlite3.connect(db_path) + conn.execute("CREATE TABLE runs (id INTEGER PRIMARY KEY, quest_name TEXT, outcome TEXT)") + conn.commit() + conn.close() - for step in steps: - quest_logger.log_step(step) + with pytest.raises(RuntimeError, match="migrate_records"): + QuestLogger(db_path=str(db_path)) - # Ensure we have a connection for this thread - quest_logger._init_connection() - # Get column names first - quest_logger._local.cursor.execute("PRAGMA table_info(steps)") - columns = quest_logger._local.cursor.fetchall() - column_names = [col[1] for col in columns] +def test_run_metadata_is_written_before_any_transition(quest_logger): + """Identity, quest provenance, and treatment must exist before execution.""" + run_id = _start_run(quest_logger, "quests/kr_1_ru/Test.qm") - # Get the indices for each column - step_idx = column_names.index("step") - action_idx = column_names.index("action") + quest_logger._local.cursor.execute( + "SELECT schema_version, quest_file, quest_checksum, quest_language, engine_revision, " + "agent_id, treatment, treatment_signature, outcome FROM runs WHERE id = ?", + (run_id,), + ) + row = quest_logger._local.cursor.fetchone() - # Check all steps were logged - quest_logger._local.cursor.execute("SELECT * FROM steps WHERE run_id = ? ORDER BY step", (run_id,)) - logged_steps = quest_logger._local.cursor.fetchall() - assert len(logged_steps) == 3 + assert row[0] == SCHEMA_VERSION + assert row[1] == "quests/kr_1_ru/Test.qm" + assert row[2] == "sha256:test" + assert row[3] == "rus" + assert row[4] == "git:test" + assert row[5] == "llm_test-agent" + assert json.loads(row[6])["harness"] == "reasoning_recent" + assert row[7] == TREATMENT["signature"] + assert row[8] is None # outcome is only known at the end - # Verify step sequence using column indices - assert logged_steps[0][step_idx] == 1 # first step - assert logged_steps[1][step_idx] == 2 # second step - assert logged_steps[2][step_idx] == 3 # third step - # Verify actions using column indices - assert logged_steps[0][action_idx] == "1" # first action - assert logged_steps[1][action_idx] == "2" # second action - assert logged_steps[2][action_idx] == "1" # third action +def test_log_transition_persists_canonical_columns(quest_logger): + run_id = _start_run(quest_logger, "quests/kr_1_ru/Test.qm") + transition = _transition(1, 2, LLMResponse(action=2, analysis="south is safer")) + quest_logger.log_transition(transition) -def test_run_summary_export_is_compact_and_single_file(tmp_path, monkeypatch, quest_logger): - """Run summary export keeps compact step schema and no per-step JSON files.""" + quest_logger._local.cursor.execute( + "SELECT transition_index, before_state, action, after_state, response, provenance, replay_status " + "FROM transitions WHERE run_id = ?", + (run_id,), + ) + row = quest_logger._local.cursor.fetchone() + + assert row[0] == 1 + before = json.loads(row[1]) + assert before["choices"][1]["text"] == "Go south" + assert before["saving"] is not None + action = json.loads(row[2]) + assert action["kind"] == "choose" + assert action["choice_index"] == 2 + assert action["performed_at_ms"] == 1735689600001 + assert json.loads(row[3])["location_id"] == "room2" + assert "south is safer" in row[4] + assert row[5] == "runtime" + assert row[6] == "pending" + + +def test_run_summary_export_is_schema_v2(tmp_path, monkeypatch, quest_logger): monkeypatch.setattr(logging_module, "RESULTS_DIR", tmp_path) - quest_logger.agent = "llm_test-agent" - quest_logger.set_quest_file("quests/kr_1_ru/Test.qm") - run_id = quest_logger.current_run_id - - agent_state = AgentState( - step=1, - location_id="room1", - observation="You are in a room", - choices=[ - {"id": "1", "text": "Go north"}, - {"id": "2", "text": "Go south"}, - ], - action="2", - llm_response=LLMResponse( - action=2, - analysis="South has better odds", - reasoning="Avoid immediate danger", - is_default=False, - prompt_tokens=12, - completion_tokens=5, - total_tokens=17, - estimated_cost_usd=0.000123, - ), + run_id = _start_run(quest_logger, "quests/kr_1_ru/Test.qm") + response = LLMResponse( + action=2, + analysis="South has better odds", + reasoning="Avoid immediate danger", + prompt_tokens=12, + completion_tokens=5, + total_tokens=17, + estimated_cost_usd=0.000123, ) - quest_logger.log_step(agent_state) - quest_logger.set_quest_outcome("FAILURE", reward=0.0) + transition = _transition(1, 2, response) + quest_logger.log_transition(transition) + quest_logger.finish_run("FAILURE", reward=0.0, terminal_snapshot=transition.after) run_dir = tmp_path / "llm_test-agent" / "Test" / f"run_{run_id}" summary_path = run_dir / "run_summary.json" assert summary_path.exists() - assert not (run_dir / "step_1.json").exists() exported = json.loads(summary_path.read_text(encoding="utf-8")) - exported_step = exported["steps"][0] - assert set(exported_step.keys()) == {"step", "location_id", "observation", "choices", "llm_decision"} - assert exported_step["choices"] == {"1": "Go north", "2": "Go south"} - assert exported_step["llm_decision"]["choice"] == {"2": "Go south"} + assert exported["schema_version"] == SCHEMA_VERSION + assert exported["quest"]["checksum"] == "sha256:test" + assert exported["treatment"]["signature"] == TREATMENT["signature"] + assert "steps" not in exported + assert "outcome" not in exported + assert "metrics" not in exported + assert "final_snapshot" not in exported + + exported_transition = exported["transitions"][0] + assert exported_transition["action"]["choice_index"] == 2 + assert exported_transition["before"]["choices"][1]["text"] == "Go south" + assert exported_transition["after"]["digest"] + assert exported["usage"]["prompt_tokens"] == 12 assert exported["usage"]["completion_tokens"] == 5 assert exported["usage"]["total_tokens"] == 17 assert exported["usage"]["estimated_cost_usd"] is not None - assert exported["metrics"]["total_steps"] == 1 - assert exported["metrics"]["repetition_count"] == 0 - assert exported["metrics"]["repetition_rate"] == 0.0 - assert exported["metrics"]["bad_decision_count"] == 1 - assert exported["metrics"]["bad_decision_rate"] == 1.0 + diagnostics = exported["transcript_diagnostics"] + assert diagnostics["total_steps"] == 1 + assert diagnostics["repetition_count"] == 0 + assert diagnostics["bad_decision_count"] == 1 + assert diagnostics["bad_decision_rate"] == 1.0 + assert exported["terminal"]["snapshot"]["location_id"] == "room2" def test_run_summary_export_tracks_repetition_rate(tmp_path, monkeypatch, quest_logger): - """Run summary export computes repetition rate from the last five actions.""" monkeypatch.setattr(logging_module, "RESULTS_DIR", tmp_path) - quest_logger.agent = "llm_test-agent" - quest_logger.set_quest_file("quests/kr_1_ru/Loop.qm") - run_id = quest_logger.current_run_id - - for step_num, action in enumerate(["1", "2", "3", "4", "5", "1"], start=1): - quest_logger.log_step( - AgentState( - step=step_num, - location_id=f"room{step_num}", - observation=f"Step {step_num}", - choices=[{"id": "1", "text": "A"}, {"id": "2", "text": "B"}], - action=action, - llm_response=LLMResponse(action=int(action)), - ) - ) - - quest_logger.set_quest_outcome("SUCCESS", reward=1.0) + run_id = _start_run(quest_logger, "quests/kr_1_ru/Loop.qm") + for index, action in enumerate([1, 2, 1, 2, 1, 2], start=1): + quest_logger.log_transition(_transition(index, action)) + quest_logger.finish_run("SUCCESS", reward=1.0) summary_path = tmp_path / "llm_test-agent" / "Loop" / f"run_{run_id}" / "run_summary.json" exported = json.loads(summary_path.read_text(encoding="utf-8")) - assert exported["metrics"]["total_steps"] == 6 - assert exported["metrics"]["repetition_window"] == 5 - assert exported["metrics"]["repetition_count"] == 1 - assert exported["metrics"]["repetition_rate"] == pytest.approx(1 / 6) - assert exported["metrics"]["bad_decision_count"] == 0 - assert exported["metrics"]["bad_decision_rate"] == 0.0 + diagnostics = exported["transcript_diagnostics"] + assert diagnostics["total_steps"] == 6 + assert diagnostics["repetition_window"] == 5 + assert diagnostics["repetition_count"] == 4 + assert diagnostics["repetition_rate"] == pytest.approx(4 / 6) + assert diagnostics["bad_decision_count"] == 0 -def test_random_player_does_not_export_json(tmp_path, monkeypatch, quest_logger): - """Random player runs should not create result artifacts in results/.""" +def test_random_player_exports_v2_json(tmp_path, monkeypatch, quest_logger): + """Random runs are recorded like every other player type, not suppressed.""" monkeypatch.setattr(logging_module, "RESULTS_DIR", tmp_path) - quest_logger.agent = "random_choice" - quest_logger.set_quest_file("quests/kr_1_ru/Test.qm") - - quest_logger.log_step( - AgentState( - step=1, - location_id="r1", - observation="obs", - choices=[{"id": "1", "text": "x"}], - action="1", - llm_response=LLMResponse(action=1), - ) + run_id = _start_run(quest_logger, "quests/kr_1_ru/Test.qm", agent_id="random_choice_t0.0_random_choice_abcdef12") + quest_logger.log_transition(_transition(1, 1)) + quest_logger.finish_run("FAILURE", reward=0.0) + + summary_path = ( + tmp_path / "random_choice_t0.0_random_choice_abcdef12" / "Test" / f"run_{run_id}" / "run_summary.json" ) - quest_logger.set_quest_outcome("FAILURE", reward=0.0) + assert summary_path.exists() + assert json.loads(summary_path.read_text(encoding="utf-8"))["schema_version"] == SCHEMA_VERSION + + +def test_metrics_count_restore_transitions_separately(quest_logger): + transitions = [ + _transition(1, 1), + QuestTransition( + index=2, + before=_snapshot("room2"), + action=QuestAction.restore(1), + after=_snapshot("room1"), + ), + _transition(3, 2), + ] - assert not any(tmp_path.rglob("run_summary.json")) + metrics = QuestLogger.calculate_metrics(transitions, "FAILURE") + assert metrics["total_steps"] == 2 + assert metrics["restore_transitions"] == 1 + assert metrics["total_transitions"] == 3 -def test_set_quest_outcome_is_first_write_wins(tmp_path, monkeypatch, quest_logger): + +def test_finish_run_is_first_write_wins(tmp_path, monkeypatch, quest_logger): """Late outcome writes (e.g., from background timeout races) must be ignored.""" monkeypatch.setattr(logging_module, "RESULTS_DIR", tmp_path) - quest_logger.agent = "llm_test-agent" - quest_logger.set_quest_file("quests/kr_1_ru/Rush.qm") - run_id = quest_logger.current_run_id - - timeout_state = { - "location_id": "1", - "text": "race in progress", - "choices": [{"id": "2", "text": "hold speed"}], - "reward": 0.0, - "done": False, - "info": {}, - } - quest_logger.set_quest_outcome("TIMEOUT", reward=0.0, final_state=timeout_state) - - # Simulate a late background write that previously overwrote timeout runs. - success_state = { - "location_id": "99", - "text": "late success", - "choices": [], - "reward": 1.0, - "done": True, - "info": {}, - } - quest_logger.set_quest_outcome("SUCCESS", reward=1.0, final_state=success_state) + run_id = _start_run(quest_logger, "quests/kr_1_ru/Rush.qm") + timeout_snapshot = _snapshot(location_id="room1", observation="race in progress") + quest_logger.finish_run("TIMEOUT", reward=0.0, terminal_snapshot=timeout_snapshot) + + late_snapshot = _snapshot(location_id="room9", observation="late success", choices=[]) + quest_logger.finish_run("SUCCESS", reward=1.0, terminal_snapshot=late_snapshot) summary_path = tmp_path / "llm_test-agent" / "Rush" / f"run_{run_id}" / "run_summary.json" exported = json.loads(summary_path.read_text(encoding="utf-8")) - assert exported["outcome"] == "TIMEOUT" - assert exported["reward"] == 0.0 - assert exported["final_state"]["done"] is False - assert exported["final_state"]["text"] == "race in progress" + assert exported["terminal"]["outcome"] == "TIMEOUT" + assert exported["terminal"]["reward"] == 0.0 + assert exported["terminal"]["snapshot"]["observation"] == "race in progress" diff --git a/llm_quest_benchmark/tests/test_import_human_trace.py b/llm_quest_benchmark/tests/test_import_human_trace.py index f438bdc..7f7c12f 100644 --- a/llm_quest_benchmark/tests/test_import_human_trace.py +++ b/llm_quest_benchmark/tests/test_import_human_trace.py @@ -15,10 +15,23 @@ def load_module(): return module -def test_import_human_trace_converts_to_run_summary(tmp_path): - module = load_module() - raw_trace = { - "schema_version": "human_trace_v1", +def _snapshot(location_id, observation, choices, saving): + return { + "location_id": location_id, + "canonical_location_id": location_id, + "observation": observation, + "params": ["money: 10"], + "choices": choices, + "game_state": "running", + "saving": saving, + } + + +def _trace(): + dock = _snapshot("42", "You are at the dock.", {"1": "Board the ship", "2": "Go home"}, {"locationId": 42}) + deck = _snapshot("43", "You are on deck.", {"1": "Sail"}, {"locationId": 43}) + return { + "schema_version": "human_trace_v2", "source": "web_play", "quest_id": "Boat", "quest_title": "Boat", @@ -26,48 +39,82 @@ def test_import_human_trace_converts_to_run_summary(tmp_path): "started_at": "2026-05-21T12:00:00.000Z", "ended_at": "2026-05-21T12:01:00.000Z", "outcome": "SUCCESS", - "steps": [ + "transitions": [ { - "step": 1, - "location_id": 42, - "observation": "You are at the dock.", - "params": ["money: 10"], - "choices": {"1": "Board the ship", "2": "Go home"}, - "human_decision": { + "index": 1, + "kind": "choose", + "before": dock, + "action": { + "kind": "choose", "choice_index": "1", "choice_text": "Board the ship", "jump_id": 7, + "performed_at_ms": None, }, - } + "after": deck, + }, + { + "index": 2, + "kind": "restore", + "before": deck, + "action": {"kind": "restore", "checkpoint_index": 1}, + "after": dock, + }, ], "terminal": {"game_state": "win", "text": "Victory"}, } + + +def test_import_human_trace_converts_to_schema_v2_record(tmp_path): + module = load_module() + raw_trace = _trace() input_path = tmp_path / "human_trace_Boat.json" output_path = tmp_path / "run_summary.json" input_path.write_text(json.dumps(raw_trace), encoding="utf-8") - summary = module.convert_trace(raw_trace, source_trace=str(input_path)) - module.write_summary(summary, output_path) + record = module.convert_trace(raw_trace, source_trace=str(input_path)) + module.write_summary(record, output_path) saved = json.loads(output_path.read_text(encoding="utf-8")) - assert saved["quest_name"] == "Boat" - assert saved["agent_id"] == "human_web" - assert saved["agent_mode"] == "human" - assert saved["outcome"] == "SUCCESS" + assert saved["schema_version"] == 2 + assert saved["quest"]["name"] == "Boat" + assert saved["run"]["agent_id"] == "human_web" + assert saved["treatment"]["harness"] == "human" + assert saved["terminal"]["outcome"] == "SUCCESS" assert saved["usage"]["total_tokens"] == 0 - assert saved["steps"][0]["observation"] == "You are at the dock." - assert saved["steps"][0]["choices"] == {"1": "Board the ship", "2": "Go home"} - assert saved["steps"][0]["llm_decision"]["choice"] == {"1": "Board the ship"} - assert saved["steps"][0]["human_decision"]["jump_id"] == 7 - assert saved["terminal"]["text"] == "Victory" + + first = saved["transitions"][0] + assert first["before"]["observation"] == "You are at the dock." + assert [c["text"] for c in first["before"]["choices"]] == ["Board the ship", "Go home"] + assert first["action"]["choice_index"] == 1 + assert first["before"]["saving"] == {"locationId": 42} + # The browser cannot observe the engine timestamp, so replay stays unavailable. + assert first["action"]["performed_at_ms"] is None + assert first["replay_status"] == "unavailable" + + +def test_import_human_trace_keeps_restore_transitions(tmp_path): + """Undone steps stay in the record; a restore is an explicit transition.""" + module = load_module() + + record = module.convert_trace(_trace()) + saved = record.to_dict() + + assert len(saved["transitions"]) == 2 + restore = saved["transitions"][1] + assert restore["action"]["kind"] == "restore" + assert restore["action"]["checkpoint_index"] == 1 + assert restore["after"]["location_id"] == "42" + assert saved["transcript_diagnostics"]["restore_transitions"] == 1 + assert saved["transcript_diagnostics"]["total_steps"] == 1 def test_import_human_trace_rejects_wrong_schema(): module = load_module() try: - module.convert_trace({"schema_version": "other", "steps": []}) + module.convert_trace({"schema_version": "human_trace_v1", "transitions": []}) except ValueError as exc: - assert "human_trace_v1" in str(exc) + assert "human_trace_v2" in str(exc) else: raise AssertionError("expected ValueError") diff --git a/llm_quest_benchmark/tests/test_leaderboard.py b/llm_quest_benchmark/tests/test_leaderboard.py index 1bae2db..f29f568 100644 --- a/llm_quest_benchmark/tests/test_leaderboard.py +++ b/llm_quest_benchmark/tests/test_leaderboard.py @@ -4,6 +4,63 @@ import pytest from llm_quest_benchmark.core.leaderboard import generate_leaderboard +from llm_quest_benchmark.harnesses.specs import build_treatment + + +def _treatment(harness: str, model: str) -> dict: + return build_treatment( + harness=harness, + model=model, + temperature=0.4, + system_template="system_role.jinja", + ).to_dict() + + +def _result_row(quest: str, model: str, harness: str, agent_id: str, outcome: str = "SUCCESS", **extra) -> dict: + treatment = _treatment(harness, model) + row = { + "quest": quest, + "model": model, + "temperature": 0.4, + "harness": harness, + "treatment": treatment, + "treatment_signature": treatment["signature"], + "agent_id": agent_id, + "attempt": 1, + "outcome": outcome, + "reward": 1.0 if outcome == "SUCCESS" else 0.0, + "error": None, + } + row.update(extra) + return row + + +def _db_run( + run_id: int, + quest_file: str, + quest_name: str, + agent_id: str, + harness: str, + model: str, + outcome: str, + usage: dict | None = None, + diagnostics: dict | None = None, + progress: dict | None = None, +) -> dict: + treatment = _treatment(harness, model) + return { + "id": run_id, + "schema_version": 2, + "quest_file": quest_file, + "quest_name": quest_name, + "agent_id": agent_id, + "treatment": treatment, + "treatment_signature": treatment["signature"], + "outcome": outcome, + "usage": usage or {}, + "transcript_diagnostics": diagnostics or {}, + "progress": progress or {}, + } def test_generate_leaderboard_aggregates_runs(tmp_path, monkeypatch): @@ -13,66 +70,48 @@ def test_generate_leaderboard_aggregates_runs(tmp_path, monkeypatch): benchmark_dir.mkdir(parents=True, exist_ok=True) results = [ - { - "quest": "quests/ru/Boat.qm", - "model": "gemini-2.5-flash", - "temperature": 0.4, - "template": "stub.jinja", - "agent_id": "llm_gemini-2.5-flash", - "attempt": 1, - "outcome": "SUCCESS", - "reward": 1.0, - "error": None, - }, - { - "quest": "quests/ru/Boat.qm", - "model": "gemini-2.5-flash", - "temperature": 0.4, - "template": "stub.jinja", - "agent_id": "llm_gemini-2.5-flash", - "attempt": 2, - "outcome": "FAILURE", - "reward": 0.0, - "error": None, - }, - { - "quest": "quests/Scout.qm", - "model": "gpt-5-mini", - "temperature": 0.4, - "template": "planner.jinja", - "agent_id": "planner_gpt-5-mini", - "attempt": 1, - "outcome": "SUCCESS", - "reward": 1.0, - "error": None, - }, + _result_row("quests/ru/Boat.qm", "gemini-2.5-flash", "minimal", "llm_gemini-2.5-flash", "SUCCESS"), + _result_row("quests/ru/Boat.qm", "gemini-2.5-flash", "minimal", "llm_gemini-2.5-flash", "FAILURE"), + _result_row("quests/Scout.qm", "gpt-5-mini", "planner", "planner_gpt-5-mini", "SUCCESS"), ] db_runs = [ - { - "id": 1, - "quest_file": "quests/ru/Boat.qm", - "quest_name": "Boat", - "agent_id": "llm_gemini-2.5-flash", - "agent_config": json.dumps({"model": "gemini-2.5-flash", "action_template": "stub.jinja"}), - "outcome": "SUCCESS", - }, - { - "id": 2, - "quest_file": "quests/ru/Boat.qm", - "quest_name": "Boat", - "agent_id": "llm_gemini-2.5-flash", - "agent_config": None, - "outcome": "FAILURE", - }, - { - "id": 3, - "quest_file": "quests/Scout.qm", - "quest_name": "Scout", - "agent_id": "planner_gpt-5-mini", - "agent_config": None, - "outcome": "SUCCESS", - }, + _db_run( + 1, + "quests/ru/Boat.qm", + "Boat", + "llm_gemini-2.5-flash", + "minimal", + "gemini-2.5-flash", + "SUCCESS", + usage={"total_tokens": 900, "estimated_cost_usd": 0.003}, + diagnostics={"total_steps": 12, "repetition_rate": 0.10}, + progress={"current": 100.0}, + ), + _db_run( + 2, + "quests/ru/Boat.qm", + "Boat", + "llm_gemini-2.5-flash", + "minimal", + "gemini-2.5-flash", + "FAILURE", + usage={"total_tokens": 600, "estimated_cost_usd": None}, + diagnostics={"total_steps": 18, "repetition_rate": 0.30}, + progress={"current": 40.0}, + ), + _db_run( + 3, + "quests/Scout.qm", + "Scout", + "planner_gpt-5-mini", + "planner", + "gpt-5-mini", + "SUCCESS", + usage={"total_tokens": 1200, "estimated_cost_usd": 0.006}, + diagnostics={"total_steps": 9, "repetition_rate": 0.0}, + progress={"current": 100.0}, + ), ] (benchmark_dir / "benchmark_summary.json").write_text( @@ -88,24 +127,6 @@ def test_generate_leaderboard_aggregates_runs(tmp_path, monkeypatch): encoding="utf-8", ) - run_summaries = { - Path("results/llm_gemini-2.5-flash/Boat/run_1/run_summary.json"): { - "usage": {"total_tokens": 900, "estimated_cost_usd": 0.003}, - "metrics": {"total_steps": 12, "repetition_rate": 0.10}, - }, - Path("results/llm_gemini-2.5-flash/Boat/run_2/run_summary.json"): { - "usage": {"total_tokens": 600, "estimated_cost_usd": None}, - "metrics": {"total_steps": 18, "repetition_rate": 0.30}, - }, - Path("results/planner_gpt-5-mini/Scout/run_3/run_summary.json"): { - "usage": {"total_tokens": 1200, "estimated_cost_usd": 0.006}, - "metrics": {"total_steps": 9, "repetition_rate": 0.0}, - }, - } - for path, payload in run_summaries.items(): - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps(payload, ensure_ascii=False), encoding="utf-8") - output_path = Path("site/leaderboard.json") leaderboard = generate_leaderboard( [str(benchmark_dir)], @@ -140,6 +161,7 @@ def test_generate_leaderboard_aggregates_runs(tmp_path, monkeypatch): assert boat_row["avg_tokens"] == pytest.approx(750.0) assert boat_row["avg_cost_usd"] == pytest.approx(0.0015) assert boat_row["repetition_rate"] == pytest.approx(0.2) + assert boat_row["avg_progress"] == pytest.approx(70.0) scout_row = next(row for row in leaderboard["results"] if row["model"] == "gpt-5-mini") assert scout_row == { @@ -152,9 +174,76 @@ def test_generate_leaderboard_aggregates_runs(tmp_path, monkeypatch): "avg_tokens": 1200.0, "avg_cost_usd": 0.006, "repetition_rate": 0.0, + "avg_progress": 100.0, } +def test_leaderboard_modes_come_from_treatment_components(tmp_path, monkeypatch): + """Mode grouping reads declared components, never the harness name.""" + monkeypatch.chdir(tmp_path) + benchmark_dir = Path("results/benchmarks/bench_modes") + benchmark_dir.mkdir(parents=True, exist_ok=True) + + cases = { + "minimal": "minimal_prompt", + "reasoning_recent": "short_context_reasoning", + "reasoning_full": "full_history_reasoning", + "memo_compact": "compact_memory_memo", + "hinted_compact": "prompt_hints", + "tool_compact": "tools_compact_memory", + "tool_hinted": "tools_hints_compact_memory", + "programmatic_memory": "tools_programmatic_memory", + "planner": "planner_loop", + "backtracking": "backtracking_loop", + "adaptive_reasoning": "adaptive_reasoning", + } + results = [_result_row("quests/Core.qm", "gpt-5-mini", harness, f"agent_{harness}") for harness in cases] + (benchmark_dir / "benchmark_summary.json").write_text( + json.dumps({"benchmark_id": "bench_modes", "agents": [], "results": results, "db_runs": []}), + encoding="utf-8", + ) + + leaderboard = generate_leaderboard( + [str(benchmark_dir)], + "site/leaderboard.json", + min_runs=0, + public_model_ids=None, + ) + + assert {row["mode"] for row in leaderboard["results"]} == set(cases.values()) + + +def test_leaderboard_marks_migrated_unknown_treatments(tmp_path, monkeypatch): + """A migrated legacy record groups as unknown, not as a guessed mode.""" + monkeypatch.chdir(tmp_path) + benchmark_dir = Path("results/benchmarks/bench_unknown") + benchmark_dir.mkdir(parents=True, exist_ok=True) + + row = _result_row("quests/Core.qm", "gpt-5-mini", "minimal", "agent") + row["treatment"] = { + "harness": "unknown", + "prompt": "unknown", + "memory": "unknown", + "tools": [], + "loop": "unknown", + "reasoning": "unknown", + "signature": "t2_unknown", + } + (benchmark_dir / "benchmark_summary.json").write_text( + json.dumps({"benchmark_id": "bench_unknown", "agents": [], "results": [row], "db_runs": []}), + encoding="utf-8", + ) + + leaderboard = generate_leaderboard( + [str(benchmark_dir)], + "site/leaderboard.json", + min_runs=0, + public_model_ids=None, + ) + + assert leaderboard["results"][0]["mode"] == "unknown" + + def test_public_leaderboard_taxonomy_has_no_legacy_labels(): repo_root = Path(__file__).resolve().parents[2] old_labels = [ @@ -172,6 +261,8 @@ def test_public_leaderboard_taxonomy_has_no_legacy_labels(): for label in old_labels: assert label not in index_html assert "leaderboard.json?v=" in index_html + # The site no longer carries a legacy mode-id alias map. + assert "MODE_ALIASES" not in index_html leaderboard = json.loads((repo_root / "site/leaderboard.json").read_text(encoding="utf-8")) mode_labels = {mode["label"] for mode in leaderboard["modes"]} @@ -197,30 +288,8 @@ def test_generate_leaderboard_filters_public_slice(tmp_path, monkeypatch): rows = [] for model in ["model-a", "model-b", "model-c"]: for quest in ["Core", "Solo"]: - rows.append( - { - "quest": f"quests/{quest}.qm", - "model": model, - "template": "stub.jinja", - "agent_id": model, - "attempt": 1, - "outcome": "SUCCESS", - "reward": 1.0, - "error": None, - } - ) - rows.append( - { - "quest": "quests/Core.qm", - "model": "low-coverage", - "template": "stub.jinja", - "agent_id": "low-coverage", - "attempt": 1, - "outcome": "SUCCESS", - "reward": 1.0, - "error": None, - } - ) + rows.append(_result_row(f"quests/{quest}.qm", model, "minimal", model)) + rows.append(_result_row("quests/Core.qm", "low-coverage", "minimal", "low-coverage")) rows = [row for row in rows if not (row["quest"] == "quests/Solo.qm" and row["model"] != "model-a")] (benchmark_dir / "benchmark_summary.json").write_text( @@ -248,11 +317,13 @@ def test_generate_leaderboard_excludes_legacy_claude_cli_runs_from_public_slice( benchmark_dir = Path("results/benchmarks/bench_claude_cli") benchmark_dir.mkdir(parents=True, exist_ok=True) + treatment = _treatment("memo_compact", "gpt-5-mini") rows = [ { "quest": "quests/Core.qm", "model": "claude:claude-haiku-4-5-20251001", - "template": "stateful_compact.jinja", + "harness": "memo_compact", + "treatment": treatment, "agent_id": "legacy-claude-cli", "attempt": attempt, "outcome": "SUCCESS", @@ -263,7 +334,8 @@ def test_generate_leaderboard_excludes_legacy_claude_cli_runs_from_public_slice( { "quest": "quests/Core.qm", "model": "anthropic:claude-haiku-4-5-20251001", - "template": "stateful_compact.jinja", + "harness": "memo_compact", + "treatment": treatment, "agent_id": "anthropic-api", "attempt": 1, "outcome": "SUCCESS", @@ -296,34 +368,10 @@ def test_generate_leaderboard_excludes_retired_exp4_variants(tmp_path, monkeypat retired_dir = Path("results/benchmarks/retired") retired_dir.mkdir(parents=True, exist_ok=True) - active_row = { - "quest": "quests/Core.qm", - "model": "gpt-5-mini", - "template": "stateful_compact.jinja", - "harness": "memo_compact", - "agent_id": "active", - "attempt": 1, - "outcome": "SUCCESS", - } + active_row = _result_row("quests/Core.qm", "gpt-5-mini", "memo_compact", "active") retired_rows = [ - { - "quest": "quests/Core.qm", - "model": "gpt-5-mini", - "template": "reasoning.jinja", - "harness": "compaction_no_memo", - "agent_id": "retired-no-memo", - "attempt": 1, - "outcome": "FAILURE", - }, - { - "quest": "quests/Core.qm", - "model": "gpt-5-mini", - "template": "memo_extended.jinja", - "harness": "memo_extended", - "agent_id": "retired-extended", - "attempt": 1, - "outcome": "FAILURE", - }, + _result_row("quests/Core.qm", "gpt-5-mini", "compaction_no_memo", "retired-no-memo", "FAILURE"), + _result_row("quests/Core.qm", "gpt-5-mini", "memo_extended", "retired-extended", "FAILURE"), ] (active_dir / "benchmark_summary.json").write_text( @@ -362,53 +410,38 @@ def test_generate_leaderboard_matches_db_runs_by_identifiers(tmp_path, monkeypat benchmark_dir.mkdir(parents=True, exist_ok=True) results = [ - { - "quest": "quests/Alpha.qm", - "model": "gpt-5-mini", - "template": "stub.jinja", - "agent_id": "llm_gpt-5-mini", - "outcome": "SUCCESS", - }, - { - "quest": "quests/Beta.qm", - "model": "gpt-5-mini", - "template": "stub.jinja", - "agent_id": "llm_gpt-5-mini", - "outcome": "SUCCESS", - }, + _result_row("quests/Alpha.qm", "gpt-5-mini", "minimal", "llm_gpt-5-mini"), + _result_row("quests/Beta.qm", "gpt-5-mini", "minimal", "llm_gpt-5-mini"), ] db_runs = [ - { - "id": 20, - "quest_file": "quests/Beta.qm", - "quest_name": "Beta", - "agent_id": "llm_gpt-5-mini", - "agent_config": json.dumps( - {"model": "gpt-5-mini", "action_template": "reasoning.jinja", "memory_mode": "full_transcript"} - ), - "outcome": "SUCCESS", - }, - { - "id": 10, - "quest_file": "quests/Alpha.qm", - "quest_name": "Alpha", - "agent_id": "llm_gpt-5-mini", - "agent_config": json.dumps( - {"model": "gpt-5-mini", "action_template": "loop_aware_reasoning.jinja", "memory_mode": "compaction"} - ), - "outcome": "SUCCESS", - }, + _db_run(20, "quests/Beta.qm", "Beta", "llm_gpt-5-mini", "reasoning_full", "gpt-5-mini", "SUCCESS"), + _db_run(10, "quests/Alpha.qm", "Alpha", "llm_gpt-5-mini", "memo_compact", "gpt-5-mini", "SUCCESS"), ] (benchmark_dir / "benchmark_summary.json").write_text( json.dumps({"benchmark_id": "bench_match", "agents": [], "results": results, "db_runs": db_runs}), encoding="utf-8", ) + # Usage/diagnostics fall back to a strict canonical run record when the DB row omits them. for run_id, quest_name, total_steps in [(10, "Alpha", 10), (20, "Beta", 20)]: path = Path("results/llm_gpt-5-mini") / quest_name / f"run_{run_id}" / "run_summary.json" path.parent.mkdir(parents=True, exist_ok=True) + db_run = next(row for row in db_runs if row["id"] == run_id) path.write_text( - json.dumps({"usage": {"total_tokens": total_steps}, "metrics": {"total_steps": total_steps}}), + json.dumps( + { + "schema_version": 2, + "run": {"id": run_id, "agent_id": "llm_gpt-5-mini"}, + "quest": {"file": f"quests/{quest_name}.qm", "name": quest_name}, + "treatment": db_run["treatment"], + "lineage": None, + "terminal": {"outcome": "SUCCESS", "reward": 1.0, "snapshot": None}, + "usage": {"total_tokens": total_steps}, + "progress": {"current": 0.0, "maximum": 100.0, "scored": False}, + "transcript_diagnostics": {"total_steps": total_steps}, + "transitions": [], + } + ), encoding="utf-8", ) @@ -422,44 +455,3 @@ def test_generate_leaderboard_matches_db_runs_by_identifiers(tmp_path, monkeypat rows = {(row["quest"], row["mode"]): row for row in leaderboard["results"]} assert rows[("Alpha", "compact_memory_memo")]["avg_steps"] == 10.0 assert rows[("Beta", "full_history_reasoning")]["avg_steps"] == 20.0 - - -def test_generate_leaderboard_uses_result_row_memory_mode_without_db_config(tmp_path, monkeypatch): - monkeypatch.chdir(tmp_path) - benchmark_dir = Path("results/benchmarks/bench_result_memory_mode") - benchmark_dir.mkdir(parents=True, exist_ok=True) - results = [ - { - "quest": "quests/Beta.qm", - "model": "gpt-5-mini", - "template": "reasoning.jinja", - "memory_mode": "full_transcript", - "agent_id": "harness_gpt-5-mini", - "outcome": "SUCCESS", - } - ] - db_runs = [ - { - "id": 20, - "quest_file": "quests/Beta.qm", - "quest_name": "Beta", - "agent_id": "harness_gpt-5-mini", - "agent_config": json.dumps({"model": "gpt-5-mini", "harness": "reasoning_full"}), - "outcome": "SUCCESS", - } - ] - (benchmark_dir / "benchmark_summary.json").write_text( - json.dumps( - {"benchmark_id": "bench_result_memory_mode", "harnesses": [], "results": results, "db_runs": db_runs} - ), - encoding="utf-8", - ) - - leaderboard = generate_leaderboard( - [str(benchmark_dir)], - "site/leaderboard.json", - min_runs=0, - public_model_ids=None, - ) - - assert leaderboard["results"][0]["mode"] == "full_history_reasoning" diff --git a/llm_quest_benchmark/tests/test_logging.py b/llm_quest_benchmark/tests/test_logging.py index 6f3f1db..8bc7370 100644 --- a/llm_quest_benchmark/tests/test_logging.py +++ b/llm_quest_benchmark/tests/test_logging.py @@ -1,51 +1,147 @@ -"""Tests for logging dataclasses""" +"""Tests for canonical schema-v2 record types""" +import json + +import pytest + +from llm_quest_benchmark.schemas.records import ( + SCHEMA_VERSION, + ProgressState, + QuestAction, + QuestSnapshot, + QuestTransition, + RunRecord, +) from llm_quest_benchmark.schemas.response import LLMResponse -from llm_quest_benchmark.schemas.state import AgentState - - -def test_agent_state_basic(): - """Test basic AgentState functionality""" - state = AgentState( - step=1, - location_id="room1", - observation="You are in a room", - choices=[{"id": "1", "text": "Go north"}, {"id": "2", "text": "Go south"}], - action="1", - llm_response=LLMResponse(action=1), + + +def _snapshot(**overrides) -> QuestSnapshot: + payload = { + "location_id": "room1", + "observation": "You are in a room", + "choices": [{"id": "11", "text": "Go north"}, {"id": "12", "text": "Go south"}], + "params_state": ["HP: 10"], + "saving": {"locationId": 1, "aleaState": [0.5, 0.25, 0.125, 7]}, + } + payload.update(overrides) + return QuestSnapshot(**payload) + + +def test_snapshot_digest_is_deterministic_and_state_sensitive(): + first = _snapshot() + second = _snapshot() + + assert first.digest == second.digest + assert first.digest != _snapshot(location_id="room2").digest + assert first.digest != _snapshot(params_state=["HP: 9"]).digest + assert first.digest != _snapshot(saving={"locationId": 1, "aleaState": [0.5, 0.25, 0.125, 8]}).digest + + +def test_snapshot_agent_observation_appends_params(): + snapshot = _snapshot() + + assert snapshot.agent_observation() == "You are in a room\n\nStatus:\nHP: 10" + + +def test_choose_transition_round_trips_through_json(): + transition = QuestTransition( + index=1, + before=_snapshot(), + action=QuestAction.choose(choice_index=2, choice_id="12", performed_at_ms=1735689600123), + after=_snapshot(location_id="room2", observation="A corridor"), + response=LLMResponse(action=2, reasoning="south is safer", total_tokens=17), + usage={"prompt_tokens": 12, "completion_tokens": 5, "total_tokens": 17, "estimated_cost_usd": None}, + progress=ProgressState(current=25.0, reached=["mission_accepted"]), + reasoning_mode="concise", + ) + + restored = QuestTransition.from_dict(json.loads(json.dumps(transition.to_dict()))) + + assert restored.action.choice_index == 2 + assert restored.action.choice_id == "12" + assert restored.action.performed_at_ms == 1735689600123 + assert restored.before.choices == transition.before.choices + assert restored.before.saving == transition.before.saving + assert restored.before.digest == transition.before.digest + assert restored.after.digest == transition.after.digest + assert restored.response.reasoning == "south is safer" + assert restored.progress.current == 25.0 + assert restored.reasoning_mode == "concise" + assert restored.is_replayable + + +def test_restore_transition_round_trips_and_is_replayable(): + transition = QuestTransition( + index=4, + before=_snapshot(location_id="dead_end"), + action=QuestAction.restore(checkpoint_index=2), + after=_snapshot(), ) - # Test basic attributes - assert state.step == 1 - assert state.location_id == "room1" - assert state.observation == "You are in a room" - assert len(state.choices) == 2 - assert state.choices[0]["text"] == "Go north" - assert state.action == "1" - assert state.llm_response.action == 1 - - -def test_agent_state_with_llm_response(): - """Test AgentState with detailed LLM response""" - state = AgentState( - step=1, - location_id="room1", - observation="You are in a room", - choices=[{"id": "1", "text": "Go north"}, {"id": "2", "text": "Go south"}], - action="1", - llm_response=LLMResponse( - action=1, analysis="The north path looks safer", reasoning="I can see better lighting in that direction" - ), + restored = QuestTransition.from_dict(json.loads(json.dumps(transition.to_dict()))) + + assert restored.action.is_restore + assert restored.action.checkpoint_index == 2 + assert restored.is_replayable + + +def test_transition_without_timestamp_is_not_replayable(): + transition = QuestTransition( + index=1, + before=_snapshot(), + action=QuestAction(kind="choose", choice_index=1, choice_id="11", performed_at_ms=None), + after=_snapshot(), ) - # Test LLM response fields - assert state.llm_response.action == 1 - assert state.llm_response.analysis == "The north path looks safer" - assert state.llm_response.reasoning == "I can see better lighting in that direction" + assert not transition.is_replayable + + +def test_snapshot_without_saving_is_not_resumable(): + assert not _snapshot(saving=None).is_resumable + assert not QuestSnapshot.unavailable().is_resumable + assert QuestSnapshot.unavailable().is_unavailable + + +def test_run_record_rejects_non_v2_payload(): + with pytest.raises(ValueError, match="migrate_records"): + RunRecord.from_dict({"schema_version": 1, "steps": []}) + + +def test_run_record_round_trips_and_reports_resumability(): + record = RunRecord( + run_id=7, + quest_file="quests/Boat.qm", + quest_name="Boat", + quest_checksum="sha256:abc", + quest_language="rus", + engine_revision="git:deadbeef", + agent_id="gpt-5-mini_t0.4_reasoning_recent_12345678", + treatment={"harness": "reasoning_recent", "signature": "t2_abc"}, + outcome="TRUNCATED", + transitions=[ + QuestTransition( + index=1, + before=_snapshot(), + action=QuestAction.choose(1, "11", 1735689600000), + after=_snapshot(location_id="room2"), + ) + ], + ) + + payload = json.loads(json.dumps(record.to_dict())) + restored = RunRecord.from_dict(payload) + + assert payload["schema_version"] == SCHEMA_VERSION + assert restored.quest_checksum == "sha256:abc" + assert restored.treatment_signature == "t2_abc" + assert restored.is_resumable + + restored.outcome = "FAILURE" + assert not restored.is_resumable def test_llm_response_to_dict(): - """Test LLMResponse dictionary conversion""" + """LLMResponse dictionary conversion drops unset optional fields""" response = LLMResponse(action=1, analysis="Let me think about this", reasoning="Based on the available information") data = response.to_dict() diff --git a/llm_quest_benchmark/tests/test_play_trace_export.py b/llm_quest_benchmark/tests/test_play_trace_export.py index 2640b8d..f080da1 100644 --- a/llm_quest_benchmark/tests/test_play_trace_export.py +++ b/llm_quest_benchmark/tests/test_play_trace_export.py @@ -11,18 +11,33 @@ def test_play_app_exports_research_trace_json(): assert "downloadJson" in source assert "Export Trace JSON" in source assert "human_trace_" in source - assert "schema_version: 'human_trace_v1'" in source + assert "schema_version: 'human_trace_v2'" in source assert "source: 'web_play'" in source -def test_trace_export_captures_observations_choices_and_terminal_state(): +def test_trace_export_captures_snapshots_choices_and_terminal_state(): source = APP_SOURCE.read_text(encoding="utf-8") - assert "observation: stripClr(gameState.text || '')" in source - assert "choices: choicesToTraceMap(choices)" in source - assert "location_id: locationId" in source - assert "params: paramsToTraceList(gameState.paramsState)" in source + assert "function traceSnapshot(" in source + assert "observation: stripClr(state.text || '')" in source + assert "choices: choicesToTraceMap(activeChoices)" in source + assert "params: paramsToTraceList(state.paramsState)" in source + assert "saving: player.getSaving()" in source assert "terminal: {" in source assert "game_state: outcome" in source - assert "traceSteps={traceSteps}" in source - assert "setTraceSteps(prev => prev.slice(0, -1))" in source + assert "traceTransitions={traceTransitions}" in source + + +def test_trace_export_retains_restore_events_instead_of_erasing_them(): + """Backtracking in the web player truncates only the active branch; the + exported trace keeps every executed transition, including the restore.""" + source = APP_SOURCE.read_text(encoding="utf-8") + + assert "kind: 'restore'" in source + assert "checkpoint_index: checkpointIndex" in source + # The old export truncated undone steps out of the trace. + assert "setTraceSteps(prev => prev.slice(0, -1))" not in source + assert "setTraceTransitions(prev => prev.slice(0, -1))" not in source + # Undo still truncates the active checkpoint branch and the UI decision path. + assert "setStepHistory(prev => prev.slice(0, -1))" in source + assert "setPath(prev => prev.slice(0, -1))" in source diff --git a/pyproject.toml b/pyproject.toml index 5d24bf6..301be84 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -30,6 +30,7 @@ dependencies = [ "openai>=1.12.0", "openrouter>=0.5.0", "typer>=0.9.0", + "tqdm>=4.66.0", "langfuse>=3.0.0", ] diff --git a/scripts/backfill_costs.py b/scripts/backfill_costs.py index 29f903f..40ed9d9 100644 --- a/scripts/backfill_costs.py +++ b/scripts/backfill_costs.py @@ -21,16 +21,7 @@ from llm_quest_benchmark.llm.client import parse_model_name # noqa: E402, I001 from llm_quest_benchmark.llm.cost import estimate_cost_usd # noqa: E402 - - -_AGENT_ID_PREFIXES = ("llm_", "planner_", "tool_") - - -def _model_name_from_agent_id(agent_id: str) -> str | None: - for prefix in _AGENT_ID_PREFIXES: - if agent_id.startswith(prefix): - return agent_id[len(prefix) :] - return None +from llm_quest_benchmark.schemas.records import SCHEMA_VERSION # noqa: E402 def backfill(results_dir: Path, dry_run: bool) -> None: @@ -45,6 +36,11 @@ def backfill(results_dir: Path, dry_run: bool) -> None: except (OSError, json.JSONDecodeError) as exc: print(f" SKIP (unreadable: {exc}): {path}") continue + + if data.get("schema_version") != SCHEMA_VERSION: + print(f" SKIP (not schema v{SCHEMA_VERSION}; run migrate-records): {path.relative_to(results_dir)}") + continue + usage = data.get("usage") or {} if usage.get("estimated_cost_usd") is not None: @@ -57,9 +53,9 @@ def backfill(results_dir: Path, dry_run: bool) -> None: skipped_no_tokens += 1 continue - agent_id = data.get("agent_id") or "" - model_name = _model_name_from_agent_id(agent_id) - if not model_name: + # The canonical treatment records the exact model that produced the run. + model_name = (data.get("treatment") or {}).get("model") + if not model_name or model_name == "unavailable": skipped_no_model += 1 print(f" SKIP (no model): {path.relative_to(results_dir)}") continue @@ -77,7 +73,7 @@ def backfill(results_dir: Path, dry_run: bool) -> None: print(f" SKIP (no price for {spec.provider}:{spec.model_id}): {path.relative_to(results_dir)}") continue - total_steps = (data.get("metrics") or {}).get("total_steps") or 1 + total_steps = (data.get("transcript_diagnostics") or {}).get("total_steps") or 1 print( f" {'[DRY]' if dry_run else 'UPDATE'} {path.relative_to(results_dir)}" f" {spec.provider}:{spec.model_id}" diff --git a/scripts/build_cohort_data.py b/scripts/build_cohort_data.py index 46bba95..be3175e 100644 --- a/scripts/build_cohort_data.py +++ b/scripts/build_cohort_data.py @@ -161,13 +161,15 @@ }, } -EXCLUDE_PATTERNS = ["random", "test", "planner", "tool"] +# Loops whose runs are not comparable with the public prompt/memory cohort. +EXCLUDED_LOOPS = {"random", "tool_select_act", "plan_act", "choose_or_restore", "interactive"} +EXCLUDE_MODEL_PATTERNS = ["test", "unavailable", "unknown"] MIN_FAMILY_STEPS = 5 def classify_agent(agent_id: str) -> str: - """Return model family string for a given agent_id.""" + """Return model family string for a recorded model id.""" a = agent_id.lower() # Claude (various prefix forms) if ( @@ -204,9 +206,12 @@ def classify_agent(agent_id: str) -> str: return "other" -def is_excluded(agent_id: str) -> bool: - a = agent_id.lower() - return any(pat in a for pat in EXCLUDE_PATTERNS) +def is_excluded(treatment: dict) -> bool: + """Exclude non-comparable treatments using declared components, not names.""" + if str(treatment.get("loop") or "") in EXCLUDED_LOOPS: + return True + model = str(treatment.get("model") or "").lower() + return any(pat in model for pat in EXCLUDE_MODEL_PATTERNS) _CYRILLIC_RE = re.compile(r"[Ѐ-ӿ]") @@ -219,10 +224,10 @@ def has_cyrillic(text: str) -> bool: def build_quest_data(conn: sqlite3.Connection, quest_name: str) -> dict: cursor = conn.cursor() - # Fetch all runs for this quest (excluding filtered agent types, all outcomes) + # Fetch all runs for this quest (excluding filtered treatments, all outcomes) cursor.execute( """ - SELECT r.id, r.agent_id, r.outcome + SELECT r.id, r.agent_id, r.outcome, r.treatment FROM runs r WHERE r.quest_name = ? """, @@ -231,19 +236,24 @@ def build_quest_data(conn: sqlite3.Connection, quest_name: str) -> dict: all_runs = cursor.fetchall() # Filter runs - runs = {} # run_id -> (agent_id, outcome) + runs = {} # run_id -> (agent_id, outcome, family) unrecognized = set() - for run_id, agent_id, outcome in all_runs: - if is_excluded(agent_id): + for run_id, agent_id, outcome, treatment_json in all_runs: + try: + treatment = json.loads(treatment_json) if treatment_json else {} + except json.JSONDecodeError: + treatment = {} + if is_excluded(treatment): continue - family = classify_agent(agent_id) + model = str(treatment.get("model") or agent_id) + family = classify_agent(model) if family == "other": - unrecognized.add(agent_id) + unrecognized.add(model) runs[run_id] = (agent_id, outcome, family) if unrecognized: - for aid in sorted(unrecognized): - print(f" [WARN] unrecognized agent_id: {aid}", file=sys.stderr) + for model in sorted(unrecognized): + print(f" [WARN] unrecognized model: {model}", file=sys.stderr) total_runs = len(runs) success_runs = sum(1 for _, (_, outcome, _) in runs.items() if outcome == "SUCCESS") @@ -260,14 +270,14 @@ def build_quest_data(conn: sqlite3.Connection, quest_name: str) -> dict: } run_ids = list(runs.keys()) - # Fetch steps for all relevant runs + # Fetch executed transitions for all relevant runs placeholders = ",".join("?" * len(run_ids)) cursor.execute( f""" - SELECT s.run_id, s.location_id, s.observation, s.choices, s.action - FROM steps s - WHERE s.run_id IN ({placeholders}) - ORDER BY s.run_id, s.step + SELECT t.run_id, t.before_state, t.action + FROM transitions t + WHERE t.run_id IN ({placeholders}) + ORDER BY t.run_id, t.transition_index """, run_ids, ) @@ -282,18 +292,22 @@ def build_quest_data(conn: sqlite3.Connection, quest_name: str) -> dict: # Track total steps per family (across all locations) for filtering family_total_steps: dict[str, int] = defaultdict(int) - for run_id, location_id, observation, choices_json, action in rows: - # Skip terminal/non-numeric actions - if not str(action).strip().isdigit(): - continue - - action_idx = int(action) - 1 # convert 1-based to 0-based - + for run_id, before_json, action_json in rows: try: - choices = json.loads(choices_json) + before = json.loads(before_json) + action = json.loads(action_json) except (json.JSONDecodeError, TypeError): continue + # Restores are backtracking-harness only and never enter cohort choices. + if action.get("kind") != "choose" or action.get("choice_index") is None: + continue + + action_idx = int(action["choice_index"]) - 1 # convert 1-based to 0-based + location_id = before.get("location_id") + observation = before.get("observation") or "" + choices = before.get("choices") or [] + if action_idx < 0 or action_idx >= len(choices): continue diff --git a/scripts/classify_failures.py b/scripts/classify_failures.py index 20b5049..104e565 100644 --- a/scripts/classify_failures.py +++ b/scripts/classify_failures.py @@ -11,9 +11,15 @@ import argparse import json import subprocess +import sys from concurrent.futures import ThreadPoolExecutor, as_completed from pathlib import Path +repo_root = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(repo_root)) + +from llm_quest_benchmark.schemas.records import RunRecord, load_run_record # noqa: E402 + CLASSIFICATION_PROMPT = """You are an expert at analyzing LLM agent failures in interactive fiction quests. Below is a condensed trace of an LLM agent playing a text-based quest. The agent failed. @@ -43,63 +49,70 @@ """ -def condense_trace(run: dict) -> str: - """Extract key info from a run, keeping it under ~3000 tokens.""" +def _action_label(transition) -> str: + if transition.action.is_restore: + return f"restore:{transition.action.checkpoint_index}" + index = transition.action.choice_index + choices = transition.before.choices + text = choices[index - 1]["text"] if index and 1 <= index <= len(choices) else "" + return f"{index}: {text[:60]}" + + +def condense_trace(record: RunRecord) -> str: + """Extract key info from a schema-v2 record, keeping it under ~3000 tokens.""" + transitions = record.transitions lines = [] - lines.append(f"Quest: {run.get('quest_name', 'unknown')}") - lines.append(f"Agent: {run.get('agent_id', 'unknown')}") - lines.append(f"Outcome: {run.get('outcome', 'unknown')}") - lines.append(f"Total steps: {len(run.get('steps', []))}") - lines.append(f"Reward: {run.get('reward', 0)}") + lines.append(f"Quest: {record.quest_name}") + lines.append(f"Agent: {record.agent_id}") + lines.append(f"Treatment: {record.treatment_signature}") + lines.append(f"Outcome: {record.outcome}") + lines.append(f"Total transitions: {len(transitions)}") + lines.append(f"Reward: {record.reward}") + lines.append(f"Progress: {record.progress.current:.1f}%") lines.append("") - steps = run.get("steps", []) # For long traces, show first 5, last 5, and sample middle - if len(steps) <= 15: - show_steps = steps + if len(transitions) <= 15: + shown = [(t, None) for t in transitions] else: - middle_idx = len(steps) // 2 - show_steps = ( - steps[:5] - + [{"_marker": f"... ({len(steps) - 10} steps omitted) ..."}] - + steps[middle_idx - 1 : middle_idx + 2] - + [{"_marker": "..."}] - + steps[-5:] + middle_idx = len(transitions) // 2 + shown = ( + [(t, None) for t in transitions[:5]] + + [(None, f"... ({len(transitions) - 10} transitions omitted) ...")] + + [(t, None) for t in transitions[middle_idx - 1 : middle_idx + 2]] + + [(None, "...")] + + [(t, None) for t in transitions[-5:]] ) - for s in show_steps: - if "_marker" in s: - lines.append(s["_marker"]) + for transition, marker in shown: + if marker is not None: + lines.append(marker) continue - step_num = s.get("step", "?") - obs = (s.get("observation") or "")[:200] - choices = s.get("choices", {}) - decision = s.get("llm_decision", {}) - chosen = decision.get("choice", {}) - reasoning = (decision.get("reasoning") or decision.get("analysis") or "")[:150] - parse_mode = decision.get("parse_mode", "") - - lines.append(f"Step {step_num}:") - lines.append(f" Observation: {obs}") - if choices: - choice_str = " | ".join(f"{k}: {v[:60]}" for k, v in list(choices.items())[:6]) + response = transition.response + reasoning = ((response.reasoning if response else None) or (response.analysis if response else None) or "")[ + :150 + ] + parse_mode = (response.parse_mode if response else "") or "" + + lines.append(f"Step {transition.index}:") + lines.append(f" Observation: {transition.before.observation[:200]}") + if transition.before.choices: + choice_str = " | ".join( + f"{i}: {c['text'][:60]}" for i, c in enumerate(transition.before.choices[:6], start=1) + ) lines.append(f" Choices: {choice_str}") - if chosen: - chosen_str = " | ".join(f"{k}: {v[:60]}" for k, v in chosen.items()) - lines.append(f" Chose: {chosen_str}") + lines.append(f" Chose: {_action_label(transition)}") if reasoning: lines.append(f" Reasoning: {reasoning}") if parse_mode and parse_mode != "json_parsed": lines.append(f" Parse mode: {parse_mode}") + if transition.reasoning_mode: + lines.append(f" Reasoning mode: {transition.reasoning_mode}") lines.append("") # Track action repetitions - actions = [] - for s in steps: - choice = s.get("llm_decision", {}).get("choice", {}) - action_key = str(sorted(choice.items())) if choice else "?" - actions.append(action_key) + actions = [_action_label(t) for t in transitions] if len(actions) > 5: from collections import Counter @@ -117,8 +130,8 @@ def condense_trace(run: dict) -> str: def classify_run(run_path: str, model: str) -> dict: """Classify a single run using claude -p.""" try: - run = json.loads(Path(run_path).read_text(encoding="utf-8")) - except (json.JSONDecodeError, OSError) as e: + run = load_run_record(run_path) + except (ValueError, OSError) as e: return {"path": run_path, "error": str(e)} trace = condense_trace(run) @@ -162,10 +175,11 @@ def classify_run(run_path: str, model: str) -> dict: return { "path": run_path, - "quest": run.get("quest_name"), - "agent": run.get("agent_id"), - "outcome": run.get("outcome"), - "steps": len(run.get("steps", [])), + "quest": run.quest_name, + "agent": run.agent_id, + "treatment_signature": run.treatment_signature, + "outcome": run.outcome, + "transitions": len(run.transitions), **classification, } except subprocess.TimeoutExpired: diff --git a/scripts/import_human_trace.py b/scripts/import_human_trace.py index 46c6e09..b3cd850 100644 --- a/scripts/import_human_trace.py +++ b/scripts/import_human_trace.py @@ -1,14 +1,39 @@ #!/usr/bin/env python3 -"""Convert a browser-exported human trace into run_summary.json format.""" +"""Convert a browser-exported human trace into a schema-v2 run_summary.json. + +Restore events exported by the web player are preserved as restore transitions; +undone steps are never erased from the record. +""" from __future__ import annotations import argparse import json +import sys from datetime import UTC, datetime from pathlib import Path from typing import Any +repo_root = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(repo_root)) + +from llm_quest_benchmark.core.logging import QuestLogger # noqa: E402 +from llm_quest_benchmark.core.provenance import quest_checksum # noqa: E402 +from llm_quest_benchmark.harnesses.specs import build_treatment # noqa: E402 +from llm_quest_benchmark.schemas.records import ( # noqa: E402 + PROVENANCE_RUNTIME, + REPLAY_UNAVAILABLE, + ProgressState, + QuestAction, + QuestSnapshot, + QuestTransition, + RunRecord, +) +from llm_quest_benchmark.schemas.response import LLMResponse # noqa: E402 + +TRACE_SCHEMA = "human_trace_v2" +UNAVAILABLE = "unavailable" + def normalize_outcome(outcome: str | None) -> str: value = (outcome or "INCOMPLETE").upper() @@ -47,68 +72,123 @@ def make_run_id(trace: dict[str, Any]) -> str: return f"human_web_{safe_quest}_{safe_time}" -def convert_step(step: dict[str, Any]) -> dict[str, Any]: - decision = step.get("human_decision") or {} - choice_index = str(decision.get("choice_index") or "") - choice_text = str(decision.get("choice_text") or "") - choices = {str(k): str(v) for k, v in (step.get("choices") or {}).items()} - if not choice_text and choice_index in choices: - choice_text = choices[choice_index] - - return { - "step": step.get("step"), - "location_id": step.get("location_id"), - "observation": step.get("observation") or "", - "params": step.get("params") or [], - "choices": choices, - "llm_decision": { - "analysis": "", - "reasoning": "human selected in web UI", - "is_default": False, - "choice": {choice_index: choice_text} if choice_index else {}, - }, - "human_decision": decision, - } - - -def convert_trace(trace: dict[str, Any], source_trace: str | None = None) -> dict[str, Any]: - if trace.get("schema_version") != "human_trace_v1": - raise ValueError("expected schema_version human_trace_v1") +def convert_snapshot(state: dict[str, Any] | None) -> QuestSnapshot: + """Map an exported web state into a canonical snapshot.""" + if not isinstance(state, dict) or not state: + return QuestSnapshot.unavailable() + + choices_map = state.get("choices") or {} + + def _choice_sort_key(item: tuple[Any, Any]) -> tuple[int, str]: + try: + return (0, f"{int(item[0]):010d}") + except (TypeError, ValueError): + return (1, str(item[0])) + + choices = [{"id": "", "text": str(text)} for _, text in sorted(choices_map.items(), key=_choice_sort_key)] + game_state = str(state.get("game_state") or "running") + saving = state.get("saving") if isinstance(state.get("saving"), dict) else None + return QuestSnapshot( + location_id=str(state.get("location_id") or ""), + observation=str(state.get("observation") or ""), + choices=choices, + params_state=[str(p) for p in (state.get("params") or [])], + done=game_state not in ("running",), + game_state=game_state, + saving=saving, + # The web player generates its own engine timestamp per jump, so choice + # ids and transition timestamps are not observable in the browser. + unavailable_fields=["choice_ids", "performed_at_ms"], + ) + + +def convert_transition(index: int, entry: dict[str, Any]) -> QuestTransition: + action_payload = entry.get("action") or {} + kind = str(action_payload.get("kind") or entry.get("kind") or "choose") + + if kind == "restore": + action = QuestAction.restore(int(action_payload.get("checkpoint_index") or 1)) + response = None + else: + choice_index = action_payload.get("choice_index") + action = QuestAction( + kind="choose", + choice_index=int(choice_index) if choice_index is not None else None, + choice_id=None, + performed_at_ms=action_payload.get("performed_at_ms"), + ) + response = LLMResponse( + action=action.choice_index or 1, + reasoning="human selected in web UI", + is_default=False, + parse_mode="human_web", + ) + + return QuestTransition( + index=index, + before=convert_snapshot(entry.get("before")), + action=action, + after=convert_snapshot(entry.get("after")), + response=response, + usage={}, + progress=ProgressState(), + provenance=PROVENANCE_RUNTIME, + # Really executed, but the browser cannot supply the engine timestamp, + # so the transition is not deterministically replayable. + replay_status=REPLAY_UNAVAILABLE, + ) + + +def convert_trace(trace: dict[str, Any], source_trace: str | None = None) -> RunRecord: + if trace.get("schema_version") != TRACE_SCHEMA: + raise ValueError(f"expected schema_version {TRACE_SCHEMA}") if trace.get("source") != "web_play": raise ValueError("expected source web_play") - steps = [convert_step(step) for step in trace.get("steps") or []] - outcome = normalize_outcome(trace.get("outcome")) - quest_name = trace.get("quest_title") or trace.get("quest_id") or "unknown" - - summary: dict[str, Any] = { - "run_id": make_run_id(trace), - "quest_name": quest_name, - "quest_id": trace.get("quest_id"), - "agent_id": "human_web", - "agent_mode": "human", - "model": "human", - "outcome": outcome, - "run_duration": duration_seconds(trace.get("started_at"), trace.get("ended_at")), - "usage": { - "prompt_tokens": 0, - "completion_tokens": 0, - "total_tokens": 0, - "estimated_cost_usd": 0, - "priced_steps": 0, - }, - "steps": steps, - "terminal": trace.get("terminal") or {}, - "source": "web_play_human_trace", - } + transitions = [ + convert_transition(index, entry) + for index, entry in enumerate(trace.get("transitions") or [], start=1) + if isinstance(entry, dict) + ] + quest_id = str(trace.get("quest_id") or "unknown") + quest_file = f"quests/{quest_id}.qm" + checksum = quest_checksum(quest_file) if Path(quest_file).exists() else UNAVAILABLE + + treatment = build_treatment( + harness="human", + model="human", + temperature=0.0, + system_template="none", + ).to_dict() + + record = RunRecord( + run_id=make_run_id(trace), + quest_file=quest_file, + quest_name=str(trace.get("quest_title") or quest_id), + quest_checksum=checksum, + quest_language="rus" if trace.get("quest_lang") == "ru" else "eng", + engine_revision=UNAVAILABLE, + agent_id="human_web", + treatment=treatment, + started_at=trace.get("started_at"), + ended_at=trace.get("ended_at"), + run_duration=duration_seconds(trace.get("started_at"), trace.get("ended_at")), + outcome=normalize_outcome(trace.get("outcome")), + reward=0.0, + transitions=transitions, + terminal_snapshot=transitions[-1].after if transitions else None, + ) + record.usage = QuestLogger.aggregate_usage(transitions) + record.transcript_diagnostics = QuestLogger.calculate_metrics(transitions, record.outcome) + record.transcript_diagnostics["source"] = "web_play_human_trace" if source_trace: - summary["source_trace"] = source_trace - return summary + record.transcript_diagnostics["source_trace"] = source_trace + return record -def write_summary(summary: dict[str, Any], output_path: Path) -> None: +def write_summary(record: RunRecord, output_path: Path) -> None: output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + output_path.write_text(json.dumps(record.to_dict(), ensure_ascii=False, indent=2) + "\n", encoding="utf-8") def main() -> None: @@ -118,8 +198,8 @@ def main() -> None: args = parser.parse_args() trace = json.loads(args.input.read_text(encoding="utf-8")) - summary = convert_trace(trace, source_trace=str(args.input)) - write_summary(summary, args.output) + record = convert_trace(trace, source_trace=str(args.input)) + write_summary(record, args.output) print(f"Wrote {args.output}") diff --git a/scripts/migrate_records.py b/scripts/migrate_records.py new file mode 100755 index 0000000..189cea5 --- /dev/null +++ b/scripts/migrate_records.py @@ -0,0 +1,57 @@ +#!/usr/bin/env python3 +"""One-time migration of legacy pre-v2 records into schema v2. + +The only legacy reader in the project. Lives outside the core package: +migration is a one-shot data-archaeology tool, not a runtime concern, and +the CLI surface stays minimal. + +Usage: + uv run scripts/migrate_records.py --source results/ --output results_v2/ + uv run scripts/migrate_records.py --source metrics.db --output metrics_v2.db + +Accepts a legacy ``run_summary.json``, a tree of them, or a legacy SQLite +database (``runs`` + ``steps``). Migration never invents a missing action, +timestamp, parameter state, engine saving, or post-state; unknowable fields +are marked unavailable and block resume. +""" + +from __future__ import annotations + +import argparse +import logging +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +from llm_quest_benchmark.core.migration import migrate_records # noqa: E402 + +logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s") +log = logging.getLogger(__name__) + + +def main() -> int: + parser = argparse.ArgumentParser(description="Convert legacy records into schema v2.") + parser.add_argument("--source", required=True, help="Legacy run_summary.json, results tree, or SQLite database.") + parser.add_argument("--output", required=True, help="Destination for the schema-v2 records.") + args = parser.parse_args() + + try: + report = migrate_records(args.source, args.output) + except (FileNotFoundError, ValueError) as exc: + log.error(f"Migration failed: {exc}") + return 1 + except Exception: # pragma: no cover - defensive + log.exception("Error during migration") + return 2 + + print(f"Migrated {report.runs_migrated} runs ({report.kind}) to {report.output}") + print(f"Transitions migrated: {report.transitions_migrated}") + print(f"Resumable runs: {report.resumable_runs}") + for skipped in report.skipped: + print(f"Skipped {skipped}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/replay_runs.py b/scripts/replay_runs.py index f5667b0..7abb1e7 100644 --- a/scripts/replay_runs.py +++ b/scripts/replay_runs.py @@ -1,18 +1,16 @@ #!/usr/bin/env python3 -"""Replay existing runs to extract per-step location_ids. +"""Verify schema-v2 run records by replaying them against the real engine. -Reads each run_summary.json, replays the recorded choice sequence through -QMPlayerEnv, and writes a location_trace.json alongside each run_summary.json. +Each run_summary.json is replayed transition by transition: recorded choose +timestamps and restore actions are re-executed and every resulting snapshot +digest is compared with the record. A replay_report.json is written alongside +each run_summary.json. -The trace list is aligned with steps[]: trace[i] is the location_id when -step i was presented to the agent. +Legacy records are not accepted here. Convert them first with: + scripts/migrate_records.py --source --output Usage: - uv run scripts/replay_runs.py [--results-dir results/] [--limit N] - -Note: The TS engine re-seeds its PRNG on each bridge startup, so replay of quests -with random branches diverges at the first stochastic choice. The trace is correct -up to that point and padded with None after. Fully deterministic quests replay 100%. + uv run scripts/replay_runs.py [--results-dir results/] [--limit N] [--force] """ import argparse @@ -23,64 +21,53 @@ repo_root = Path(__file__).resolve().parents[1] sys.path.insert(0, str(repo_root)) +from llm_quest_benchmark.core.replay import ReplayError, replay_record, verify_environment # noqa: E402 from llm_quest_benchmark.environments.qm import QMPlayerEnv # noqa: E402 - - -def _extract_choice(step: dict) -> str | None: - """Extract 1-based choice key from llm_decision.choice dict.""" - decision = step.get("llm_decision") or {} - choice = decision.get("choice") - if not choice or not isinstance(choice, dict): - return None - return next(iter(choice)) +from llm_quest_benchmark.schemas.records import load_run_record # noqa: E402 def replay_run(summary_path: Path) -> dict: - """Replay a single run and return the location trace. - - Returns a dict with keys: - - run_id - - quest_file - - location_trace: list[str | None] aligned with steps[] - - error: str | None (set if replay failed partway through) - """ - data = json.loads(summary_path.read_text(encoding="utf-8")) - run_id = data.get("run_id") - quest_file = data.get("quest_file") - steps = data.get("steps") or [] - + """Replay one recorded run and return its verification report.""" result = { - "run_id": run_id, - "quest_file": quest_file, + "run_id": None, + "quest_file": None, + "verified_transitions": 0, + "total_transitions": 0, "location_trace": [], + "status": "failed", "error": None, } - if not quest_file or not steps: - result["error"] = "missing quest_file or steps" + try: + record = load_run_record(str(summary_path)) + except (ValueError, OSError) as exc: + result["error"] = str(exc) + return result + + result["run_id"] = record.run_id + result["quest_file"] = record.quest_file + result["total_transitions"] = len(record.transitions) + # The recorded trace needs no replay: each transition carries its own state. + result["location_trace"] = [t.before.location_id for t in record.transitions] + if record.transitions: + result["location_trace"].append(record.transitions[-1].after.location_id) + + if not record.transitions: + result["status"] = "skipped" + result["error"] = "record has no transitions" return result env = None try: - env = QMPlayerEnv(quest_file) - env.reset() - result["location_trace"].append(env.state["location_id"]) - - # Replay all but the last step (last step has no outgoing choice to take) - for step in steps[:-1]: - choice = _extract_choice(step) - if choice is None: - break - env.step(choice) - result["location_trace"].append(env.state["location_id"]) - - # Pad to full length with None if replay stopped early - while len(result["location_trace"]) < len(steps): - result["location_trace"].append(None) - except Exception as e: - result["error"] = str(e) - while len(result["location_trace"]) < len(steps): - result["location_trace"].append(None) + verify_environment(record, record.quest_file) + env = QMPlayerEnv(record.quest_file, language=record.quest_language) + replay = replay_record(env, record) + result["verified_transitions"] = replay.verified_transitions + result["status"] = "verified" + except ReplayError as exc: + result["error"] = str(exc) + except Exception as exc: # noqa: BLE001 - report engine/setup failures verbatim + result["error"] = str(exc) finally: if env is not None: env.close() @@ -92,6 +79,7 @@ def main() -> None: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--results-dir", default="results", help="Path to results directory") parser.add_argument("--limit", type=int, default=0, help="Max runs to process (0 = all)") + parser.add_argument("--force", action="store_true", help="Re-verify runs that already have a report") args = parser.parse_args() results_dir = Path(args.results_dir) @@ -105,29 +93,33 @@ def main() -> None: print(f"Replaying {len(summaries)} runs from {results_dir}") - success = failed = skipped = 0 + verified = failed = skipped = 0 for path in summaries: - trace_path = path.parent / "location_trace.json" - if trace_path.exists(): + report_path = path.parent / "replay_report.json" + if report_path.exists() and not args.force: skipped += 1 continue print(f" {path.relative_to(results_dir)}", end=" ... ", flush=True) - trace = replay_run(path) + report = replay_run(path) - tmp = trace_path.with_suffix(".tmp") - tmp.write_text(json.dumps(trace, ensure_ascii=False, indent=2), encoding="utf-8") - tmp.replace(trace_path) + tmp = report_path.with_suffix(".tmp") + tmp.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8") + tmp.replace(report_path) - if trace["error"]: - print(f"PARTIAL ({trace['error'][:60]})") - failed += 1 + if report["status"] == "verified": + print(f"OK ({report['verified_transitions']}/{report['total_transitions']} transitions)") + verified += 1 + elif report["status"] == "skipped": + print(f"SKIPPED ({report['error']})") + skipped += 1 else: - covered = sum(1 for x in trace["location_trace"] if x is not None) - print(f"OK ({covered}/{len(trace['location_trace'])} steps)") - success += 1 + print(f"FAILED ({str(report['error'])[:80]})") + failed += 1 - print(f"\nDone: {success} ok, {failed} partial/failed, {skipped} skipped (trace exists)") + print(f"\nDone: {verified} verified, {failed} failed, {skipped} skipped") + if failed: + sys.exit(1) if __name__ == "__main__": diff --git a/scripts/select_runs_for_analysis.py b/scripts/select_runs_for_analysis.py index 0ecc1ae..7a97a6d 100644 --- a/scripts/select_runs_for_analysis.py +++ b/scripts/select_runs_for_analysis.py @@ -18,25 +18,30 @@ import argparse import glob import json +import sys from collections import defaultdict from pathlib import Path +repo_root = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(repo_root)) + +from llm_quest_benchmark.schemas.records import RunRecord, load_run_record # noqa: E402 + JUNK_QUESTS = {"test_quest", "quest_1", "repeatable_quest", "repeatable", "nonexistent"} -ANALYSIS_OUTCOMES = {"FAILURE", "TIMEOUT"} +ANALYSIS_OUTCOMES = {"FAILURE", "TIMEOUT", "TRUNCATED"} MAX_PER_QUEST = 50 -def load_run(path: str) -> dict | None: +def load_run(path: str) -> RunRecord | None: try: - with open(path, encoding="utf-8") as f: - return json.load(f) - except (json.JSONDecodeError, OSError): + return load_run_record(path) + except (ValueError, OSError): return None -def has_reasoning(run: dict) -> bool: - for step in run.get("steps", []): - r = (step.get("llm_decision") or {}).get("reasoning") or "" +def has_reasoning(run: RunRecord) -> bool: + for transition in run.transitions: + r = (transition.response.reasoning if transition.response else "") or "" if r and "error" not in r.lower()[:20]: return True return False @@ -58,12 +63,12 @@ def select_runs(results_dir: Path) -> list[dict]: if run is None: continue - quest = run.get("quest_name", "") + quest = run.quest_name if quest in JUNK_QUESTS: skipped_junk += 1 continue - outcome = run.get("outcome", "") + outcome = run.outcome or "" if outcome == "ERROR": skipped_error += 1 continue @@ -76,11 +81,12 @@ def select_runs(results_dir: Path) -> list[dict]: { "path": path, "quest": quest, - "agent": run.get("agent_id", ""), + "agent": run.agent_id, + "treatment_signature": run.treatment_signature, "outcome": outcome, - "steps": len(run.get("steps", [])), + "transitions": len(run.transitions), "has_reasoning": has_reasoning(run), - "run_id": run.get("run_id", 0), + "run_id": run.run_id if isinstance(run.run_id, int) else 0, } ) @@ -153,7 +159,14 @@ def main(): "quests": len(set(r["quest"] for r in selected)), "agents": len(set(r["agent"] for r in selected)), "runs": [ - {"path": r["path"], "quest": r["quest"], "agent": r["agent"], "outcome": r["outcome"], "steps": r["steps"]} + { + "path": r["path"], + "quest": r["quest"], + "agent": r["agent"], + "treatment_signature": r["treatment_signature"], + "outcome": r["outcome"], + "transitions": r["transitions"], + } for r in selected ], } diff --git a/site/index.html b/site/index.html index 1092b52..3f276c0 100644 --- a/site/index.html +++ b/site/index.html @@ -31,7 +31,11 @@ .mode-prompt_hints { background: #db61a2; color: var(--bg); } .mode-tools_compact_memory { background: var(--green); color: var(--bg); } .mode-tools_hints_compact_memory { background: #56d4dd; color: var(--bg); } +.mode-tools_programmatic_memory { background: #2ea043; color: var(--bg); } .mode-planner_loop { background: var(--orange); color: var(--bg); } +.mode-backtracking_loop { background: #f0883e; color: var(--bg); } +.mode-adaptive_reasoning { background: #bc8cff; color: var(--bg); } +.mode-unknown { background: var(--border); color: var(--text); } .success-bar { height: 6px; border-radius: 3px; background: var(--border); overflow: hidden; min-width: 60px; } .success-fill { height: 100%; border-radius: 3px; transition: width 0.3s; } .filter-btn { background: var(--surface); border: 1px solid var(--border); color: var(--text); padding: 0.3em 0.8em; border-radius: 6px; font-size: 0.85rem; cursor: pointer; transition: all 0.15s; } @@ -129,7 +133,9 @@
Success Rate by Model and Mode
let filterQuest = 'all'; let viewMode = 'summary'; -const LEADERBOARD_DATA_VERSION = '2026-05-06-taxonomy-v2'; +const LEADERBOARD_DATA_VERSION = '2026-08-26-schema-v2'; +// Mode ids come from schema-v2 treatment components (prompt, memory, tools, +// loop, reasoning). Unknown ids fall back to the label supplied in the data. const TAXONOMY_LABELS = { minimal_prompt:'Minimal prompt', short_context_reasoning:'Short-context reasoning', @@ -138,20 +144,11 @@
Success Rate by Model and Mode
prompt_hints:'Prompt hints', tools_compact_memory:'Tools + compact memory', tools_hints_compact_memory:'Tools + hints + compact memory', + tools_programmatic_memory:'Tools + programmatic memory', planner_loop:'Planner loop', -}; -const MODE_ALIASES = { - a:'minimal_prompt', - baseline:'minimal_prompt', - b:'short_context_reasoning', - prompted:'short_context_reasoning', - c:'compact_memory_memo', - knowledge:'compact_memory_memo', - d:'planner_loop', - planner:'planner_loop', - e:'tools_compact_memory', - 'tool-aug':'tools_compact_memory', - tool_augmented:'tools_compact_memory', + backtracking_loop:'Backtracking loop', + adaptive_reasoning:'Adaptive reasoning', + unknown:'Unknown treatment', }; const MODE_CLASSES = { minimal_prompt:'mode-minimal_prompt', @@ -161,7 +158,11 @@
Success Rate by Model and Mode
prompt_hints:'mode-prompt_hints', tools_compact_memory:'mode-tools_compact_memory', tools_hints_compact_memory:'mode-tools_hints_compact_memory', + tools_programmatic_memory:'mode-tools_programmatic_memory', planner_loop:'mode-planner_loop', + backtracking_loop:'mode-backtracking_loop', + adaptive_reasoning:'mode-adaptive_reasoning', + unknown:'mode-unknown', }; const COLORS = { minimal_prompt:'#c9d1d9', @@ -171,7 +172,11 @@
Success Rate by Model and Mode
prompt_hints:'#db61a2', tools_compact_memory:'#3fb950', tools_hints_compact_memory:'#56d4dd', + tools_programmatic_memory:'#2ea043', planner_loop:'#d29922', + backtracking_loop:'#f0883e', + adaptive_reasoning:'#bc8cff', + unknown:'#6e7681', }; function successColor(v) { return v >= 0.8 ? 'var(--green)' : v >= 0.4 ? 'var(--orange)' : 'var(--red)'; } @@ -179,32 +184,23 @@
Success Rate by Model and Mode
function modelLabel(id) { return MODEL_BY_ID[id]?.label || id; } function modeLabel(id) { return MODE_BY_ID[id]?.label || id; } -function canonicalModeId(id, label) { - const rawId = String(id || '').trim(); - const rawLabel = String(label || '').trim(); - const key = rawId.toLowerCase().replace(/\s+/g, '_'); - return TAXONOMY_LABELS[rawId] ? rawId : MODE_ALIASES[key] || MODE_ALIASES[rawLabel.toLowerCase()] || rawId; -} - function normalizeLeaderboard(data) { - const rawModeLabels = Object.fromEntries((data.modes || []).map(mode => [mode.id, mode.label])); const modeSeen = new Set(); - const modes = (data.modes || []).map(mode => { - const id = canonicalModeId(mode.id, mode.label); - return {id, label: TAXONOMY_LABELS[id] || mode.label || id}; - }).filter(mode => { + const modes = (data.modes || []).map(mode => ({ + id: mode.id, + label: TAXONOMY_LABELS[mode.id] || mode.label || mode.id, + })).filter(mode => { if (modeSeen.has(mode.id)) return false; modeSeen.add(mode.id); return true; }); const knownModes = new Set(modes.map(mode => mode.id)); const results = (data.results || []).map(row => { - const mode = canonicalModeId(row.mode, rawModeLabels[row.mode]); - if (!knownModes.has(mode)) { - modes.push({id: mode, label: TAXONOMY_LABELS[mode] || mode}); - knownModes.add(mode); + if (!knownModes.has(row.mode)) { + modes.push({id: row.mode, label: TAXONOMY_LABELS[row.mode] || row.mode}); + knownModes.add(row.mode); } - return {...row, mode}; + return row; }); return {...data, modes, results}; } diff --git a/site/play/app.js b/site/play/app.js index b04f631..b0f12cb 100644 --- a/site/play/app.js +++ b/site/play/app.js @@ -584,10 +584,24 @@ function choicesToTraceMap(choices) { function paramsToTraceList(paramsState) { return (paramsState || []).filter(p => p && p.trim()).map(p => stripClr(p)); } + +// One canonical snapshot: full engine saving plus the state the player saw. +function traceSnapshot(player, state, canonicalLocationId) { + const activeChoices = (state.choices || []).filter(c => c.active); + return { + location_id: String(player.getSaving().locationId), + canonical_location_id: canonicalLocationId != null ? String(canonicalLocationId) : null, + observation: stripClr(state.text || ''), + params: paramsToTraceList(state.paramsState), + choices: choicesToTraceMap(activeChoices), + game_state: state.gameState || 'running', + saving: player.getSaving() + }; +} function buildHumanTrace({ quest, outcome, - steps, + transitions, terminalText, startedAt }) { @@ -598,7 +612,7 @@ function buildHumanTrace({ dead: 'FAILURE' }[outcome] || 'INCOMPLETE'; return { - schema_version: 'human_trace_v1', + schema_version: 'human_trace_v2', source: 'web_play', quest_id: quest.id, quest_title: quest.title || quest.id, @@ -607,12 +621,13 @@ function buildHumanTrace({ started_at: startedAt || now, ended_at: now, outcome: outcomeLabel, - steps, + // Chronological log of every executed transition, including restores. + // Backtracking truncates only the active branch, never this history. + transitions, terminal: { game_state: outcome, text: stripClr(terminalText || '') }, - undo_events: [], metadata: { app_url: PLAY_URL, user_agent: navigator.userAgent || '', @@ -687,7 +702,7 @@ function EndScreen({ mediaState, audioEnabled, families, - traceSteps, + traceTransitions, startedAt, onPlayAgain, onTryAnother @@ -744,7 +759,7 @@ function EndScreen({ const trace = buildHumanTrace({ quest, outcome, - steps: traceSteps, + transitions: traceTransitions, terminalText: endText, startedAt }); @@ -833,7 +848,7 @@ function QuestPlay({ const [stepHistory, setStepHistory] = useState([]); const [stepNum, setStepNum] = useState(0); const [path, setPath] = useState([]); - const [traceSteps, setTraceSteps] = useState([]); + const [traceTransitions, setTraceTransitions] = useState([]); const [ended, setEnded] = useState(null); const [endText, setEndText] = useState(''); const [endMediaState, setEndMediaState] = useState(null); @@ -864,7 +879,7 @@ function QuestPlay({ setCanonicalPlayer(canonical); setGameState(p.getState()); setStepNum(1); - setTraceSteps([]); + setTraceTransitions([]); startedAtRef.current = new Date().toISOString(); setLoading(false); }).catch(err => { @@ -900,8 +915,7 @@ function QuestPlay({ } function handleChoice(choice, activeChoices) { const locationId = canonicalPlayer ? canonicalPlayer.getSaving().locationId : player.getSaving().locationId; - const choices = gameState.choices || []; - const choiceIndex = Math.max(0, choices.findIndex(c => c.jumpId === choice.jumpId)); + const choiceIndex = Math.max(0, activeChoices.findIndex(c => c.jumpId === choice.jumpId)); const choiceNorm = canonicalChoiceNorm(choice); const cohortLoc = getCohortLoc(locationId); const isBranching = activeChoices.length >= 2; @@ -917,25 +931,32 @@ function QuestPlay({ hasCohortData, playerChoiceNorm: isBranching ? choiceNorm : null }]); + const before = traceSnapshot(player, gameState, locationId); setStepHistory(prev => [...prev, { player: player.getSaving(), - canonicalPlayer: canonicalPlayer ? canonicalPlayer.getSaving() : null + canonicalPlayer: canonicalPlayer ? canonicalPlayer.getSaving() : null, + trace: before }]); - setTraceSteps(prev => [...prev, { + player.performJump(choice.jumpId); + if (canonicalPlayer) canonicalPlayer.performJump(choice.jumpId); + const nextState = player.getState(); + const afterLocationId = canonicalPlayer ? canonicalPlayer.getSaving().locationId : player.getSaving().locationId; + setTraceTransitions(prev => [...prev, { + index: prev.length + 1, step: stepNum, - location_id: locationId, - observation: stripClr(gameState.text || ''), - params: paramsToTraceList(gameState.paramsState), - choices: choicesToTraceMap(choices), - human_decision: { + kind: 'choose', + before, + action: { + kind: 'choose', choice_index: String(choiceIndex + 1), choice_text: stripClr(choice.text || ''), - jump_id: choice.jumpId - } + jump_id: choice.jumpId, + // QMPlayer.performJump generates its own engine timestamp, so the exact + // value is not observable here and is never invented. + performed_at_ms: null + }, + after: traceSnapshot(player, nextState, afterLocationId) }]); - player.performJump(choice.jumpId); - if (canonicalPlayer) canonicalPlayer.performJump(choice.jumpId); - const nextState = player.getState(); const gs = nextState.gameState; const isTerminal = gs === 'win' || gs === 'fail' || gs === 'dead'; if (isTerminal) { @@ -950,16 +971,34 @@ function QuestPlay({ } function handleBack() { if (stepHistory.length === 0) return; - const prevSaving = stepHistory[stepHistory.length - 1]; - player.loadSaving(prevSaving.player || prevSaving); - if (canonicalPlayer && prevSaving.canonicalPlayer) { - canonicalPlayer.loadSaving(prevSaving.canonicalPlayer); + const checkpoint = stepHistory[stepHistory.length - 1]; + const beforeState = ended ? endMediaState || player.getState() : gameState; + const beforeLocationId = canonicalPlayer ? canonicalPlayer.getSaving().locationId : player.getSaving().locationId; + const before = traceSnapshot(player, beforeState, beforeLocationId); + const checkpointIndex = stepHistory.length; + player.loadSaving(checkpoint.player || checkpoint); + if (canonicalPlayer && checkpoint.canonicalPlayer) { + canonicalPlayer.loadSaving(checkpoint.canonicalPlayer); } + const restoredState = player.getState(); + const restoredLocationId = canonicalPlayer ? canonicalPlayer.getSaving().locationId : player.getSaving().locationId; + // The restore is itself a transition: undone steps stay in the exported + // trace, and only the active checkpoint branch is truncated. + setTraceTransitions(prev => [...prev, { + index: prev.length + 1, + step: stepNum, + kind: 'restore', + before, + action: { + kind: 'restore', + checkpoint_index: checkpointIndex + }, + after: checkpoint.trace || traceSnapshot(player, restoredState, restoredLocationId) + }]); setStepHistory(prev => prev.slice(0, -1)); - setGameState(player.getState()); + setGameState(restoredState); setStepNum(n => Math.max(1, n - 1)); setPath(prev => prev.slice(0, -1)); - setTraceSteps(prev => prev.slice(0, -1)); setObsKey(k => k + 1); setEnded(null); setEndMediaState(null); @@ -1002,7 +1041,7 @@ function QuestPlay({ mediaState: endMediaState, audioEnabled: audioEnabled, families: cohortData && cohortData.model_families || [], - traceSteps: traceSteps, + traceTransitions: traceTransitions, startedAt: startedAtRef.current, onPlayAgain: () => { player.start(); @@ -1010,7 +1049,7 @@ function QuestPlay({ setGameState(player.getState()); setStepNum(1); setPath([]); - setTraceSteps([]); + setTraceTransitions([]); setStepHistory([]); setEnded(null); setEndText(''); diff --git a/site/play/app.jsx b/site/play/app.jsx index c2394e7..fe187be 100644 --- a/site/play/app.jsx +++ b/site/play/app.jsx @@ -517,11 +517,25 @@ function paramsToTraceList(paramsState) { return (paramsState || []).filter(p => p && p.trim()).map(p => stripClr(p)); } -function buildHumanTrace({ quest, outcome, steps, terminalText, startedAt }) { +// One canonical snapshot: full engine saving plus the state the player saw. +function traceSnapshot(player, state, canonicalLocationId) { + const activeChoices = (state.choices || []).filter(c => c.active); + return { + location_id: String(player.getSaving().locationId), + canonical_location_id: canonicalLocationId != null ? String(canonicalLocationId) : null, + observation: stripClr(state.text || ''), + params: paramsToTraceList(state.paramsState), + choices: choicesToTraceMap(activeChoices), + game_state: state.gameState || 'running', + saving: player.getSaving(), + }; +} + +function buildHumanTrace({ quest, outcome, transitions, terminalText, startedAt }) { const now = new Date().toISOString(); const outcomeLabel = { win: 'SUCCESS', fail: 'FAILURE', dead: 'FAILURE' }[outcome] || 'INCOMPLETE'; return { - schema_version: 'human_trace_v1', + schema_version: 'human_trace_v2', source: 'web_play', quest_id: quest.id, quest_title: quest.title || quest.id, @@ -530,12 +544,13 @@ function buildHumanTrace({ quest, outcome, steps, terminalText, startedAt }) { started_at: startedAt || now, ended_at: now, outcome: outcomeLabel, - steps, + // Chronological log of every executed transition, including restores. + // Backtracking truncates only the active branch, never this history. + transitions, terminal: { game_state: outcome, text: stripClr(terminalText || ''), }, - undo_events: [], metadata: { app_url: PLAY_URL, user_agent: navigator.userAgent || '', @@ -603,7 +618,7 @@ function buildShareText(questTitle, outcomeLabel, path, cohortWinRate) { return lines.join('\n'); } -function EndScreen({ outcome, cohortWinRate, path, quest, endText, mediaState, audioEnabled, families, traceSteps, startedAt, onPlayAgain, onTryAnother }) { +function EndScreen({ outcome, cohortWinRate, path, quest, endText, mediaState, audioEnabled, families, traceTransitions, startedAt, onPlayAgain, onTryAnother }) { const [shareStatus, setShareStatus] = useState(''); const outcomeLabel = { win: 'SUCCESS', fail: 'FAILURE', dead: 'DEAD' }[outcome] || 'FAILURE'; const questTitle = quest.title || quest.id; @@ -641,7 +656,7 @@ function EndScreen({ outcome, cohortWinRate, path, quest, endText, mediaState, a } function handleExportTrace() { - const trace = buildHumanTrace({ quest, outcome, steps: traceSteps, terminalText: endText, startedAt }); + const trace = buildHumanTrace({ quest, outcome, transitions: traceTransitions, terminalText: endText, startedAt }); const stamp = new Date().toISOString().replace(/[-:]/g, '').replace(/\..*/, 'Z'); downloadJson(trace, 'human_trace_' + safeFilenamePart(quest.id) + '_' + stamp + '.json'); setShareStatus('Downloaded trace JSON.'); @@ -692,7 +707,7 @@ function QuestPlay({ quest, cohortData, onQuit }) { const [stepHistory, setStepHistory] = useState([]); const [stepNum, setStepNum] = useState(0); const [path, setPath] = useState([]); - const [traceSteps, setTraceSteps] = useState([]); + const [traceTransitions, setTraceTransitions] = useState([]); const [ended, setEnded] = useState(null); const [endText, setEndText] = useState(''); const [endMediaState, setEndMediaState] = useState(null); @@ -719,7 +734,7 @@ function QuestPlay({ quest, cohortData, onQuit }) { setCanonicalPlayer(canonical); setGameState(p.getState()); setStepNum(1); - setTraceSteps([]); + setTraceTransitions([]); startedAtRef.current = new Date().toISOString(); setLoading(false); }) @@ -757,8 +772,7 @@ function QuestPlay({ quest, cohortData, onQuit }) { function handleChoice(choice, activeChoices) { const locationId = canonicalPlayer ? canonicalPlayer.getSaving().locationId : player.getSaving().locationId; - const choices = gameState.choices || []; - const choiceIndex = Math.max(0, choices.findIndex(c => c.jumpId === choice.jumpId)); + const choiceIndex = Math.max(0, activeChoices.findIndex(c => c.jumpId === choice.jumpId)); const choiceNorm = canonicalChoiceNorm(choice); const cohortLoc = getCohortLoc(locationId); const isBranching = activeChoices.length >= 2; @@ -776,26 +790,35 @@ function QuestPlay({ quest, cohortData, onQuit }) { playerChoiceNorm: isBranching ? choiceNorm : null, }]); + const before = traceSnapshot(player, gameState, locationId); setStepHistory(prev => [...prev, { player: player.getSaving(), canonicalPlayer: canonicalPlayer ? canonicalPlayer.getSaving() : null, + trace: before, }]); - setTraceSteps(prev => [...prev, { + + player.performJump(choice.jumpId); + if (canonicalPlayer) canonicalPlayer.performJump(choice.jumpId); + + const nextState = player.getState(); + const afterLocationId = canonicalPlayer ? canonicalPlayer.getSaving().locationId : player.getSaving().locationId; + setTraceTransitions(prev => [...prev, { + index: prev.length + 1, step: stepNum, - location_id: locationId, - observation: stripClr(gameState.text || ''), - params: paramsToTraceList(gameState.paramsState), - choices: choicesToTraceMap(choices), - human_decision: { + kind: 'choose', + before, + action: { + kind: 'choose', choice_index: String(choiceIndex + 1), choice_text: stripClr(choice.text || ''), jump_id: choice.jumpId, + // QMPlayer.performJump generates its own engine timestamp, so the exact + // value is not observable here and is never invented. + performed_at_ms: null, }, + after: traceSnapshot(player, nextState, afterLocationId), }]); - player.performJump(choice.jumpId); - if (canonicalPlayer) canonicalPlayer.performJump(choice.jumpId); - const nextState = player.getState(); const gs = nextState.gameState; const isTerminal = gs === 'win' || gs === 'fail' || gs === 'dead'; @@ -812,16 +835,37 @@ function QuestPlay({ quest, cohortData, onQuit }) { function handleBack() { if (stepHistory.length === 0) return; - const prevSaving = stepHistory[stepHistory.length - 1]; - player.loadSaving(prevSaving.player || prevSaving); - if (canonicalPlayer && prevSaving.canonicalPlayer) { - canonicalPlayer.loadSaving(prevSaving.canonicalPlayer); + const checkpoint = stepHistory[stepHistory.length - 1]; + const beforeState = ended ? (endMediaState || player.getState()) : gameState; + const beforeLocationId = canonicalPlayer ? canonicalPlayer.getSaving().locationId : player.getSaving().locationId; + const before = traceSnapshot(player, beforeState, beforeLocationId); + const checkpointIndex = stepHistory.length; + + player.loadSaving(checkpoint.player || checkpoint); + if (canonicalPlayer && checkpoint.canonicalPlayer) { + canonicalPlayer.loadSaving(checkpoint.canonicalPlayer); } + + const restoredState = player.getState(); + const restoredLocationId = canonicalPlayer ? canonicalPlayer.getSaving().locationId : player.getSaving().locationId; + // The restore is itself a transition: undone steps stay in the exported + // trace, and only the active checkpoint branch is truncated. + setTraceTransitions(prev => [...prev, { + index: prev.length + 1, + step: stepNum, + kind: 'restore', + before, + action: { + kind: 'restore', + checkpoint_index: checkpointIndex, + }, + after: checkpoint.trace || traceSnapshot(player, restoredState, restoredLocationId), + }]); + setStepHistory(prev => prev.slice(0, -1)); - setGameState(player.getState()); + setGameState(restoredState); setStepNum(n => Math.max(1, n - 1)); setPath(prev => prev.slice(0, -1)); - setTraceSteps(prev => prev.slice(0, -1)); setObsKey(k => k + 1); setEnded(null); setEndMediaState(null); @@ -856,7 +900,7 @@ function QuestPlay({ quest, cohortData, onQuit }) { mediaState={endMediaState} audioEnabled={audioEnabled} families={(cohortData && cohortData.model_families) || []} - traceSteps={traceSteps} + traceTransitions={traceTransitions} startedAt={startedAtRef.current} onPlayAgain={() => { player.start(); @@ -864,7 +908,7 @@ function QuestPlay({ quest, cohortData, onQuit }) { setGameState(player.getState()); setStepNum(1); setPath([]); - setTraceSteps([]); + setTraceTransitions([]); setStepHistory([]); setEnded(null); setEndText(''); diff --git a/site/traces.html b/site/traces.html index a941df1..ba55fc6 100644 --- a/site/traces.html +++ b/site/traces.html @@ -256,8 +256,11 @@
📄
-

Drop run_summary.json here

+

Drop a schema-v2 run_summary.json here

or click to select a file

+

+ Older records: convert once with llm-quest migrate-records --source <path> --output <path> +

@@ -379,32 +382,52 @@

return Number(n).toLocaleString('en-US', { maximumFractionDigits: digits != null ? digits : 2 }); } + function showError(message) { + console.error(message); + const container = document.getElementById('steps-container'); + document.getElementById('timeline').style.display = 'block'; + document.getElementById('steps-label').textContent = ''; + container.innerHTML = '
' + esc(message) + '
'; + } + function fmtDur(secs) { if (secs == null) return '-'; if (secs < 60) return secs.toFixed(1) + 's'; return (secs / 60).toFixed(1) + 'min'; } - function getChosenKey(choice) { - if (!choice || typeof choice !== 'object') return null; - return Object.keys(choice)[0] || null; - } - - function isTerminalStep(step) { - return step.is_terminal === true; + function choicesMap(snapshot) { + const out = {}; + ((snapshot && snapshot.choices) || []).forEach((choice, i) => { + out[String(i + 1)] = choice.text || ''; + }); + return out; } function render(data) { - const steps = data.steps || []; - const metrics = data.metrics || {}; + if (data.schema_version !== 2) { + showError('This run record is not schema v2. Convert it first: ' + + 'llm-quest migrate-records --source --output '); + return; + } + + // Schema-v2 records are grouped into canonical domains; nothing but + // schema_version lives at the top level. + const transitions = data.transitions || []; + const diagnostics = data.transcript_diagnostics || {}; const usage = data.usage || {}; + const progress = data.progress || {}; + const run = data.run || {}; + const quest = data.quest || {}; + const terminal = data.terminal || {}; // Show header document.getElementById('run-header').style.display = 'block'; - document.getElementById('hdr-quest').textContent = data.quest_name || data.quest_file || 'Unknown Quest'; - document.getElementById('hdr-agent').textContent = data.agent_id || ''; + document.getElementById('hdr-quest').textContent = quest.name || quest.file || 'Unknown Quest'; + document.getElementById('hdr-agent').textContent = + (run.agent_id || '') + ((data.treatment && data.treatment.signature) ? ' · ' + data.treatment.signature : ''); - const outcome = (data.outcome || '').toUpperCase(); + const outcome = (terminal.outcome || '').toUpperCase(); const outcomeBadge = document.getElementById('hdr-outcome'); outcomeBadge.textContent = outcome || 'UNKNOWN'; outcomeBadge.className = 'outcome-badge outcome-' + (outcome || 'UNKNOWN'); @@ -412,10 +435,13 @@

// Stats pills const totalTokens = usage.total_tokens || (usage.prompt_tokens || 0) + (usage.completion_tokens || 0); const cost = usage.estimated_cost_usd; + const restores = (diagnostics.restore_transitions != null) ? diagnostics.restore_transitions : 0; const pills = [ - { label: 'Steps', value: metrics.total_steps != null ? metrics.total_steps : steps.length }, - { label: 'Reward', value: data.reward != null ? fmt(data.reward, 0) : '-' }, - { label: 'Duration', value: fmtDur(data.run_duration) }, + { label: 'Steps', value: diagnostics.total_steps != null ? diagnostics.total_steps : transitions.length }, + { label: 'Restores', value: restores }, + { label: 'Progress', value: progress.scored ? fmt(progress.current, 1) + '%' : 'unscored' }, + { label: 'Reward', value: terminal.reward != null ? fmt(terminal.reward, 0) : '-' }, + { label: 'Duration', value: fmtDur(run.duration) }, { label: 'Total Tokens', value: totalTokens ? fmt(totalTokens, 0) : '-' }, { label: 'Cost', value: cost != null ? ('$' + cost.toFixed(4)) : '-' }, ]; @@ -425,32 +451,37 @@

// Timeline document.getElementById('timeline').style.display = 'block'; - document.getElementById('steps-label').textContent = steps.length + ' steps'; + document.getElementById('steps-label').textContent = transitions.length + ' transitions'; const container = document.getElementById('steps-container'); container.innerHTML = ''; - steps.forEach((step, idx) => { - const dec = step.llm_decision || {}; - const chosenKey = getChosenKey(dec.choice); - const chosenText = chosenKey ? (dec.choice[chosenKey] || '') : ''; - const isTerminal = isTerminalStep(step); + transitions.forEach((step, idx) => { + const dec = step.response || {}; + const action = step.action || {}; + const isRestore = action.kind === 'restore'; + const chosenKey = (!isRestore && action.choice_index != null) ? String(action.choice_index) : null; + const choices = choicesMap(step.before); + const chosenText = chosenKey ? (choices[chosenKey] || '') : ''; + const isTerminal = !!(step.after && step.after.done); const isError = dec.parse_mode === 'error_default'; const isSingleAuto = dec.reasoning === 'auto_single_choice'; - // Collapsed preview: chosen action or error label - const previewText = chosenText || (chosenKey ? 'Choice ' + chosenKey : 'No choice'); + // Collapsed preview: restored checkpoint, chosen action, or error label + const previewText = isRestore + ? ('Restored checkpoint ' + action.checkpoint_index) + : (chosenText || (chosenKey ? 'Choice ' + chosenKey : 'No choice')); // Tokens - const pt = dec.prompt_tokens || 0; - const ct = dec.completion_tokens || 0; + const stepUsage = step.usage || {}; + const pt = stepUsage.prompt_tokens || 0; + const ct = stepUsage.completion_tokens || 0; const stepTokens = pt + ct; const item = document.createElement('div'); item.className = 'step-item' + (isTerminal ? ' step-terminal' : ''); // Build choices HTML - const choices = step.choices || {}; const choicesHtml = Object.keys(choices).length === 0 ? 'No choices available' : '
    ' + Object.entries(choices).map(([k, v]) => @@ -490,21 +521,25 @@

    reasoningHtml += `
    Tool Results
    ${trHtml}
    `; } + const progressPct = (step.progress && step.progress.current != null) ? fmt(step.progress.current, 1) + '%' : null; item.innerHTML = `
    - #${step.step} + #${esc(step.index)} ${esc(previewText)} - ${isTerminal ? '' + esc((data.outcome || 'END').toUpperCase()) + '' : ''} + ${isRestore ? 'RESTORE' : ''} + ${isTerminal ? '' + esc((terminal.outcome || 'END').toUpperCase()) + '' : ''} ${isError ? 'ERROR' : ''} ${isSingleAuto ? 'auto' : ''} + ${step.reasoning_mode ? `${esc(step.reasoning_mode)}` : ''} + ${progressPct ? `${esc(progressPct)}` : ''} ${stepTokens > 0 ? `${fmt(stepTokens, 0)} tok` : ''} ▶
    Observation
    -
    ${esc(step.observation || '')}
    +
    ${esc((step.before && step.before.observation) || '')}
    Choices
    ${choicesHtml} diff --git a/uv.lock b/uv.lock index 0b38b58..201535a 100644 --- a/uv.lock +++ b/uv.lock @@ -545,6 +545,7 @@ dependencies = [ { name = "pyyaml" }, { name = "rich" }, { name = "sqlalchemy" }, + { name = "tqdm" }, { name = "typer" }, ] @@ -578,6 +579,7 @@ requires-dist = [ { name = "rich", specifier = ">=13.0.0" }, { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.4.0" }, { name = "sqlalchemy", specifier = ">=2.0.0" }, + { name = "tqdm", specifier = ">=4.66.0" }, { name = "typer", specifier = ">=0.9.0" }, { name = "types-python-dateutil", marker = "extra == 'dev'" }, { name = "types-pyyaml", marker = "extra == 'dev'" },