Add programmatic memory harness

configs/benchmarks/programmatic_memory_pilot.yaml

@@ -0,0 +1,74 @@
+# Small pilot comparison for docs/PROGRAMMATIC_MEMORY_PROPOSAL.md.
+# Holds model, temperature, quest set, quest_timeout, and max_steps constant
+# across the five harnesses in the proposal's "Primary comparison" table.
+#
+# This is an EXPLORATORY BUNDLED-HARNESS COMPARISON, not an isolated
+# single-dimension treatment: programmatic_memory simultaneously removes
+# CompactionMemory (compaction) and replaces tool_compact's clipped
+# quest_history search with full read/search, and its tool/prompt path also
+# differs from reasoning_recent's (tool-select-then-act loop, calculator and
+# scratchpad tools, memo-producing prompt) rather than varying retrieval
+# fidelity alone. A result from this pilot cannot attribute a success-rate or
+# cost change to programmatic retrieval specifically -- it can only say
+# whether the programmatic_memory harness, as a whole, out- or
+# under-performs each existing baseline harness on this quest set. Isolating
+# retrieval fidelity from compaction would need a separate ablation family
+# (e.g. compaction on/off with retrieval fixed, and retrieval fidelity varied
+# with compaction fixed); that is out of scope for this pilot.
+#
+# Quests are reused from existing benchmark configs (exp3/exp5/exp6,
+# memory_modes_pilot) and the fake-provider e2e smoke quest rather than
+# cherry-picked for this comparison: Boat.qm is a short stateful puzzle
+# (present locally, always runnable), Banket_eng.qm and Borzukhan_eng.qm are
+# longer multi-turn quests already exercised together in
+# tests/integration/test_mode_agents_e2e.py. This is a pilot scale to validate
+# the matrix and artifacts before increasing repetitions or quest count.
+name: programmatic_memory_pilot
+quests:
+  - quests/Boat.qm
+  - quests/sr_2_1_2121_eng/Banket_eng.qm
+  - quests/sr_2_1_2121_eng/Borzukhan_eng.qm
+agents:
+  # Minimal bounded-context baseline: recent context only, no external history.
+  - model: openrouter:google/gemini-3-flash-preview
+    harness: reasoning_recent
+    temperature: 0.4
+    runs: 3
+
+  # Capacity-heavy baseline: full transcript in every prompt, no external history.
+  - model: openrouter:google/gemini-3-flash-preview
+    harness: reasoning_full
+    temperature: 0.4
+    runs: 3
+
+  # Summary baseline: recent context plus LLM-compacted summary/memo.
+  - model: openrouter:google/gemini-3-flash-preview
+    harness: memo_compact
+    temperature: 0.4
+    runs: 3
+    compaction_interval: 10
+
+  # Current closest tool baseline: compacted memory plus clipped keyword search.
+  - model: openrouter:google/gemini-3-flash-preview
+    harness: tool_compact
+    temperature: 0.4
+    runs: 3
+    compaction_interval: 10
+
+  # Proposed treatment: recent bounded context plus full read/search retrieval,
+  # no LLM compaction.
+  - model: openrouter:google/gemini-3-flash-preview
+    harness: programmatic_memory
+    temperature: 0.4
+    runs: 3
+
+debug: false
+quest_timeout: 600
+# Shared step cap: reasoning_recent/reasoning_full make one model call per
+# decision while tool_compact/programmatic_memory can make two, so without a
+# shared max_steps a faster one-call harness could execute more quest actions
+# before quest_timeout than a slower one, confounding success/timeout
+# comparisons independent of the memory/retrieval difference under test.
+max_steps: 60
+max_workers: 2
+output_dir: results/benchmarks

docs/ARCHITECTURE.md

@@ -59,6 +59,12 @@ planning choices change behavior.
   `FullTranscriptMemory`, and `CompactionMemory`.
 - `llm_quest_benchmark/harnesses/tools.py`: Calculator, scratchpad, and quest
   history helpers used by tool harnesses.
+- `llm_quest_benchmark/harnesses/trajectory.py`: `Trajectory`, a run-local
+  retrieval/index view holding references to canonical executed `AgentState`
+  objects for `programmatic_memory`, with bounded deterministic `read`/`search`.
+  `QuestRunner` emits the same object to the harness, callbacks, and
+  `QuestLogger`; `QuestLogger` serializes the persisted `run_summary.json`
+  trace.
 - `llm_quest_benchmark/harnesses/factory.py`: `create_harness()` and the
   canonical harness registry.
 - `llm_quest_benchmark/players/human.py`,
@@ -104,6 +110,8 @@ and benchmark configuration parsing do not require API keys.
   - `planner.jinja`: Planner loop prompt.
   - `tool_augmented.jinja`, `tool_augmented_hints.jinja`: Tool prompts with
     compact memory, optionally with hints.
+  - `programmatic_memory.jinja`: Tool prompt for bounded recent context plus
+    full-fidelity `history_read`/`history_search` retrieval, no compaction.
 
 ## Persistence
 
@@ -134,3 +142,24 @@ The harness names above are canonical snake_case identifiers used in YAML
 configs, the CLI, result artifacts, and documentation. Public labels can be
 friendlier, but experiment records should preserve the canonical names so runs
 remain comparable.
+
+## Experimental Harnesses (Not Yet Public)
+
+| Label | Harness name | Template | Memory | Tools | Loop |
+|---|---|---|---|---|---|
+| Programmatic memory (experimental) | `programmatic_memory` | `programmatic_memory.jinja` | `DefaultMemory` | calculator, scratchpad, history_read, history_search | tool-select-then-act |
+
+`programmatic_memory` is an experimental treatment (see
+`docs/PROGRAMMATIC_MEMORY_PROPOSAL.md`): it replaces `tool_compact`'s clipped
+`quest_history` keyword search with bounded deterministic reads/searches over a
+run-local `Trajectory` view of canonical executed `AgentState` objects, and
+replaces `CompactionMemory` with `DefaultMemory` so no LLM compaction and no
+full transcript run in the background. `QuestRunner` constructs each
+`AgentState` once after an action executes, then sends that exact object to the
+harness retrieval view, callbacks, and `QuestLogger` for persistence. It reuses
+`ToolCompactHarness`'s tool-select-then-act call budget, so the model gets at
+most one retrieval call before its final action, matching `tool_compact`.
+`DefaultMemory` is the single bounded recent-context source in its select-turn
+prompt; the trajectory contributes no separate recent-context block, only
+on-demand `history_read`/`history_search` retrieval. It is not yet part of the
+public leaderboard taxonomy.