rlm+mempalace

bench_BrowseComp-Plus/run_replicate_rlm_benchmark.py

@@ -195,6 +195,7 @@ def resolve_output_file(
 
 
 # Subagent models used across baselines (gpt-5.4-* and gpt-5-mini share the "mini" bucket).
+# Keep bucket rules in sync with bench_Oolong_real/run_replicate_rlm_benchmark.py.
 _SUBCALL_NANO_MARKERS: tuple[str, ...] = ("gpt-5.4-nano",)
 _SUBCALL_MINI_MARKERS: tuple[str, ...] = ("gpt-5.4-mini", "gpt-5-mini")
 
@@ -205,6 +206,9 @@ def _increment_subcall_bucket(counts: dict[str, int], model: str) -> None:
         counts["nano"] += 1
     elif any(marker in m for marker in _SUBCALL_MINI_MARKERS):
         counts["mini"] += 1
+    elif m and m != "unknown":
+        # Subagent LM call with a model id that does not contain our markers (still count).
+        counts["mini"] += 1
 
 
 def make_subcall_counter() -> tuple[dict[str, int], Callable[[int, str, str], None]]:

bench_Oolong_real/README.md

@@ -66,6 +66,48 @@ Run the root-only baseline (no RLM):
 python bench_Oolong_real/run_benchmark.py --limit 1 --baseline
 ```
 
+### MemPalace PoC (ephemeral search baseline)
+
+Install the optional extra from the repo root (or `pip install -e ../mempalace` plus a compatible `chromadb`):
+
+```bash
+uv pip install -e ".[mempalace-poc]"
+```
+
+**Lenient mode** (default for `--palace-poc`): the full transcript remains in `context['context_window_text']` and is also indexed in a **temporary** Chroma palace per example. The REPL gains `search_memories(query, wing=..., room=..., n_results=...)` for semantic retrieval. The palace directory is deleted after each `completion`.
+
+```bash
+python bench_Oolong_real/run_benchmark.py --limit 1 --palace-poc
+```
+
+**Strict mode** (`--palace-poc-strict`): `context_window_text` is **omitted** from `context`; the model must use `search_memories` to read the transcript. Harder eval; use for ablations only after lenient mode works.
+
+```bash
+python bench_Oolong_real/run_benchmark.py --limit 1 --palace-poc-strict
+```
+
+**Verbose lobby** (`--palace-poc-verbose`): only with palace PoC flags; prints a wing/room tree and the first 50 characters of each indexed drawer (whitespace collapsed) to stdout **right after** the palace is populated and **before** the root RLM run starts. Useful for inspecting chunking.
+
+```bash
+python bench_Oolong_real/run_benchmark.py --limit 1 --palace-poc --palace-poc-verbose
+```
+
+**By-speaker ingest** (`--palace-poc-by-speaker`): requires palace PoC. Parses the Oolong transcript into **`_preamble`** plus **grouped dialogue drawers**: short back-and-forth is merged until a minimum character threshold (default 250, configurable via `build_ephemeral_palace_tools(..., min_drawer_chars=...)`). Each drawer’s Chroma **`room`** is usually the **dominant** speaker (most UTF-8 text in that bundle); **`_mixed`** is used when two speakers tie. All participants are listed in metadata (`speakers_json` / `speaker_slugs`); drawer **text** keeps `Name:` labels. **`search_memories(..., room='<slug>')`** runs a primary `room=<slug>` search and a wing-wide pass, then merges hits so a secondary speaker still surfaces drawers filed under another dominant room. Episode **line** spans are stored as zero-padded `line_start` / `line_end` strings (and optional `line_ranges` JSON) on chunks when applicable. **`list_taxonomy()`** lists **`_preamble` first**, then **`_mixed`** (with a speaker hint when known), then other rooms. Use `room='_preamble'` for instructions/mapping.
+
+```bash
+python bench_Oolong_real/run_benchmark.py --limit 1 --palace-poc --palace-poc-by-speaker
+```
+
+**By-block ingest** (`--palace-poc-by-block`): mutually exclusive with `--palace-poc-by-speaker`. Splits the episode into overlapping **dense** (non-empty) line windows (default 75 lines, 7-line overlap) as **`room = block_001`**, `block_002`, …. **Inside each room**, **interaction grouping** appends lines in order (full `Speaker:` lines preserved) **without** starting a new drawer on every speaker change; a new drawer starts once the running buffer reaches **`min_drawer_chars`** (same parameter as by-speaker grouping, default 250, via `build_ephemeral_palace_tools(..., min_drawer_chars=...)`), or at the end of the window—so rapid back-and-forth stays in one searchable “event.” Each drawer’s metadata includes **`speakers_json`** (all `Name:` speakers in that drawer, first-appearance order), **`speaker_slugs`**, and primary **`speaker`** (first in that list). **`line_start`** / **`line_end`** are zero-padded **1-based indices into the episode’s non-empty lines**. When a drawer closes because it reached ``min_drawer_chars`` and more lines remain in that temporal window, the **triggering line** is also prepended to the **next** drawer (one-line overlap across drawer boundaries). A drawer is split into multiple Chroma rows only if it exceeds the palace **chunk size** (shared line span and speaker metadata on each part). **`list_taxonomy()`** orders **`_preamble`** then blocks **numerically**. Temporal `room` names are time windows, not cast slugs.
+
+```bash
+python bench_Oolong_real/run_benchmark.py --limit 1 --palace-poc --palace-poc-by-block
+```
+
+`run_multi_model_benchmark.py` supports the same flags and writes a `condition` field (`default` | `palace_poc` | `palace_poc_strict` | `palace_poc_by_speaker` | `palace_poc_strict_by_speaker` | `palace_poc_by_block` | `palace_poc_strict_by_block`) into each JSONL row.
+
+`--palace-poc` cannot be combined with `--baseline`.
+
 Run a specific example id with the subagent-encouraging prompt:
 
 ```bash
@@ -270,6 +312,35 @@ Each JSONL line is one trial with fields: `task_id`, `query`, `baseline`, `trial
 
 RLM trajectory logs go under `bench_Oolong_real/logs/`.
 
+## `run_rlm_mempalace_benchmark.py`
+
+Frozen **replicate-scale** suite (same 50 `example_id`s and baselines as `run_replicate_rlm_benchmark.py` by default) with **RLM + ephemeral MemPalace always enabled**. Default transcript ingest is **`by_block`** (temporal block rooms); use `--palace-poc-by-speaker` for grouped-by-speaker rooms. **`--palace-poc-strict`** omits `context_window_text` from the REPL context so the model must use `search_memories`. There is **no** `--palace-poc-verbose`; MemPalace indexing stays quiet (`verbose=False`).
+
+**Episode modes** (mutually exclusive):
+
+- **Default** — 50 single-episode IDs; JSONL rows use **`suite`: `single_50`**.
+- **`--allow_two_episodes`** — 50 two-episode IDs; **`suite`: `two_50`**.
+- **`--mixed_25_episodes`** — first **25** single-episode IDs, then first **25** two-episode IDs (deterministic order), using the default validation JSONLs only; **`--data_path` is not allowed** in this mode. Rows use **`suite`: `mixed_25_25`**.
+
+Per trial, palace teardown runs in a **`finally`** block so temp Chroma state is cleaned up even if `completion` raises.
+
+Results go under **`bench_Oolong_real/replicate_rlm_mempalace_benchmarks/<baseline>[_depthN]/`**, separate from non-palace replicate runs (`replicate_rlm_benchmarks/`). All modes share the same directory layout and `results.jsonl` naming. Each JSONL row includes **`mempalace`**, **`ingest`**, **`condition`**, **`suite`**, **`num_episode`** (length of the example’s `episodes` list, typically 1 or 2), **`runner`**, and **`benchmark_kind`**.
+
+Requires `OPENAI_API_KEY2` (same as other Oolong runners). Example:
+
+```bash
+export OPENAI_API_KEY2=...
+uv run python bench_Oolong_real/run_rlm_mempalace_benchmark.py --baseline mini-root_mini-sub --max_depth 2
+```
+
+Mixed 25+25 example:
+
+```bash
+uv run python bench_Oolong_real/run_rlm_mempalace_benchmark.py --mixed_25_episodes --baseline mini-root_mini-sub
+```
+
+Use `--timestamped` / `--system_prompt` the same way as `run_replicate_rlm_benchmark.py`. With `--resume`, skips use **`(task_id, baseline, trial, condition, suite)`** so different strict/ingest settings and episode suites do not collide. Rows written before **`suite`** existed resume with an empty suite token for those lines only.
+
 ## Subagent model selection
 
 The core [RLM](cci:2://file:///Users/dannyyu/Desktop/rlm/rlm/core/rlm.py:64:0-1181:20) library now supports separate child-model configuration with:

bench_Oolong_real/rlm_mempalace_benchmarks/mini-root_mini-sub/latest.jsonl

@@ -0,0 +1 @@
+results.jsonl