Skip to content

docs: preserve RPAP benchmark status sync #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
LaloLalo1999 wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

docs: preserve RPAP benchmark status sync #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
60206e7
docs(coding-agent): sync MemSWE benchmark status
LaloLalo1999 Jul 9, 2026
cdaf112
Merge remote-tracking branch 'origin/main' into tmp-merge-rpap-luO5RS
LaloLalo1999 Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,14 @@ npm --prefix packages/coding-agent run memswe:smoke
npm --prefix packages/coding-agent run memswe:smoke -- --task-id=repo-gamma-invoice-export-001
npm --prefix packages/coding-agent run memswe:smoke -- --all-tasks --continue-on-task-failure
npm --prefix packages/coding-agent run memswe:smoke -- --condition=no_memory
npm --prefix packages/coding-agent run memswe:smoke -- --condition=full_context --repetition-index=1
```

The smoke runner defaults to the deterministic faux provider and makes no real model/provider API calls unless real mode is explicitly enabled. With `MEMSWE_ALLOW_REAL_MODEL=1` and `--agent-mode=real` it drives a real provider (default `omniroute`, model `azure/deepseek-v4-flash`, via `OMNIROUTE_API_KEY`/`OMNIROUTE_BASE_URL`; legacy `--agent-mode=minimax-real` remains for MiniMax); the Honcho memory adapter can run against Honcho Cloud (`https://api.honcho.dev`, cloud-first with `HONCHO_API_URL` self-host fallback). In all modes it copies MemSWE fixtures into temporary worktrees, keeps hidden/protected verifier assets harness-side, emits artifacts under `.memswe-runs/<timestamp>/<task-id>/`, and writes `suite-summary.json` for all-task runs. See [packages/coding-agent/README.md](packages/coding-agent/README.md#memswe--pap-membench-harness) and [packages/coding-agent/docs/memswe-benchmark-status.html](packages/coding-agent/docs/memswe-benchmark-status.html) for current command, artifact, and readiness details.
The smoke runner defaults to the deterministic faux provider and makes no real model/provider API calls unless real mode is explicitly enabled. With `MEMSWE_ALLOW_REAL_MODEL=1` and `--agent-mode=real` it drives a real provider (default `omniroute`, model `azure/deepseek-v4-flash`, via `OMNIROUTE_API_KEY`/`OMNIROUTE_BASE_URL`; legacy `--agent-mode=minimax-real` remains for MiniMax); the Honcho memory adapter can run against Honcho Cloud (`https://api.honcho.dev`, cloud-first with `HONCHO_API_URL` self-host fallback). In all modes it copies MemSWE fixtures into temporary worktrees, keeps hidden/protected verifier assets harness-side, emits artifacts under `.memswe-runs/<timestamp>/<task-id>/`, and writes `suite-summary.json` for all-task runs.

The evidence export added at `7dca44e1` covers a 240-cell matrix: 10 canonical tasks x 6 scoped conditions (`no_memory`, `full_context`, `hindsight`, `honcho`, `zep`, and `supermemory`) x 4 repetitions. All 240 exported cells report `task_success_visible=1` and `otel_trace_complete=pass`. These are deliberately narrow claims: visible success is saturated and does not distinguish the conditions; hidden success was not established; task-specific memory trace predicates remain unsupported by the current runner; and RAM was not measured. Provider lifecycle execution also does not, by itself, prove that recalled provider memory influenced the graded agent session.

The fixed-runtime rule remains central: comparisons must hold model, prompt template, tools, fixture visibility, verifier rules, repetition policy, and scoring constant while changing only the memory condition/provider. See [packages/coding-agent/README.md](packages/coding-agent/README.md#memswe--pap-membench-harness), the [benchmark status page](packages/coding-agent/docs/memswe-benchmark-status.html), and the [PAP evidence index](docs/pap-sunday-2026-07-05/README.md) for current commands, artifacts, limitations, and charts.

* **[@earendil-works/pi-coding-agent](packages/coding-agent)**: Interactive coding agent CLI
* **[@earendil-works/pi-agent-core](packages/agent)**: Agent runtime with tool calling and state management
Expand Down
34 changes: 34 additions & 0 deletions docs/pap-sunday-2026-07-05/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# PAP delivery evidence index

This directory contains presentation-ready evidence for the MemSWE/PAP-membench delivery at `pi-memswe` commit `7dca44e1`.

## Current evidence

The committed matrix contains 240 cells:

- 10 canonical tasks from the sibling `../memswe` benchmark repository;
- 6 scoped conditions: `no_memory`, `full_context`, `hindsight`, `honcho`, `zep`, and `supermemory`;
- 4 repetitions per task and condition.

All 240 cells report `task_success_visible=1` and `otel_trace_complete=pass`. The first result is a saturated visible-only signal; the second confirms harness trace completeness and successful trace flush. Neither establishes hidden-test success or task-specific memory correctness.

## Deliverables

- [`charts/README.md`](charts/README.md): data-source notes, aggregate table, and limitations.
- [`charts/memswe-cell-matrix.csv`](charts/memswe-cell-matrix.csv): one normalized row per matrix cell.
- [`charts/memswe-aggregate-by-system.csv`](charts/memswe-aggregate-by-system.csv) and [`charts/memswe-aggregate-by-system.json`](charts/memswe-aggregate-by-system.json): per-condition aggregates.
- [`charts/chart-success-rate.png`](charts/chart-success-rate.png): visible-success ceiling across conditions.
- [`charts/chart-latency.png`](charts/chart-latency.png): mean and median task latency.
- [`charts/chart-cost-tokens.png`](charts/chart-cost-tokens.png): mean cost and input/output tokens.
- [`charts/memswe-flow-diagram.md`](charts/memswe-flow-diagram.md): harness execution flow.

## Interpretation limits

- Hidden success is unsupported by the committed matrix evidence.
- RAM/peak-memory usage was not measured.
- The current runner emits sibling-defined task trace predicates as `not_evaluable`; `otel_trace_complete` is a separate diagnostic predicate.
- Aggregate injected-memory share is zero, so the matrix cannot support memory-utilization claims.
- `full_context` prepends prior transcripts, and Honcho has an explicit graded recall/readback path. The generic Hindsight, Zep, and Supermemory condition paths record provider lifecycle probes but do not generally inject provider recall into the graded prompt.
- Comparisons are valid only when model, prompt template, tools, fixture visibility, verifier policy, repetition policy, and scoring remain fixed while the memory condition changes.

Benchmark definitions, task assets, verifier policy, and run-record schema remain owned by the sibling `../memswe` repository. This repository owns the pi execution harness, condition adapters, trace capture, and exported delivery artifacts.
62 changes: 37 additions & 25 deletions packages/coding-agent/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,60 +102,69 @@ Then just talk to pi. By default, pi gives the model four tools: `read`, `write`

## MemSWE / PAP-membench Harness

This fork is the fixed pi runtime/harness for MemSWE experiments. The benchmark source of truth is the sibling `../memswe` repository: task descriptors, fixtures, session prompts, fact lifecycle, verifier policy, trace predicates, and run-record schema belong there. This package supplies the current runner around pi's coding-agent SDK and must keep the runtime fixed while memory conditions/providers vary.
This fork is the fixed pi runtime/harness for MemSWE experiments. The benchmark source of truth is the sibling `../memswe` repository: task descriptors, fixtures, session prompts, fact lifecycle, verifier policy, trace predicates, and run-record schema belong there. This package owns execution, condition adapters, trace capture, verifier orchestration, and run artifacts; changes to benchmark identity belong in `memswe`.

The smoke runner defaults to the faux provider with tools disabled, and can run narrow real-model plumbing checks when explicitly requested. It is meant to audit harness plumbing, verifier isolation, artifact emission, and condition setup before scaled real-model pilots.
The smoke runner defaults to the faux provider with tools disabled, and can run narrow real-model plumbing checks when explicitly requested. It is meant to audit harness plumbing, verifier isolation, artifact emission, and condition setup before scaled real-model pilots. A valid comparison must keep the model, prompt template, tools, fixture visibility, verifier rules, repetition policy, and scoring fixed while changing only the memory condition/provider.
### Commands

Run from the `pi-memswe` repo root:

```bash
# Default gamma smoke: repo-gamma-invoice-export-001, condition no_memory.
# Default gamma plumbing smoke: repo-gamma-invoice-export-001, no_memory, rep 1.
npm --prefix packages/coding-agent run memswe:smoke

# Explicit task.
npm --prefix packages/coding-agent run memswe:smoke -- --task-id=repo-gamma-invoice-export-001

# Discover and run all MemSWE task descriptors, continuing after expected task failures.
# Discover all sibling-repo task descriptors. This all-task path is faux-only.
npm --prefix packages/coding-agent run memswe:smoke -- --all-tasks --continue-on-task-failure

# Explicit condition.
npm --prefix packages/coding-agent run memswe:smoke -- --condition=no_memory
# Explicit condition and 1-based repetition index.
npm --prefix packages/coding-agent run memswe:smoke -- --condition=full_context --repetition-index=1

# Local Hindsight AMS/API smoke. This may call the configured local Hindsight LLM
# provider during retain/recall and should only be run after explicit token approval.
# Provider lifecycle probes. These may call configured services or their LLMs.
npm --prefix packages/coding-agent run memswe:hindsight-smoke
npm --prefix packages/coding-agent run memswe:honcho-smoke
npm --prefix packages/coding-agent run memswe:zep-smoke
npm --prefix packages/coding-agent run memswe:supermemory-smoke

# Task-aware Hindsight condition through the MemSWE runner; use --agent-mode=real only with explicit approval.
npm --prefix packages/coding-agent run memswe:smoke -- --task-id=repo-tau-constraint-recall-001 --condition=hindsight --agent-mode=faux-text
```

Condition support is intentionally narrow:
Supported condition IDs are `no_memory`, `full_context`, `repository_docs`, `filesystem`, `localrag`, `zep`, `supermemory`, `honcho`, `graphiti`, `mem0`, `letta`, and `hindsight`.

- `no_memory`: implemented default baseline for the faux/no-edit smoke runner.
- `repository_docs`: scaffolded materialization path that writes `docs/agent-project-memory/memswe-facts.md` into the temp fixture and records the copied memory doc as an artifact. Treat it as harness plumbing, not a completed benchmark-ready local-memory baseline.
- `full_context`: replays prior non-graded session transcripts before the graded prompt.
- Provider conditions (`filesystem`, `localrag`, `zep`, `supermemory`, `honcho`, `graphiti`, `mem0`, `letta`, `hindsight`): run the corresponding adapter lifecycle smoke and record `condition.memory_system = <provider>` in the run record. Missing cloud/local credentials return provider `status: "skipped"` instead of failing the whole condition setup.
- `no_memory` is the implemented default baseline and injects no memory context.
- `full_context` assembles declared seeded history and prior non-graded transcripts into a preamble for the graded prompt.
- `repository_docs` writes valid facts to `docs/agent-project-memory/memswe-facts.md` in the temporary fixture. The agent still needs tools and instructions that can consume that file; materialization alone is not proof of use.
- `honcho` has a graded path that seeds a run-scoped persistent workspace, injects a recall preamble, writes the graded conclusion back, and requires a positive readback gate.
- The other provider conditions (`filesystem`, `localrag`, `zep`, `supermemory`, `graphiti`, `mem0`, `letta`, and `hindsight`) run their adapter lifecycle probe and record its export before the graded session. Missing cloud/local credentials return provider `status: "skipped"` instead of failing condition setup. The lifecycle result is not generally injected into the graded prompt, so treat these cells as provider/harness integration evidence, not proof that provider recall improved task performance.
- Any other `--condition=...` value is invalid and exits before task execution.

`memswe:hindsight-smoke` is separate provider-readiness plumbing for the Hindsight API. The MemSWE runner's `--condition=hindsight` path invokes the same lifecycle smoke task-aware: the current task id and run id are passed through so a repo-tau canary seeds/recalls repo-tau facts, not the default gamma fixture. Because retain/recall can use the configured Hindsight LLM provider, treat Hindsight smokes as real local AMS/API calls that may incur model/token usage; do not run them without explicit token scope and approval.

`memswe:smoke` defaults to deterministic `--agent-mode=faux-text`. Real-model plumbing checks are available as `--agent-mode=real` (default provider `omniroute` / model `azure/deepseek-v4-flash`, via `OMNIROUTE_API_KEY` and optional `OMNIROUTE_BASE_URL`; override with `MEMSWE_LLM_PROVIDER`/`MEMSWE_LLM_MODEL`) or the legacy `--agent-mode=minimax-real`, which selects pi provider `minimax` / model `MiniMax-M3`. Real modes require `MEMSWE_ALLOW_REAL_MODEL=1`, disable `--all-tasks`, and should only be used after pilot approval. Note the endpoint distinction: the Hindsight/LiteLLM smoke uses the MiniMax token-plan endpoint `https://api.minimax.io/v1`, while pi's built-in MiniMax-M3 entry uses the provider's Anthropic-compatible endpoint `https://api.minimax.io/anthropic`.
`memswe:smoke` defaults to deterministic `--agent-mode=faux-text`. Real-model plumbing checks are available as `--agent-mode=real` (default provider `omniroute` / model `azure/deepseek-v4-flash`, via `OMNIROUTE_API_KEY` and optional `OMNIROUTE_BASE_URL`; override with `MEMSWE_LLM_PROVIDER`/`MEMSWE_LLM_MODEL`) or the legacy `--agent-mode=minimax-real`, which selects pi provider `minimax` / model `MiniMax-M3`. Real modes require `MEMSWE_ALLOW_REAL_MODEL=1`, disable `--all-tasks`, and should only be used after pilot approval. Note the endpoint distinction: the Hindsight/LiteLLM smoke uses the MiniMax token-plan endpoint `https://api.minimax.io/v1`, while pi's built-in MiniMax-M3 entry uses the provider's Anthropic-compatible endpoint `https://api.minimax.io/anthropic`. Provider lifecycle probes can also incur service or model usage; run them only with explicit credential and token scope.

`memswe:report` aggregates ignored `.memswe-runs/**` artifacts into a static browser report at `.memswe-runs/reports/latest/index.html`, with `run-summary.json` beside it. The report includes run records, verifier counts, suite summaries, Hindsight lifecycle smokes, and optional MiniMax/Hermes diagnostic review JSON files from `.memswe-runs/reviews/`.

Expected behavior with the faux/no-edit runner:
### Committed 240-cell evidence export

- The default gamma smoke should pass visible/protected verification.
- In `--all-tasks --continue-on-task-failure`, non-gamma tasks may fail verification explicitly because the faux agent acknowledges prompts and does not edit files. Those failures are expected evidence that verifiers are active, not evidence of real-model quality.
- Hidden verifiers remain isolated and are skipped unless an explicit hidden-run policy is used. Do not expose hidden/protected verifier internals to the agent runtime.
Commit `7dca44e1` adds charts and normalized CSV/JSON built from the latest run record for each of 10 canonical tasks x 6 scoped conditions (`no_memory`, `full_context`, `hindsight`, `honcho`, `zep`, `supermemory`) x 4 repetitions. The export reports:

- 240/240 cells with `task_success_visible=1`;
- 240/240 cells with `otel_trace_complete=pass`;
- per-condition latency, cost, and token aggregates.

Interpret these results narrowly. Visible success is at a 1.00 ceiling for every condition, so it cannot establish a memory-system advantage. Hidden verifiers were not part of the committed matrix evidence, `task_success_hidden` is therefore unsupported, and the runner emits benchmark-specific trace predicates as `not_evaluable`. `otel_trace_complete` only confirms the expected harness spans and successful trace flush. RAM is not recorded. The export also reports zero injected-memory share, so it cannot support utilization or provider-memory-attribution claims.

The committed evidence is under [`docs/pap-sunday-2026-07-05/charts/`](../../docs/pap-sunday-2026-07-05/charts/). Generated `.memswe-runs/` data remains ignored.

### Artifacts

Each task run writes artifacts under:

```text
.memswe-runs/<timestamp>/<task-id>/
.memswe-runs/<timestamp>/<task-id>-<condition-id>-r<repetition-index>/
```

Important files:
Expand All @@ -167,11 +176,14 @@ Important files:
- `verifier-results.json`: harness-side visible/protected verifier command results.
- `skipped-hidden-verifiers.json`: hidden verifier metadata skipped by default.
- `agent.patch`, `worktree-diff.patch`, `changed-files.json`: no-op patch artifacts for faux/no-edit runs; real-model pilots must persist real diffs here.
- `condition-result.json`: selected memory condition and condition-specific artifact paths.
- `repository-docs/memswe-facts.md`: only present for the `repository_docs` scaffold.
- `provider-smoke-<condition>.json`: present for provider conditions, including task-aware Hindsight lifecycle evidence.
- `condition-result.json`: selected memory condition, memory-system label, and condition-specific artifact paths.
- `full-context-transcript.json`: prior transcript bundle for `full_context`.
- `repository-docs/memswe-facts.md`: materialized valid facts for `repository_docs`.
- `provider-smoke-<condition>.json`: lifecycle result for provider conditions other than the Honcho graded path, including task-aware Hindsight evidence.
- `honcho-graded-memory.json` and `honcho-readback.json`: run-scoped seeded/recall and post-session readback evidence for Honcho.
- `memswe-trace.json`: local trace artifact used by the `otel_trace_complete` diagnostic predicate.

The Hindsight local smoke writes `.memswe-runs/<timestamp>/hindsight-local-smoke/hindsight-smoke-result.json` when run directly, while `--condition=hindsight` writes a per-task `provider-smoke-hindsight.json` beside the run record. Both include request/response trace events, predicate results, and structured failure guidance.
Standalone provider smokes write provider-specific lifecycle artifacts. They are readiness evidence, not MemSWE task scores. The Hindsight local smoke writes `.memswe-runs/<timestamp>/hindsight-local-smoke/hindsight-smoke-result.json` when run directly, while `--condition=hindsight` writes a per-task `provider-smoke-hindsight.json` beside the run record. Both include request/response trace events, predicate results, and structured failure guidance.

The report generator writes `.memswe-runs/reports/<timestamp>/index.html` and refreshes `.memswe-runs/reports/latest/index.html`. Generated report artifacts stay ignored; commit the generator, not the reports.

Expand All @@ -185,7 +197,7 @@ The suite summary lists task IDs, pass/fail status, run-record path, failed phas

### Validation

Use these checks after harness edits or docs updates that change command references:
Use these checks after harness edits. Documentation-only changes can use link and static-content checks without running model/provider flows:

```bash
# Targeted runner/helper tests, from packages/coding-agent.
Expand All @@ -197,7 +209,7 @@ cd ../..
npm run check
```

Do not run real model/provider pilots from this smoke path unless API-token scope, local trace readiness, and hidden/protected policy are explicitly approved.
Do not run real model/provider pilots unless API-token scope, local trace readiness, and hidden/protected policy are explicitly approved. Never interpret visible-only success, provider setup, or a complete harness trace as hidden-test success or task-specific memory correctness.

---

Expand Down
Loading