From 60206e7f4746dd06de6fa89be800143285ce1f52 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduardo=20Javier=20Garc=C3=ADa=20L=C3=B3pez?= <6173685+LaloLalo1999@users.noreply.github.com> Date: Thu, 9 Jul 2026 16:52:33 -0600 Subject: [PATCH] docs(coding-agent): sync MemSWE benchmark status --- README.md | 7 +- docs/pap-sunday-2026-07-05/README.md | 34 +++ packages/coding-agent/README.md | 63 ++-- .../docs/memswe-benchmark-status.html | 289 ++++++++---------- 4 files changed, 206 insertions(+), 187 deletions(-) create mode 100644 docs/pap-sunday-2026-07-05/README.md diff --git a/README.md b/README.md index 08719d75897..b028adfe048 100644 --- a/README.md +++ b/README.md @@ -31,9 +31,14 @@ npm --prefix packages/coding-agent run memswe:smoke npm --prefix packages/coding-agent run memswe:smoke -- --task-id=repo-gamma-invoice-export-001 npm --prefix packages/coding-agent run memswe:smoke -- --all-tasks --continue-on-task-failure npm --prefix packages/coding-agent run memswe:smoke -- --condition=no_memory +npm --prefix packages/coding-agent run memswe:smoke -- --condition=full_context --repetition-index=1 ``` -The smoke runner uses the deterministic faux provider and does not call real model/provider APIs. It copies MemSWE fixtures into temporary worktrees, keeps hidden/protected verifier assets harness-side, emits artifacts under `.memswe-runs///`, and writes `suite-summary.json` for all-task runs. See [packages/coding-agent/README.md](packages/coding-agent/README.md#memswe--pap-membench-harness) and [packages/coding-agent/docs/memswe-benchmark-status.html](packages/coding-agent/docs/memswe-benchmark-status.html) for current command, artifact, and readiness details. +The runner defaults to the deterministic faux provider, but it also has an explicitly gated real-model mode and condition adapters. It copies MemSWE fixtures into temporary worktrees, keeps hidden/protected verifier assets harness-side, and emits run records and supporting artifacts under `.memswe-runs/`. + +At committed HEAD `7dca44e1`, the project includes an evidence export for a 240-cell matrix: 10 canonical tasks x 6 scoped conditions (`no_memory`, `full_context`, `hindsight`, `honcho`, `zep`, and `supermemory`) x 4 repetitions. All 240 exported cells report `task_success_visible=1` and `otel_trace_complete=pass`. These are deliberately narrow claims: visible success is saturated and does not distinguish the conditions; hidden success was not established; task-specific memory trace predicates remain unsupported by the current runner; and RAM was not measured. Provider lifecycle execution also does not, by itself, prove that recalled provider memory influenced the graded agent session. + +The fixed-runtime rule remains central: comparisons must hold model, prompt template, tools, fixture visibility, verifier rules, repetition policy, and scoring constant while changing only the memory condition/provider. See [packages/coding-agent/README.md](packages/coding-agent/README.md#memswe--pap-membench-harness), the [benchmark status page](packages/coding-agent/docs/memswe-benchmark-status.html), and the [PAP evidence index](docs/pap-sunday-2026-07-05/README.md) for current commands, artifacts, limitations, and charts. * **[@earendil-works/pi-coding-agent](packages/coding-agent)**: Interactive coding agent CLI * **[@earendil-works/pi-agent-core](packages/agent)**: Agent runtime with tool calling and state management diff --git a/docs/pap-sunday-2026-07-05/README.md b/docs/pap-sunday-2026-07-05/README.md new file mode 100644 index 00000000000..7fa93a5f7a9 --- /dev/null +++ b/docs/pap-sunday-2026-07-05/README.md @@ -0,0 +1,34 @@ +# PAP delivery evidence index + +This directory contains presentation-ready evidence for the MemSWE/PAP-membench delivery at `pi-memswe` commit `7dca44e1`. + +## Current evidence + +The committed matrix contains 240 cells: + +- 10 canonical tasks from the sibling `../memswe` benchmark repository; +- 6 scoped conditions: `no_memory`, `full_context`, `hindsight`, `honcho`, `zep`, and `supermemory`; +- 4 repetitions per task and condition. + +All 240 cells report `task_success_visible=1` and `otel_trace_complete=pass`. The first result is a saturated visible-only signal; the second confirms harness trace completeness and successful trace flush. Neither establishes hidden-test success or task-specific memory correctness. + +## Deliverables + +- [`charts/README.md`](charts/README.md): data-source notes, aggregate table, and limitations. +- [`charts/memswe-cell-matrix.csv`](charts/memswe-cell-matrix.csv): one normalized row per matrix cell. +- [`charts/memswe-aggregate-by-system.csv`](charts/memswe-aggregate-by-system.csv) and [`charts/memswe-aggregate-by-system.json`](charts/memswe-aggregate-by-system.json): per-condition aggregates. +- [`charts/chart-success-rate.png`](charts/chart-success-rate.png): visible-success ceiling across conditions. +- [`charts/chart-latency.png`](charts/chart-latency.png): mean and median task latency. +- [`charts/chart-cost-tokens.png`](charts/chart-cost-tokens.png): mean cost and input/output tokens. +- [`charts/memswe-flow-diagram.md`](charts/memswe-flow-diagram.md): harness execution flow. + +## Interpretation limits + +- Hidden success is unsupported by the committed matrix evidence. +- RAM/peak-memory usage was not measured. +- The current runner emits sibling-defined task trace predicates as `not_evaluable`; `otel_trace_complete` is a separate diagnostic predicate. +- Aggregate injected-memory share is zero, so the matrix cannot support memory-utilization claims. +- `full_context` prepends prior transcripts, and Honcho has an explicit graded recall/readback path. The generic Hindsight, Zep, and Supermemory condition paths record provider lifecycle probes but do not generally inject provider recall into the graded prompt. +- Comparisons are valid only when model, prompt template, tools, fixture visibility, verifier policy, repetition policy, and scoring remain fixed while the memory condition changes. + +Benchmark definitions, task assets, verifier policy, and run-record schema remain owned by the sibling `../memswe` repository. This repository owns the pi execution harness, condition adapters, trace capture, and exported delivery artifacts. diff --git a/packages/coding-agent/README.md b/packages/coding-agent/README.md index 334f084af27..05ebbed00a2 100644 --- a/packages/coding-agent/README.md +++ b/packages/coding-agent/README.md @@ -102,57 +102,64 @@ Then just talk to pi. By default, pi gives the model four tools: `read`, `write` ## MemSWE / PAP-membench Harness -This fork is the fixed pi runtime/harness for MemSWE experiments. The benchmark source of truth is the sibling `../memswe` repository: task descriptors, fixtures, session prompts, fact lifecycle, verifier policy, trace predicates, and run-record schema belong there. This package supplies the current runner around pi's coding-agent SDK and must keep the runtime fixed while memory conditions/providers vary. +This fork is the fixed pi runtime/harness for MemSWE experiments. The benchmark source of truth is the sibling `../memswe` repository: task descriptors, fixtures, session prompts, fact lifecycle, verifier policy, trace predicates, and run-record schema belong there. This package owns execution, condition adapters, trace capture, verifier orchestration, and run artifacts; changes to benchmark identity belong in `memswe`. -The current smoke runner uses the faux provider, disables agent tools, and does not call real model/provider APIs. It is meant to audit harness plumbing, verifier isolation, artifact emission, and condition setup before real-model pilots. +The runner defaults to the deterministic faux provider. Real-model execution is available only through an explicit environment gate. A valid comparison must keep the model, prompt template, tools, fixture visibility, verifier rules, repetition policy, and scoring fixed while changing only the memory condition/provider. ### Commands Run from the `pi-memswe` repo root: ```bash -# Default gamma smoke: repo-gamma-invoice-export-001, condition no_memory. +# Default gamma plumbing smoke: repo-gamma-invoice-export-001, no_memory, rep 1. npm --prefix packages/coding-agent run memswe:smoke # Explicit task. npm --prefix packages/coding-agent run memswe:smoke -- --task-id=repo-gamma-invoice-export-001 -# Discover and run all MemSWE task descriptors, continuing after expected task failures. +# Discover all sibling-repo task descriptors. This all-task path is faux-only. npm --prefix packages/coding-agent run memswe:smoke -- --all-tasks --continue-on-task-failure -# Explicit condition. -npm --prefix packages/coding-agent run memswe:smoke -- --condition=no_memory +# Explicit condition and 1-based repetition index. +npm --prefix packages/coding-agent run memswe:smoke -- --condition=full_context --repetition-index=1 -# Local Hindsight AMS/API smoke. This may call the configured local Hindsight LLM -# provider during retain/recall and should only be run after explicit token approval. +# Provider lifecycle probes. These may call configured services or their LLMs. npm --prefix packages/coding-agent run memswe:hindsight-smoke +npm --prefix packages/coding-agent run memswe:honcho-smoke +npm --prefix packages/coding-agent run memswe:zep-smoke +npm --prefix packages/coding-agent run memswe:supermemory-smoke ``` -Condition support is intentionally narrow: +Supported condition IDs are `no_memory`, `full_context`, `repository_docs`, `filesystem`, `localrag`, `zep`, `supermemory`, `honcho`, `graphiti`, `mem0`, `letta`, and `hindsight`. -- `no_memory`: implemented default baseline for the faux/no-edit smoke runner. -- `repository_docs`: scaffolded materialization path that writes `docs/agent-project-memory/memswe-facts.md` into the temp fixture and records the copied memory doc as an artifact. Treat it as harness plumbing, not a completed benchmark-ready local-memory baseline. -- `full_context` and `hindsight`: accepted as known condition IDs, but fail closed as not implemented in the current runner. -- Any other `--condition=...` value is invalid and exits before task execution. +- `no_memory` injects no memory context. +- `full_context` assembles declared seeded history and prior non-graded transcripts into a preamble for the graded prompt. +- `repository_docs` writes valid facts to `docs/agent-project-memory/memswe-facts.md` in the temporary fixture. The agent still needs tools and instructions that can consume that file; materialization alone is not proof of use. +- `honcho` has a graded path that seeds a run-scoped persistent workspace, injects a recall preamble, writes the graded conclusion back, and requires a positive readback gate. +- The other provider conditions run their adapter lifecycle probe and record its export before the graded session. In the current runner, that lifecycle result is not generally injected into the graded prompt. Treat these cells as provider/harness integration evidence, not proof that provider recall improved task performance. -`memswe:hindsight-smoke` is separate from `memswe:smoke`: it targets a local Hindsight API, creates/resets a smoke bank, seeds task-YAML-derived gamma facts, performs retain/recall/delete, and writes `hindsight-smoke-result.json`. Because retain/recall can use the configured Hindsight LLM provider, treat this as a real local AMS/API smoke that may incur model/token usage; do not run it without explicit token scope and approval. - -`memswe:smoke` defaults to deterministic `--agent-mode=faux-text`. A narrow real-model plumbing check is available as `--agent-mode=minimax-real`, which selects pi provider `minimax` / model `MiniMax-M3`, requires both `MEMSWE_ALLOW_REAL_MODEL=1` and `MINIMAX_API_KEY` in the environment, disables agent tools, and is blocked for `--all-tasks` to avoid accidental multi-task spend. Note the endpoint distinction: the Hindsight/LiteLLM smoke uses the MiniMax token-plan endpoint `https://api.minimax.io/v1`, while pi's built-in MiniMax-M3 entry uses the provider's Anthropic-compatible endpoint `https://api.minimax.io/anthropic`. +`memswe:smoke` defaults to `--agent-mode=faux-text`. `--agent-mode=real` requires `MEMSWE_ALLOW_REAL_MODEL=1`, an API key (`MEMSWE_LLM_API_KEY` or `OMNIROUTE_API_KEY`), and the configured provider/model/base URL. `--agent-mode=minimax-real` is the older narrow MiniMax path. Real modes disable agent tools in this runner, and `--all-tasks` is intentionally restricted to faux mode to prevent accidental multi-task spend. Provider lifecycle probes can also incur service or model usage; run them only with explicit credential and token scope. `memswe:report` aggregates ignored `.memswe-runs/**` artifacts into a static browser report at `.memswe-runs/reports/latest/index.html`, with `run-summary.json` beside it. The report includes run records, verifier counts, suite summaries, Hindsight lifecycle smokes, and optional MiniMax/Hermes diagnostic review JSON files from `.memswe-runs/reviews/`. -Expected behavior with the faux/no-edit runner: +### Committed 240-cell evidence export + +Commit `7dca44e1` adds charts and normalized CSV/JSON built from the latest run record for each of 10 canonical tasks x 6 scoped conditions (`no_memory`, `full_context`, `hindsight`, `honcho`, `zep`, `supermemory`) x 4 repetitions. The export reports: + +- 240/240 cells with `task_success_visible=1`; +- 240/240 cells with `otel_trace_complete=pass`; +- per-condition latency, cost, and token aggregates. + +Interpret these results narrowly. Visible success is at a 1.00 ceiling for every condition, so it cannot establish a memory-system advantage. Hidden verifiers were not part of the committed matrix evidence, `task_success_hidden` is therefore unsupported, and the runner emits benchmark-specific trace predicates as `not_evaluable`. `otel_trace_complete` only confirms the expected harness spans and successful trace flush. RAM is not recorded. The export also reports zero injected-memory share, so it cannot support utilization or provider-memory-attribution claims. -- The default gamma smoke should pass visible/protected verification. -- In `--all-tasks --continue-on-task-failure`, non-gamma tasks may fail verification explicitly because the faux agent acknowledges prompts and does not edit files. Those failures are expected evidence that verifiers are active, not evidence of real-model quality. -- Hidden verifiers remain isolated and are skipped unless an explicit hidden-run policy is used. Do not expose hidden/protected verifier internals to the agent runtime. +The committed evidence is under [`docs/pap-sunday-2026-07-05/charts/`](../../docs/pap-sunday-2026-07-05/charts/). Generated `.memswe-runs/` data remains ignored. ### Artifacts Each task run writes artifacts under: ```text -.memswe-runs/// +.memswe-runs//--r/ ``` Important files: @@ -164,10 +171,14 @@ Important files: - `verifier-results.json`: harness-side visible/protected verifier command results. - `skipped-hidden-verifiers.json`: hidden verifier metadata skipped by default. - `agent.patch`, `worktree-diff.patch`, `changed-files.json`: no-op patch artifacts for faux/no-edit runs; real-model pilots must persist real diffs here. -- `condition-result.json`: selected memory condition and condition-specific artifact paths. -- `repository-docs/memswe-facts.md`: only present for the `repository_docs` scaffold. +- `condition-result.json`: selected memory condition, memory-system label, and condition-specific artifact paths. +- `full-context-transcript.json`: prior transcript bundle for `full_context`. +- `repository-docs/memswe-facts.md`: materialized valid facts for `repository_docs`. +- `provider-smoke-.json`: lifecycle result for provider conditions other than the Honcho graded path. +- `honcho-graded-memory.json` and `honcho-readback.json`: run-scoped seeded/recall and post-session readback evidence for Honcho. +- `memswe-trace.json`: local trace artifact used by the `otel_trace_complete` diagnostic predicate. -The Hindsight local smoke writes `.memswe-runs//hindsight-local-smoke/hindsight-smoke-result.json` with request/response trace events, predicate results, and structured failure guidance. It is provider-readiness evidence only, not a MemSWE task score. +Standalone provider smokes write provider-specific lifecycle artifacts. They are readiness evidence, not MemSWE task scores. The report generator writes `.memswe-runs/reports//index.html` and refreshes `.memswe-runs/reports/latest/index.html`. Generated report artifacts stay ignored; commit the generator, not the reports. @@ -181,7 +192,7 @@ The suite summary lists task IDs, pass/fail status, run-record path, failed phas ### Validation -Use these checks after harness edits or docs updates that change command references: +Use these checks after harness edits. Documentation-only changes can use link and static-content checks without running model/provider flows: ```bash # Targeted runner/helper tests, from packages/coding-agent. @@ -193,7 +204,7 @@ cd ../.. npm run check ``` -Do not run real model/provider pilots from this smoke path unless API-token scope, local trace readiness, and hidden/protected policy are explicitly approved. +Do not run real model/provider pilots unless API-token scope, local trace readiness, and hidden/protected policy are explicitly approved. Never interpret visible-only success, provider setup, or a complete harness trace as hidden-test success or task-specific memory correctness. --- diff --git a/packages/coding-agent/docs/memswe-benchmark-status.html b/packages/coding-agent/docs/memswe-benchmark-status.html index 4b597f73ed0..e0a8346525b 100644 --- a/packages/coding-agent/docs/memswe-benchmark-status.html +++ b/packages/coding-agent/docs/memswe-benchmark-status.html @@ -260,211 +260,185 @@
- Default gamma smoke: passing - All-task loop: scaffolded - Working tree not committed - Generated 2026-06-17 03:26 UTC + 240 committed evidence cells + Harness traces complete + Visible-only success ceiling + Updated 2026-07-09

MemSWE / PAP-membench benchmark status

-

Consolidated status of the pi-memswe fixed runtime/harness fork: current faux-provider smoke commands, guarded MiniMax real-model plumbing checks, all-task suite loop, condition scaffold, artifacts, and next steps without overclaiming benchmark performance.

+

Artifact-backed status of the fixed pi runtime/harness fork at 7dca44e1, including the 10-task × 6-condition × 4-repetition evidence matrix and the limits of what that matrix establishes.

-
Default smoke run
-
PASS
-
Gamma task, faux agent, no_memory
+
Matrix cells
+
240
+
10 tasks × 6 conditions × 4 reps
-
All-task loop
-
6 tasks
-
Continues and writes suite summary
+
Canonical tasks
+
10
+
Selected from the sibling benchmark repo
-
Conditions
-
2 scaffolded
-
no_memory verified; repository_docs materializes docs
+
Scoped conditions
+
6
+
2 baselines + 4 provider labels
-
Repo check
-
PASS
-
npm run check
+
Repetitions
+
4
+
Per task and condition
-

What has been completed

+

What is established

- Fixed-runtime harness role documented - pi-memswe is treated as the fixed runtime/harness fork. The benchmark source of truth remains ../memswe; memory condition/provider behavior is the evaluated variable. + Benchmark/runtime ownership is separated + ../memswe owns tasks, fixtures, sessions, facts, verifier policy, trace predicates, and schemas. This fork owns execution, adapters, traces, and artifacts.
- Smoke runner - packages/coding-agent/scripts/memswe-smoke-runner.ts creates temp fixture worktrees, runs pi createAgentSession(...) sessions, captures agent artifacts, runs harness-side verifiers, validates run-record shape, and emits run-record.json. Default mode remains deterministic --agent-mode=faux-text. + Condition and repetition orchestration exists + The runner accepts a 1-based repetition index and condition IDs for baselines and provider adapters, creates run-scoped workspaces, invokes the fixed pi session path, and validates uam-run.v0.1 output against the sibling schema.
- Verifier environment and patch artifacts - The runner initializes a git baseline, captures agent.patch, worktree-diff.patch, and changed-files.json, creates a Python venv, runs task setup commands, and keeps hidden/protected assets out of agent-visible context. + Isolation and deterministic evidence exist + Fixtures are copied into temporary worktrees; patch, message, verifier, condition, and trace artifacts are retained. Hidden/protected assets remain harness-controlled.
- All-task suite loop - --all-tasks --continue-on-task-failure discovers six MemSWE task descriptors, records pass/fail per task, and writes .memswe-runs/<timestamp>/suite-summary.json. Faux/no-edit failures on non-gamma tasks are expected verifier evidence. + 240-cell export is committed + The chart bundle contains one normalized row for every selected task/condition/repetition cell plus per-system aggregates and presentation-ready charts.
- Condition scaffold - --condition=no_memory is the verified default. repository_docs currently materializes remembered facts into project docs and records a condition artifact, but remains harness plumbing rather than a completed benchmark baseline. full_context and hindsight fail closed in memswe:smoke; the separate memswe:hindsight-smoke command is a real local AMS/API probe. + Trace plumbing is complete for the export + All 240 cells report otel_trace_complete=pass, meaning required harness spans were present and the configured trace flush succeeded.
-

Current repository state

+

Evidence boundaries

- + - - - + + + - - - + + + - - - + + + + + + + + + + + + +
RepoStatusNotes
ClaimStatusInterpretation
pi-memswemodifiedCurrent branch tracks origin/main. Latest local commit is 4854e7a9. Before this docs pass, one script file was already modified; this status page and READMEs are additional uncommitted docs changes.Visible success240/240Every exported cell passed its visible verifier. The signal is saturated at 1.00 for every condition.
memsweread-only dependencyBenchmark/spec repo at ../memswe remains source of truth for tasks, fixtures, sessions, facts, verifier metadata, and run-record schema.Trace completeness240/240Harness trace structure and flush completed. This is not a task-specific memory predicate.
memorybench-memswereference onlyReference patterns may inform provider isolation and reports, but this harness is not coupled to MemoryBench.Hidden successunsupportedThe committed export does not establish hidden-verifier results.
Memory behaviornot evaluableDeclared task predicates are emitted as not_evaluable; injected-memory share is zero in the aggregate export.
RAMnot measuredNo committed matrix field supports memory-consumption comparisons.
- Scope note: this page describes current local harness status. It does not claim benchmark-spec ownership, real-model performance, Hindsight readiness, or a completed repository_docs baseline. + Conclusion: the matrix demonstrates repeatable harness execution, visible-verifier completion, trace capture, and cost/latency/token collection. It does not demonstrate that one memory system improves correctness.
-

Latest verified harness results

-

Representative commands run from pi-memswe. The first two use only the faux provider and harness-side verifiers; the MiniMax command is a separate guarded provider-plumbing smoke:

- 
npm --prefix packages/coding-agent run memswe:smoke
-npm --prefix packages/coding-agent run memswe:smoke -- --all-tasks --continue-on-task-failure
-MEMSWE_ALLOW_REAL_MODEL=1 MINIMAX_API_KEY=<redacted> npm --prefix packages/coding-agent run memswe:smoke -- --task-id=repo-gamma-invoice-export-001 --condition=no_memory --agent-mode=minimax-real
-cd packages/coding-agent && node ../../node_modules/vitest/dist/cli.js --run test/suite/memswe-smoke-runner.test.ts
-npm run check
-
-
-

Smoke runner behavior

-
    -
  • Default gamma: passes visible/protected verification with faux/no-edit agent behavior.
    • -
  • MiniMax real-model mode: available only with explicit --agent-mode=minimax-real; selects pi provider minimax / model MiniMax-M3, requires MEMSWE_ALLOW_REAL_MODEL=1 plus an environment API key, disables agent tools, and is blocked for --all-tasks to avoid accidental multi-task spend.
    • -
  • All-task suite: six task descriptors discovered; latest summary recorded five expected verifier failures and one gamma pass.
    • -
  • Hidden verifier policy: hidden verifiers remain isolated and skipped by default unless explicitly enabled by harness policy.
    • -
  • Condition scaffold: no_memory default writes condition-result.json; repository_docs additionally writes a copied project-memory doc artifact.
    • -
  • Run records: runner performs local shape validation before writing run-record.json.
    • -
-
-
-

Validation commands

-
    -
  • cd packages/coding-agent && node ../../node_modules/vitest/dist/cli.js --run test/suite/memswe-smoke-runner.test.ts
    • -
  • npm run check from repo root.
    • -
  • All-task smoke is expected to exit nonzero when non-gamma faux/no-edit tasks fail verification; use --continue-on-task-failure to collect suite-summary.json.
    • -
-
-
-

Representative artifact paths

- 
pi-memswe/.memswe-runs/2026-06-16T22-04-26-416Z/suite-summary.json
-pi-memswe/.memswe-runs/2026-06-17T03-26-40-597Z/repo-gamma-invoice-export-001/run-record.json
-pi-memswe/.memswe-runs/2026-06-16T23-29-10-817Z/repo-gamma-invoice-export-001/run-record.json
-pi-memswe/.memswe-runs/2026-06-16T23-29-10-817Z/repo-gamma-invoice-export-001/condition-result.json
-pi-memswe/.memswe-runs/2026-06-16T23-29-10-817Z/repo-gamma-invoice-export-001/repository-docs/memswe-facts.md
-pi-memswe/.memswe-runs/<timestamp>/<task-id>/agent.patch
-pi-memswe/.memswe-runs/<timestamp>/<task-id>/worktree-diff.patch
-pi-memswe/.memswe-runs/<timestamp>/<task-id>/changed-files.json
+

Matrix summary

+

Each condition has 40 exported cells. Visible success is identical; secondary metrics differ but should be treated as descriptive because the primary correctness signal is saturated.

+ + + + + + + + + + +
ConditionVisible successMean latencyMean costMean tokens
no_memory1.0020.4 s$0.000122599
full_context1.003.8 s$0.000118716
hindsight1.0018.8 s$0.000113567
honcho1.0014.5 s$0.000109649
zep1.0023.0 s$0.000145678
supermemory1.0015.0 s$0.000086470
+

Source: docs/pap-sunday-2026-07-05/charts/memswe-aggregate-by-system.json. Latest run record per cell was selected; repetition 98/99 sanity runs were excluded.

-

Benchmark status

+

Fixed-runtime controls

    -
  • Task schema exists: MemSWE task descriptors use uam-task.v0.1 in ../memswe.
    • -
  • Run-record schema exists: harness-emitted records target uam-run.v0.1; the runner currently performs local shape validation.
    • -
  • Phase 0 task inventory exists: six task descriptors are discoverable by the harness.
    • -
  • Worked smoke target: repo-gamma-invoice-export-001 is runnable and passing through the current faux/no-edit smoke harness.
    • -
  • Verifiers are deterministic-first: visible/protected tests run separately from the agent-visible fixture; hidden verifiers are isolated and skipped by default.
    • -
  • Trace predicates exist: current faux-agent runs mark them not_evaluable because no real memory/tool-use traces are generated yet.
    • +
  • Same model and pricing configuration.
    • +
  • Same graded prompt and prompt template.
    • +
  • Same tool availability and fixture visibility.
    • +
  • Same visible/hidden/protected verifier policy.
    • +
  • Same repetition count and scoring rules.
    • +
  • Only memory delivery/provider may vary.
+

These are experimental invariants. The committed chart CSV does not itself carry every control field, so the underlying run records remain the audit source.

-

Harness status

+

Condition semantics

    -
  • Runtime path: pi coding-agent SDK via createAgentSession(...).
    • -
  • Default agent model: deterministic faux provider, no paid/real model calls unless --agent-mode=minimax-real is explicitly selected.
    • -
  • Current tools: no agent tools in smoke mode; verifiers run harness-side.
    • -
  • Fixture isolation: each task fixture is copied into a temporary worktree with an initialized git baseline.
    • -
  • Verifier isolation: visible/protected verifier files are copied into the temp worktree; hidden verifier metadata is skipped unless explicitly included.
    • -
  • All-task execution: --all-tasks --continue-on-task-failure iterates all six descriptors and emits suite-summary.json.
    • -
  • Condition handling: no_memory is implemented as the default; repository_docs writes project-memory docs as a scaffold; full_context and hindsight are known but not implemented.
    • -
  • Artifact emission: agent events/messages/final response, setup result, patch artifacts, condition result, verifier results, skipped hidden list, and run record.
    • +
  • no_memory: no memory preamble.
    • +
  • full_context: prior declared transcripts are prepended to the graded prompt.
    • +
  • honcho: run-scoped persistent workspace, recall preamble, post-session write, and positive readback gate.
    • +
  • hindsight, zep, supermemory: current generic provider path records lifecycle-probe output; it does not generally inject recalled provider content into the graded prompt.
+

Therefore, provider-labeled cells outside the explicit Honcho path are integration cells, not clean evidence of memory-conditioned agent behavior.

-

Memory-system readiness

+

Current interfaces and artifacts

- + - - - - + + + - - - - + + + - - - - + + + - - - - + + + - - - - - - - - - - + + +
System / conditionReadinessWhat works nowRemaining work
SurfaceCurrent behaviorCaveat
no_memoryimplemented smoke baselineDefault condition; agent sees current prompt and fixture state only, with no memory injection.Needs full multi-session orchestration beyond current single graded-session smoke.--conditionBaselines plus filesystem/local RAG and seven provider IDs.A known ID does not imply graded recall injection.
repository_docsscaffold onlyWrites current valid facts to docs/agent-project-memory/memswe-facts.md in the temp worktree and records the doc artifact.Not benchmark-ready: still needs fact materialization policy review, supersede/forget handling, session sequencing, and validation as a local-memory baseline.--repetition-indexPositive, 1-based repetition carried into run ID and artifact path.The caller orchestrates the four-repetition matrix.
full_contextknown but not implementedRecognized as a condition ID in the CLI scaffold.Implement transcript assembly/injection and fixed-runtime controls.run-record.jsonSchema-validated condition, session, reward, metrics, predicates, and output locations.Many advanced memory metrics are currently null or zero.
hindsightpilot targetRecognized as a condition ID only in memswe:smoke. Separate memswe:hindsight-smoke can make real local Hindsight retain/recall/delete API calls and may incur configured model/token usage.Local Docker/API pilot requires explicit token scope, bank reset proof, retain/recall/delete behavior, settle polling, trace export, and complete local trace readiness.memswe-trace.jsonBenchmark, memory-prepare, agent-session, verifier, and scoring trace structure.Completeness is separate from semantic predicate correctness.
filesystem / ragreference baseline patternsExisting MemoryBench providers can inform simple file/RAG baselines.Adapt to code-session facts, lifecycle state, scope IDs, and run-record-compatible traces.
supermemory, mem0, zepexternal-service candidatesProvider adapters exist in MemoryBench reference repo.Require API/service setup and explicit inclusion; not immediate local Phase 0 conditions.Patch/verifier artifactsAgent patch, worktree diff, changed files, verifier results, and skipped-hidden list.Hidden assets must never enter the agent-visible fixture.
@@ -474,64 +448,59 @@

Memory-system readiness

Known gaps and risks

-

Harness gaps

+

Measurement gaps

    -
  • All-task loop exists, but current faux/no-edit behavior is only a plumbing check; it is not a real agent performance run.
    • -
  • Current smoke executes a single graded session per task, not full multi-session memory-condition execution.
    • -
  • Faux agent does not edit files, so trace predicates remain diagnostic/not evaluable.
    • -
  • repository_docs materialization is scaffold-only and needs policy/forget validation before being claimed as a baseline.
    • -
  • Hidden tests are correctly isolated by default, but hidden-run mode still needs explicit policy around when it is allowed.
    • +
  • Visible correctness is saturated and cannot rank systems.
    • +
  • No hidden-success conclusion is supported.
    • +
  • No RAM or peak-memory measurements are present.
    • +
  • Injected-memory share is zero in the committed aggregate.
    • +
  • Task-specific trace predicates are not evaluated.
-

Benchmark gaps

+

Integration gaps

    -
  • Need stronger aggregation across task, condition, model, and repetition.
    • -
  • Need real patch capture and diff constraints for agent runs.
    • -
  • Need normalized memory trace JSONL for retrieval/write/delete events.
    • -
  • Need provider adapter contract implemented in pi-memswe.
    • -
  • Need Hindsight local reset/delete behavior proven before claiming AMS readiness.
    • +
  • Generic provider lifecycle success can coexist with no provider recall in the graded prompt.
    • +
  • Normalized retrieval/write/delete/use evidence is incomplete.
    • +
  • All-task execution remains faux-only; real matrices require external orchestration of individual cells.
    • +
  • The chart CSV omits some experimental-control metadata available in full run records.
    • +
  • Ignored raw run artifacts must be archived separately for full reproducibility.
-

Recommended next steps

+

Required next evidence

    -
  1. Keep README/status docs current. Update this page after harness milestones and include validation output with each docs pass.
    2. -
  2. Promote smoke phases into explicit harness phases. Split current script into load_task, materialize_fixture, prepare_environment, run_agent_session, verify_artifacts, and emit_run_record.
    3. -
  3. Finish local baselines. Keep no_memory fixed, then complete and validate repository_docs before claiming it as a benchmark-ready condition.
    4. -
  4. Implement full-context replay. Assemble session history without changing model, prompt template, tools, fixture visibility, verifier rules, repetition policy, or scoring.
    5. -
  5. Run full session sequence for one task. Execute s1, s2, and s3 with condition-specific history delivery, not only the graded session.
    6. -
  6. Capture real patch/tool artifacts. For real agent runs, persist final diff, tool calls, stdout/stderr, final answer, injected context spans, and trace IDs.
    7. -
  7. Make trace predicates evaluable. Connect predicate checks to retrieved memories, injected content, tool arguments, final response, and code diffs.
    8. -
  8. Hindsight pilot spike. Use npm --prefix packages/coding-agent run memswe:hindsight-smoke only after explicit token/API approval. It creates/resets a smoke bank, seeds task-YAML-derived gamma facts, runs retain/recall/delete against local Hindsight, and records provider-facing request/response traces and failure guidance.
    9. -
  9. Real model / MiniMax pilot. Requires explicit API token approval plus complete local artifact/trace readiness; do not use this smoke path for unapproved provider calls.
    10. -
  10. Aggregate reporting. Add a report builder over .memswe-runs/ that produces per-condition/task/model/repetition summaries aligned with MemSWE measurement docs.
    11. +
  11. Inject and capture normalized recall output for every provider condition used in a graded comparison.
    12. +
  12. Evaluate each sibling-defined trace predicate with evidence references.
    13. +
  13. Run the approved hidden/protected policy from harness-controlled paths and report hidden results separately.
    14. +
  14. Add RAM/peak-memory instrumentation if resource efficiency is an RPAP claim.
    15. +
  15. Archive immutable run records with model, prompt, tools, fixture, verifier, and code-revision identifiers.
    16. +
  16. Use a harder task set or stronger scoring vector to avoid the visible-success ceiling.
-

Immediate command checklist

- 
# from pi-memswe
+      
Command and artifact reference

+      
# deterministic plumbing smoke
 npm --prefix packages/coding-agent run memswe:smoke
-npm --prefix packages/coding-agent run memswe:smoke -- --condition=no_memory
+npm --prefix packages/coding-agent run memswe:smoke -- --condition=full_context --repetition-index=1
+
+# faux-only discovery loop
+npm --prefix packages/coding-agent run memswe:smoke -- --all-tasks --continue-on-task-failure
 
-# targeted helper/runner tests
+# targeted runner tests (no provider calls)
 cd packages/coding-agent
 node ../../node_modules/vitest/dist/cli.js --run test/suite/memswe-smoke-runner.test.ts
-cd ../..
-
-# full repo gate
-npm run check
-
-git status --short --branch
 
-git diff -- README.md packages/coding-agent/README.md packages/coding-agent/docs/memswe-benchmark-status.html

+# committed evidence
+docs/pap-sunday-2026-07-05/charts/memswe-cell-matrix.csv
+docs/pap-sunday-2026-07-05/charts/memswe-aggregate-by-system.json
-

Generated from current local artifacts and command output in /Users/eduardojaviergarcialopez/AgenticEngineering/PAP-membench. This document is intentionally artifact-backed; update it after each harness milestone or benchmark pilot.

+

Status is grounded in committed HEAD 7dca44e1, the chart bundle under docs/pap-sunday-2026-07-05/charts/, and the current runner implementation. Update this page whenever condition semantics or the evidence matrix changes.