diff --git a/README.md b/README.md index 11077e68c1a..f4c51edb6f4 100644 --- a/README.md +++ b/README.md @@ -31,9 +31,14 @@ npm --prefix packages/coding-agent run memswe:smoke npm --prefix packages/coding-agent run memswe:smoke -- --task-id=repo-gamma-invoice-export-001 npm --prefix packages/coding-agent run memswe:smoke -- --all-tasks --continue-on-task-failure npm --prefix packages/coding-agent run memswe:smoke -- --condition=no_memory +npm --prefix packages/coding-agent run memswe:smoke -- --condition=full_context --repetition-index=1 ``` -The smoke runner defaults to the deterministic faux provider and makes no real model/provider API calls unless real mode is explicitly enabled. With `MEMSWE_ALLOW_REAL_MODEL=1` and `--agent-mode=real` it drives a real provider (default `omniroute`, model `azure/deepseek-v4-flash`, via `OMNIROUTE_API_KEY`/`OMNIROUTE_BASE_URL`; legacy `--agent-mode=minimax-real` remains for MiniMax); the Honcho memory adapter can run against Honcho Cloud (`https://api.honcho.dev`, cloud-first with `HONCHO_API_URL` self-host fallback). In all modes it copies MemSWE fixtures into temporary worktrees, keeps hidden/protected verifier assets harness-side, emits artifacts under `.memswe-runs///`, and writes `suite-summary.json` for all-task runs. See [packages/coding-agent/README.md](packages/coding-agent/README.md#memswe--pap-membench-harness) and [packages/coding-agent/docs/memswe-benchmark-status.html](packages/coding-agent/docs/memswe-benchmark-status.html) for current command, artifact, and readiness details. +The smoke runner defaults to the deterministic faux provider and makes no real model/provider API calls unless real mode is explicitly enabled. With `MEMSWE_ALLOW_REAL_MODEL=1` and `--agent-mode=real` it drives a real provider (default `omniroute`, model `azure/deepseek-v4-flash`, via `OMNIROUTE_API_KEY`/`OMNIROUTE_BASE_URL`; legacy `--agent-mode=minimax-real` remains for MiniMax); the Honcho memory adapter can run against Honcho Cloud (`https://api.honcho.dev`, cloud-first with `HONCHO_API_URL` self-host fallback). In all modes it copies MemSWE fixtures into temporary worktrees, keeps hidden/protected verifier assets harness-side, emits artifacts under `.memswe-runs///`, and writes `suite-summary.json` for all-task runs. + +The evidence export added at `7dca44e1` covers a 240-cell matrix: 10 canonical tasks x 6 scoped conditions (`no_memory`, `full_context`, `hindsight`, `honcho`, `zep`, and `supermemory`) x 4 repetitions. All 240 exported cells report `task_success_visible=1` and `otel_trace_complete=pass`. These are deliberately narrow claims: visible success is saturated and does not distinguish the conditions; hidden success was not established; task-specific memory trace predicates remain unsupported by the current runner; and RAM was not measured. Provider lifecycle execution also does not, by itself, prove that recalled provider memory influenced the graded agent session. + +The fixed-runtime rule remains central: comparisons must hold model, prompt template, tools, fixture visibility, verifier rules, repetition policy, and scoring constant while changing only the memory condition/provider. See [packages/coding-agent/README.md](packages/coding-agent/README.md#memswe--pap-membench-harness), the [benchmark status page](packages/coding-agent/docs/memswe-benchmark-status.html), and the [PAP evidence index](docs/pap-sunday-2026-07-05/README.md) for current commands, artifacts, limitations, and charts. * **[@earendil-works/pi-coding-agent](packages/coding-agent)**: Interactive coding agent CLI * **[@earendil-works/pi-agent-core](packages/agent)**: Agent runtime with tool calling and state management diff --git a/docs/pap-sunday-2026-07-05/README.md b/docs/pap-sunday-2026-07-05/README.md new file mode 100644 index 00000000000..7fa93a5f7a9 --- /dev/null +++ b/docs/pap-sunday-2026-07-05/README.md @@ -0,0 +1,34 @@ +# PAP delivery evidence index + +This directory contains presentation-ready evidence for the MemSWE/PAP-membench delivery at `pi-memswe` commit `7dca44e1`. + +## Current evidence + +The committed matrix contains 240 cells: + +- 10 canonical tasks from the sibling `../memswe` benchmark repository; +- 6 scoped conditions: `no_memory`, `full_context`, `hindsight`, `honcho`, `zep`, and `supermemory`; +- 4 repetitions per task and condition. + +All 240 cells report `task_success_visible=1` and `otel_trace_complete=pass`. The first result is a saturated visible-only signal; the second confirms harness trace completeness and successful trace flush. Neither establishes hidden-test success or task-specific memory correctness. + +## Deliverables + +- [`charts/README.md`](charts/README.md): data-source notes, aggregate table, and limitations. +- [`charts/memswe-cell-matrix.csv`](charts/memswe-cell-matrix.csv): one normalized row per matrix cell. +- [`charts/memswe-aggregate-by-system.csv`](charts/memswe-aggregate-by-system.csv) and [`charts/memswe-aggregate-by-system.json`](charts/memswe-aggregate-by-system.json): per-condition aggregates. +- [`charts/chart-success-rate.png`](charts/chart-success-rate.png): visible-success ceiling across conditions. +- [`charts/chart-latency.png`](charts/chart-latency.png): mean and median task latency. +- [`charts/chart-cost-tokens.png`](charts/chart-cost-tokens.png): mean cost and input/output tokens. +- [`charts/memswe-flow-diagram.md`](charts/memswe-flow-diagram.md): harness execution flow. + +## Interpretation limits + +- Hidden success is unsupported by the committed matrix evidence. +- RAM/peak-memory usage was not measured. +- The current runner emits sibling-defined task trace predicates as `not_evaluable`; `otel_trace_complete` is a separate diagnostic predicate. +- Aggregate injected-memory share is zero, so the matrix cannot support memory-utilization claims. +- `full_context` prepends prior transcripts, and Honcho has an explicit graded recall/readback path. The generic Hindsight, Zep, and Supermemory condition paths record provider lifecycle probes but do not generally inject provider recall into the graded prompt. +- Comparisons are valid only when model, prompt template, tools, fixture visibility, verifier policy, repetition policy, and scoring remain fixed while the memory condition changes. + +Benchmark definitions, task assets, verifier policy, and run-record schema remain owned by the sibling `../memswe` repository. This repository owns the pi execution harness, condition adapters, trace capture, and exported delivery artifacts. diff --git a/packages/coding-agent/README.md b/packages/coding-agent/README.md index 4f4674d20b9..706c083ed78 100644 --- a/packages/coding-agent/README.md +++ b/packages/coding-agent/README.md @@ -102,60 +102,69 @@ Then just talk to pi. By default, pi gives the model four tools: `read`, `write` ## MemSWE / PAP-membench Harness -This fork is the fixed pi runtime/harness for MemSWE experiments. The benchmark source of truth is the sibling `../memswe` repository: task descriptors, fixtures, session prompts, fact lifecycle, verifier policy, trace predicates, and run-record schema belong there. This package supplies the current runner around pi's coding-agent SDK and must keep the runtime fixed while memory conditions/providers vary. +This fork is the fixed pi runtime/harness for MemSWE experiments. The benchmark source of truth is the sibling `../memswe` repository: task descriptors, fixtures, session prompts, fact lifecycle, verifier policy, trace predicates, and run-record schema belong there. This package owns execution, condition adapters, trace capture, verifier orchestration, and run artifacts; changes to benchmark identity belong in `memswe`. -The smoke runner defaults to the faux provider with tools disabled, and can run narrow real-model plumbing checks when explicitly requested. It is meant to audit harness plumbing, verifier isolation, artifact emission, and condition setup before scaled real-model pilots. +The smoke runner defaults to the faux provider with tools disabled, and can run narrow real-model plumbing checks when explicitly requested. It is meant to audit harness plumbing, verifier isolation, artifact emission, and condition setup before scaled real-model pilots. A valid comparison must keep the model, prompt template, tools, fixture visibility, verifier rules, repetition policy, and scoring fixed while changing only the memory condition/provider. ### Commands Run from the `pi-memswe` repo root: ```bash -# Default gamma smoke: repo-gamma-invoice-export-001, condition no_memory. +# Default gamma plumbing smoke: repo-gamma-invoice-export-001, no_memory, rep 1. npm --prefix packages/coding-agent run memswe:smoke # Explicit task. npm --prefix packages/coding-agent run memswe:smoke -- --task-id=repo-gamma-invoice-export-001 -# Discover and run all MemSWE task descriptors, continuing after expected task failures. +# Discover all sibling-repo task descriptors. This all-task path is faux-only. npm --prefix packages/coding-agent run memswe:smoke -- --all-tasks --continue-on-task-failure -# Explicit condition. -npm --prefix packages/coding-agent run memswe:smoke -- --condition=no_memory +# Explicit condition and 1-based repetition index. +npm --prefix packages/coding-agent run memswe:smoke -- --condition=full_context --repetition-index=1 -# Local Hindsight AMS/API smoke. This may call the configured local Hindsight LLM -# provider during retain/recall and should only be run after explicit token approval. +# Provider lifecycle probes. These may call configured services or their LLMs. npm --prefix packages/coding-agent run memswe:hindsight-smoke +npm --prefix packages/coding-agent run memswe:honcho-smoke +npm --prefix packages/coding-agent run memswe:zep-smoke +npm --prefix packages/coding-agent run memswe:supermemory-smoke # Task-aware Hindsight condition through the MemSWE runner; use --agent-mode=real only with explicit approval. npm --prefix packages/coding-agent run memswe:smoke -- --task-id=repo-tau-constraint-recall-001 --condition=hindsight --agent-mode=faux-text ``` -Condition support is intentionally narrow: +Supported condition IDs are `no_memory`, `full_context`, `repository_docs`, `filesystem`, `localrag`, `zep`, `supermemory`, `honcho`, `graphiti`, `mem0`, `letta`, and `hindsight`. -- `no_memory`: implemented default baseline for the faux/no-edit smoke runner. -- `repository_docs`: scaffolded materialization path that writes `docs/agent-project-memory/memswe-facts.md` into the temp fixture and records the copied memory doc as an artifact. Treat it as harness plumbing, not a completed benchmark-ready local-memory baseline. -- `full_context`: replays prior non-graded session transcripts before the graded prompt. -- Provider conditions (`filesystem`, `localrag`, `zep`, `supermemory`, `honcho`, `graphiti`, `mem0`, `letta`, `hindsight`): run the corresponding adapter lifecycle smoke and record `condition.memory_system = ` in the run record. Missing cloud/local credentials return provider `status: "skipped"` instead of failing the whole condition setup. +- `no_memory` is the implemented default baseline and injects no memory context. +- `full_context` assembles declared seeded history and prior non-graded transcripts into a preamble for the graded prompt. +- `repository_docs` writes valid facts to `docs/agent-project-memory/memswe-facts.md` in the temporary fixture. The agent still needs tools and instructions that can consume that file; materialization alone is not proof of use. +- `honcho` has a graded path that seeds a run-scoped persistent workspace, injects a recall preamble, writes the graded conclusion back, and requires a positive readback gate. +- The other provider conditions (`filesystem`, `localrag`, `zep`, `supermemory`, `graphiti`, `mem0`, `letta`, and `hindsight`) run their adapter lifecycle probe and record its export before the graded session. Missing cloud/local credentials return provider `status: "skipped"` instead of failing condition setup. The lifecycle result is not generally injected into the graded prompt, so treat these cells as provider/harness integration evidence, not proof that provider recall improved task performance. - Any other `--condition=...` value is invalid and exits before task execution. `memswe:hindsight-smoke` is separate provider-readiness plumbing for the Hindsight API. The MemSWE runner's `--condition=hindsight` path invokes the same lifecycle smoke task-aware: the current task id and run id are passed through so a repo-tau canary seeds/recalls repo-tau facts, not the default gamma fixture. Because retain/recall can use the configured Hindsight LLM provider, treat Hindsight smokes as real local AMS/API calls that may incur model/token usage; do not run them without explicit token scope and approval. -`memswe:smoke` defaults to deterministic `--agent-mode=faux-text`. Real-model plumbing checks are available as `--agent-mode=real` (default provider `omniroute` / model `azure/deepseek-v4-flash`, via `OMNIROUTE_API_KEY` and optional `OMNIROUTE_BASE_URL`; override with `MEMSWE_LLM_PROVIDER`/`MEMSWE_LLM_MODEL`) or the legacy `--agent-mode=minimax-real`, which selects pi provider `minimax` / model `MiniMax-M3`. Real modes require `MEMSWE_ALLOW_REAL_MODEL=1`, disable `--all-tasks`, and should only be used after pilot approval. Note the endpoint distinction: the Hindsight/LiteLLM smoke uses the MiniMax token-plan endpoint `https://api.minimax.io/v1`, while pi's built-in MiniMax-M3 entry uses the provider's Anthropic-compatible endpoint `https://api.minimax.io/anthropic`. +`memswe:smoke` defaults to deterministic `--agent-mode=faux-text`. Real-model plumbing checks are available as `--agent-mode=real` (default provider `omniroute` / model `azure/deepseek-v4-flash`, via `OMNIROUTE_API_KEY` and optional `OMNIROUTE_BASE_URL`; override with `MEMSWE_LLM_PROVIDER`/`MEMSWE_LLM_MODEL`) or the legacy `--agent-mode=minimax-real`, which selects pi provider `minimax` / model `MiniMax-M3`. Real modes require `MEMSWE_ALLOW_REAL_MODEL=1`, disable `--all-tasks`, and should only be used after pilot approval. Note the endpoint distinction: the Hindsight/LiteLLM smoke uses the MiniMax token-plan endpoint `https://api.minimax.io/v1`, while pi's built-in MiniMax-M3 entry uses the provider's Anthropic-compatible endpoint `https://api.minimax.io/anthropic`. Provider lifecycle probes can also incur service or model usage; run them only with explicit credential and token scope. `memswe:report` aggregates ignored `.memswe-runs/**` artifacts into a static browser report at `.memswe-runs/reports/latest/index.html`, with `run-summary.json` beside it. The report includes run records, verifier counts, suite summaries, Hindsight lifecycle smokes, and optional MiniMax/Hermes diagnostic review JSON files from `.memswe-runs/reviews/`. -Expected behavior with the faux/no-edit runner: +### Committed 240-cell evidence export -- The default gamma smoke should pass visible/protected verification. -- In `--all-tasks --continue-on-task-failure`, non-gamma tasks may fail verification explicitly because the faux agent acknowledges prompts and does not edit files. Those failures are expected evidence that verifiers are active, not evidence of real-model quality. -- Hidden verifiers remain isolated and are skipped unless an explicit hidden-run policy is used. Do not expose hidden/protected verifier internals to the agent runtime. +Commit `7dca44e1` adds charts and normalized CSV/JSON built from the latest run record for each of 10 canonical tasks x 6 scoped conditions (`no_memory`, `full_context`, `hindsight`, `honcho`, `zep`, `supermemory`) x 4 repetitions. The export reports: + +- 240/240 cells with `task_success_visible=1`; +- 240/240 cells with `otel_trace_complete=pass`; +- per-condition latency, cost, and token aggregates. + +Interpret these results narrowly. Visible success is at a 1.00 ceiling for every condition, so it cannot establish a memory-system advantage. Hidden verifiers were not part of the committed matrix evidence, `task_success_hidden` is therefore unsupported, and the runner emits benchmark-specific trace predicates as `not_evaluable`. `otel_trace_complete` only confirms the expected harness spans and successful trace flush. RAM is not recorded. The export also reports zero injected-memory share, so it cannot support utilization or provider-memory-attribution claims. + +The committed evidence is under [`docs/pap-sunday-2026-07-05/charts/`](../../docs/pap-sunday-2026-07-05/charts/). Generated `.memswe-runs/` data remains ignored. ### Artifacts Each task run writes artifacts under: ```text -.memswe-runs/// +.memswe-runs//--r/ ``` Important files: @@ -167,11 +176,14 @@ Important files: - `verifier-results.json`: harness-side visible/protected verifier command results. - `skipped-hidden-verifiers.json`: hidden verifier metadata skipped by default. - `agent.patch`, `worktree-diff.patch`, `changed-files.json`: no-op patch artifacts for faux/no-edit runs; real-model pilots must persist real diffs here. -- `condition-result.json`: selected memory condition and condition-specific artifact paths. -- `repository-docs/memswe-facts.md`: only present for the `repository_docs` scaffold. -- `provider-smoke-.json`: present for provider conditions, including task-aware Hindsight lifecycle evidence. +- `condition-result.json`: selected memory condition, memory-system label, and condition-specific artifact paths. +- `full-context-transcript.json`: prior transcript bundle for `full_context`. +- `repository-docs/memswe-facts.md`: materialized valid facts for `repository_docs`. +- `provider-smoke-.json`: lifecycle result for provider conditions other than the Honcho graded path, including task-aware Hindsight evidence. +- `honcho-graded-memory.json` and `honcho-readback.json`: run-scoped seeded/recall and post-session readback evidence for Honcho. +- `memswe-trace.json`: local trace artifact used by the `otel_trace_complete` diagnostic predicate. -The Hindsight local smoke writes `.memswe-runs//hindsight-local-smoke/hindsight-smoke-result.json` when run directly, while `--condition=hindsight` writes a per-task `provider-smoke-hindsight.json` beside the run record. Both include request/response trace events, predicate results, and structured failure guidance. +Standalone provider smokes write provider-specific lifecycle artifacts. They are readiness evidence, not MemSWE task scores. The Hindsight local smoke writes `.memswe-runs//hindsight-local-smoke/hindsight-smoke-result.json` when run directly, while `--condition=hindsight` writes a per-task `provider-smoke-hindsight.json` beside the run record. Both include request/response trace events, predicate results, and structured failure guidance. The report generator writes `.memswe-runs/reports//index.html` and refreshes `.memswe-runs/reports/latest/index.html`. Generated report artifacts stay ignored; commit the generator, not the reports. @@ -185,7 +197,7 @@ The suite summary lists task IDs, pass/fail status, run-record path, failed phas ### Validation -Use these checks after harness edits or docs updates that change command references: +Use these checks after harness edits. Documentation-only changes can use link and static-content checks without running model/provider flows: ```bash # Targeted runner/helper tests, from packages/coding-agent. @@ -197,7 +209,7 @@ cd ../.. npm run check ``` -Do not run real model/provider pilots from this smoke path unless API-token scope, local trace readiness, and hidden/protected policy are explicitly approved. +Do not run real model/provider pilots unless API-token scope, local trace readiness, and hidden/protected policy are explicitly approved. Never interpret visible-only success, provider setup, or a complete harness trace as hidden-test success or task-specific memory correctness. --- diff --git a/packages/coding-agent/docs/memswe-benchmark-status.html b/packages/coding-agent/docs/memswe-benchmark-status.html index 358b00fd7f7..0bb123154ee 100644 --- a/packages/coding-agent/docs/memswe-benchmark-status.html +++ b/packages/coding-agent/docs/memswe-benchmark-status.html @@ -260,100 +260,114 @@
- Default gamma smoke: passing - All-task loop: scaffolded - Working tree not committed - Generated 2026-06-17 03:26 UTC + 240 committed evidence cells + Harness traces complete + Visible-only success ceiling + Updated 2026-07-09

MemSWE / PAP-membench benchmark status

-

Consolidated status of the pi-memswe fixed runtime/harness fork: current faux-provider smoke commands, guarded MiniMax real-model plumbing checks, all-task suite loop, condition scaffold, artifacts, and next steps without overclaiming benchmark performance.

+

Artifact-backed status of the fixed pi runtime/harness fork at 7dca44e1, including the 10-task × 6-condition × 4-repetition evidence matrix and the limits of what that matrix establishes.

-
Default smoke run
-
PASS
-
Gamma task, faux agent, no_memory
+
Matrix cells
+
240
+
10 tasks × 6 conditions × 4 reps
-
All-task loop
-
6 tasks
-
Continues and writes suite summary
+
Canonical tasks
+
10
+
Selected from the sibling benchmark repo
-
Conditions
-
2 scaffolded
-
no_memory verified; repository_docs materializes docs
+
Scoped conditions
+
6
+
2 baselines + 4 provider labels
-
Repo check
-
PASS
-
npm run check
+
Repetitions
+
4
+
Per task and condition
-

What has been completed

+

What is established

- Fixed-runtime harness role documented - pi-memswe is treated as the fixed runtime/harness fork. The benchmark source of truth remains ../memswe; memory condition/provider behavior is the evaluated variable. + Benchmark/runtime ownership is separated + ../memswe owns tasks, fixtures, sessions, facts, verifier policy, trace predicates, and schemas. This fork owns execution, adapters, traces, and artifacts.
- Smoke runner - packages/coding-agent/scripts/memswe-smoke-runner.ts creates temp fixture worktrees, runs pi createAgentSession(...) sessions, captures agent artifacts, runs harness-side verifiers, validates run-record shape, and emits run-record.json. Default mode remains deterministic --agent-mode=faux-text. + Condition and repetition orchestration exists + The runner accepts a 1-based repetition index and condition IDs for baselines and provider adapters, creates run-scoped workspaces, invokes the fixed pi session path, and validates uam-run.v0.1 output against the sibling schema.
- Verifier environment and patch artifacts - The runner initializes a git baseline, captures agent.patch, worktree-diff.patch, and changed-files.json, creates a Python venv, runs task setup commands, and keeps hidden/protected assets out of agent-visible context. + Isolation and deterministic evidence exist + Fixtures are copied into temporary worktrees; patch, message, verifier, condition, and trace artifacts are retained. Hidden/protected assets remain harness-controlled.
- All-task suite loop - --all-tasks --continue-on-task-failure discovers six MemSWE task descriptors, records pass/fail per task, and writes .memswe-runs/<timestamp>/suite-summary.json. Faux/no-edit failures on non-gamma tasks are expected verifier evidence. + 240-cell export is committed + The chart bundle contains one normalized row for every selected task/condition/repetition cell plus per-system aggregates and presentation-ready charts.
Condition scaffold --condition=no_memory is the verified default. repository_docs currently materializes remembered facts into project docs and records a condition artifact, but remains harness plumbing rather than a completed benchmark baseline. full_context now replays prior non-graded session transcripts before the graded prompt; hindsight and the provider memory conditions run as guarded provider-lifecycle smokes, and the separate memswe:hindsight-smoke command is a real local AMS/API probe.
+
+ Trace plumbing is complete for the export + All 240 cells report otel_trace_complete=pass, meaning required harness spans were present and the configured trace flush succeeded. +
-

Current repository state

+

Evidence boundaries

- + - - - + + + + + + + + - - - + + + - - - + + + + + + + +
RepoStatusNotes
ClaimStatusInterpretation
pi-memswemodifiedCurrent branch tracks origin/main. Latest local commit is 4854e7a9. Before this docs pass, one script file was already modified; this status page and READMEs are additional uncommitted docs changes.Visible success240/240Every exported cell passed its visible verifier. The signal is saturated at 1.00 for every condition.
Trace completeness240/240Harness trace structure and flush completed. This is not a task-specific memory predicate.
memsweread-only dependencyBenchmark/spec repo at ../memswe remains source of truth for tasks, fixtures, sessions, facts, verifier metadata, and run-record schema.Hidden successunsupportedThe committed export does not establish hidden-verifier results.
memorybench-memswereference onlyReference patterns may inform provider isolation and reports, but this harness is not coupled to MemoryBench.Memory behaviornot evaluableDeclared task predicates are emitted as not_evaluable; injected-memory share is zero in the aggregate export.
RAMnot measuredNo committed matrix field supports memory-consumption comparisons.
- Scope note: this page describes current local harness status. It does not claim benchmark-spec ownership, real-model performance, Hindsight readiness, or a completed repository_docs baseline. + Conclusion: the matrix demonstrates repeatable harness execution, visible-verifier completion, trace capture, and cost/latency/token collection. It does not demonstrate that one memory system improves correctness.
@@ -398,10 +412,33 @@

Representative artifact paths

pi-memswe/.memswe-runs/<timestamp>/<task-id>/changed-files.json +
+

Matrix summary

+

Each condition has 40 exported cells. Visible success is identical; secondary metrics differ but should be treated as descriptive because the primary correctness signal is saturated.

+ + + + + + + + + + +
ConditionVisible successMean latencyMean costMean tokens
no_memory1.0020.4 s$0.000122599
full_context1.003.8 s$0.000118716
hindsight1.0018.8 s$0.000113567
honcho1.0014.5 s$0.000109649
zep1.0023.0 s$0.000145678
supermemory1.0015.0 s$0.000086470
+

Source: docs/pap-sunday-2026-07-05/charts/memswe-aggregate-by-system.json. Latest run record per cell was selected; repetition 98/99 sanity runs were excluded.

+
+
-

Benchmark status

+

Fixed-runtime controls and harness status

    +
  • Same model and pricing configuration.
    • +
  • Same graded prompt and prompt template.
    • +
  • Same tool availability and fixture visibility.
    • +
  • Same visible/hidden/protected verifier policy.
    • +
  • Same repetition count and scoring rules.
    • +
  • Only memory delivery/provider may vary.
  • Task schema exists: MemSWE task descriptors use uam-task.v0.1 in ../memswe.
  • Run-record schema exists: harness-emitted records target uam-run.v0.1; the runner currently performs local shape validation.
  • Phase 0 task inventory exists: six task descriptors are discoverable by the harness.
    • @@ -409,9 +446,10 @@

    Benchmark status

  • Verifiers are deterministic-first: visible/protected tests run separately from the agent-visible fixture; hidden verifiers are isolated and skipped by default.
  • Trace predicates exist: faux-agent runs mark them not_evaluable; real-mode/provider runs emit memswe-trace-export.json and evaluate trace completeness/flush (pass/fail) when tracing is enabled.
+

These are experimental invariants. The committed chart CSV does not itself carry every control field, so the underlying run records remain the audit source.

-

Harness status

+

Condition semantics

  • Runtime path: pi coding-agent SDK via createAgentSession(...).
  • Default agent model: deterministic faux provider, no paid/real model calls unless real mode is explicitly selected (--agent-mode=real, default provider omniroute / azure/deepseek-v4-flash; legacy --agent-mode=minimax-real).
    • @@ -419,51 +457,70 @@

    Harness status

  • Fixture isolation: each task fixture is copied into a temporary worktree with an initialized git baseline.
  • Verifier isolation: visible/protected verifier files are copied into the temp worktree; hidden verifier metadata is skipped unless explicitly included.
  • All-task execution: --all-tasks --continue-on-task-failure iterates all six descriptors and emits suite-summary.json.
    • -
  • Condition handling: no_memory (default) and repository_docs are implemented; full_context replays prior non-graded session transcripts before the graded prompt; hindsight and the provider memory conditions (filesystem, localrag, zep, supermemory, honcho, graphiti, mem0, letta) run as guarded provider-lifecycle smokes.
    • +
  • no_memory: the default injects no memory preamble.
    • +
  • repository_docs: materializes remembered facts into project docs, but remains harness plumbing rather than a completed benchmark baseline.
    • +
  • full_context: prior declared transcripts are prepended to the graded prompt.
    • +
  • honcho: run-scoped persistent workspace, recall preamble, post-session write, and positive readback gate.
    • +
  • Other provider conditions: filesystem, localrag, hindsight, zep, supermemory, graphiti, mem0, and letta record guarded lifecycle-probe output; they do not generally inject recalled provider content into the graded prompt.
  • Artifact emission: agent events/messages/final response, setup result, patch artifacts, condition result, verifier results, skipped hidden list, and run record.
+

Therefore, provider-labeled cells outside the explicit Honcho path are integration cells, not clean evidence of memory-conditioned agent behavior.

-

Memory-system readiness

+

Current interfaces and artifacts

- + - - - - + + + - - - - + + + - - - + + + + + + + + + + + + + + + + + - - - - + + + - - + - - - + + + + + + + @@ -474,64 +531,59 @@

Memory-system readiness

Known gaps and risks

-

Harness gaps

+

Measurement gaps

    -
  • All-task loop exists, but current faux/no-edit behavior is only a plumbing check; it is not a real agent performance run.
    • -
  • Current smoke executes a single graded session per task, not full multi-session memory-condition execution.
    • -
  • Faux agent does not edit files, so trace predicates remain diagnostic/not evaluable.
    • -
  • repository_docs materialization is scaffold-only and needs policy/forget validation before being claimed as a baseline.
    • -
  • Hidden tests are correctly isolated by default, but hidden-run mode still needs explicit policy around when it is allowed.
    • +
  • Visible correctness is saturated and cannot rank systems.
    • +
  • No hidden-success conclusion is supported.
    • +
  • No RAM or peak-memory measurements are present.
    • +
  • Injected-memory share is zero in the committed aggregate.
    • +
  • Task-specific trace predicates are not evaluated.
-

Benchmark gaps

+

Integration gaps

    -
  • Need stronger aggregation across task, condition, model, and repetition.
    • -
  • Need real patch capture and diff constraints for agent runs.
    • -
  • Need normalized memory trace JSONL for retrieval/write/delete events.
    • -
  • Need provider adapter contract implemented in pi-memswe.
    • -
  • Need Hindsight local reset/delete behavior proven before claiming AMS readiness.
    • +
  • Generic provider lifecycle success can coexist with no provider recall in the graded prompt.
    • +
  • Normalized retrieval/write/delete/use evidence is incomplete.
    • +
  • All-task execution remains faux-only; real matrices require external orchestration of individual cells.
    • +
  • The chart CSV omits some experimental-control metadata available in full run records.
    • +
  • Ignored raw run artifacts must be archived separately for full reproducibility.
-

Recommended next steps

+

Required next evidence

    -
  1. Keep README/status docs current. Update this page after harness milestones and include validation output with each docs pass.
    2. -
  2. Promote smoke phases into explicit harness phases. Split current script into load_task, materialize_fixture, prepare_environment, run_agent_session, verify_artifacts, and emit_run_record.
    3. -
  3. Finish local baselines. Keep no_memory fixed, then complete and validate repository_docs before claiming it as a benchmark-ready condition.
    4. -
  4. Implement full-context replay. Assemble session history without changing model, prompt template, tools, fixture visibility, verifier rules, repetition policy, or scoring.
    5. -
  5. Run full session sequence for one task. Execute s1, s2, and s3 with condition-specific history delivery, not only the graded session.
    6. -
  6. Capture real patch/tool artifacts. For real agent runs, persist final diff, tool calls, stdout/stderr, final answer, injected context spans, and trace IDs.
    7. -
  7. Make trace predicates evaluable. Connect predicate checks to retrieved memories, injected content, tool arguments, final response, and code diffs.
    8. -
  8. Hindsight pilot spike. Use npm --prefix packages/coding-agent run memswe:hindsight-smoke only after explicit token/API approval. It creates/resets a smoke bank, seeds task-YAML-derived gamma facts, runs retain/recall/delete against local Hindsight, and records provider-facing request/response traces and failure guidance.
    9. -
  9. Real model / MiniMax pilot. Requires explicit API token approval plus complete local artifact/trace readiness; do not use this smoke path for unapproved provider calls.
    10. -
  10. Aggregate reporting. Add a report builder over .memswe-runs/ that produces per-condition/task/model/repetition summaries aligned with MemSWE measurement docs.
    11. +
  11. Inject and capture normalized recall output for every provider condition used in a graded comparison.
    12. +
  12. Evaluate each sibling-defined trace predicate with evidence references.
    13. +
  13. Run the approved hidden/protected policy from harness-controlled paths and report hidden results separately.
    14. +
  14. Add RAM/peak-memory instrumentation if resource efficiency is an RPAP claim.
    15. +
  15. Archive immutable run records with model, prompt, tools, fixture, verifier, and code-revision identifiers.
    16. +
  16. Use a harder task set or stronger scoring vector to avoid the visible-success ceiling.
-

Immediate command checklist

- 
# from pi-memswe
+      
Command and artifact reference

+      
# deterministic plumbing smoke
 npm --prefix packages/coding-agent run memswe:smoke
-npm --prefix packages/coding-agent run memswe:smoke -- --condition=no_memory
+npm --prefix packages/coding-agent run memswe:smoke -- --condition=full_context --repetition-index=1
+
+# faux-only discovery loop
+npm --prefix packages/coding-agent run memswe:smoke -- --all-tasks --continue-on-task-failure
 
-# targeted helper/runner tests
+# targeted runner tests (no provider calls)
 cd packages/coding-agent
 node ../../node_modules/vitest/dist/cli.js --run test/suite/memswe-smoke-runner.test.ts
-cd ../..
-
-# full repo gate
-npm run check
-
-git status --short --branch
 
-git diff -- README.md packages/coding-agent/README.md packages/coding-agent/docs/memswe-benchmark-status.html

+# committed evidence
+docs/pap-sunday-2026-07-05/charts/memswe-cell-matrix.csv
+docs/pap-sunday-2026-07-05/charts/memswe-aggregate-by-system.json
-

Generated from current local artifacts and command output in /Users/eduardojaviergarcialopez/AgenticEngineering/PAP-membench. This document is intentionally artifact-backed; update it after each harness milestone or benchmark pilot.

+

Status is grounded in committed HEAD 7dca44e1, the chart bundle under docs/pap-sunday-2026-07-05/charts/, and the current runner implementation. Update this page whenever condition semantics or the evidence matrix changes.

System / conditionReadinessWhat works nowRemaining work
SurfaceCurrent behaviorCaveat
no_memoryimplemented smoke baselineDefault condition; agent sees current prompt and fixture state only, with no memory injection.Needs full multi-session orchestration beyond current single graded-session smoke.--conditionBaselines plus filesystem/local RAG and seven provider IDs.A known ID does not imply graded recall injection.
repository_docsscaffold onlyWrites current valid facts to docs/agent-project-memory/memswe-facts.md in the temp worktree and records the doc artifact.Not benchmark-ready: still needs fact materialization policy review, supersede/forget handling, session sequencing, and validation as a local-memory baseline.--repetition-indexPositive, 1-based repetition carried into run ID and artifact path.The caller orchestrates the four-repetition matrix.
full_contexttranscript replay implementedReplays prior non-graded session transcripts before the graded prompt and sets memory_system=full_context so run-records validate.run-record.jsonSchema-validated condition, session, reward, metrics, predicates, and output locations.Many advanced memory metrics are currently null or zero.
memswe-trace.jsonBenchmark, memory-prepare, agent-session, verifier, and scoring trace structure.Completeness is separate from semantic predicate correctness.
Patch/verifier artifactsAgent patch, worktree diff, changed files, verifier results, and skipped-hidden list.Hidden assets must never enter the agent-visible fixture.
full_context
transcript replay implemented		Replays prior non-graded session transcripts before the graded prompt and sets memory_system=full_context so run records validate. Still needs fixed-runtime controls and validation as a graded baseline.
hindsightpilot targetThe runner's --condition=hindsight path now invokes the Hindsight lifecycle smoke task-aware (task/run id passed through). The separate memswe:hindsight-smoke command can make real local Hindsight retain/recall/delete API calls and may incur configured model/token usage.Local Docker/API pilot requires explicit token scope, bank reset proof, retain/recall/delete behavior, settle polling, trace export, and complete local trace readiness.hindsight
pilot target		The runner's --condition=hindsight path invokes the Hindsight lifecycle smoke task-aware. The separate memswe:hindsight-smoke command can make real local Hindsight retain/recall/delete API calls and may incur configured model/token usage.Local Docker/API pilots require explicit token scope and lifecycle, settle, and trace evidence.
filesystem / localragguarded lifecycle smokefilesystem / localrag
guarded lifecycle smoke		 Each runs its adapter lifecycle smoke via --condition=<id>, recording condition.memory_system in the run record; missing local assets return provider status: "skipped" rather than failing setup. Adapt to code-session facts, lifecycle state, scope IDs, and run-record-compatible traces; validate as graded baselines.
honcho, zep, supermemory, graphiti, mem0, lettaguarded lifecycle smoke (credentialed)Each has an adapter lifecycle smoke wired into --condition=<id>; without cloud/local credentials the provider returns status: "skipped" instead of failing the condition. The honcho adapter targets Honcho Cloud (https://api.honcho.dev) cloud-first with HONCHO_API_URL self-host fallback and requires a positive peer/conclusion readback to pass.honcho
credentialed graded path		Targets Honcho Cloud (https://api.honcho.dev) cloud-first with HONCHO_API_URL self-host fallback. The graded path seeds and recalls a run-scoped workspace, writes the conclusion back, and requires positive readback.Requires API/service setup and explicit inclusion.
zep, supermemory, graphiti, mem0, letta
guarded lifecycle smoke		Each has an adapter lifecycle smoke wired into --condition=<id>; without cloud/local credentials the provider returns status: "skipped" instead of failing the condition. Require API/service setup and explicit inclusion; not immediate local Phase 0 graded baselines.