Skip to content

feat(benchmarks): add concurrent-write driver - #1304

Merged
phernandez merged 3 commits into
mainfrom
benchmarks/concurrent-write-driver
Aug 23, 2026
Merged

feat(benchmarks): add concurrent-write driver#1304
phernandez merged 3 commits into
mainfrom
benchmarks/concurrent-write-driver

Conversation

@phernandez

@phernandez phernandez commented Aug 23, 2026

Copy link
Copy Markdown
Member

Summary

Ports the concurrent-write convergence benchmark from basic-memory-benchmarks into Core's canonical /benchmarks package.

  • Adds deterministic multi-session MCP create/edit workloads and marker-based convergence checks.
  • Records per-operation latency/errors plus file, entity, observation, relation, and lost/doubled-write integrity.
  • Shares the warm MCP runtime with the existing local retrieval provider.
  • Updates the benchmark docs and command surface for the Core-owned package.

Why

The benchmark suite now lives in Core, while standalone PR basic-memory-benchmarks#42 was opened against the old repository. Keeping the driver there would strand issue #1248 as that repository is deprecated.

This port also fixes all three unresolved review findings from the standalone PR:

  1. The convergence verdict is captured before the optional full reindex mutates derived state.
  2. A timed-out MCP request is cancelled, its stdio subprocess is terminated by the transport, and the client thread must exit before verification continues.
  3. Concurrent-write runs require a local git checkout and record its exact resolved SHA.

Implementation

  • Adds bm_runtime.py for an explicitly managed warm MCP stdio session.
  • Adds concurrent_write.py for deterministic planning, concurrent execution, settle detection, integrity checks, and artifacts.
  • Adds bm-bench run concurrent-write plus smoke/load just recipes.
  • Refactors bm-local to use the shared MCP runtime.
  • Documents Core as the canonical benchmark location and requires BM_LOCAL_PATH for reproducible concurrent-write runs.
  • Rejects dirty target checkouts and calculates throughput from concurrent operations only, excluding sequential hub setup.

Verification

  • just test — 168 tests passed.
  • just lint — passed.
  • Changed-file pyright — 0 errors.
  • just --list — passed; both concurrent-write recipes parse.
  • Real MCP smoke on the final fix: 4 writers, 100 notes, 4 hubs — converged with 194/194 operations and every integrity check passing before reindex.
  • Package-wide pyright still reports two pre-existing errors in test_baseline_providers.py and test_mem0_normalization.py; neither file is changed here.

Risk

The main behavioral change outside the new command is the shared lifecycle wrapper used by bm-local. Its normal startup/call/stop contract is unchanged; shutdown is stricter because it now fails instead of silently leaving a live session behind.

Refs #1248

Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez

Copy link
Copy Markdown
Member Author

@codex review

@phernandez phernandez changed the title feat(core): add concurrent-write benchmark driver feat(benchmarks): add concurrent-write driver Aug 23, 2026
@phernandez

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 37b61771d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread benchmarks/src/basic_memory_benchmarks/concurrent_write.py Outdated
Comment thread benchmarks/src/basic_memory_benchmarks/concurrent_write.py Outdated
Comment thread benchmarks/src/basic_memory_benchmarks/bm_runtime.py Outdated
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 365bf175bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread benchmarks/src/basic_memory_benchmarks/concurrent_write.py Outdated
Comment thread benchmarks/src/basic_memory_benchmarks/concurrent_write.py Outdated
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez

Copy link
Copy Markdown
Member Author

@codex review

1 similar comment
@phernandez

Copy link
Copy Markdown
Member Author

@codex review

@phernandez
phernandez merged commit a9f4e08 into main Aug 23, 2026
21 checks passed
@phernandez
phernandez deleted the benchmarks/concurrent-write-driver branch August 23, 2026 17:14

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b60830c250

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

manifest = ConcurrentWriteManifest(
run_id=config.run_id,
created_at_utc=utc_now_iso(),
benchmark_git_sha=git_sha(Path(".")) or "unknown",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Resolve the benchmark SHA from the harness checkout

When bm-bench is invoked from outside the benchmark checkout, git_sha(Path(".")) records the caller's repository HEAD—or "unknown" outside a Git repository—rather than the SHA of the benchmark code being executed. This silently misattributes published comparison artifacts; resolve the repository containing the harness package and fail if its exact revision cannot be captured.

AGENTS.md reference: benchmarks/AGENTS.md:L82-L85

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant