feat(benchmarks): add concurrent-write driver - #1304
Conversation
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 37b61771d2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 365bf175bb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
1 similar comment
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b60830c250
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| manifest = ConcurrentWriteManifest( | ||
| run_id=config.run_id, | ||
| created_at_utc=utc_now_iso(), | ||
| benchmark_git_sha=git_sha(Path(".")) or "unknown", |
There was a problem hiding this comment.
Resolve the benchmark SHA from the harness checkout
When bm-bench is invoked from outside the benchmark checkout, git_sha(Path(".")) records the caller's repository HEAD—or "unknown" outside a Git repository—rather than the SHA of the benchmark code being executed. This silently misattributes published comparison artifacts; resolve the repository containing the harness package and fail if its exact revision cannot be captured.
AGENTS.md reference: benchmarks/AGENTS.md:L82-L85
Useful? React with 👍 / 👎.
Summary
Ports the concurrent-write convergence benchmark from
basic-memory-benchmarksinto Core's canonical/benchmarkspackage.Why
The benchmark suite now lives in Core, while standalone PR basic-memory-benchmarks#42 was opened against the old repository. Keeping the driver there would strand issue #1248 as that repository is deprecated.
This port also fixes all three unresolved review findings from the standalone PR:
Implementation
bm_runtime.pyfor an explicitly managed warm MCP stdio session.concurrent_write.pyfor deterministic planning, concurrent execution, settle detection, integrity checks, and artifacts.bm-bench run concurrent-writeplus smoke/loadjustrecipes.bm-localto use the shared MCP runtime.BM_LOCAL_PATHfor reproducible concurrent-write runs.Verification
just test— 168 tests passed.just lint— passed.pyright— 0 errors.just --list— passed; both concurrent-write recipes parse.pyrightstill reports two pre-existing errors intest_baseline_providers.pyandtest_mem0_normalization.py; neither file is changed here.Risk
The main behavioral change outside the new command is the shared lifecycle wrapper used by
bm-local. Its normal startup/call/stop contract is unchanged; shutdown is stricter because it now fails instead of silently leaving a live session behind.Refs #1248