Track C++ Google Benchmark microbenchmarks via ASV

[vasil-pashov]

What

Adds C++ microbenchmark tracking alongside the existing Python ASV benchmarks.
Each Google Benchmark function becomes a separate ASV class so results appear
in the same dashboard with the same tooling.

New files

• python/benchmarks/cpp_microbenchmarks.py — ASV module that runs the C++
  benchmark binary (ARCTICDB_CPP_BENCHMARKS_BINARY) and exposes each function
  as an ASV track_* class. Parameter variants (BM_foo/10000/1) become ASV
  params; the unit = "ms" attribute is set so graphs show the correct Y-axis
  label. Per-benchmark flag overrides are supported via BENCHMARK_FLAGS.
• .github/workflows/benchmark_cpp_microbenchmarks.yml — callable workflow
  that builds the benchmarks binary, then runs asv run --bench "^cpp_microbenchmarks\.".
  Results are stored in a separate ArcticDB library
  (cpp_microbenchmarks_asv_results) to avoid key conflicts with Python results
  stored under the same 8-char commit hash.

Changes to analysis_workflow.yml

• New benchmark_cpp_microbenchmarks job runs in parallel with benchmark_commits,
  each on its own EC2 runner — no shared machine contention.
• continue-on-error: true so C++ regressions appear as a visible failed job
  but do not block PRs or the Python benchmark publish step.
• The publish job's if condition uses always() so a C++ failure does not
  prevent Python results from being published.
• run-asv-check-script now builds the C++ benchmarks binary before running
  asv_checks.py. Without the binary cpp_microbenchmarks.py registers zero
  classes, causing asv_checks.py to rewrite benchmarks.json without C++
  entries and fail the hash-change check on every PR.

Why

C++ micro-benchmark results were previously invisible. This change surfaces
regressions in CI alongside the Python benchmark history without blocking
existing workflows.

Pre-existing behaviour unchanged

Python ASV benchmarks (benchmark_commits) are unmodified. The publish job and
regression detection logic for Python results are unchanged.

[claude]

The rm command fails (exits 1) if no files match the glob — which would happen if ASV placed results in an unexpected directory or if master_commit_hash.txt is empty. Because bash globbing passes the unexpanded pattern as-is to rm when there are no matches, rm always sees a "file not found" and exits non-zero. The || exit 1 guard then immediately aborts the step.

Use rm -f to silently succeed when the file is missing:

Suggested change

	rm python/.asv/results/**/$(cat master_commit_hash.txt)*.json || exit 1
	rm -f python/.asv/results/**/$(cat master_commit_hash.txt)*.json

Also note that ** requires shopt -s globstar to traverse recursively in bash; without it only the immediate directory is searched. Check that the shell default in the container has globstar enabled, or switch to find:

find python/.asv/results -name "$(cat master_commit_hash.txt)*.json" -delete

[claude]

This module-level code runs a subprocess at import time. If the binary exists but --benchmark_list_tests exits non-zero (e.g., a crash or mislinked build), result.check_returncode() inside discover_benchmark_names() raises CalledProcessError, which propagates unhandled here and causes the entire module import to fail — with a confusing traceback rather than "no benchmarks registered".

Consider wrapping the discovery to degrade gracefully:

try:
    _names = discover_benchmark_names()
except Exception as exc:
    import warnings
    warnings.warn(f"cpp_microbenchmarks: benchmark discovery failed: {exc}", RuntimeWarning)
    _names = []

for func_name, variants in group_benchmarks(_names).items():
    globals()[to_identifier(func_name)] = make_benchmark_class(func_name, variants)

[claude]

ArcticDB Code Review Summary

PR: Track C++ Google Benchmark microbenchmarks via ASV (#2943)
Reviewed: full diff (delta fetch unavailable; all commits reviewed)

API & Compatibility

• No breaking changes to public Python API (Arctic, Library, NativeVersionStore)
• Deprecation protocol not applicable (no API removed/renamed)
• On-disk format unchanged
• Protobuf schema unchanged
• Key types and ValueType enum unchanged

Memory & Safety

• No C++ memory management involved (Python/CI-only change)
• RAII not applicable
• No GIL concerns

Correctness

• WARNING: rm without -f in Benchmark against master step (line 120): rm python/.asv/results/will exit 1 if no files match. Also, ** recursive glob requires shopt -s globstar which is not set. Use rm -f or find -delete. (Still not addressed.)
• FIXED: github.ref_name sanitization - replace(github.ref_name, '/', '_') is now used in the library name, preventing invalid names from branch names containing /.
• FIXED: Import-time CalledProcessError - discover_benchmark_names() returns [] on non-zero return code; combined with the binary-exists guard, all failure modes handled gracefully.
• OK: needs.start_ec2_runner.result == 'success' guard correctly prevents benchmark_commit from running when EC2 start fails.
• OK: run_benchmark correctly handles iteration vs. aggregate rows, unit conversion, and temp-file cleanup via finally: os.unlink.
• OK: Closure capture in make_benchmark_class is correct (new scope per call).
• OK: gbench_escape correctly handles ECMAScript regex escaping.

Code Quality

• No duplicated logic; new workflow follows patterns from existing benchmark_commits job.
• always() && !cancelled() pattern on benchmark_commit is consistent with other workflows.
• always() on stop-ec2-runner and publish jobs correctly ensures cleanup/publishing even on failure.
• Typo fixes (supress to suppress) in both workflow files.
• cpp_microbenchmarks.py module docstring clearly explains env var, fallback behaviour, and BENCHMARK_FLAGS override.

Testing

• Infrastructure/CI change - no behavioural change to ArcticDB data path; no functional tests required.
• Existing Python ASV benchmarks unaffected.

Build & Dependencies

• FIXED: actions/checkout@v4 - Upgraded from @v3.3.0 (Node.js 16) to @v4 (Node.js 20).
• INFO: ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true remains in benchmark_cpp_microbenchmarks.yml - now needed for mozilla-actions/sccache-action@v0.0.9 rather than checkout. Consider upgrading sccache-action to eliminate the flag entirely.
• OK: asv added to publish step pip install - correctly required for the new ASV check.
• OK: C++ benchmarks build step correctly ensures benchmark discovery works during the ASV hash-change check.

Security

• FIXED: Missing permissions: blocks - Both analysis_workflow.yml and benchmark_cpp_microbenchmarks.yml now declare permissions: contents: read. Resolves all GitHub Advanced Security findings.
• No hardcoded credentials (all via secrets).
• inputs.commit used as SHA^! in shell - hex-only commit hashes are injection-safe.

PR Title & Description

• Title is clear and imperative.
• Description explains what changed and why.
• No public API changes.
• No linked GitHub issue referenced.
• PR is not labelled (consider adding enhancement or ci).

Documentation

• No public API docstrings affected.
• cpp_microbenchmarks.py module docstring clearly explains the environment variable, fallback behaviour, and BENCHMARK_FLAGS override mechanism.

[poodlewars]

asv is already in the Testing deps?

[poodlewars]

This was already done in the other workflow file?

[poodlewars]

There's a lot of duplication with the analysis workflow. Can't we just fold the new steps in to that one?

[poodlewars]

Why do we need a totally separate library to store these? Are they presented properly in the UI?

[poodlewars]

We should make sure we blow up with a clear error message explaining that you need to build the benchmarks binary, if this is pointing at a binary that doesn't exist. Otherwise it will be hard for people to update the ASV .json file.

[poodlewars]

I'm fine with the script to convert the benchmarks to ASV format, but think it should all be folded in to the existing analysis workflow rather than creating a new one.