Skip to content

docs: twelve-repository sync, query and graph benchmarks / 十二仓库增量同步、查询与图谱一致性数据 - #17

Merged
NianJiuZst merged 2 commits into
NianJiuZst:mainfrom
columbia11:claude/twelve-repo-benchmarks
Nov 2, 2026
Merged

NianJiuZst merged 2 commits into
NianJiuZst:mainfrom
columbia11:claude/twelve-repo-benchmarks

Conversation

@columbia11

Copy link
Copy Markdown
Contributor

概要

用合并后的 main cd69b83 对照原版 d2e0389,测了全部 12 个仓库的全量索引、增量同步、查询延迟和精确率/召回率,替换 README.md 和 README.zh-CN.md 中"实测性能"一节的旧数据。只改文档,没有代码改动。

  • 删除的旧内容:六仓库 edb403e 全量索引表、"Second-phase languages" 小节(数据并入新表)、b827b1d 的增量同步表和查询延迟表(只有 4 个仓库,且来自另一台机器)。
  • 新增:同一台机器、同一套配置下的 12 仓库全量索引、增量同步、查询延迟、图谱一致性四张表,以及配置表;保留"性能工程更新"小节。
  • 新增 docs/benchmarks/2026-11-02-twelve-repo.json:全部样本、查询列表、cgr sync 报告、分类型边数和配置。
  • 两份 README 末尾的"性能测试范围"从四个仓库改为十二个仓库。
  • 旧的 docs/BENCHMARKS*.md 和 docs/benchmarks/2026-11-01-*.json 作为历史记录保留,没有删。

Summary

Measures full indexing, incremental CLI sync, query latency and graph agreement for all twelve repositories with merged main cd69b83 against upstream d2e0389, and replaces the old data in the "Measured performance" sections of both READMEs. Documentation only.

  • Removed: the six-repository edb403e table, the "Second-phase languages" subsection (folded into the new tables), and the four-repository b827b1d sync and query tables, which came from a different machine.
  • Added: four twelve-repository tables (full index, sync, query, graph agreement) from one machine and configuration, plus a configuration table. "Performance engineering update" is kept.
  • Added docs/benchmarks/2026-11-02-twelve-repo.json with every sample, query list, cgr sync report, per-kind edge count and the configuration.
  • The "Performance scope" limitation now says twelve repositories instead of four.
  • The older docs/BENCHMARKS*.md reports and docs/benchmarks/2026-11-01-*.json stay as history.

测试配置 / Configuration

项目 / Item 配置 / Value
机器 / Machine AMD Ryzen 9 8940HX (16 cores, 32 threads), Windows 11 Home on AC power; WSL2 Ubuntu 24.04, Linux 6.18.33.2-microsoft-standard-WSL2, 15 GiB RAM
cgr cd69b83, cargo build --release --locked -p cgr (fat LTO, codegen-units = 1), rustc 1.99.1 (58cf8f9c3 2026-10-13); binary SHA-256 906cf11f3abc67d4790364c19d601068b1459cbc115616e2a58548eefcc9c9eb
原版 / Upstream colbymchenry/codegraph d2e0389, production dist/ with the linux-x64 native kernel prebuild; Node v24.18.0 as node --liftoff-only --disable-warning=ExperimentalWarning
线程与内存 / Workers CGR_THREADS=2, CGR_MEMORY_MB=2048, CODEGRAPH_PARSE_WORKERS=2, CODEGRAPH_RESOLVE_WORKERS=2, telemetry off (restricted tier, same as docs/BENCHMARKS.md)
计时规则 / Timing 逐条执行;计时命令之间 ≥ 30 s;每条执行前可用内存 ≥ 6 GiB;热文件缓存;耗时含进程启动和退出;峰值 RSS 取自 wait4 / one command at a time, ≥ 30 s gap, ≥ 6 GiB available, warm caches, wall time includes process start/exit, peak RSS from wait4
语料 / Corpus 12 个固定提交(README 表中有链接);双方索引相同文件,SHA-256 源码摘要一致 / twelve pinned commits (linked in the README); identical file sets and SHA-256 source digests on both sides
全量索引 / Full index 每边 1 次预热 + 3 次计时,交替顺序。样本来自 f4b9685,它与 cd69b83 源码相同,只差 README / 1 warmup + 3 timed rebuilds per side, alternating; samples from f4b9685, source-identical to cd69b83
增量同步 / Sync 无修改 / 追加注释 / 单文件符号改名,每类每边 3 次,交替顺序;每类结束后恢复原始字节并同步。Express、Gin、Tokio、Django 沿用 docs/BENCHMARKS.md 中的文件和符号,其余 8 个仓库各选一个核心源文件 / no change, comment edit, single-file rename; 3 timed runs per category and side; original bytes restored after each category; Express/Gin/Tokio/Django reuse the files and symbols from docs/BENCHMARKS.md
查询 / Query 热查询:20 个固定问题,先预热 8 个,再 20 × 3 共 60 次,最近秩百分位,调用进程内 explore 处理函数;新进程 CLI:第一个问题每边 3 次。Express、Gin、Tokio、Django 沿用 docs/benchmarks/2026-11-01-main-retest.json 的问题,其余 8 个仓库按相同形式用固定种子从生产代码的函数/方法/类型名生成 / warm: 20 fixed queries, 8 warm-up, then 20 × 3 in-process; CLI: first query × 3; four repos reuse the existing query lists, eight use seeded queries of the same shape
精确率/召回率 / Precision/recall 边键 (源稳定 ID, 目标稳定 ID, 类型, 行, UTF-16 列);P = 共有 / cgr,R = 共有 / 原版;门槛 P ≥ 97%、R ≥ 95% / edge key (src stable ID, dst stable ID, kind, line, UTF-16 column); gate P ≥ 97%, R ≥ 95%

结果 / Results

等权几何平均加速比 / Equal-weight geometric-mean speedups:

项目 / Metric 加速比 / Speedup
全量索引 / Full index 6.63×
无修改同步 / No-change sync 9.23×
追加注释同步 / Comment-edit sync 7.04×
单文件改名同步 / Single-file rename sync 3.14×
热查询 P50 / Warm explore P50 3.92×
新进程 CLI 查询 / New-process CLI explore 13.12×
  • 全量索引峰值 RSS 中位数降低 55.2%–87.6%。 / Median peak RSS is 55.2%–87.6% lower.
  • 精确率/召回率门槛通过 4/12:cJSON、MediatR、Slim、Sinatra。 / The precision/recall gate passes in 4 of 12 repositories.
  • 正确性检查:12 个仓库恢复后两边的严格边集合都与初始一致,cgr status --verify --strict 返回 0,cgr debug diff 输出 identical。 / In all twelve repositories both sides return to their initial strict edge sets after restoration, and cgr status --verify --strict and cgr debug diff pass.

需要注意 / Notes

  • 单文件改名可能退回全量重建。 12 个仓库中有 8 个,改名后 cgr sync 标记的受影响引用超过 30% 上限,因此重建了完整索引;例如 Tokio 65,414 个引用中有 62,419 个被标记。Tokio 因此比原版慢(753.5 → 897.4 ms,0.84×)。Tokio 和 Express 用的编辑与已删除的 b827b1d 表相同:旧表中 Tokio 改名只要 71.4 ms,远低于旧 README 里 Tokio 的全量索引(edb403e,555.3 ms),说明当时走的是增量同步;在 cd69b83 上同一编辑退回了全量重建。由于其他任务暂停,这次没有调查原因。README 中这些格子用 † 标出。
    Single-file renames can fall back to a full rebuild. In 8 of 12 repositories the rename marks more than 30% of references as affected (Tokio: 62,419 of 65,414), so cgr sync rebuilds the index, and Tokio is slower than upstream (0.84×). Tokio and Express use the same edit as the removed b827b1d table, where the Tokio rename took 71.4 ms, far below the old README's Tokio full index (edb403e, 555.3 ms), so it was incremental; at cd69b83 it falls back to a full rebuild. Not investigated here; marked † in the README.
  • cJSON 和 MediatR 的热查询 P95 略慢(19.60 → 21.37 ms,17.99 → 19.20 ms)。 / Warm P95 is slightly slower in cJSON and MediatR.
  • 机器不同。 这次的机器与旧表(Apple M5)不同,绝对耗时不能与旧数据直接比较。 / This is a different machine from the old Apple M5 tables, so absolute times are not comparable with the removed data.
  • 回答长度单位不同。 原版统计 UTF-16 码元,cgr 统计 UTF-8 字节。 / Upstream answer sizes are UTF-16 code units; cgr sizes are UTF-8 bytes.

Replace the six-repository edb403e table, the second-phase table and
the b827b1d sync/query tables in both READMEs with one twelve-repository
set measured on cd69b83 against upstream d2e0389: full index,
incremental CLI sync, query latency and graph agreement, all on one
machine and configuration. Add the structured evidence.
The development section still reported the four-repository b827b1d
retest as current; cite the cd69b83 checks and evidence instead.
@NianJiuZst
NianJiuZst merged commit 33040e9 into NianJiuZst:main Nov 2, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants