Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,27 @@
# Changelog

## Unreleased — engine performance follow-through

- Add event-path sync, bounded watcher events and source-free snapshot compaction after a
quiet period. Keep full scanning/rebuild fallbacks for broad or structural changes.
- Spill accumulated kernel buffers to immutable temporary mappings; cap resolver caches and
indexing workers by the memory tuning budget. Use the configured pool for table construction
and lower indexing-worker priority on Linux/macOS/Windows.
- Preallocate resolution outcomes, fuse hook trigger scans, and link event keys in bounded,
deterministic parallel chunks with selective incremental reconnection.
- Keep package/module representatives ordered by source path and track package/declaration
import dependencies, preserving incremental/full equality when package membership changes.
Engine revision 2 refreshes older dependency records on rebuild.
- Compress BM25 postings, persist global document lengths, and add adaptive radix edge sorting.
**Snapshot format 2 requires rebuilding old indexes with `cgr index`.**
- Use bounded source text/line-offset views; add MCP backpressure and cooperative cancellation,
coalesce background rebuilds, release watcher resources on LRU eviction, and bound shutdown.
- Bound prompt-hook discovery; repair smoke-helper stdin and timer cleanup on macOS Bash 3.2.
- Document implementation, correctness validation and remaining boundaries in
[English and Chinese](docs/PERFORMANCE-IMPLEMENTATION.md). The historical 6.40× comparison is
unchanged; a new benchmark was refused by the resource preflight, so no additional speedup
is claimed.

## 0.1.0 — first release (milestones M0–M4)

**Engine**
Expand Down Expand Up @@ -32,4 +54,4 @@
smoke test, static musl build. Release workflow for six targets with checksums; install scripts.

Not in this release: C/C++, C#, PHP, Ruby, Swift and the other upstream languages; Vue/Svelte;
code signing and notarisation; npm/PyPI wrappers; Homebrew tap. See docs/STATUS.md.
code signing and notarisation; npm/PyPI wrappers; Homebrew tap. See the README's current limitations.
6 changes: 5 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

58 changes: 41 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,22 @@ New-process CLI measurements include startup. The warm-query P50 speedup geometr
**5.85×**. These measurements cover repository analysis and retrieval; no LLM task-completion
or token-savings benchmark was run.

### Performance engineering update

The next engine update adds event-path synchronization, source-free snapshot compaction after
a quiet period, extraction-buffer spill to temporary mappings, preallocated resolution outcomes,
fused hook scanning, selective/parallel event synthesis, compressed BM25 postings and adaptive
radix edge sorting. Query source windows use one bounded text buffer with lazy line offsets.
MCP now has bounded admission, cooperative cancellation and bounded project/watcher resources;
prompt-hook discovery and the smoke helper's stdin/timer handling have been fixed.

These changes use **snapshot format 2**. Run `cgr index /path/to/project` to rebuild an old index.
The historical speedups above still refer to `fcc59af`. A new guarded benchmark was refused
because available memory and swap did not meet the design's prerequisites, so this update has
no additional measured speedup claim. See the bilingual
[implementation and validation record](docs/PERFORMANCE-IMPLEMENTATION.md) for exact changes,
resource readings and remaining work.

## Installation

### Build and install from source
Expand Down Expand Up @@ -371,7 +387,7 @@ flags for downstream ranking and analysis.
| Variable | Behavior |
|---|---|
| `CGR_THREADS` | Indexing worker count; the detected memory budget can reduce it |
| `CGR_MEMORY_MB` | Memory tuning budget for worker/batch sizing; not a hard process RSS limit |
| `CGR_MEMORY_MB` | Tuning budget for workers, batches, kernel spill and resolver caches; not an RSS cap |
| `CGR_MCP_TOOLS` | Comma-separated list of exposed MCP tools; `CODEGRAPH_MCP_TOOLS` is a fallback |
| `CGR_EXPLORE_DEDUP=1` | Skip source already returned in the same durable MCP conversation |
| `CGR_NO_PROMPT_HOOK=1` | Disable the optional Claude Code prompt hook at runtime |
Expand All @@ -398,8 +414,9 @@ the manifest; readers retain a stable snapshot, with shared locks protecting fil

Incremental updates compare timestamps/sizes and content hashes, extract changed files, then
re-resolve references affected by changed definitions, imports, exports or hierarchy. One delta
is computed against the base; deltas do not accumulate as an unbounded stack. Thresholds merge
or rebuild the graph when a change becomes too broad. The correctness target is that the
is computed against the base; deltas do not accumulate as an unbounded stack. Quiet-period
compaction rewrites the live snapshot without re-extraction; broad reference changes or config
changes can still rebuild the graph. The correctness target is that the
incremental view equals a fresh full build, checked by canonical graph comparisons.

| Crate | Responsibility |
Expand Down Expand Up @@ -463,12 +480,19 @@ compares the current graph with a fresh full build. The benchmark wrapper adds m
and swap preflight checks and process-tree monitoring; its prerequisites may require closing
other applications before a run.

The 2026-11-01 local validation passed **116 tests, with one ignored**, and incremental/full
comparisons on all four benchmark repositories. This records that local revision's result;
Linux/Windows runs and current hosted CI were not verified in that measurement.
The performance implementation update passed **137 tests, with one ignored**, on macOS,
plus release compilation, Clippy, formatting and the CLI/MCP/hook smoke test. All four real
projects passed full/edit/restore graph equivalence and matched the prior engine's graph dumps.
The historical benchmark revision passed 116 tests. Current verification details and the benchmark preflight refusal are in
[the implementation record](docs/PERFORMANCE-IMPLEMENTATION.md). Linux/Windows runtime checks
and hosted CI results are separate from this local validation.

Code revision `f82506e` passed all nine [hosted CI checks](https://github.com/NianJiuZst/CodeGrap-Rust/actions/runs/41286327676),
including Linux/macOS/Windows tests and end-to-end smoke, lint, kernel reproducibility and the
static Linux build. Hosted correctness results are not used as performance measurements.

The CI configuration includes formatting, Clippy, kernel reproducibility, multi-platform tests
and release-build checks. The existing smoke helper has known issues described below.
and release-build checks. The smoke helper covers the CLI, MCP handshake and prompt hook.
To refresh the kernel, use `scripts/sync-kernel.sh <upstream-checkout> <commit>` and verify the
resulting patches and extraction tests. Vendored formatting is kept separate from workspace
formatting checks.
Expand All @@ -479,16 +503,16 @@ formatting checks.
explore responses differ from upstream. The benchmark has a per-repository coverage table.
- **Static-analysis limits:** ambiguity can leave references unresolved; dynamic behavior is
covered only by implemented patterns. Inspect important call paths before relying on them.
- **Memory tuning:** `CGR_MEMORY_MB` sizes some resources but does not bound all accumulated
extraction results or resolution tables. Large repositories can exceed that value.
- **Prompt-hook discovery:** the optional hook can scan broadly when started in an unindexed
directory. A root-directory probe was stopped after 88 seconds. Use `install --no-hook` or
`CGR_NO_PROMPT_HOOK=1` when relying on CLI/MCP retrieval without this hook.
- **MCP execution budget:** a timeout response does not cancel a query already running in the
worker pool. The current service should not be treated as a hard-isolated query executor.
- **Smoke helper:** `scripts/smoke.sh` currently loses stdin for its background MCP command,
and its timer can retain the output pipe. Direct-stdin MCP validation passed; the helper's
failed run is recorded in the benchmark notes.
- **Memory tuning:** kernel-buffer spill and resolver cache limits release heap memory, but
facts, graph builders and shared tables still scale with repository size. The tuning budget
does not enforce a process RSS cap.
- **Incremental scope:** filesystem event probing is local, but inventory assembly and the
base-relative delta still do broader in-memory work. Structural changes can trigger wider
resolution/synthesis; background compaction with delta rebasing remains M6 work.
- **Execution deadlines:** query/hook cancellation is cooperative; a blocking filesystem call
cannot be forcibly interrupted by the worker deadline.
- **New performance targets:** the latest optimizations have correctness coverage; their
throughput, P95/P99 latency and large-repository RSS still need a new guarded measurement.

## Contributing

Expand Down
41 changes: 30 additions & 11 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,18 @@ Node.js 运行时或数据库服务。原生索引保存在项目的 `.cgr/` 目
热查询测量进程内的 explore 处理函数,不含 MCP 传输与排队;新进程 CLI 测量包含进程启动。
热查询 P50 加速比的几何平均为 **5.85×**。这些数据衡量代码分析与检索,本次没有运行 LLM 任务完成或 Token 节省测试。

### 本轮性能工程更新

后续开发已加入按事件路径同步、静默期纯数据快照合并、提取缓冲临时映射降级、解析结果预分配、
hook 融合扫描、事件合成的局部/并行连接、BM25 倒排压缩和自适应基数排序。查询源码窗口改用
单份有界文本与延迟行偏移。MCP 增加排队上限、协作取消和项目/监听资源回收;提示钩子的发现范围、
smoke 脚本的 stdin 与计时器问题也已修复。

快照格式升级为 **version 2**,升级后运行 `cgr index /path/to/project` 重建旧索引。
上方历史加速数据仍对应 `fcc59af`。本轮正式复测因可用内存和 Swap 不满足设计要求而未启动,
暂不增加新的性能提升百分比。具体实现、验证、资源读数与待办见
[中英双语优化记录](docs/PERFORMANCE-IMPLEMENTATION.md)。

## 安装

### 从源码构建并安装
Expand Down Expand Up @@ -332,7 +344,7 @@ cgr serve --http 127.0.0.1:7878 --path /absolute/path/to/project
| 变量 | 行为 |
|---|---|
| `CGR_THREADS` | 索引工作线程数,检测到的内存预算可能降低实际数量 |
| `CGR_MEMORY_MB` | 用于线程与批次大小的内存调优预算,不是进程 RSS 硬上限 |
| `CGR_MEMORY_MB` | 用于线程、批次、kernel 临时映射和解析缓存的调优预算,不是 RSS 硬上限 |
| `CGR_MCP_TOOLS` | 逗号分隔的 MCP 工具列表,兼容回退到 `CODEGRAPH_MCP_TOOLS` |
| `CGR_EXPLORE_DEDUP=1` | 在同一个持久 MCP 对话中跳过已返回源码 |
| `CGR_NO_PROMPT_HOOK=1` | 运行时禁用可选的 Claude Code 提示钩子 |
Expand All @@ -357,7 +369,8 @@ CLI / MCP ← 查询处理 ← base + delta 视图 ← mmap 段 ← 发布快照
发布阶段写入新的不可变文件,再切换 manifest;读者持有稳定快照,通过共享锁防止所用文件被 GC 提前回收。

增量更新比较时间戳、大小及内容哈希,提取变化文件,并重新解析受定义、导入、导出或继承关系影响的引用。
一个 delta 始终针对 base 计算,不会形成无限叠加的层;变更过大时按阈值合并或重建。
一个 delta 始终针对 base 计算,不会形成无限叠加的层;静默期合并直接重写存活快照,不重新提取。
受影响引用过多或配置变化时仍可全量重建。
正确性目标是增量视图与全量重建一致,通过 canonical 图谱对比验证。

| Crate | 职责 |
Expand Down Expand Up @@ -416,24 +429,30 @@ scripts/bench-local.sh --help
`debug diff` 比较当前图谱和全量重建结果。benchmark 包装脚本提供内存、磁盘、Swap 前置检查及进程树监控,
其运行条件可能需要先关闭其他应用。

2026-11-01 的本机验证结果为 **116 项测试通过、1 项忽略**,四个真实仓库的增量与全量对比均通过。
该结果针对当时的本机版本;本轮测量没有验证 Linux/Windows 运行情况或当前远程 CI。
本轮性能实现已在 macOS 通过 **137 项测试、1 项忽略**,以及 release 编译、Clippy、格式和 CLI/MCP/提示钩子 smoke 检查。
四个真实项目的初始/编辑/恢复图谱均与全量一致,图谱输出也与旧引擎相同。历史 benchmark 版本曾通过 116 项测试。
当前验证细节及性能复测的前置检查结果见[优化记录](docs/PERFORMANCE-IMPLEMENTATION.md);
Linux/Windows 运行情况与远程 CI 应单独确认。

代码版本 `f82506e` 已通过全部 9 项[远程 CI 检查](https://github.com/NianJiuZst/CodeGrap-Rust/actions/runs/41286327676):
Linux/macOS/Windows 测试与端到端 smoke、格式和 Clippy、内核来源复现、静态 Linux 构建。
远程正确性检查不作为性能测量结果。

CI 配置包含格式检查、Clippy、内核可重复同步、多平台测试与发布构建检查。
现有 smoke 辅助脚本仍有下述已知问题。更新内核时使用
Smoke 脚本覆盖 CLI、MCP 握手与提示钩子。更新内核时使用
`scripts/sync-kernel.sh <upstream-checkout> <commit>`,并验证补丁和提取测试;vendoring 代码保留上游格式,
与 workspace 自有代码的格式检查分开处理。

## 当前限制

- **覆盖度对齐**:语言范围、框架路由、合成边和 explore 返回内容与上游仍有差异,实测文档提供逐仓库对照。
- **静态分析边界**:歧义引用可能保持未解析,动态行为仅覆盖已实现的模式;重要调用链仍应结合源码核查。
- **内存调优**:`CGR_MEMORY_MB` 没有限制全部累计提取结果和引用解析表,大型仓库的 RSS 可能超过该值。
- **提示钩子发现范围**:从未索引目录启动时,可选钩子可能广泛扫描;根目录探测在 88 秒后被终止。
可用 `install --no-hook` 或 `CGR_NO_PROMPT_HOOK=1`,通过 CLI/MCP 获取上下文而不启用该钩子。
- **MCP 执行预算**:超时响应不会取消线程池中已开始的查询,当前服务不提供查询执行的硬隔离。
- **Smoke 辅助脚本**:`scripts/smoke.sh` 的后台 MCP 命令会丢失 stdin,计时器也可能持有输出管道。
独立 stdin MCP 验证已通过,辅助脚本的失败结果保留在实测说明中。
- **内存调优**:kernel 临时映射和解析缓存上限可以释放堆内存,但 facts、图谱构建数组和共享表仍随项目增长;
调优预算不能保证进程 RSS 不超过该值。
- **增量范围**:文件系统事件探测已局部化,内存清单和相对 base 的 delta 仍有较广的处理范围;
结构变化可能扩大解析/合成。后台合并与 delta 变基仍属于 M6。
- **执行期限**:查询和提示钩子采用协作取消,阻塞的文件系统调用不能被线程期限强制中断。
- **新增性能目标**:本轮优化已有正确性覆盖,吞吐、P95/P99 延迟和大仓库 RSS 仍需受保护的正式复测。

## 参与贡献

Expand Down
4 changes: 2 additions & 2 deletions crates/cg-core/src/brand.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ pub const UPSTREAM_CONFIG_FILE: &str = "codegraph.json";
pub const CONFIG_FILE: &str = "cgr.json";
/// Version of the resolution rules: changing resolver behaviour bumps this, which
/// marks older indexes as "built by an older engine" (design doc 5.8).
pub const ENGINE_VERSION: u32 = 1;
pub const ENGINE_VERSION: u32 = 2;
/// Version of the on-disk segment format; a mismatch forces a full rebuild.
pub const FORMAT_VERSION: u32 = 1;
pub const FORMAT_VERSION: u32 = 2;
/// The upstream commit whose kernel is vendored (see crates/cg-kernel/UPSTREAM_COMMIT).
pub const UPSTREAM_KERNEL_COMMIT: &str = include_str!("../../cg-kernel/UPSTREAM_COMMIT");

Expand Down
Loading
Loading