Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,34 @@
# Changelog

## Unreleased — language coverage and Rust macro items

- Vendor the upstream C/C++, C#, PHP, Ruby and Swift walkers and the Swift grammar through
kernel patch P5 (`scripts/sync-kernel.sh` re-vendors them; each language has a cargo feature).
A `.h` header with C++ constructs is parsed with the C++ grammar, as upstream does.
- Facts extractors for the five languages (scopes, typed parameters/locals/fields, imports,
namespaces), and resolution rules: `#include` by relative path and path suffix, C# namespaces
including the enclosing ones, PHP namespaced `use` names, Ruby `require`/`require_relative`,
Swift modules per SwiftPM target; implicit-`this` member calls, call-name normalisation
(PHP `->`/`parent::`/`self::`, C# `?.`/`base.`, Ruby `@ivar`, C++ arity suffixes), nested type
paths, qualified callables, and members declared in other files (partial classes, extensions,
reopened classes, out-of-line C++ definitions). Per-family builtin and common-method lists
keep untyped receivers from linking to same-named project methods.
- Rust: items inside conditional wrapper macros (`cfg_rt! { pub struct X; impl X { .. } }`,
Tokio's `cfg_*!` family) are extracted from a position-preserving view with only the wrapper
tokens blanked. Node identities and positions of everything else are unchanged; the original
text is used again if blanking introduces a syntax error. On Tokio this adds ~740 nodes and
~2,200 resolved call references.
- Self-review fixes: depth bounds (iterative helpers) in the five extractors; deterministic
tie-breaking for cross-file members and qualified names; C++ `::a::f` / `::f()` resolved in the
global namespace only (no self-links from `test::open` wrappers), `::std::..` as builtin, and
unqualified `get`/`swap`/`move`/.. in C++ files as standard functions; PHP `parent::__construct()`
and namespaced names; Ruby single-segment `require` limited to load-path roots; an explicit Swift
standard-type list with nested-type shadowing; Objective-C headers recorded as file nodes; the
`cfg_*!` pre-pass only blanks closed wrappers and only when it does not add syntax errors.
- Fix `file_node_only` building a node row 8 bytes larger than the kernel ABI row (a debug
assertion panic on any kernel failure in debug/test builds).
- **Engine rules version 5**: indexes built by an older engine are rebuilt on the next sync.

## Unreleased — engine performance follow-through

- Add event-path sync, bounded watcher events and source-free snapshot compaction after a
Expand Down
55 changes: 55 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

46 changes: 44 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,37 @@ the 97% / 95% gate in any of the six repositories; graph outputs differ and upst
semantic ground truth. See the [six-repository report](docs/BENCHMARKS-SIX-REPO-2026-11-01.md)
and [structured evidence](docs/benchmarks/2026-11-01-six-repo-retest.json).

### Second-phase languages

The **2026-11-02 twelve-repository run** of `cgr` **`f4b9685`** and upstream **`d2e0389`**
measured a **6.63× geometric-mean full-index speedup** (6.66× on the six C, C++, C#, PHP, Ruby
and Swift repositories) and **55.2%–87.6% lower median peak RSS** on a second machine (AMD
Ryzen 9 8940HX, 32 threads, 15 GiB, WSL2 Ubuntu 24.04, Node v24.18.0). The six repositories
above are re-measured here so that every row shares one machine; absolute times are not
comparable with the table above, where upstream itself is about 1.7–1.9× faster.

| Repository | Files | Upstream full index | `cgr` full index | Speedup | Peak RSS: upstream → `cgr` |
|---|---:|---:|---:|---:|---:|
| Express | 142 | 990.8 ms | 162.3 ms | **6.10×** | 218.7 → 97.9 MiB |
| Gin | 99 | 1,242.5 ms | 165.0 ms | **7.53×** | 235.1 → 67.8 MiB |
| Tokio | 814 | 4,637.9 ms | 882.0 ms | **5.26×** | 701.3 → 108.7 MiB |
| Django | 2,984 | 18,467.9 ms | 2,392.8 ms | **7.72×** | 1,936.6 → 239.4 MiB |
| Zod | 522 | 4,276.2 ms | 689.2 ms | **6.20×** | 598.2 → 142.0 MiB |
| OkHttp | 690 | 6,677.3 ms | 939.8 ms | **7.10×** | 823.0 → 151.1 MiB |
| cJSON (C) | 68 | 1,046.5 ms | 125.5 ms | **8.34×** | 307.3 → 66.5 MiB |
| fmt (C++) | 80 | 4,731.2 ms | 497.9 ms | **9.50×** | 542.0 → 107.3 MiB |
| MediatR (C#) | 159 | 1,046.3 ms | 171.3 ms | **6.11×** | 228.0 → 98.0 MiB |
| Slim (PHP) | 125 | 831.1 ms | 122.9 ms | **6.76×** | 209.2 → 61.6 MiB |
| Sinatra (Ruby) | 147 | 806.5 ms | 137.9 ms | **5.85×** | 206.3 → 70.8 MiB |
| Alamofire (Swift) | 111 | 2,048.6 ms | 450.4 ms | **4.55×** | 403.9 → 107.3 MiB |

Same method as above: one warmup and three timed CLI rebuilds per side, two workers, a 2 GiB
`cgr` memory budget, at least 30 seconds between runs, warm OS caches, the same files and bytes
on both sides, and upstream's native kernel loaded (`linux-x64` prebuild). On this machine `main`
(`2c35e7a`) indexes Express, Gin, Django, Zod and OkHttp within the three-sample spread of this
branch; Tokio is 669.5 → 882.0 ms because this branch also extracts the items inside `cfg_*!`
wrapper macros (739 more nodes). Sync and query latency were not re-measured.

The sections below retain the earlier **`b827b1d`** sync and query measurements. The new
retest completed only Express sync and did not complete query latency tests.

Expand Down Expand Up @@ -227,6 +258,12 @@ with watching enabled, it performs that synchronization automatically after a de
| Java | `.java` |
| Rust | `.rs` |
| Kotlin | `.kt`, `.kts` |
| C | `.c`, `.h` (a `.h` with C++ constructs is parsed as C++) |
| C++ | `.cpp`, `.cc`, `.cxx`, `.hpp`, `.hxx`, `.metal`, `.cu`, `.cuh` |
| C# | `.cs` |
| PHP | `.php`, `.module`, `.install`, `.theme`, `.inc` |
| Ruby | `.rb`, `.rake` |
| Swift | `.swift` |

Extraction records source structure, symbols and references. Resolution uses available scope,
import, receiver-type, module and package information. Coverage depends on language constructs
Expand Down Expand Up @@ -282,8 +319,13 @@ Not covered: MyBatis XML, Angular templates, Vue / Svelte ecosystem stores and r
events and the Fabric / Expo native side, Tauri events, GoFrame, and Python parameter-type flow
(upstream implements it for Ruby only).

C/C++, C#, PHP, Ruby, Swift and the other upstream language families are outside this version's
supported extraction set. Vue/Svelte-specific extraction is also outside the current scope.
C/C++, C#, PHP, Ruby and Swift use the upstream extraction walkers with this project's own
resolution rules (includes, namespaces, `require`, SwiftPM modules, implicit `this` members).
Against upstream `d2e0389` on strict edge keys, cJSON, MediatR, Slim and Sinatra meet the
97% / 95% gate; fmt (94.1% / 92.8%) and Alamofire (96.5% / 93.8%) do not; much of the gap is
call sites where the two pick different overloads or scopes (see [docs/LANGUAGES-P5.md](docs/LANGUAGES-P5.md)). These
languages have no framework route rules yet. Dart, Scala, Lua, Luau and R are detected but not extracted.
Vue/Svelte-specific extraction is also outside the current scope.

## Command-line reference

Expand Down
8 changes: 7 additions & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,12 @@ MCP 服务运行且启用监听时,会在防抖后自动同步源码变化。
| Java | `.java` |
| Rust | `.rs` |
| Kotlin | `.kt`、`.kts` |
| C | `.c`、`.h`(含 C++ 语法的 `.h` 按 C++ 解析) |
| C++ | `.cpp`、`.cc`、`.cxx`、`.hpp`、`.hxx`、`.metal`、`.cu`、`.cuh` |
| C# | `.cs` |
| PHP | `.php`、`.module`、`.install`、`.theme`、`.inc` |
| Ruby | `.rb`、`.rake` |
| Swift | `.swift` |

提取阶段记录源码结构、符号和引用;引用解析阶段使用可获得的作用域、导入、接收者类型、模块及包信息。
覆盖度受语言构造和项目配置影响,当前实现是静态代码图谱,不能替代完整编译器的类型检查。
Expand Down Expand Up @@ -246,7 +252,7 @@ Go 接口分派包含无歧义的提升方法;方法集推断仍是有界的
尚未覆盖:MyBatis XML、Angular 模板、Vue / Svelte 生态的 store 与路由、React Native 事件与 Fabric/Expo 原生侧、
Tauri 事件、GoFrame,以及 Python 参数类型流(上游仅对 Ruby 实现)。

当前版本暂不支持 C/C++、C#、PHP、Ruby、Swift 等上游语言,也不包含 Vue/Svelte 专用提取逻辑。
C/C++、C#、PHP、Ruby、Swift 使用上游的提取 walker,并配有本项目自己的解析规则(头文件包含、命名空间、`require`、SwiftPM 模块、隐式 `this` 成员);其解析质量只在样例仓库上做过检查,尚未与上游 oracle 对比,也还没有框架路由规则。Dart、Scala、Lua、Luau、R 只做语言识别,不做提取。当前也不包含 Vue/Svelte 专用提取逻辑。

## 命令行参考

Expand Down
2 changes: 1 addition & 1 deletion crates/cg-core/src/brand.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ pub const UPSTREAM_CONFIG_FILE: &str = "codegraph.json";
pub const CONFIG_FILE: &str = "cgr.json";
/// Version of the resolution rules: changing resolver behaviour bumps this, which
/// marks older indexes as "built by an older engine" (design doc 5.8).
pub const ENGINE_VERSION: u32 = 9;
pub const ENGINE_VERSION: u32 = 10;
/// Version of the on-disk segment format; a mismatch forces a full rebuild.
pub const FORMAT_VERSION: u32 = 2;
/// The upstream commit whose kernel is vendored (see crates/cg-kernel/UPSTREAM_COMMIT).
Expand Down
34 changes: 28 additions & 6 deletions crates/cg-core/src/kinds.rs
Original file line number Diff line number Diff line change
Expand Up @@ -139,9 +139,9 @@ impl RefKind {
}
}

/// Languages. `Unknown` is 0 so zeroed records mean "no language". Walkers exist only
/// for the MVP set (design doc 1.3); the rest are detected so configuration stays
/// forward compatible.
/// Languages. `Unknown` is 0 so zeroed records mean "no language". Walkers exist for
/// the MVP set (design doc 1.3) and C/C++, C#, PHP, Ruby and Swift; the rest are
/// detected so configuration stays forward compatible.
#[repr(u8)]
#[derive(Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, Debug)]
pub enum Lang {
Expand All @@ -166,6 +166,8 @@ pub enum Lang {
Lua = 18,
Luau = 19,
R = 20,
/// Detected by content in `.h` headers so they are not fed to the C walker; no walker.
ObjC = 21,
}

/// Languages whose symbols may reference one another by name (design doc 6.5: "same
Expand Down Expand Up @@ -210,6 +212,7 @@ impl Lang {
Lang::Lua,
Lang::Luau,
Lang::R,
Lang::ObjC,
];

/// Upstream `Language` string; also the kernel's language argument.
Expand All @@ -236,6 +239,7 @@ impl Lang {
Lang::Lua => "lua",
Lang::Luau => "luau",
Lang::R => "r",
Lang::ObjC => "objc",
}
}

Expand All @@ -251,9 +255,27 @@ impl Lang {
Lang::ALL.iter().copied().find(|l| *l as u8 == v).unwrap_or(Lang::Unknown)
}

/// Whether this build has a walker for the language (MVP set, design doc 1.3 + D2).
/// Whether this build has a walker for the language (MVP set, design doc 1.3 + D2,
/// plus the second-phase C/C++, C#, PHP, Ruby and Swift walkers).
pub fn is_supported(self) -> bool {
matches!(self, Lang::TypeScript | Lang::Tsx | Lang::JavaScript | Lang::Jsx | Lang::Python | Lang::Go | Lang::Java | Lang::Rust | Lang::Kotlin)
matches!(
self,
Lang::TypeScript
| Lang::Tsx
| Lang::JavaScript
| Lang::Jsx
| Lang::Python
| Lang::Go
| Lang::Java
| Lang::Rust
| Lang::Kotlin
| Lang::C
| Lang::Cpp
| Lang::CSharp
| Lang::Php
| Lang::Ruby
| Lang::Swift
)
}

pub fn family(self) -> LangFamily {
Expand All @@ -271,7 +293,7 @@ impl Lang {
Lang::Dart => LangFamily::Dart,
Lang::Lua | Lang::Luau => LangFamily::Lua,
Lang::R => LangFamily::R,
Lang::Unknown => LangFamily::Unknown,
Lang::Unknown | Lang::ObjC => LangFamily::Unknown,
}
}

Expand Down
2 changes: 1 addition & 1 deletion crates/cg-extract/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ license.workspace = true

[dependencies]
cg-core = { workspace = true }
cg-kernel = { workspace = true, features = ["lang-ts", "lang-python", "lang-go", "lang-java", "lang-rust", "lang-kotlin"] }
cg-kernel = { workspace = true, features = ["lang-ts", "lang-python", "lang-go", "lang-java", "lang-rust", "lang-kotlin", "lang-c", "lang-csharp", "lang-php", "lang-ruby", "lang-swift"] }
cg-scan = { workspace = true }
tree-sitter = { workspace = true }
rayon = { workspace = true }
Expand Down
Loading