Skip to content

Add C/C++, C#, PHP, Ruby and Swift; extract Rust items inside cfg_*! wrappers; engine rules 10 - #9

Merged
NianJiuZst merged 13 commits into
NianJiuZst:mainfrom
columbia11:claude/ecstatic-ride-msxkg5
Nov 2, 2026
Merged

NianJiuZst merged 13 commits into
NianJiuZst:mainfrom
columbia11:claude/ecstatic-ride-msxkg5

Conversation

@columbia11

@columbia11 columbia11 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

What this does
Language coverage (issue item 2). Kernel patch P5 vendors the upstream C/C++, C#, PHP, Ruby and Swift walkers and the Swift grammar (scripts/sync-kernel.sh re-vendors; each language has a cargo feature). A .h header with C++ constructs is parsed with the C++ grammar (upstream looksLikeCpp rules); an Objective-C header is recorded as a file node instead of being forced through the C grammar. Each language gets a facts extractor with focused tests and resolution rules:

  • #include "a/b.h" by includer-relative path, then path suffix; <...> by suffix only, else external
  • C# namespaces as packages; the enclosing namespaces are visible without using; using X = A.B.C / using static bind declarations
  • PHP use A\B\C to classes/interfaces/traits/functions/namespaces; $this->m() (emitted bare by the kernel) as an implicit-this member; parent::, self::, static::, ->, ?-> normalised; parent::__construct() reaches the parent class
  • Ruby require / require_relative by path suffix (single-segment require 'x' only under lib/, spec/, test/, app/ or the root, so require 'logger' stays the standard library); @ivar.m → self.@ivar.m; self.class.m → self.m; modules hold members
  • Swift modules per SwiftPM target; constructors of standard types (Data(), Result { try .. }, NSNumber(..)) are builtins even when the project has an extension of them, unless a project struct/enum/protocol/typealias of that name shadows it (a nested one only inside its enclosing declaration); extensions read their type's fields; Type.staticMember.method() chains; argument-label overloads
  • C++ ::a::f is an absolute qualified name and ::f() a global-namespace function (only from the calling file or the files it includes); ::std::.. and unqualified get/swap/move/.. in C++ files are standard functions; a namespace opened by a macro (FMT_BEGIN_NAMESPACE) is recognised, so fmt::format(..) calls and bases such as fmt::detail::buffer<T> resolve; a call through a field chain (pipe.write_end.close()) keeps the chain and is typed
  • members declared in other files (C# partial classes, Swift extensions, reopened Ruby classes, out-of-line C++ definitions) are found; ties between candidates are broken deterministically (caller file, then included/imported files, then path and line); per-family builtin / common-method lists keep untyped receivers from linking to same-named project methods
  • a known receiver type with no such member is no longer resolved through the unique-global-name fallback (found while spot-checking: Delegator.target in Sinatra linked to a test class method)

JS (one small, separate commit on top of #12, easy to drop): this.m() in a function with no enclosing class and no prototype owner (an object-literal method in _.mixin(Input.prototype, Emitter, {..}), a getter passed to defineGetter) reaches m when the file defines exactly one function of that name. Calls on named receivers keep #12's no-guess rule.

Rust call recall (issue item 1, Tokio). Items inside conditional wrapper macros (cfg_rt! { pub struct X; impl X { .. } }, Tokio's cfg_*! family) are opaque token trees to tree-sitter, so the kernel extracted neither the declarations nor the calls inside them. cg-extract/src/macro_mask.rs now blanks only the wrapper's name! { and closing } in a position-preserving view (newlines kept, every other byte unchanged) before the kernel runs, so line/column positions and node ids of everything else are identical. Only wrappers that really close are blanked, and the masked text is used only when it does not add syntax errors. cfg_if!, macro definitions and other macros stay opaque; nothing is expanded.
Fixes found along the way. file_node_only built a node row 8 bytes larger than the kernel ABI row (debug-assertion panic on any kernel failure in debug/test builds). cgr bench sync now knows the new languages' comment/function/rename/import edits.
Engine rules version 10 (main is 9 after #12); indexes built by an older engine rebuild on the next sync. Format stays 2.

README evaluation (upstream oracle)
Upstream colbymchenry/codegraph @ d2e0389 as the oracle, both sides indexing the same files, strict edge keys (source, target, kind, line, column), gate P ≥ 97% and R ≥ 95%.

Repository Language P R cgr / upstream / shared Gate
cJSON C 99.84 99.60 3,757 / 3,766 / 3,751 pass
MediatR C# 98.17 97.28 5,685 / 5,737 / 5,581 pass
Slim PHP 98.21 95.39 4,527 / 4,661 / 4,446 pass
Sinatra Ruby 98.62 96.30 2,613 / 2,676 / 2,577 pass
fmt C++ 94.07 92.79 20,638 / 20,923 / 19,414 fail
Alamofire Swift (+ JS in docs/) 96.46 93.78 12,164 / 12,512 / 11,734 fail

Where fmt and Alamofire fall short, much of the gap is call sites where the two sides pick different targets. Where the sampled cases were clear upstream mistakes, this PR does not copy them:

  • fmt: 680 of the 1,224 cgr-only edges are same-site disagreements. About 340 are clear upstream errors: 210 fmt::format/fmt::print calls upstream links to the text_style overloads in color.h, and ~130 constructor calls where upstream takes the first declared constructor regardless of arity (bigint(42) → bigint(), uint128() → uint128(hi, lo), basic_string_view(s, n) → the default constructor). The other ~340 are template overload picks (fill_n, format_decimal, write2digits, ...) where either side may be right. At the gate cgr-only can be at most ~615 edges, so even with only the ~340 clear errors counted as unavoidable, the remaining cgr-only edges would have to fall from ~884 to ~275 while shared edges grow by 463.
  • Alamofire: Swift alone is P 97.41 / R 96.29 (passes). The gap is the vendored jquery.min.js / typeahead.jquery.js under docs/: minified code with many same-named nested functions, where the two sides pick different targets (~126 edges, not checked one by one) or upstream links calls this project leaves unresolved (~130).

Against current main (2c35e7a), same oracle:

  • Express 64.15/87.49 → 64.18/87.75 and Zod 92.19/92.82 → 92.18/92.86. The only differences are the JS this.m() rule above: +6 / +13 shared edges, plus 2 + 2 edges upstream lacks that were checked correct (Express this.host → the host getter; Zod { callee: this.parse } → ZodMiniType::parse). Gin is identical; OkHttp and Django differ by under 0.05 points.
  • Tokio 87.40/86.52 → 78.84/88.30: the cfg_*! items extracted here do not exist upstream, so their edges are not in the oracle.

Measured performance (full index)
Same method as the README table: one warmup and three timed CLI rebuilds per side, two workers, a 2 GiB cgr memory budget, at least 30 seconds between runs, warm OS caches, the same files and bytes on both sides, upstream d2e0389 with its native kernel loaded. Measured 2026-11-02 on a second machine (AMD Ryzen 9 8940HX, 32 threads, 15 GiB, WSL2 Ubuntu 24.04, Node v24.18.0), so the six original repositories are re-measured and absolute times are not comparable with the README's existing table (upstream itself is about 1.7–1.9× faster there). Geometric-mean speedup 6.63× over all twelve (6.66× on the six new-language repositories); median peak RSS 55.2%–87.6% lower. Added to the README under "Second-phase languages".

Repository Files Upstream full index cgr full index Speedup Peak RSS: upstream → cgr
Express 142 990.8 ms 162.3 ms 6.10× 218.7 → 97.9 MiB
Gin 99 1,242.5 ms 165.0 ms 7.53× 235.1 → 67.8 MiB
Tokio 814 4,637.9 ms 882.0 ms 5.26× 701.3 → 108.7 MiB
Django 2,984 18,467.9 ms 2,392.8 ms 7.72× 1,936.6 → 239.4 MiB
Zod 522 4,276.2 ms 689.2 ms 6.20× 598.2 → 142.0 MiB
OkHttp 690 6,677.3 ms 939.8 ms 7.10× 823.0 → 151.1 MiB
cJSON (C) 68 1,046.5 ms 125.5 ms 8.34× 307.3 → 66.5 MiB
fmt (C++) 80 4,731.2 ms 497.9 ms 9.50× 542.0 → 107.3 MiB
MediatR (C#) 159 1,046.3 ms 171.3 ms 6.11× 228.0 → 98.0 MiB
Slim (PHP) 125 831.1 ms 122.9 ms 6.76× 209.2 → 61.6 MiB
Sinatra (Ruby) 147 806.5 ms 137.9 ms 5.85× 206.3 → 70.8 MiB
Alamofire (Swift) 111 2,048.6 ms 450.4 ms 4.55× 403.9 → 107.3 MiB

Against main (2c35e7a) on the same machine: Express 158.6, Gin 153.8, Django 2,387.6, Zod 694.4 and OkHttp 860.0 ms, all within the three-sample spread of this branch. Tokio is 669.5 → 882.0 ms (+32%) because this branch also extracts the items inside cfg_*! wrapper macros. Sync and query latency were not re-measured.

Validation

  • cargo test --workspace (incl. 6 × 100 random-edit incremental==full sequences); CI-scope cargo fmt --check and cargo clippy --workspace --exclude cg-kernel --all-targets -- -D warnings
  • scripts/smoke.sh; scripts/verify-repos.sh --edits 40 on fmt, Alamofire and Express (incremental == full rebuild, determinism, checksums)
  • scripts/sync-kernel.sh reproduces the committed crates/cg-kernel tree byte-for-byte from upstream 9d3abb8 + P1/P2/P4/P5-js-ts-graph/P5-second-phase-languages (the last patch regenerated on top of Fix JavaScript and TypeScript graph resolution gaps #12's P5-js-ts-graph)
  • Windows: x86_64-pc-windows-gnu (mingw-w64) cross-build succeeds including the vendored Kotlin and Swift C grammars; the binary was not executed (no Wine in the build environment) — the msvc target and install.ps1 still need a real Windows run

Not done / left for follow-up

  • fmt and Alamofire do not meet the gate (above); Express global precision and Django routes (issue item 1) are untouched.
  • macOS install/run not tested; Windows binary not executed (see above).
  • The new languages have no framework route rules yet; Dart, Scala, Lua, Luau and R are detected but not extracted.
  • Each facts extractor documents its judgement calls in its header (e.g. C++ auto x = Foo(1) constructor-vs-function by name shape). Details and caveats: docs/LANGUAGES-P5.md.

columbia11 and others added 6 commits November 1, 2026 16:37
Vendor the upstream walkers and the Swift grammar through a new kernel
patch, enable the features in cg-extract, mark the languages supported
and add stub facts extractors for them.
…t; extract Rust items inside cfg wrappers

* facts extractors for C#, PHP and Ruby (scopes, typed bindings, imports,
  namespaces); C/C++ and Swift follow
* module resolution: C/C++ #include by relative path and path suffix,
  C# namespaces (with parent namespaces), PHP namespaced names,
  Ruby require/require_relative, Swift modules per SwiftPM target
* resolver: implicit-this members for the new families, call-name
  normalisation (PHP ->, parent::, self::; C# ?. base.; Ruby @ivar;
  C++ arity suffix), nested type paths, qualified callable lookup,
  partial classes / extensions / reopened classes / out-of-line members,
  per-family builtins and common methods
* Rust: conditional item wrappers (cfg_rt! { .. }) are blanked in a
  position-preserving view before the kernel runs, so the items inside
  become nodes; node ids of everything else are unchanged
* engine rules version 5
…, document results

* facts extractors for C/C++ and Swift with tests; .h headers with C++
  constructs are parsed as C++ (upstream looksLikeCpp rules)
* a known receiver type without the member no longer falls back to a
  unique global name; Swift constructor calls of standard types are
  builtins even when the project extends the type
* bench sync knows the new languages' comment, function, rename and
  import edits; incremental == full rebuild verified on six repos
* fix file_node_only building a node row 8 bytes larger than the ABI
* README (en/zh), CHANGELOG and docs/LANGUAGES-P5.md
Robustness: depth bounds and iterative helpers in the C/C++, C#, PHP, Ruby
and Swift facts extractors; the cfg_*! pre-pass only blanks wrappers that
really close and is kept only when it adds no syntax errors; a pre-pass
timeout marks the file as timed out. Objective-C headers are recorded as
file nodes instead of being parsed with the C grammar.

Resolution: deterministic tie-breaking for cross-file members and qualified
names (caller file, then included/imported files, then path and line);
C++ `::a::f` and `::f()` resolve in the global namespace only, `::std::..`
is builtin, and unqualified get/swap/move/.. in C++ files are standard
functions; PHP parent::__construct() and namespaced names; Ruby
single-segment require limited to load-path roots; Swift standard types
from an explicit list (incl. Result, Error), shadowed by project types,
nested ones only inside their enclosing declaration.

The sample repositories were diffed reference-by-reference against the
pre-review graph and every gained or retargeted edge was checked against
source; docs/LANGUAGES-P5.md records the results.
Keep both sides: upstream's overload, framework and dynamic-dispatch
resolution and the C/C++, C#, PHP, Ruby and Swift rules. Call paths added
for the new languages now pass CallArgs through member_call and
enclosing_member. Engine rules 8, above both 5 and upstream's 7, so indexes
built by either rebuild.
@columbia11 columbia11 changed the title Add C/C++, C#, PHP, Ruby and Swift; extract Rust items inside cfg_*! wrappers; engine rules 5 Add C/C++, C#, PHP, Ruby and Swift; extract Rust items inside cfg_*! wrappers; engine rules 8 Nov 2, 2026
C/C++: port preParse, `using` declarations, constructors by arity,
return and alias types, macro-opened namespaces and override edges.
Ruby: bare calls as method calls, module members by name.
Swift: argument-label overloads, chained calls, extension fields,
nested-type and `Self` scoping, static-member chains.
A base written as fmt::detail::buffer<T>, with fmt opened by a macro, now
resolves through the same scoped lookup as other C++ types, so inherited
members of such bases are found. A call on a member of a member
(pipe.write_end.close()) keeps its field chain instead of the bare
method name the kernel gives it.
…ction

In a method of an object literal (_.mixin(Input.prototype, Emitter, {..})),
this has no enclosing class or prototype owner. The receiver is still the
object the method belongs to, not an arbitrary value, so a this.m() call
now reaches m when the file defines exactly one function of that name.
Calls on named receivers keep the existing no-guess rule.
P5-second-phase-languages.patch is rebuilt on top of main's
P5-js-ts-graph.patch and now carries the later Swift and C++ walker
changes; sync-kernel.sh reproduces crates/cg-kernel byte for byte.
README and docs/LANGUAGES-P5.md give the upstream-oracle comparison
for the six new-language repositories and against current main.
rustfmt for three long lines in resolve.rs.
@columbia11 columbia11 changed the title Add C/C++, C#, PHP, Ruby and Swift; extract Rust items inside cfg_*! wrappers; engine rules 8 Add C/C++, C#, PHP, Ruby and Swift; extract Rust items inside cfg_*! wrappers; engine rules 10 Nov 2, 2026
Twelve repositories measured on one machine with the README method:
one warmup and three timed rebuilds per side, two workers, a 2 GiB
budget and 30-second gaps, upstream d2e0389 with its native kernel.
@NianJiuZst
NianJiuZst merged commit cd69b83 into NianJiuZst:main Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants