diff --git a/CHANGELOG.md b/CHANGELOG.md index 36a7459..702ee16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,27 @@ # Changelog +## Unreleased — engine performance follow-through + +- Add event-path sync, bounded watcher events and source-free snapshot compaction after a + quiet period. Keep full scanning/rebuild fallbacks for broad or structural changes. +- Spill accumulated kernel buffers to immutable temporary mappings; cap resolver caches and + indexing workers by the memory tuning budget. Use the configured pool for table construction + and lower indexing-worker priority on Linux/macOS/Windows. +- Preallocate resolution outcomes, fuse hook trigger scans, and link event keys in bounded, + deterministic parallel chunks with selective incremental reconnection. +- Keep package/module representatives ordered by source path and track package/declaration + import dependencies, preserving incremental/full equality when package membership changes. + Engine revision 2 refreshes older dependency records on rebuild. +- Compress BM25 postings, persist global document lengths, and add adaptive radix edge sorting. + **Snapshot format 2 requires rebuilding old indexes with `cgr index`.** +- Use bounded source text/line-offset views; add MCP backpressure and cooperative cancellation, + coalesce background rebuilds, release watcher resources on LRU eviction, and bound shutdown. +- Bound prompt-hook discovery; repair smoke-helper stdin and timer cleanup on macOS Bash 3.2. +- Document implementation, correctness validation and remaining boundaries in + [English and Chinese](docs/PERFORMANCE-IMPLEMENTATION.md). The historical 6.40× comparison is + unchanged; a new benchmark was refused by the resource preflight, so no additional speedup + is claimed. + ## 0.1.0 — first release (milestones M0–M4) **Engine** @@ -32,4 +54,4 @@ smoke test, static musl build. Release workflow for six targets with checksums; install scripts. Not in this release: C/C++, C#, PHP, Ruby, Swift and the other upstream languages; Vue/Svelte; -code signing and notarisation; npm/PyPI wrappers; Homebrew tap. See docs/STATUS.md. +code signing and notarisation; npm/PyPI wrappers; Homebrew tap. See the README's current limitations. diff --git a/Cargo.lock b/Cargo.lock index 665b1b0..7a056fa 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -206,11 +206,12 @@ dependencies = [ name = "cg-extract" version = "0.1.0" dependencies = [ + "aho-corasick", "cg-core", "cg-kernel", "cg-scan", - "libc", "memchr", + "memmap2", "rayon", "regex", "rustc-hash", @@ -275,6 +276,7 @@ dependencies = [ "cg-store", "fst", "globset", + "memchr", "rustc-hash", "serde_json", "smallvec", @@ -355,6 +357,7 @@ dependencies = [ "proptest", "rayon", "rustc-hash", + "serde_json", "tempfile", "tracing", ] @@ -380,6 +383,7 @@ dependencies = [ "regex", "rusqlite", "serde_json", + "tempfile", "tracing", "tracing-subscriber", ] diff --git a/README.md b/README.md index c33528d..0d91356 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,22 @@ New-process CLI measurements include startup. The warm-query P50 speedup geometr **5.85×**. These measurements cover repository analysis and retrieval; no LLM task-completion or token-savings benchmark was run. +### Performance engineering update + +The next engine update adds event-path synchronization, source-free snapshot compaction after +a quiet period, extraction-buffer spill to temporary mappings, preallocated resolution outcomes, +fused hook scanning, selective/parallel event synthesis, compressed BM25 postings and adaptive +radix edge sorting. Query source windows use one bounded text buffer with lazy line offsets. +MCP now has bounded admission, cooperative cancellation and bounded project/watcher resources; +prompt-hook discovery and the smoke helper's stdin/timer handling have been fixed. + +These changes use **snapshot format 2**. Run `cgr index /path/to/project` to rebuild an old index. +The historical speedups above still refer to `fcc59af`. A new guarded benchmark was refused +because available memory and swap did not meet the design's prerequisites, so this update has +no additional measured speedup claim. See the bilingual +[implementation and validation record](docs/PERFORMANCE-IMPLEMENTATION.md) for exact changes, +resource readings and remaining work. + ## Installation ### Build and install from source @@ -371,7 +387,7 @@ flags for downstream ranking and analysis. | Variable | Behavior | |---|---| | `CGR_THREADS` | Indexing worker count; the detected memory budget can reduce it | -| `CGR_MEMORY_MB` | Memory tuning budget for worker/batch sizing; not a hard process RSS limit | +| `CGR_MEMORY_MB` | Tuning budget for workers, batches, kernel spill and resolver caches; not an RSS cap | | `CGR_MCP_TOOLS` | Comma-separated list of exposed MCP tools; `CODEGRAPH_MCP_TOOLS` is a fallback | | `CGR_EXPLORE_DEDUP=1` | Skip source already returned in the same durable MCP conversation | | `CGR_NO_PROMPT_HOOK=1` | Disable the optional Claude Code prompt hook at runtime | @@ -398,8 +414,9 @@ the manifest; readers retain a stable snapshot, with shared locks protecting fil Incremental updates compare timestamps/sizes and content hashes, extract changed files, then re-resolve references affected by changed definitions, imports, exports or hierarchy. One delta -is computed against the base; deltas do not accumulate as an unbounded stack. Thresholds merge -or rebuild the graph when a change becomes too broad. The correctness target is that the +is computed against the base; deltas do not accumulate as an unbounded stack. Quiet-period +compaction rewrites the live snapshot without re-extraction; broad reference changes or config +changes can still rebuild the graph. The correctness target is that the incremental view equals a fresh full build, checked by canonical graph comparisons. | Crate | Responsibility | @@ -463,12 +480,19 @@ compares the current graph with a fresh full build. The benchmark wrapper adds m and swap preflight checks and process-tree monitoring; its prerequisites may require closing other applications before a run. -The 2026-11-01 local validation passed **116 tests, with one ignored**, and incremental/full -comparisons on all four benchmark repositories. This records that local revision's result; -Linux/Windows runs and current hosted CI were not verified in that measurement. +The performance implementation update passed **137 tests, with one ignored**, on macOS, +plus release compilation, Clippy, formatting and the CLI/MCP/hook smoke test. All four real +projects passed full/edit/restore graph equivalence and matched the prior engine's graph dumps. +The historical benchmark revision passed 116 tests. Current verification details and the benchmark preflight refusal are in +[the implementation record](docs/PERFORMANCE-IMPLEMENTATION.md). Linux/Windows runtime checks +and hosted CI results are separate from this local validation. + +Code revision `f82506e` passed all nine [hosted CI checks](https://github.com/NianJiuZst/CodeGrap-Rust/actions/runs/41286327676), +including Linux/macOS/Windows tests and end-to-end smoke, lint, kernel reproducibility and the +static Linux build. Hosted correctness results are not used as performance measurements. The CI configuration includes formatting, Clippy, kernel reproducibility, multi-platform tests -and release-build checks. The existing smoke helper has known issues described below. +and release-build checks. The smoke helper covers the CLI, MCP handshake and prompt hook. To refresh the kernel, use `scripts/sync-kernel.sh ` and verify the resulting patches and extraction tests. Vendored formatting is kept separate from workspace formatting checks. @@ -479,16 +503,16 @@ formatting checks. explore responses differ from upstream. The benchmark has a per-repository coverage table. - **Static-analysis limits:** ambiguity can leave references unresolved; dynamic behavior is covered only by implemented patterns. Inspect important call paths before relying on them. -- **Memory tuning:** `CGR_MEMORY_MB` sizes some resources but does not bound all accumulated - extraction results or resolution tables. Large repositories can exceed that value. -- **Prompt-hook discovery:** the optional hook can scan broadly when started in an unindexed - directory. A root-directory probe was stopped after 88 seconds. Use `install --no-hook` or - `CGR_NO_PROMPT_HOOK=1` when relying on CLI/MCP retrieval without this hook. -- **MCP execution budget:** a timeout response does not cancel a query already running in the - worker pool. The current service should not be treated as a hard-isolated query executor. -- **Smoke helper:** `scripts/smoke.sh` currently loses stdin for its background MCP command, - and its timer can retain the output pipe. Direct-stdin MCP validation passed; the helper's - failed run is recorded in the benchmark notes. +- **Memory tuning:** kernel-buffer spill and resolver cache limits release heap memory, but + facts, graph builders and shared tables still scale with repository size. The tuning budget + does not enforce a process RSS cap. +- **Incremental scope:** filesystem event probing is local, but inventory assembly and the + base-relative delta still do broader in-memory work. Structural changes can trigger wider + resolution/synthesis; background compaction with delta rebasing remains M6 work. +- **Execution deadlines:** query/hook cancellation is cooperative; a blocking filesystem call + cannot be forcibly interrupted by the worker deadline. +- **New performance targets:** the latest optimizations have correctness coverage; their + throughput, P95/P99 latency and large-repository RSS still need a new guarded measurement. ## Contributing diff --git a/README.zh-CN.md b/README.zh-CN.md index 63d4b85..42489bc 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -79,6 +79,18 @@ Node.js 运行时或数据库服务。原生索引保存在项目的 `.cgr/` 目 热查询测量进程内的 explore 处理函数,不含 MCP 传输与排队;新进程 CLI 测量包含进程启动。 热查询 P50 加速比的几何平均为 **5.85×**。这些数据衡量代码分析与检索,本次没有运行 LLM 任务完成或 Token 节省测试。 +### 本轮性能工程更新 + +后续开发已加入按事件路径同步、静默期纯数据快照合并、提取缓冲临时映射降级、解析结果预分配、 +hook 融合扫描、事件合成的局部/并行连接、BM25 倒排压缩和自适应基数排序。查询源码窗口改用 +单份有界文本与延迟行偏移。MCP 增加排队上限、协作取消和项目/监听资源回收;提示钩子的发现范围、 +smoke 脚本的 stdin 与计时器问题也已修复。 + +快照格式升级为 **version 2**,升级后运行 `cgr index /path/to/project` 重建旧索引。 +上方历史加速数据仍对应 `fcc59af`。本轮正式复测因可用内存和 Swap 不满足设计要求而未启动, +暂不增加新的性能提升百分比。具体实现、验证、资源读数与待办见 +[中英双语优化记录](docs/PERFORMANCE-IMPLEMENTATION.md)。 + ## 安装 ### 从源码构建并安装 @@ -332,7 +344,7 @@ cgr serve --http 127.0.0.1:7878 --path /absolute/path/to/project | 变量 | 行为 | |---|---| | `CGR_THREADS` | 索引工作线程数,检测到的内存预算可能降低实际数量 | -| `CGR_MEMORY_MB` | 用于线程与批次大小的内存调优预算,不是进程 RSS 硬上限 | +| `CGR_MEMORY_MB` | 用于线程、批次、kernel 临时映射和解析缓存的调优预算,不是 RSS 硬上限 | | `CGR_MCP_TOOLS` | 逗号分隔的 MCP 工具列表,兼容回退到 `CODEGRAPH_MCP_TOOLS` | | `CGR_EXPLORE_DEDUP=1` | 在同一个持久 MCP 对话中跳过已返回源码 | | `CGR_NO_PROMPT_HOOK=1` | 运行时禁用可选的 Claude Code 提示钩子 | @@ -357,7 +369,8 @@ CLI / MCP ← 查询处理 ← base + delta 视图 ← mmap 段 ← 发布快照 发布阶段写入新的不可变文件,再切换 manifest;读者持有稳定快照,通过共享锁防止所用文件被 GC 提前回收。 增量更新比较时间戳、大小及内容哈希,提取变化文件,并重新解析受定义、导入、导出或继承关系影响的引用。 -一个 delta 始终针对 base 计算,不会形成无限叠加的层;变更过大时按阈值合并或重建。 +一个 delta 始终针对 base 计算,不会形成无限叠加的层;静默期合并直接重写存活快照,不重新提取。 +受影响引用过多或配置变化时仍可全量重建。 正确性目标是增量视图与全量重建一致,通过 canonical 图谱对比验证。 | Crate | 职责 | @@ -416,11 +429,17 @@ scripts/bench-local.sh --help `debug diff` 比较当前图谱和全量重建结果。benchmark 包装脚本提供内存、磁盘、Swap 前置检查及进程树监控, 其运行条件可能需要先关闭其他应用。 -2026-11-01 的本机验证结果为 **116 项测试通过、1 项忽略**,四个真实仓库的增量与全量对比均通过。 -该结果针对当时的本机版本;本轮测量没有验证 Linux/Windows 运行情况或当前远程 CI。 +本轮性能实现已在 macOS 通过 **137 项测试、1 项忽略**,以及 release 编译、Clippy、格式和 CLI/MCP/提示钩子 smoke 检查。 +四个真实项目的初始/编辑/恢复图谱均与全量一致,图谱输出也与旧引擎相同。历史 benchmark 版本曾通过 116 项测试。 +当前验证细节及性能复测的前置检查结果见[优化记录](docs/PERFORMANCE-IMPLEMENTATION.md); +Linux/Windows 运行情况与远程 CI 应单独确认。 + +代码版本 `f82506e` 已通过全部 9 项[远程 CI 检查](https://github.com/NianJiuZst/CodeGrap-Rust/actions/runs/41286327676): +Linux/macOS/Windows 测试与端到端 smoke、格式和 Clippy、内核来源复现、静态 Linux 构建。 +远程正确性检查不作为性能测量结果。 CI 配置包含格式检查、Clippy、内核可重复同步、多平台测试与发布构建检查。 -现有 smoke 辅助脚本仍有下述已知问题。更新内核时使用 +Smoke 脚本覆盖 CLI、MCP 握手与提示钩子。更新内核时使用 `scripts/sync-kernel.sh `,并验证补丁和提取测试;vendoring 代码保留上游格式, 与 workspace 自有代码的格式检查分开处理。 @@ -428,12 +447,12 @@ CI 配置包含格式检查、Clippy、内核可重复同步、多平台测试 - **覆盖度对齐**:语言范围、框架路由、合成边和 explore 返回内容与上游仍有差异,实测文档提供逐仓库对照。 - **静态分析边界**:歧义引用可能保持未解析,动态行为仅覆盖已实现的模式;重要调用链仍应结合源码核查。 -- **内存调优**:`CGR_MEMORY_MB` 没有限制全部累计提取结果和引用解析表,大型仓库的 RSS 可能超过该值。 -- **提示钩子发现范围**:从未索引目录启动时,可选钩子可能广泛扫描;根目录探测在 88 秒后被终止。 - 可用 `install --no-hook` 或 `CGR_NO_PROMPT_HOOK=1`,通过 CLI/MCP 获取上下文而不启用该钩子。 -- **MCP 执行预算**:超时响应不会取消线程池中已开始的查询,当前服务不提供查询执行的硬隔离。 -- **Smoke 辅助脚本**:`scripts/smoke.sh` 的后台 MCP 命令会丢失 stdin,计时器也可能持有输出管道。 - 独立 stdin MCP 验证已通过,辅助脚本的失败结果保留在实测说明中。 +- **内存调优**:kernel 临时映射和解析缓存上限可以释放堆内存,但 facts、图谱构建数组和共享表仍随项目增长; + 调优预算不能保证进程 RSS 不超过该值。 +- **增量范围**:文件系统事件探测已局部化,内存清单和相对 base 的 delta 仍有较广的处理范围; + 结构变化可能扩大解析/合成。后台合并与 delta 变基仍属于 M6。 +- **执行期限**:查询和提示钩子采用协作取消,阻塞的文件系统调用不能被线程期限强制中断。 +- **新增性能目标**:本轮优化已有正确性覆盖,吞吐、P95/P99 延迟和大仓库 RSS 仍需受保护的正式复测。 ## 参与贡献 diff --git a/crates/cg-core/src/brand.rs b/crates/cg-core/src/brand.rs index 2f65417..3489294 100644 --- a/crates/cg-core/src/brand.rs +++ b/crates/cg-core/src/brand.rs @@ -17,9 +17,9 @@ pub const UPSTREAM_CONFIG_FILE: &str = "codegraph.json"; pub const CONFIG_FILE: &str = "cgr.json"; /// Version of the resolution rules: changing resolver behaviour bumps this, which /// marks older indexes as "built by an older engine" (design doc 5.8). -pub const ENGINE_VERSION: u32 = 1; +pub const ENGINE_VERSION: u32 = 2; /// Version of the on-disk segment format; a mismatch forces a full rebuild. -pub const FORMAT_VERSION: u32 = 1; +pub const FORMAT_VERSION: u32 = 2; /// The upstream commit whose kernel is vendored (see crates/cg-kernel/UPSTREAM_COMMIT). pub const UPSTREAM_KERNEL_COMMIT: &str = include_str!("../../cg-kernel/UPSTREAM_COMMIT"); diff --git a/crates/cg-core/src/budget.rs b/crates/cg-core/src/budget.rs index 20f7057..bbbea31 100644 --- a/crates/cg-core/src/budget.rs +++ b/crates/cg-core/src/budget.rs @@ -1,5 +1,5 @@ -//! Memory budget and thread count (design doc 3.3). Every cache, queue and batch size in -//! the engine derives from `MemBudget`; nothing grows without a bound. +//! Memory tuning budget and thread count (design doc 3.3). Worker counts, batch sizes, +//! spill thresholds and resolver cache limits derive from this budget. It is not an RSS cap. use crate::brand::env; @@ -25,7 +25,7 @@ impl MemBudget { pub fn from_total(total: u64, threads: usize) -> MemBudget { let total = total.max(256 * MB); let by_mem = (total / 4 / PER_THREAD_BYTES).max(1) as usize; - MemBudget { total, extract: total / 4, resolve: total * 35 / 100, synth: total * 15 / 100, threads: threads.min(by_mem).max(1) } + MemBudget { total, extract: total / 4, resolve: total / 100 * 35, synth: total / 100 * 15, threads: threads.min(by_mem).max(1) } } /// Detect from the environment, honouring `CGR_MEMORY_MB` / `CGR_THREADS`. @@ -34,16 +34,29 @@ impl MemBudget { let threads = env::get(env::THREADS, None).and_then(|v| v.parse::().ok()).filter(|n| *n > 0).unwrap_or(cores); let total = env::get(env::MEMORY_MB, None) .and_then(|v| v.parse::().ok()) - .map(|mb| mb * MB) + .map(|mb| mb.saturating_mul(MB)) .unwrap_or_else(|| available_memory().map(|a| a * 6 / 10).unwrap_or(2048 * MB)); MemBudget::from_total(total, threads) } + /// Apply command-line overrides without allowing an explicit thread count to bypass + /// the memory-derived worker limit. + pub fn configured(memory_mb: Option, threads: Option) -> MemBudget { + let detected = Self::detect(); + Self::from_total(memory_mb.map(|mb| mb.saturating_mul(MB)).unwrap_or(detected.total), threads.unwrap_or(detected.threads)) + } + /// Source bytes one extraction batch may hold (a fraction of the extract share, since a /// file's tables are several times its source). pub fn extract_batch_bytes(&self) -> u64 { (self.extract / 8).clamp(8 * MB, 256 * MB) } + + /// One worker's optional resolver cache share. Clearing a cache only recomputes a + /// lookup; it never truncates candidate sets or changes resolution semantics. + pub fn worker_cache_bytes(&self) -> usize { + (self.resolve / self.threads.max(1) as u64 / 8).clamp(64 << 10, 16 << 20) as usize + } } /// Memory this process could still use, in bytes (design doc 3.3): the smaller of the @@ -57,7 +70,11 @@ pub fn available_memory() -> Option { { macos_available() } - #[cfg(not(any(target_os = "linux", target_os = "macos")))] + #[cfg(windows)] + { + windows_available() + } + #[cfg(not(any(target_os = "linux", target_os = "macos", windows)))] { None } @@ -94,22 +111,51 @@ fn linux_available() -> Option { } #[cfg(target_os = "macos")] +#[allow(deprecated)] // libc still exposes the Mach VM ABI used here; avoid another runtime dependency. fn macos_available() -> Option { - // vm_stat reports pages; free + inactive + speculative is what the system can hand out. - let out = std::process::Command::new("vm_stat").output().ok()?; - let text = String::from_utf8_lossy(&out.stdout); - let page: u64 = text - .lines() - .next() - .and_then(|l| l.split("page size of ").nth(1)) - .and_then(|l| l.split_whitespace().next()) - .and_then(|n| n.parse().ok()) - .unwrap_or(16384); - let pages = |key: &str| -> u64 { - text.lines().find(|l| l.starts_with(key)).and_then(|l| l.split(':').nth(1)).and_then(|v| v.trim().trim_end_matches('.').parse().ok()).unwrap_or(0) - }; - let free = pages("Pages free") + pages("Pages inactive") + pages("Pages speculative"); - Some(free * page) + extern "C" { + fn mach_port_deallocate(task: libc::mach_port_t, name: libc::mach_port_t) -> libc::kern_return_t; + } + // Use Mach directly: detecting a budget must not spawn vm_stat on a hot build path. + // SAFETY: the kernel fills the correctly sized plain VM statistics struct we own. + unsafe { + let host = libc::mach_host_self(); + let mut vm: libc::vm_statistics64 = std::mem::zeroed(); + let mut count = libc::HOST_VM_INFO64_COUNT; + let result = libc::host_statistics64(host, libc::HOST_VM_INFO64, (&mut vm as *mut libc::vm_statistics64).cast(), &mut count); + let page = libc::sysconf(libc::_SC_PAGESIZE); + mach_port_deallocate(libc::mach_task_self(), host); + if result != libc::KERN_SUCCESS || page <= 0 { + return None; + } + Some((vm.free_count as u64 + vm.inactive_count as u64 + vm.speculative_count as u64) * page as u64) + } +} + +#[cfg(windows)] +fn windows_available() -> Option { + #[repr(C)] + struct MemoryStatus { + length: u32, + load: u32, + total_phys: u64, + avail_phys: u64, + total_page: u64, + avail_page: u64, + total_virtual: u64, + avail_virtual: u64, + avail_extended: u64, + } + #[link(name = "kernel32")] + extern "system" { + fn GlobalMemoryStatusEx(status: *mut MemoryStatus) -> i32; + } + // SAFETY: Windows checks length, then fills this documented MEMORYSTATUSEX layout. + unsafe { + let mut status: MemoryStatus = std::mem::zeroed(); + status.length = std::mem::size_of::() as u32; + (GlobalMemoryStatusEx(&mut status) != 0).then_some(status.avail_phys) + } } /// Peak resident set size of this process in MiB (best effort, 0 if unknown). @@ -140,6 +186,39 @@ pub fn current_rss_mb() -> u64 { 0 } +/// Lower only the indexing worker's scheduling priority, leaving the MCP / query threads +/// responsive. macOS uses thread QoS because process-wide nice would also lower readers. +pub fn lower_thread_priority() { + #[cfg(target_os = "linux")] + // SAFETY: on Linux, who=0 changes the calling thread's priority. + unsafe { + libc::setpriority(libc::PRIO_PROCESS, 0, 10); + } + #[cfg(target_os = "macos")] + { + extern "C" { + fn pthread_set_qos_class_self_np(class: u32, relative: i32) -> i32; + } + // QOS_CLASS_UTILITY, from the macOS pthread/qos.h ABI. + // SAFETY: changes the scheduling class of this worker only. + unsafe { + pthread_set_qos_class_self_np(0x11, 0); + } + } + #[cfg(windows)] + { + #[link(name = "kernel32")] + extern "system" { + fn GetCurrentThread() -> *mut std::ffi::c_void; + fn SetThreadPriority(thread: *mut std::ffi::c_void, priority: i32) -> i32; + } + // SAFETY: the pseudo-handle is valid for this thread; -1 is BELOW_NORMAL. + unsafe { + SetThreadPriority(GetCurrentThread(), -1); + } + } +} + #[cfg(test)] mod tests { use super::*; @@ -156,4 +235,12 @@ mod tests { let b = MemBudget::from_total(256 * MB, 16); assert_eq!(b.threads, 1); } + #[test] + fn explicit_threads_still_respect_the_memory_share() { + let b = MemBudget::configured(Some(256), Some(1024)); + assert_eq!(b.total, 256 * MB); + assert_eq!(b.threads, 1); + #[cfg(any(target_os = "linux", target_os = "macos", windows))] + assert!(available_memory().is_some()); + } } diff --git a/crates/cg-core/src/cancel.rs b/crates/cg-core/src/cancel.rs new file mode 100644 index 0000000..cde6448 --- /dev/null +++ b/crates/cg-core/src/cancel.rs @@ -0,0 +1,115 @@ +//! Cooperative deadlines shared by the query engine and its transports. Cancellation is +//! scoped to a worker, so concurrent requests never cancel one another. + +use std::cell::RefCell; +use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, +}; +use std::time::{Duration, Instant}; + +#[derive(Clone, Debug)] +pub struct Cancellation { + cancelled: Arc, + deadline: Instant, +} + +#[derive(Debug)] +pub struct Cancelled; + +thread_local! { + static CURRENT: RefCell> = const { RefCell::new(None) }; +} + +impl Cancellation { + pub fn new(timeout: Duration) -> Self { + Self { cancelled: Arc::new(AtomicBool::new(false)), deadline: Instant::now() + timeout } + } + pub fn cancel(&self) { + self.cancelled.store(true, Ordering::Relaxed); + } + pub fn is_cancelled(&self) -> bool { + self.cancelled.load(Ordering::Relaxed) || Instant::now() >= self.deadline + } + pub fn run(&self, f: impl FnOnce() -> T) -> T { + struct Restore(Option<(Cancellation, u16)>); + impl Drop for Restore { + fn drop(&mut self) { + CURRENT.with(|c| *c.borrow_mut() = self.0.take()); + } + } + let _restore = Restore(CURRENT.with(|c| c.replace(Some((self.clone(), 0))))); + check_now(); + f() + } +} + +/// Cheap checkpoints may be placed in hot graph accessors. Read the clock every 128 +/// accesses, rather than adding a clock read to every edge. +#[inline] +pub fn check() { + let cancelled = CURRENT.with(|c| { + let mut c = c.borrow_mut(); + match c.as_mut() { + Some((token, remaining)) if *remaining == 0 => { + *remaining = 127; + token.is_cancelled() + } + Some((_, remaining)) => { + *remaining -= 1; + false + } + None => false, + } + }); + if cancelled { + std::panic::resume_unwind(Box::new(Cancelled)); + } +} + +pub fn check_now() { + let cancelled = CURRENT.with(|c| c.borrow().as_ref().is_some_and(|(t, _)| t.is_cancelled())); + if cancelled { + std::panic::resume_unwind(Box::new(Cancelled)); + } +} + +#[cfg(test)] +mod tests { + use super::*; + #[test] + fn cancelled_work_never_starts_and_scope_is_restored() { + let token = Cancellation::new(Duration::from_secs(5)); + token.cancel(); + let mut ran = false; + let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| token.run(|| ran = true))); + assert!(r.unwrap_err().is::()); + assert!(!ran); + check_now(); + Cancellation::new(Duration::from_secs(5)).run(check_now); + } + #[test] + fn deadline_is_checked_without_a_watchdog() { + let token = Cancellation::new(Duration::ZERO); + assert!(std::panic::catch_unwind(|| token.run(|| 1)).unwrap_err().is::()); + } + #[test] + fn running_worker_observes_external_cancellation() { + let token = Cancellation::new(Duration::from_secs(5)); + let worker = token.clone(); + let (tx, rx) = std::sync::mpsc::channel(); + let thread = std::thread::spawn(move || { + std::panic::catch_unwind(|| { + worker.run(|| { + tx.send(()).unwrap(); + loop { + check(); + } + }) + }) + }); + rx.recv_timeout(Duration::from_secs(2)).unwrap(); + token.cancel(); + assert!(thread.join().unwrap().unwrap_err().is::()); + } +} diff --git a/crates/cg-core/src/lib.rs b/crates/cg-core/src/lib.rs index b7a4bc7..45f255d 100644 --- a/crates/cg-core/src/lib.rs +++ b/crates/cg-core/src/lib.rs @@ -5,6 +5,7 @@ pub mod brand; pub mod budget; +pub mod cancel; pub mod config; pub mod error; pub mod fileflags; diff --git a/crates/cg-extract/Cargo.toml b/crates/cg-extract/Cargo.toml index 24f0af9..c4d73ac 100644 --- a/crates/cg-extract/Cargo.toml +++ b/crates/cg-extract/Cargo.toml @@ -18,9 +18,6 @@ serde_json = { workspace = true } smallvec = { workspace = true } tracing = { workspace = true } memchr = { workspace = true } - -[target.'cfg(unix)'.dependencies] -libc = { workspace = true } - -[dev-dependencies] +memmap2 = { workspace = true } tempfile = { workspace = true } +aho-corasick = { workspace = true } diff --git a/crates/cg-extract/src/facts/python.rs b/crates/cg-extract/src/facts/python.rs index e3cfff3..3746fbf 100644 --- a/crates/cg-extract/src/facts/python.rs +++ b/crates/cg-extract/src/facts/python.rs @@ -256,7 +256,7 @@ fn assignment(cx: &mut Cx, n: Node, scope: u32) { .bindings .iter() .any(|b| b.scope == cs && b.name == nm && b.kind == BindKind::Field && (!b.decl_type.is_empty() || !b.init_ctor.is_empty())); - if !already || !(decl.is_empty() && init.is_empty()) { + if !(already && decl.is_empty() && init.is_empty()) { cx.add_binding(cs, &nm, BindKind::Field, &decl, &init, line); } } diff --git a/crates/cg-extract/src/hook_plan.rs b/crates/cg-extract/src/hook_plan.rs new file mode 100644 index 0000000..c304a2e --- /dev/null +++ b/crates/cg-extract/src/hook_plan.rs @@ -0,0 +1,89 @@ +//! One literal scan per source file, shared by every framework / synthesis scanner. + +use crate::FileHook; +use aho_corasick::AhoCorasick; +use cg_core::Lang; +use smallvec::SmallVec; +use std::collections::BTreeMap; +use std::sync::Arc; + +pub(crate) struct HookPlan { + matcher: Option, + owners: Vec>, +} + +impl HookPlan { + pub fn new(hooks: &[Arc]) -> Self { + let mut literals: BTreeMap<&str, Vec> = BTreeMap::new(); + for (i, hook) in hooks.iter().enumerate() { + for literal in hook.triggers() { + literals.entry(literal).or_default().push(i); + } + } + let matcher = (!literals.is_empty()).then(|| AhoCorasick::new(literals.keys()).expect("valid literal hook triggers")); + Self { matcher, owners: literals.into_values().collect() } + } + + pub fn select(&self, hooks: &[Arc], lang: Lang, rel: &str, src: &str) -> SmallVec<[bool; 16]> { + let enabled: SmallVec<[bool; 16]> = hooks.iter().map(|h| h.applies(lang, rel)).collect(); + let mut selected: SmallVec<[bool; 16]> = hooks.iter().zip(&enabled).map(|(h, enabled)| *enabled && h.triggers().is_empty()).collect(); + let mut left = enabled.iter().zip(&selected).filter(|(a, b)| **a && !**b).count(); + if left == 0 { + return selected; + } + if let Some(matcher) = &self.matcher { + // Overlap matters: one literal may be a suffix/prefix of another hook's literal. + for hit in matcher.find_overlapping_iter(src) { + for &i in &self.owners[hit.pattern().as_usize()] { + if enabled[i] && !selected[i] { + selected[i] = true; + left -= 1; + } + } + if left == 0 { + break; + } + } + } + selected + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::{HookCtx, HookOut}; + + struct Hook(&'static [&'static str], bool); + impl FileHook for Hook { + fn name(&self) -> &'static str { + "test" + } + fn triggers(&self) -> &'static [&'static str] { + self.0 + } + fn applies(&self, _: Lang, _: &str) -> bool { + self.1 + } + fn run(&self, _: &HookCtx<'_>, _: &mut HookOut) {} + } + + #[test] + fn fused_scan_matches_literal_semantics_including_overlap_and_shared_triggers() { + let hooks: Vec> = vec![ + Arc::new(Hook(&["app.get", "@route"], true)), + Arc::new(Hook(&["get"], true)), + Arc::new(Hook(&["app.get"], true)), + Arc::new(Hook(&[], true)), + Arc::new(Hook(&["app.get"], false)), + ]; + let plan = HookPlan::new(&hooks); + for src in ["", "app.get('/x')", "@route hi", "forget", "世界 app.get app.get"] { + let expected: Vec<_> = hooks + .iter() + .map(|h| h.applies(Lang::JavaScript, "x.js") && (h.triggers().is_empty() || h.triggers().iter().any(|t| src.contains(t)))) + .collect(); + assert_eq!(&plan.select(&hooks, Lang::JavaScript, "x.js", src)[..], expected); + } + } +} diff --git a/crates/cg-extract/src/lib.rs b/crates/cg-extract/src/lib.rs index 413d6f7..70aa50f 100644 --- a/crates/cg-extract/src/lib.rs +++ b/crates/cg-extract/src/lib.rs @@ -6,9 +6,11 @@ pub mod post; pub mod rows; pub use facts::{BindKind, BindingFact, ExportFact, Facts, ImportFact, ImportKind, ScopeFact, ScopeKind}; -pub use pipeline::{extract_all, extract_bytes, extract_one, extract_stream, file_node_only, fileflags, ExtractConfig, FileBatch}; +pub use pipeline::{extract_all, extract_bytes, extract_one, extract_stream, file_node_only, fileflags, try_extract_stream, ExtractConfig, FileBatch}; pub use post::{node_digest, ExtraNode, ExtraRef, FileHook, HookCtx, HookOut, SiteFact}; pub use rows::{EdgeView, FileRows, NodeView, RefView}; +mod hook_plan; +pub mod spill; /// Wire-format version of the vendored kernel (recorded in every segment). pub const KERNEL_ABI_VERSION: u8 = cg_kernel::KERNEL_ABI_VERSION; diff --git a/crates/cg-extract/src/pipeline.rs b/crates/cg-extract/src/pipeline.rs index fb7e675..905ac86 100644 --- a/crates/cg-extract/src/pipeline.rs +++ b/crates/cg-extract/src/pipeline.rs @@ -31,7 +31,8 @@ pub struct ExtractConfig { /// Lower thread priority (Unix nice 10) so a build does not starve the editor. pub low_priority: bool, /// Framework extract hooks and synthesis site scanners. - pub hooks: Vec>, + hooks: Vec>, + hook_plan: Option>, } impl ExtractConfig { @@ -45,8 +46,16 @@ impl ExtractConfig { overrides: BTreeMap::new(), low_priority: false, hooks: Vec::new(), + hook_plan: None, } } + + /// Compile all hook trigger literals once for the extraction pool. Replacing hooks + /// also replaces the plan, so no stale automaton can survive a configuration change. + pub fn set_hooks(&mut self, hooks: Vec>) { + self.hook_plan = Some(Arc::new(crate::hook_plan::HookPlan::new(&hooks))); + self.hooks = hooks; + } } /// Everything extraction learned about one file. @@ -57,8 +66,9 @@ pub struct FileBatch { pub mtime_ns: i64, pub hash: [u8; 16], pub flags: u8, - /// Kernel tables (or a file-node-only table if the file could not be extracted). + /// Resident kernel tables. After spilling, use `rows()` to access the mapped tables. pub emit: EmitOut, + pub(crate) spilled: Option, pub extra_nodes: Vec, pub extra_refs: Vec, pub sites: Vec, @@ -69,7 +79,7 @@ pub struct FileBatch { impl FileBatch { pub fn rows(&self) -> FileRows<'_> { - FileRows::new(&self.emit) + self.spilled.as_ref().map_or_else(|| FileRows::new(&self.emit), |s| s.rows()) } } @@ -251,8 +261,9 @@ pub fn extract_bytes(rel: &str, bytes: &[u8], mtime_ns: i64, lang_hint: Lang, cf let rows = FileRows::new(&emit); let ctx = HookCtx { rel, lang, src: &src, rows, facts: &facts }; let kcount = rows.node_count() as u32; - for hook in &cfg.hooks { - if !hook.applies(lang, rel) || (!hook.triggers().is_empty() && !hook.triggers().iter().any(|t| src.contains(t))) { + let selected = cfg.hook_plan.as_ref().expect("set_hooks builds the trigger plan").select(&cfg.hooks, lang, rel, &src); + for (hook, selected) in cfg.hooks.iter().zip(selected) { + if !selected { continue; } let mut out = HookOut::default(); @@ -284,7 +295,21 @@ pub fn extract_bytes(rel: &str, bytes: &[u8], mtime_ns: i64, lang_hint: Lang, cf } } } - FileBatch { rel: rel.to_owned(), lang, size: bytes.len() as u32, mtime_ns, hash: h, flags, emit, extra_nodes, extra_refs, sites, facts, errors } + FileBatch { + rel: rel.to_owned(), + lang, + size: bytes.len() as u32, + mtime_ns, + hash: h, + flags, + emit, + spilled: None, + extra_nodes, + extra_refs, + sites, + facts, + errors, + } } fn build_pool(cfg: &ExtractConfig) -> rayon::ThreadPool { @@ -295,26 +320,29 @@ fn build_pool(cfg: &ExtractConfig) -> rayon::ThreadPool { .thread_name(|i| format!("cg-extract-{i}")) .start_handler(move |_| { if low { - lower_thread_priority(); + cg_core::budget::lower_thread_priority(); } }) .build() .expect("extract thread pool") } -/// Unix nice 10 for the calling thread (Linux applies `who = 0` to the thread; other -/// platforms keep their default priority until the platform-specific calls land in M6). -fn lower_thread_priority() { - #[cfg(target_os = "linux")] - // SAFETY: setpriority only changes scheduling attributes of the calling thread. - unsafe { - libc::setpriority(libc::PRIO_PROCESS, 0, 10); - } -} - /// Extract all `entries`, feeding results to `sink` in input order. Files that vanished /// are reported as `None`. pub fn extract_stream(entries: &[FileEntry], cfg: &ExtractConfig, mut sink: impl FnMut(usize, Option)) { + let result = try_extract_stream(entries, cfg, |i, b| { + sink(i, b); + Ok::<_, std::convert::Infallible>(()) + }); + match result { + Ok(()) => {} + Err(never) => match never {}, + } +} + +/// Fallible sink variant. Stops before starting the next batch if a spill or downstream +/// write fails, instead of retaining the rest of the repository in memory. +pub fn try_extract_stream(entries: &[FileEntry], cfg: &ExtractConfig, mut sink: impl FnMut(usize, Option) -> Result<(), E>) -> Result<(), E> { let pool = build_pool(cfg); let mut start = 0usize; while start < entries.len() { @@ -326,10 +354,11 @@ pub fn extract_stream(entries: &[FileEntry], cfg: &ExtractConfig, mut sink: impl } let results: Vec> = pool.install(|| entries[start..end].par_iter().map(|e| extract_one(e, cfg)).collect()); for (i, r) in results.into_iter().enumerate() { - sink(start + i, r); + sink(start + i, r)?; } start = end; } + Ok(()) } /// Convenience: all batches in memory, aligned with `entries`. diff --git a/crates/cg-extract/src/rows.rs b/crates/cg-extract/src/rows.rs index ef293dc..86ee76f 100644 --- a/crates/cg-extract/src/rows.rs +++ b/crates/cg-extract/src/rows.rs @@ -17,7 +17,17 @@ fn u16_at(b: &[u8], at: usize) -> u16 { /// One file's tables. #[derive(Clone, Copy)] pub struct FileRows<'a> { - pub emit: &'a EmitOut, + pub emit: TableSlices<'a>, +} + +/// Kernel ABI buffers, borrowed either from the worker output or an immutable spill map. +#[derive(Clone, Copy)] +pub struct TableSlices<'a> { + pub meta: &'a [u8], + pub nodes: &'a [u8], + pub edges: &'a [u8], + pub refs: &'a [u8], + pub arena: &'a [u8], } #[inline] @@ -27,25 +37,28 @@ fn opt_u32(v: u32) -> Option { impl<'a> FileRows<'a> { pub fn new(emit: &'a EmitOut) -> Self { + Self::from_slices(TableSlices { meta: &emit.meta, nodes: &emit.nodes, edges: &emit.edges, refs: &emit.refs, arena: &emit.arena }) + } + pub fn from_slices(emit: TableSlices<'a>) -> Self { debug_assert!(emit.meta.len() >= META_SIZE); FileRows { emit } } pub fn node_count(&self) -> usize { - u32_at(&self.emit.meta, 4) as usize + u32_at(self.emit.meta, 4) as usize } pub fn edge_count(&self) -> usize { - u32_at(&self.emit.meta, 8) as usize + u32_at(self.emit.meta, 8) as usize } pub fn ref_count(&self) -> usize { - u32_at(&self.emit.meta, 12) as usize + u32_at(self.emit.meta, 12) as usize } /// JSON array of extraction errors, if any were recorded. pub fn errors_json(&self) -> Option<&'a str> { - let off = u32_at(&self.emit.meta, 20); + let off = u32_at(self.emit.meta, 20); if off == NONE { return None; } - let len = u32_at(&self.emit.meta, 24) as usize; + let len = u32_at(self.emit.meta, 24) as usize; self.s(off, len as u32) } diff --git a/crates/cg-extract/src/spill.rs b/crates/cg-extract/src/spill.rs new file mode 100644 index 0000000..9a05ce6 --- /dev/null +++ b/crates/cg-extract/src/spill.rs @@ -0,0 +1,109 @@ +//! Release accumulated kernel buffers to immutable temporary mappings when their resident +//! share reaches the extraction budget. One file backs a whole chunk, avoiding a descriptor +//! per source file. Facts remain resident; this is a degradation path, not an RSS hard limit. + +use crate::rows::{FileRows, TableSlices}; +use crate::FileBatch; +use memmap2::Mmap; +use std::io::{self, Write}; +use std::ops::Range; +use std::path::Path; +use std::sync::Arc; +use tempfile::NamedTempFile; + +struct Backing { + // Drop the mapping before deleting the file (also required on Windows). + map: Mmap, + _file: NamedTempFile, +} + +pub(crate) struct SpilledTables { + backing: Arc, + ranges: [Range; 5], +} + +impl SpilledTables { + pub(crate) fn rows(&self) -> FileRows<'_> { + let bytes = &self.backing.map; + let [meta, nodes, edges, refs, arena] = self.ranges.each_ref().map(|r| &bytes[r.clone()]); + FileRows::from_slices(TableSlices { meta, nodes, edges, refs, arena }) + } +} + +/// Heap capacity of the kernel buffers; includes spare capacity that will be released. +pub fn resident_bytes(batch: &FileBatch) -> u64 { + let e = &batch.emit; + [&e.meta, &e.nodes, &e.edges, &e.refs, &e.arena].iter().map(|v| v.capacity() as u64).sum() +} + +/// Spill all resident tables in `batches` as one immutable chunk. No buffers are released +/// until the write and map have both succeeded, so an I/O failure leaves the input intact. +pub fn spill<'a>(batches: impl IntoIterator, dir: &Path) -> io::Result { + let batches: Vec<_> = batches.into_iter().filter(|b| b.spilled.is_none()).collect(); + if batches.is_empty() { + return Ok(0); + } + std::fs::create_dir_all(dir)?; + let mut file = tempfile::Builder::new().prefix("extract-").suffix(".tmp").tempfile_in(dir)?; + let mut ranges = Vec::with_capacity(batches.len()); + let mut offset = 0usize; + let mut released = 0u64; + for b in &batches { + released += resident_bytes(b); + let e = &b.emit; + let mut parts: [Range; 5] = std::array::from_fn(|_| 0..0); + for (slot, bytes) in parts.iter_mut().zip([&e.meta, &e.nodes, &e.edges, &e.refs, &e.arena]) { + file.write_all(bytes)?; + *slot = offset..offset + bytes.len(); + offset += bytes.len(); + } + ranges.push(parts); + } + file.flush()?; + // SAFETY: the private tempfile is never modified or truncated after this mapping and + // its owner lives as long as every slice into it. No source file is mapped here. + let map = unsafe { Mmap::map(file.as_file())? }; + let backing = Arc::new(Backing { map, _file: file }); + for (b, ranges) in batches.into_iter().zip(ranges) { + b.spilled = Some(SpilledTables { backing: backing.clone(), ranges }); + b.emit.meta = Vec::new(); + b.emit.nodes = Vec::new(); + b.emit.edges = Vec::new(); + b.emit.refs = Vec::new(); + b.emit.arena = Vec::new(); + } + Ok(released) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::{extract_bytes, ExtractConfig}; + use cg_core::Lang; + + #[test] + fn spill_preserves_every_abi_buffer_and_shares_one_file() { + let dir = tempfile::tempdir().unwrap(); + let cfg = ExtractConfig::new(dir.path()); + let mut batches: Vec<_> = + ["a.ts", "b.ts"].iter().map(|rel| extract_bytes(rel, b"export function foo() { return foo(); }", 0, Lang::TypeScript, &cfg)).collect(); + let before: Vec<_> = batches + .iter() + .map(|b| { + let e = b.rows().emit; + [e.meta, e.nodes, e.edges, e.refs, e.arena].map(<[u8]>::to_vec) + }) + .collect(); + assert!(spill(&mut batches, dir.path()).unwrap() > 0); + assert_eq!(std::fs::read_dir(dir.path()).unwrap().count(), 1); + for (b, expected) in batches.iter().zip(before) { + assert_eq!(resident_bytes(b), 0); + let e = b.rows().emit; + assert_eq!([e.meta, e.nodes, e.edges, e.refs, e.arena].map(<[u8]>::to_vec), expected); + assert!(b.rows().node_count() > 1); + } + assert_eq!(spill(&mut batches, dir.path()).unwrap(), 0); + drop(batches); + assert_eq!(std::fs::read_dir(dir.path()).unwrap().count(), 0); + } +} diff --git a/crates/cg-install/src/cursor.rs b/crates/cg-install/src/cursor.rs index 98cbb92..9973f10 100644 --- a/crates/cg-install/src/cursor.rs +++ b/crates/cg-install/src/cursor.rs @@ -77,7 +77,10 @@ mod tests { let req = Request { bin: "/opt/cgr".into(), local: true, project: dir.path().join("proj"), hook: false, permissions: false, dry_run: false }; Cursor.install(&env, &req).unwrap(); let text = std::fs::read_to_string(dir.path().join("proj/.cursor/mcp.json")).unwrap(); - assert!(text.contains(&dir.path().join("proj").to_string_lossy().into_owned())); + // Compare the decoded argument: Windows backslashes are escaped in JSON text. + let value: serde_json::Value = serde_json::from_str(&text).unwrap(); + let args = value["mcpServers"][brand::MCP_SERVER_NAME]["args"].as_array().unwrap(); + assert!(args.contains(&json!(req.project.to_string_lossy()))); } #[test] diff --git a/crates/cg-mcp/src/server.rs b/crates/cg-mcp/src/server.rs index a0a6120..c2cdf39 100644 --- a/crates/cg-mcp/src/server.rs +++ b/crates/cg-mcp/src/server.rs @@ -3,6 +3,7 @@ use crate::defs; use cg_core::brand; +use cg_core::cancel::{Cancellation, Cancelled}; use cg_query::explore::History; use cg_query::strings; use cg_query::tools::{self, StatusInfo, ToolEnv, ToolOutput}; @@ -68,17 +69,21 @@ type ProjectHandles = (Arc, Arc>, Arc); pub struct Server { cfg: Config, cache: Mutex, + project_lifecycle: Mutex<()>, roots: Mutex, roots_cv: Condvar, client_roots: AtomicBool, protocol: Mutex, - pool_tx: Mutex>>, + pool_tx: Mutex>>, + requests: Mutex>, outbound: OnceLock>, shutdown_flag: AtomicBool, shutdown_hooks: Mutex>>, status_hook: Mutex>, rebuild_hook: Mutex>, + rebuilding: Arc>>, open_hook: Mutex>, + close_hook: Mutex>, next_req: AtomicU64, /// Tool calls that were accepted and have not been answered yet (see [`Server::drain`]). inflight: (Mutex, Condvar), @@ -137,7 +142,7 @@ pub fn file_uri_to_path(uri: &str) -> Option { impl Server { pub fn new(cfg: Config) -> Arc { - let (tx, rx) = mpsc::channel::(); + let (tx, rx) = mpsc::sync_channel::(POOL_SIZE * 4); let rx = Arc::new(Mutex::new(rx)); for i in 0..POOL_SIZE { let rx = Arc::clone(&rx); @@ -152,18 +157,22 @@ impl Server { Arc::new(Server { cfg, cache: Mutex::new(Cache::default()), + project_lifecycle: Mutex::new(()), roots: Mutex::new(RootsState::default()), roots_cv: Condvar::new(), client_roots: AtomicBool::new(false), protocol: Mutex::new(SUPPORTED_PROTOCOLS[0].to_owned()), pool_tx: Mutex::new(Some(tx)), + requests: Mutex::new(HashMap::new()), inflight: (Mutex::new(0), Condvar::new()), outbound: OnceLock::new(), shutdown_flag: AtomicBool::new(false), shutdown_hooks: Mutex::new(Vec::new()), status_hook: Mutex::new(None), rebuild_hook: Mutex::new(None), + rebuilding: Arc::new(Mutex::new(None)), open_hook: Mutex::new(None), + close_hook: Mutex::new(None), next_req: AtomicU64::new(1), started: Instant::now(), }) @@ -186,6 +195,11 @@ impl Server { *self.open_hook.lock().unwrap_or_else(|e| e.into_inner()) = Some(h); } + /// Release per-project resources when its cached snapshot is evicted. + pub fn set_close_hook(&self, h: OpenHook) { + *self.close_hook.lock().unwrap_or_else(|e| e.into_inner()) = Some(h); + } + /// Open the project at `start` ahead of the first tool call (so it is watched from the start). pub fn preopen(&self, start: &Path) { let _ = self.project_for(Some(&start.to_string_lossy())); @@ -219,6 +233,9 @@ impl Server { return; } *self.pool_tx.lock().unwrap_or_else(|e| e.into_inner()) = None; + for token in self.requests.lock().unwrap_or_else(|e| e.into_inner()).values() { + token.cancel(); + } let hooks: Vec<_> = self.shutdown_hooks.lock().unwrap_or_else(|e| e.into_inner()).drain(..).collect(); for h in hooks { h(); @@ -227,7 +244,7 @@ impl Server { fn submit(&self, job: Job) -> bool { match self.pool_tx.lock().unwrap_or_else(|e| e.into_inner()).as_ref() { - Some(tx) => tx.send(job).is_ok(), + Some(tx) => tx.try_send(job).is_ok(), None => false, } } @@ -303,6 +320,9 @@ impl Server { }; let Some(store) = Store::locate(&start) else { return Err(self.guidance(&start, explicit)) }; let root = store.root.clone(); + // Keep open/evict callbacks in cache mutation order. Otherwise a later eviction + // could stop a watcher before an earlier open callback has actually started it. + let _lifecycle = self.project_lifecycle.lock().unwrap_or_else(|e| e.into_inner()); let mut cache = self.cache.lock().unwrap_or_else(|e| e.into_inner()); cache.tick += 1; let tick = cache.tick; @@ -322,12 +342,19 @@ impl Server { None => (Arc::new(Mutex::new(History::new())), Arc::new(AtomicBool::new(false))), }; cache.map.insert(root.clone(), Entry { project: p.clone(), history: history.clone(), noticed: noticed.clone(), last_used: tick }); + let mut evicted = None; if cache.map.len() > MAX_PROJECTS { if let Some(oldest) = cache.map.iter().min_by_key(|(_, e)| e.last_used).map(|(k, _)| k.clone()) { cache.map.remove(&oldest); + evicted = Some(oldest); } } drop(cache); + if let Some(root) = evicted { + if let Some(close) = self.close_hook.lock().unwrap_or_else(|e| e.into_inner()).clone() { + close(&root); + } + } if first_open { if let Some(h) = self.open_hook.lock().unwrap_or_else(|e| e.into_inner()).clone() { h(&root); @@ -336,13 +363,53 @@ impl Server { Ok((p, history, noticed)) } Err(e) => { - cache.map.remove(&root); + let had_project = cache.map.remove(&root).is_some(); drop(cache); - if let Some(h) = self.rebuild_hook.lock().unwrap_or_else(|e| e.into_inner()).clone() { - let r = root.clone(); - std::thread::spawn(move || h(&r)); + if had_project { + if let Some(close) = self.close_hook.lock().unwrap_or_else(|e| e.into_inner()).clone() { + close(&root); + } } - Err(ToolOutput::err(format!("The index could not be opened ({e}). It is being rebuilt in the background; retry in a few seconds. If this persists the user can run `{} index`.", brand::BIN_NAME))) + let rebuilding = self.request_rebuild(&root); + let note = if rebuilding { + "It is being rebuilt in the background; retry in a few seconds." + } else { + "A rebuild is needed; the background rebuild worker may be busy with another project." + }; + Err(ToolOutput::err(format!("The index could not be opened ({e}). {note} If this persists the user can run `{} index`.", brand::BIN_NAME))) + } + } + } + + /// Admit one background build for repeated old-format / corrupt-index requests. + fn request_rebuild(&self, root: &Path) -> bool { + if self.is_shutdown() { + return false; + } + let mut slot = self.rebuilding.lock().unwrap_or_else(|e| e.into_inner()); + if let Some(active) = slot.as_ref() { + return active == root; + } + let Some(hook) = self.rebuild_hook.lock().unwrap_or_else(|e| e.into_inner()).clone() else { + return false; + }; + *slot = Some(root.to_path_buf()); + let state = self.rebuilding.clone(); + let root = root.to_path_buf(); + std::thread::spawn(move || { + let _ = std::panic::catch_unwind(AssertUnwindSafe(|| hook(&root))); + *state.lock().unwrap_or_else(|e| e.into_inner()) = None; + }); + true + } + + fn clear_projects(&self) { + let _lifecycle = self.project_lifecycle.lock().unwrap_or_else(|e| e.into_inner()); + let roots: Vec<_> = self.cache.lock().unwrap_or_else(|e| e.into_inner()).map.drain().map(|(root, _)| root).collect(); + let close = self.close_hook.lock().unwrap_or_else(|e| e.into_inner()).clone(); + if let Some(close) = close { + for root in roots { + close(&root); } } } @@ -389,6 +456,14 @@ impl Server { pub fn handle(self: &Arc, msg: Value, reply: Arc) { let method = msg.get("method").and_then(|m| m.as_str()).map(str::to_owned); let id = msg.get("id").cloned(); + if method.as_deref() == Some("notifications/cancelled") && id.is_none() { + if let Some(id) = msg.pointer("/params/requestId") { + if let Some(token) = self.requests.lock().unwrap_or_else(|e| e.into_inner()).get(&id.to_string()) { + token.cancel(); + } + } + return; + } match (method, id) { (Some(m), Some(id)) => self.handle_request(&m, id, msg.get("params").cloned().unwrap_or(Value::Null), reply), (Some(m), None) => self.handle_notification(&m), @@ -446,34 +521,158 @@ impl Server { return; }; let args = params.get("arguments").cloned().unwrap_or_else(|| json!({})); + let token = Cancellation::new(CALL_TIMEOUT); + let key = id.to_string(); + { + let mut requests = self.requests.lock().unwrap_or_else(|e| e.into_inner()); + if requests.contains_key(&key) { + reply.send(json!({"jsonrpc":"2.0", "id":id, "error":{"code":-32600,"message":"A request with this id is already running."}})); + return; + } + requests.insert(key.clone(), token.clone()); + } let (tx, rx) = mpsc::channel::(); let me = Arc::clone(self); + let worker_token = token.clone(); let submitted = self.submit(Box::new(move || { - let r = std::panic::catch_unwind(AssertUnwindSafe(|| me.run_tool(&name, &args))); + let r = std::panic::catch_unwind(AssertUnwindSafe(|| worker_token.run(|| me.run_tool(&name, &args)))); let out = match r { Ok(o) => o, + Err(e) if e.is::() => ToolOutput::ok("This call was cancelled or exceeded its deadline. Narrow the query and try again."), Err(_) => { - me.cache.lock().unwrap_or_else(|e| e.into_inner()).map.clear(); + me.clear_projects(); ToolOutput::err("Internal error while answering (the cached snapshot was dropped). Retry once.") } }; let _ = tx.send(tool_result(out)); })); if !submitted { - reply.send(json!({"jsonrpc": "2.0", "id": id, "result": tool_result(ToolOutput::err("The server is shutting down."))})); + self.requests.lock().unwrap_or_else(|e| e.into_inner()).remove(&key); + reply.send(json!({"jsonrpc": "2.0", "id": id, "result": tool_result(ToolOutput::ok("The server is busy or shutting down. Retry after the current calls finish."))})); return; } // The watchdog answers with a success-shaped note if the call overruns its budget. let in_flight = InFlight::enter(self); + let me = Arc::clone(self); std::thread::spawn(move || { let _in_flight = in_flight; let result = match rx.recv_timeout(CALL_TIMEOUT) { Ok(v) => v, - Err(_) => tool_result(ToolOutput::ok( - "This call took longer than 30 seconds and was stopped. Narrow the query — name specific symbols or files — and try again.", - )), + Err(_) => { + token.cancel(); + tool_result(ToolOutput::ok("This call exceeded its 30 second deadline; cancellation was requested. Narrow the query and try again.")) + } }; + me.requests.lock().unwrap_or_else(|e| e.into_inner()).remove(&key); reply.send(json!({"jsonrpc": "2.0", "id": id, "result": result})); }); } } + +#[cfg(test)] +mod admission_tests { + use super::*; + + struct ChannelReply(mpsc::Sender); + impl Reply for ChannelReply { + fn send(&self, v: Value) { + let _ = self.0.send(v); + } + } + + struct Release(Arc<(Mutex, Condvar)>); + impl Drop for Release { + fn drop(&mut self) { + *self.0 .0.lock().unwrap() = true; + self.0 .1.notify_all(); + } + } + + fn blocked_pool() -> (Arc, Release) { + let server = Server::new(Config::default()); + let gate = Arc::new((Mutex::new(false), Condvar::new())); + let release = Release(gate.clone()); + let (tx, rx) = mpsc::channel(); + for _ in 0..POOL_SIZE { + let gate = gate.clone(); + let tx = tx.clone(); + assert!(server.submit(Box::new(move || { + tx.send(()).unwrap(); + let mut open = gate.0.lock().unwrap(); + while !*open { + open = gate.1.wait(open).unwrap(); + } + }))); + } + for _ in 0..POOL_SIZE { + rx.recv_timeout(Duration::from_secs(2)).unwrap(); + } + (server, release) + } + + #[test] + fn full_queue_rejects_immediately_and_keeps_control_requests_responsive() { + let (server, release) = blocked_pool(); + for _ in 0..POOL_SIZE * 4 { + assert!(server.submit(Box::new(|| {}))); + } + assert!(!server.submit(Box::new(|| panic!("overflow job must not run")))); + let (tx, rx) = mpsc::channel(); + let reply = Arc::new(ChannelReply(tx)); + server.handle(json!({"jsonrpc":"2.0", "id":1, "method":"tools/call", "params":{"name":"explore","arguments":{"query":"foo"}}}), reply.clone()); + let busy = rx.recv_timeout(Duration::from_secs(1)).unwrap(); + assert!(busy["result"]["content"][0]["text"].as_str().unwrap().contains("busy")); + assert!(server.requests.lock().unwrap().is_empty()); + server.handle(json!({"jsonrpc":"2.0", "id":2, "method":"ping"}), reply); + assert_eq!(rx.recv_timeout(Duration::from_secs(1)).unwrap()["id"], 2); + drop(release); + server.shutdown(); + } + + #[test] + fn repeated_rebuild_requests_share_one_background_worker() { + let server = Server::new(Config::default()); + let gate = Arc::new((Mutex::new(false), Condvar::new())); + let release = Release(gate.clone()); + let (tx, rx) = mpsc::channel(); + server.set_rebuild_hook(Arc::new(move |_| { + tx.send(()).unwrap(); + let mut open = gate.0.lock().unwrap(); + while !*open { + open = gate.1.wait(open).unwrap(); + } + })); + assert!(server.request_rebuild(Path::new("one"))); + rx.recv_timeout(Duration::from_secs(2)).unwrap(); + for _ in 0..50 { + assert!(server.request_rebuild(Path::new("one"))); + } + assert!(!server.request_rebuild(Path::new("two"))); + assert!(rx.try_recv().is_err()); + drop(release); + server.shutdown(); + } + + #[test] + fn cancelled_queued_call_never_opens_a_project_and_answers_once() { + let (server, release) = blocked_pool(); + let opened = Arc::new(AtomicU64::new(0)); + let count = opened.clone(); + server.set_open_hook(Arc::new(move |_| { + count.fetch_add(1, Ordering::SeqCst); + })); + let (tx, rx) = mpsc::channel(); + let reply = Arc::new(ChannelReply(tx)); + server + .handle(json!({"jsonrpc":"2.0", "id":"cancel-me", "method":"tools/call", "params":{"name":"explore","arguments":{"query":"foo"}}}), reply.clone()); + server.handle(json!({"jsonrpc":"2.0", "method":"notifications/cancelled", "params":{"requestId":"cancel-me"}}), reply); + drop(release); + let result = rx.recv_timeout(Duration::from_secs(2)).unwrap(); + assert!(result["result"]["content"][0]["text"].as_str().unwrap().contains("cancelled")); + server.drain(Duration::from_secs(1)); + assert_eq!(opened.load(Ordering::SeqCst), 0); + assert!(server.requests.lock().unwrap().is_empty()); + assert!(rx.try_recv().is_err()); + server.shutdown(); + } +} diff --git a/crates/cg-mcp/src/stdio.rs b/crates/cg-mcp/src/stdio.rs index 4b05179..386c11c 100644 --- a/crates/cg-mcp/src/stdio.rs +++ b/crates/cg-mcp/src/stdio.rs @@ -27,7 +27,7 @@ fn watch_parent(server: Arc) { { let original = unsafe { libc::getppid() }; std::thread::spawn(move || loop { - std::thread::sleep(Duration::from_millis(1000)); + std::thread::sleep(Duration::from_millis(200)); if server.is_shutdown() { break; } @@ -68,9 +68,9 @@ pub fn serve_stdio(server: Arc) -> std::io::Result<()> { Err(e) => reply.send(serde_json::json!({"jsonrpc": "2.0", "id": null, "error": {"code": -32700, "message": format!("Parse error: {e}")}})), } } - // stdin closed: answer what was already accepted (a call is capped at 30 s), then stop the - // watcher and release locks - server.drain(Duration::from_secs(35)); + // Give the final small request a chance to answer, then cancel outstanding work and + // stop all watchers against a shared deadline. EOF must not wait for a 30 s query. + server.drain(Duration::from_millis(200)); server.shutdown(); Ok(()) } diff --git a/crates/cg-mcp/tests/server.rs b/crates/cg-mcp/tests/server.rs index 66e3104..68f8439 100644 --- a/crates/cg-mcp/tests/server.rs +++ b/crates/cg-mcp/tests/server.rs @@ -159,3 +159,25 @@ fn project_path_selects_another_index() { assert!(text_of(&out.by_id(8).unwrap()).contains("only_in_second")); assert!(!text_of(&out.by_id(9).unwrap()).contains("def only_in_second"), "the default project must not see the other index"); } + +#[test] +fn project_lru_evicts_the_corresponding_watcher_resource() { + let dirs: Vec<_> = (0..9).map(|_| project()).collect(); + let (server, _) = server_for(dirs[0].path(), None); + let watched = Arc::new(Mutex::new(std::collections::BTreeSet::new())); + let opened = watched.clone(); + server.set_open_hook(Arc::new(move |root| { + opened.lock().unwrap().insert(root.to_path_buf()); + })); + let closed = watched.clone(); + server.set_close_hook(Arc::new(move |root| { + closed.lock().unwrap().remove(root); + })); + for dir in &dirs { + server.preopen(dir.path()); + assert!(watched.lock().unwrap().len() <= 8); + } + assert_eq!(watched.lock().unwrap().len(), 8); + assert!(!watched.lock().unwrap().contains(dirs[0].path())); + server.shutdown(); +} diff --git a/crates/cg-query/Cargo.toml b/crates/cg-query/Cargo.toml index ca2a5e3..a33a2c8 100644 --- a/crates/cg-query/Cargo.toml +++ b/crates/cg-query/Cargo.toml @@ -6,6 +6,7 @@ edition.workspace = true license.workspace = true [dependencies] +memchr = { workspace = true } cg-core = { workspace = true } cg-store = { workspace = true } fst = { workspace = true } diff --git a/crates/cg-query/src/explore/mod.rs b/crates/cg-query/src/explore/mod.rs index cfc1923..981e5ea 100644 --- a/crates/cg-query/src/explore/mod.rs +++ b/crates/cg-query/src/explore/mod.rs @@ -124,15 +124,9 @@ impl Graph { (rec.rec.size as u64, rec.rec.mtime_ns) } - fn read_lines(&self, f: FileId) -> Option> { + fn read_lines(&self, f: FileId) -> Option { let abs = self.abs_path(self.path(f))?; - let bytes = std::fs::read(abs).ok()?; - let text = String::from_utf8_lossy(&bytes); - let mut lines: Vec = text.split('\n').map(|l| l.strip_suffix('\r').unwrap_or(l).to_owned()).collect(); - if lines.last().is_some_and(|l| l.is_empty()) { - lines.pop(); - } - Some(lines) + crate::source::SourceView::read(&abs, 1 << 20).ok() } /// Run the explore pipeline for `query`. @@ -423,7 +417,7 @@ impl Graph { } } else { for (n, s) in &listed { - let must = exact.contains(n) || flow.path_nodes.contains(n) || (grp.pinned.is_some() && pinned_nodes.contains(n) && false); + let must = exact.contains(n) || flow.path_nodes.contains(n); ctx.for_node(*n, *s, must, &mut wants); } for a in &grp.anchors { diff --git a/crates/cg-query/src/explore/window.rs b/crates/cg-query/src/explore/window.rs index dba737f..cbdc0c0 100644 --- a/crates/cg-query/src/explore/window.rs +++ b/crates/cg-query/src/explore/window.rs @@ -122,13 +122,14 @@ pub fn range_cost(cost: &[u32], a: u32, b: u32) -> usize { (cost[b] - cost[a - 1]) as usize } -pub fn line_costs(lines: &[String]) -> Vec { +pub fn line_costs(lines: &crate::source::SourceView) -> Vec { let mut cost = Vec::with_capacity(lines.len() + 1); cost.push(0u32); let mut acc = 0u32; - for (i, l) in lines.iter().enumerate() { + for i in 0..lines.len() { + cg_core::cancel::check(); let digits = ((i + 1) as f64).log10().floor() as u32 + 1; - acc += l.len() as u32 + digits + 2; + acc += lines[i].len() as u32 + digits + 2; cost.push(acc); } cost @@ -170,7 +171,7 @@ pub fn plan(wants: Vec, gap: u32, cost: &[u32], cap: usize) -> Vec /// Render ranges as `\t` blocks joined by gap markers (named when the hole holds /// indexed symbols). -pub fn render(g: &Graph, f: cg_core::FileId, lines: &[String], ranges: &[Range]) -> String { +pub fn render(g: &Graph, f: cg_core::FileId, lines: &crate::source::SourceView, ranges: &[Range]) -> String { let mut out = String::new(); let mut prev_end: Option = None; // symbols for named gaps diff --git a/crates/cg-query/src/graph.rs b/crates/cg-query/src/graph.rs index 6f63143..2eebbae 100644 --- a/crates/cg-query/src/graph.rs +++ b/crates/cg-query/src/graph.rs @@ -45,7 +45,8 @@ impl Graph { if norm.is_empty() { return 1.0; } - let sum: u64 = norm.iter().map(|v| *v as u64).sum(); + // The fallback only serves snapshots produced before the statistic was persisted. + let sum: u64 = if self.view.base.meta.total_tokens > 0 { self.view.base.meta.total_tokens } else { norm.iter().map(|v| *v as u64).sum() }; (sum as f32 / norm.len() as f32).max(1.0) }) } diff --git a/crates/cg-query/src/lib.rs b/crates/cg-query/src/lib.rs index 266c69e..5f891c2 100644 --- a/crates/cg-query/src/lib.rs +++ b/crates/cg-query/src/lib.rs @@ -5,6 +5,7 @@ pub mod explore; pub mod graph; pub mod project; pub mod search; +pub mod source; pub mod strings; pub mod tools; pub mod traverse; diff --git a/crates/cg-query/src/search.rs b/crates/cg-query/src/search.rs index 90dca94..bdb96ee 100644 --- a/crates/cg-query/src/search.rs +++ b/crates/cg-query/src/search.rs @@ -401,10 +401,7 @@ impl Graph { for (seg, _) in self.layers() { if let Some(f) = &seg.tokens_fst { if let Some(r) = f.get(&t.lower) { - let off = &seg.p.token_post_off; - if (r as usize) + 1 < off.len() { - df += (off[r as usize + 1] - off[r as usize]) as usize; - } + df += seg.p.token_df.get(r as usize).copied().unwrap_or(0) as usize; } } } @@ -562,7 +559,7 @@ impl Graph { let avg = self.graph_avg(); for w in &words { let mut df = 0usize; - let mut lists: Vec<(&Seg, bool, &[cg_store::PostEntry], usize)> = Vec::new(); + let mut lists: Vec<(&Seg, bool, &[u8], usize)> = Vec::new(); for (seg, d) in self.layers() { let Some(fst) = &seg.tokens_fst else { continue }; let Some(rank) = fst.get(w) else { continue }; @@ -571,8 +568,10 @@ impl Graph { if r + 1 >= off.len() { continue; } - let list = &seg.p.token_post[off[r] as usize..off[r + 1] as usize]; - df += list.len(); + let Some(list) = seg.p.token_post.get(off[r] as usize..off[r + 1] as usize) else { + continue; + }; + df += seg.p.token_df.get(r).copied().unwrap_or(0) as usize; lists.push((seg, d, list, r)); } if df == 0 { @@ -581,7 +580,8 @@ impl Graph { let idf = (1.0 + (n_docs - df as f32 + 0.5) / (df as f32 + 0.5)).ln(); let n_base = self.view.base.p.nodes_hot.len() as u32; for (seg, d, list, _) in lists { - for e in list { + for e in cg_store::index::TokenPostings::new(list) { + cg_core::cancel::check(); if !self.live(e.node, d) { continue; } diff --git a/crates/cg-query/src/source.rs b/crates/cg-query/src/source.rs new file mode 100644 index 0000000..1e41392 --- /dev/null +++ b/crates/cg-query/src/source.rs @@ -0,0 +1,81 @@ +//! A bounded, current-on-disk source view. Source files can be modified in place by an editor, +//! so unlike immutable index segments they are read into owned bytes rather than mmap'd. + +use std::io::Read; +use std::ops::Index; +use std::path::Path; +use std::sync::OnceLock; + +pub struct SourceView { + text: String, + starts: OnceLock>, +} + +impl SourceView { + pub fn read(path: &Path, max_bytes: usize) -> std::io::Result { + cg_core::cancel::check_now(); + let mut bytes = Vec::new(); + std::fs::File::open(path)?.take((max_bytes as u64).saturating_add(1)).read_to_end(&mut bytes)?; + cg_core::cancel::check_now(); + if bytes.len() > max_bytes { + return Err(std::io::Error::other("source exceeds the per-file read budget")); + } + let text = String::from_utf8(bytes).unwrap_or_else(|e| String::from_utf8_lossy(e.as_bytes()).into_owned()); + Ok(Self { text, starts: OnceLock::new() }) + } + fn starts(&self) -> &[usize] { + self.starts.get_or_init(|| { + let mut starts = vec![0]; + for i in memchr::memchr_iter(b'\n', self.text.as_bytes()) { + if i + 1 < self.text.len() { + starts.push(i + 1); + } + } + if self.text.is_empty() { + starts.clear(); + } + starts + }) + } + pub fn len(&self) -> usize { + self.starts().len() + } + pub fn is_empty(&self) -> bool { + self.len() == 0 + } +} + +impl Index for SourceView { + type Output = str; + fn index(&self, i: usize) -> &str { + let starts = self.starts(); + let end = starts.get(i + 1).copied().unwrap_or(self.text.len()); + let line = &self.text[starts[i]..end]; + match line.strip_suffix('\n') { + Some(line) => line.strip_suffix('\r').unwrap_or(line), + None => line, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + #[test] + fn reads_current_content_without_per_line_strings_and_bounds_growth() { + let file = tempfile::NamedTempFile::new().unwrap(); + std::fs::write(file.path(), "α\r\n\nlast\n").unwrap(); + let s = SourceView::read(file.path(), 32).unwrap(); + assert_eq!(s.len(), 3); + assert_eq!(&s[0], "α"); + assert_eq!(&s[1], ""); + assert_eq!(&s[2], "last"); + std::fs::write(file.path(), "new\n").unwrap(); + assert_eq!(&SourceView::read(file.path(), 32).unwrap()[0], "new"); + assert!(SourceView::read(file.path(), 2).is_err()); + std::fs::write(file.path(), "").unwrap(); + assert!(SourceView::read(file.path(), 32).unwrap().is_empty()); + std::fs::write(file.path(), "last\r").unwrap(); + assert_eq!(&SourceView::read(file.path(), 32).unwrap()[0], "last\r"); + } +} diff --git a/crates/cg-query/src/strings.rs b/crates/cg-query/src/strings.rs index 146027d..cc56bd6 100644 --- a/crates/cg-query/src/strings.rs +++ b/crates/cg-query/src/strings.rs @@ -38,9 +38,9 @@ const SERVER_INSTRUCTIONS: &str = r#"# Codegraph — code intelligence over an i Codegraph is a memory-mapped knowledge graph of every symbol, edge, and file in the workspace — pre-computed structure you would otherwise re-derive by reading files (cached intelligence: thousands of parse/trace decisions you -don't pay to re-reason each run). It indexes 30+ languages -(TypeScript/JavaScript, Python, Go, Rust, Java, C#, C/C++, PHP, Ruby, Swift, -Kotlin, and more) — don't assume a language here isn't covered. Reads are +don't pay to re-reason each run). This build indexes TypeScript/JavaScript, +Python, Go, Rust, Java and Kotlin (including TSX/JSX). +Other language files are not extracted by this build. Reads are sub-millisecond; the index lags writes by ~1s through the file watcher. Reach for it BEFORE *and* while writing or editing code — not just for questions: one call returns the verbatim source PLUS who calls it and what it affects, so you edit with the @@ -96,7 +96,7 @@ calls; a grep/read exploration is dozens. const SERVER_INSTRUCTIONS_NO_ROOT_INDEX: &str = r#"# Codegraph — available (per-project; pass projectPath) Codegraph is a knowledge graph of a codebase's symbols, edges, and -files (30+ languages): one `codegraph_explore` call returns the verbatim, line-numbered source +files (TypeScript/JavaScript, Python, Go, Rust, Java and Kotlin, including TSX/JSX): one `codegraph_explore` call returns the verbatim, line-numbered source of the relevant symbols PLUS the call paths between them and a blast-radius summary — replacing a grep + Read loop with one round-trip. diff --git a/crates/cg-resolve/src/driver.rs b/crates/cg-resolve/src/driver.rs index 0f8cb18..a0f1326 100644 --- a/crates/cg-resolve/src/driver.rs +++ b/crates/cg-resolve/src/driver.rs @@ -77,6 +77,39 @@ pub fn resolve_refs<'e, 'v>(e: &'e Engine<'v>, files: &[FileId], round: Round_, parts.into_iter().flatten().collect() } +/// Full builds write directly into disjoint file ranges in the final outcome array. There +/// is no per-file result vector followed by a flattened round vector and another copy. +fn resolve_into(e: &Engine<'_>, files: &[FileId], round: Round, outcomes: &mut [Outcome]) { + let mut files = files.to_vec(); + files.sort_unstable_by_key(|f| e.view.ref_range(*f).start); + let mut jobs = Vec::with_capacity(files.len()); + let mut remaining = outcomes; + let mut cursor = 0usize; + for f in files { + let range = e.view.ref_range(f); + if range.is_empty() { + continue; + } + let (_, rest) = remaining.split_at_mut(range.start as usize - cursor); + let (slots, rest) = rest.split_at_mut(range.len()); + jobs.push((f, slots)); + remaining = rest; + cursor = range.end as usize; + } + jobs.into_par_iter().for_each_init(WorkerCx::new, |wcx, (file, slots)| { + let refs = e.view.refs_of_file(file); + if !refs.iter().any(|r| in_round(r.kind, round)) { + return; + } + let cx = FileCx::new(e, file); + for (slot, reference) in slots.iter_mut().zip(refs) { + if in_round(reference.kind, round) { + *slot = cx.resolve_ref(wcx, reference); + } + } + }); +} + #[derive(Clone, Copy, PartialEq, Eq)] pub enum Round_ { One, @@ -161,16 +194,18 @@ pub fn resolve_all(e: &mut Engine<'_>) -> ResolveOutput { let files: Vec = view.files().collect(); let mut outcomes = vec![Outcome::PENDING; n_refs]; - let r1 = resolve_refs(e, &files, Round_::One, None); - let pairs = hierarchy_pairs(e, &r1); + resolve_into(e, &files, Round::One, &mut outcomes); + let pairs = outcomes + .iter() + .enumerate() + .filter_map(|(id, o)| { + let r = view.reference(id as u32); + (o.status == status::RESOLVED && matches!(RefKind::from_wire(r.kind), Some(RefKind::Edge(EdgeKind::Extends | EdgeKind::Implements)))) + .then_some((r.from, NodeIdx(o.target))) + }) + .collect::>(); e.set_hierarchy(pairs.into_iter()); - for (id, o) in &r1 { - outcomes[*id as usize] = *o; - } - let r2 = resolve_refs(e, &files, Round_::Two, None); - for (id, o) in &r2 { - outcomes[*id as usize] = *o; - } + resolve_into(e, &files, Round::Two, &mut outcomes); let mut edges = Vec::new(); for (id, o) in outcomes.iter().enumerate() { @@ -181,10 +216,7 @@ pub fn resolve_all(e: &mut Engine<'_>) -> ResolveOutput { let mut import_targets = Vec::new(); for f in view.files() { for t in &e.imp_targets[f.idx()] { - import_targets.push(match t { - crate::modules::ImportTarget::File(tf) => tf.0, - _ => NONE, - }); + import_targets.push(e.modules.target_file(*t).map_or(NONE, |file| file.0)); } } let mut export_targets = Vec::new(); diff --git a/crates/cg-resolve/src/engine.rs b/crates/cg-resolve/src/engine.rs index b349a7e..74f6871 100644 --- a/crates/cg-resolve/src/engine.rs +++ b/crates/cg-resolve/src/engine.rs @@ -31,6 +31,7 @@ pub enum Bound { } pub struct Engine<'v> { + pub budget: cg_core::budget::MemBudget, pub view: &'v View, pub project: &'v ProjectInfo, pub modules: Modules<'v>, @@ -54,6 +55,10 @@ fn is_top_level(view: &View, n: NodeIdx) -> bool { impl<'v> Engine<'v> { pub fn build(view: &'v View, project: &'v ProjectInfo) -> Engine<'v> { + Self::build_with_budget(view, project, cg_core::budget::MemBudget::detect()) + } + + pub fn build_with_budget(view: &'v View, project: &'v ProjectInfo, budget: cg_core::budget::MemBudget) -> Engine<'v> { let modules = Modules::build(view, project); let n_files = modules.file_pkg.len().max(view.n_files()); let files: Vec = view.files().collect(); @@ -82,6 +87,7 @@ impl<'v> Engine<'v> { } let mut e = Engine { + budget, view, project, modules, diff --git a/crates/cg-resolve/src/index.rs b/crates/cg-resolve/src/index.rs index 72641d3..f3f0a48 100644 --- a/crates/cg-resolve/src/index.rs +++ b/crates/cg-resolve/src/index.rs @@ -8,20 +8,39 @@ use crate::project::ProjectInfo; use cg_core::budget::{peak_rss_mb, MemBudget}; use cg_core::config::ProjectConfig; use cg_core::{CgError, Result}; -use cg_extract::{extract_stream, fileflags, ExtractConfig, FileBatch, FileHook}; +use cg_extract::{fileflags, try_extract_stream, ExtractConfig, FileBatch, FileHook}; use cg_scan::{scan, ScanOptions}; use cg_store::{build_draft, finish, Draft, FinishInput, ProjectMeta, Seg, Store, View}; use std::path::Path; use std::sync::Arc; use std::time::{Duration, Instant}; -#[derive(Clone, Debug, Default)] +#[derive(Clone, Debug)] pub struct IndexOptions { pub threads: Option, pub memory_mb: Option, pub low_priority: bool, } +impl Default for IndexOptions { + fn default() -> Self { + Self { threads: None, memory_mb: None, low_priority: true } + } +} + +pub fn indexing_pool(budget: MemBudget, low_priority: bool) -> Result { + rayon::ThreadPoolBuilder::new() + .num_threads(budget.threads) + .stack_size(16 << 20) + .start_handler(move |_| { + if low_priority { + cg_core::budget::lower_thread_priority(); + } + }) + .build() + .map_err(|e| CgError::other(format!("thread pool: {e}"))) +} + #[derive(Clone, Debug, Default)] pub struct IndexReport { pub files: usize, @@ -38,6 +57,8 @@ pub struct IndexReport { pub threads: usize, pub budget_mb: u64, pub peak_rss_mb: u64, + /// Kernel heap capacity released to immutable temporary mappings during extraction. + pub spilled_kernel_bytes: u64, pub timings: Vec<(&'static str, Duration)>, pub frameworks: Vec, pub warnings: Vec, @@ -53,6 +74,9 @@ impl IndexReport { s.push_str(&format!(" {:<14} {:>9.1} ms\n", name, d.as_secs_f64() * 1e3)); } s.push_str(&format!(" {:<14} {:>9.1} ms peak RSS {} MB\n", "total", self.total().as_secs_f64() * 1e3, self.peak_rss_mb)); + if self.spilled_kernel_bytes > 0 { + s.push_str(&format!(" kernel spill: {:.1} MiB of heap buffers released to temporary maps\n", self.spilled_kernel_bytes as f64 / (1 << 20) as f64)); + } s } } @@ -87,15 +111,8 @@ pub struct Built { /// Build a complete base segment for the project at `root`. pub fn build_seg(root: &Path, opts: &IndexOptions) -> Result { let mut phase = Phase::new(); - let mut budget = MemBudget::detect(); - if let Some(mb) = opts.memory_mb { - budget = MemBudget::from_total(mb << 20, budget.threads); - } - if let Some(t) = opts.threads { - budget.threads = t.max(1); - } - let pool = - rayon::ThreadPoolBuilder::new().num_threads(budget.threads).stack_size(16 << 20).build().map_err(|e| CgError::other(format!("thread pool: {e}")))?; + let budget = MemBudget::configured(opts.memory_mb, opts.threads); + let pool = indexing_pool(budget, opts.low_priority)?; let mut report = IndexReport { threads: budget.threads, budget_mb: budget.total >> 20, ..Default::default() }; let config = ProjectConfig::load(root); @@ -117,17 +134,25 @@ pub fn build_seg(root: &Path, opts: &IndexOptions) -> Result { cfg.batch_bytes = budget.extract_batch_bytes(); cfg.overrides = config.extensions.clone(); cfg.low_priority = opts.low_priority; - cfg.hooks = hooks_for(&project); + cfg.set_hooks(hooks_for(&project)); let deprio = deprioritize_matcher(root, &config); let mut batches: Vec = Vec::with_capacity(scanned.files.len()); - extract_stream(&scanned.files, &cfg, |i, b| { + let mut resident = 0u64; + let spill_dir = root.join(cg_core::brand::DATA_DIR).join("tmp"); + try_extract_stream(&scanned.files, &cfg, |i, b| { if let Some(mut b) = b { if deprio.as_ref().is_some_and(|g| g.matched_path_or_any_parents(&scanned.files[i].rel, false).is_ignore()) { b.flags |= fileflags::DEPRIORITIZED; } + resident += cg_extract::spill::resident_bytes(&b); batches.push(b); + if resident >= budget.extract / 2 { + report.spilled_kernel_bytes += cg_extract::spill::spill(&mut batches, &spill_dir)?; + resident = 0; + } } - }); + Ok::<_, std::io::Error>(()) + })?; report.files = batches.len(); report.source_bytes = batches.iter().map(|b| b.size as u64).sum(); report.parse_error_files = batches.iter().filter(|b| b.flags & fileflags::PARSE_ERROR != 0).count(); @@ -142,7 +167,7 @@ pub fn build_seg(root: &Path, opts: &IndexOptions) -> Result { let seg = Arc::new(seg); let view = View::new(seg.clone()).with_root(root.to_path_buf()); let (out, synth) = { - let mut engine = Engine::build(&view, &project); + let mut engine = pool.install(|| Engine::build_with_budget(&view, &project, budget)); phase.lap("tables"); let out = pool.install(|| resolve_all(&mut engine)); phase.lap("resolve"); @@ -202,8 +227,8 @@ pub fn deprioritize_matcher(root: &Path, config: &ProjectConfig) -> Option Result { let store = Store::at(root); store.create()?; - store.cleanup_tmp(); let guard = store.write_lock(Duration::from_secs(30))?; + store.cleanup_tmp(); index_with_guard(&store, &guard, root, opts) } diff --git a/crates/cg-resolve/src/modules.rs b/crates/cg-resolve/src/modules.rs index d38f61e..8aa2f56 100644 --- a/crates/cg-resolve/src/modules.rs +++ b/crates/cg-resolve/src/modules.rs @@ -73,7 +73,10 @@ impl<'v> Modules<'v> { rs_mod_to_file: FxHashMap::default(), rs_file_mod: FxHashMap::default(), }; - let files: Vec = view.files().collect(); + let mut files: Vec = view.files().collect(); + // Delta file ids are appended, unlike a fresh build's path-ordered ids. Package + // representatives and first-match module aliases must use the same path order. + files.sort_unstable_by(|a, b| view.file(*a).path().cmp(view.file(*b).path())); for &f in &files { let fr = view.file(f); let path = fr.path(); @@ -98,6 +101,17 @@ impl<'v> Modules<'v> { self.view.file(f).path() } + /// Stable file dependency recorded for an import, including package representatives + /// and declarations. A package's representative may change when a file is added. + pub fn target_file(&self, target: ImportTarget) -> Option { + match target { + ImportTarget::File(file) => Some(file), + ImportTarget::Node(node) => Some(self.view.hot(node).file), + ImportTarget::Pkg(pkg) => self.pkgs[pkg.0 as usize].files.first().copied(), + ImportTarget::External | ImportTarget::NotFound => None, + } + } + // ----------------------------------------------------------------- indexing fn index_python(&mut self, f: FileId, path: &str) { // Absolute imports name a module relative to its *package root*: climb while the diff --git a/crates/cg-resolve/src/project.rs b/crates/cg-resolve/src/project.rs index b361609..f348f4e 100644 --- a/crates/cg-resolve/src/project.rs +++ b/crates/cg-resolve/src/project.rs @@ -91,8 +91,14 @@ fn read_json(path: &Path) -> Option { impl ProjectInfo { pub fn load(root: &Path, config: &ProjectConfig) -> ProjectInfo { let files = find_config_files(root, config, 8); + Self::load_known(root, &files) + } + + /// A watcher already knows the configuration file set. Ordinary source edits need no + /// directory walk to rediscover it; configuration and directory events use `load`. + pub fn load_known(root: &Path, files: &[String]) -> ProjectInfo { let mut info = ProjectInfo { root: root.to_path_buf(), ..Default::default() }; - for rel in &files { + for rel in files { let abs = root.join(rel); let Ok(bytes) = std::fs::read(&abs) else { continue }; info.config_hashes.insert(rel.clone(), hash::hash64(&bytes)); diff --git a/crates/cg-resolve/src/resolve.rs b/crates/cg-resolve/src/resolve.rs index f92bc58..2de5e18 100644 --- a/crates/cg-resolve/src/resolve.rs +++ b/crates/cg-resolve/src/resolve.rs @@ -125,29 +125,59 @@ pub struct FileCx<'e, 'v> { type FieldList = Vec<(Box, Option)>; pub struct WorkerCx<'e, 'v> { - cache: Vec<(FileId, Rc>)>, + cache: Vec<(FileId, Rc>, usize)>, + cache_bytes: usize, field_cache: FxHashMap, + field_cache_bytes: usize, } impl<'e, 'v> WorkerCx<'e, 'v> { pub fn new() -> Self { - WorkerCx { cache: Vec::new(), field_cache: FxHashMap::default() } + WorkerCx { cache: Vec::new(), cache_bytes: 0, field_cache: FxHashMap::default(), field_cache_bytes: 0 } } pub fn file_cx(&mut self, e: &'e Engine<'v>, f: FileId) -> Rc> { - if let Some(pos) = self.cache.iter().position(|(id, _)| *id == f) { + if let Some(pos) = self.cache.iter().position(|(id, _, _)| *id == f) { let item = self.cache.remove(pos); let rc = item.1.clone(); self.cache.push(item); return rc; } let rc = Rc::new(FileCx::new(e, f)); - if self.cache.len() >= 6 { - self.cache.remove(0); + let bytes = rc.heap_bytes(); + let limit = e.budget.worker_cache_bytes(); + if bytes > limit { + return rc; + } + while self.cache.len() >= 6 || self.cache_bytes.saturating_add(bytes) > limit { + let (_, _, evicted) = self.cache.remove(0); + self.cache_bytes -= evicted; } - self.cache.push((f, rc.clone())); + self.cache.push((f, rc.clone(), bytes)); + self.cache_bytes += bytes; rc } + + fn cache_field(&mut self, t: NodeIdx, name: &str, value: Option, limit: usize) { + if name.len().saturating_add(256) > limit { + return; + } + if self.field_cache_bytes.saturating_add(name.len() + 256) > limit { + self.field_cache = FxHashMap::default(); + self.field_cache_bytes = 0; + } + let old_map_capacity = self.field_cache.capacity(); + let list = self.field_cache.entry(t).or_default(); + let old_capacity = list.capacity(); + list.push((name.into(), value)); + self.field_cache_bytes += name.len() + + (list.capacity() - old_capacity) * std::mem::size_of::<(Box, Option)>() + + (self.field_cache.capacity() - old_map_capacity) * (std::mem::size_of::<(NodeIdx, FieldList)>() + 8); + if self.field_cache_bytes > limit { + self.field_cache = FxHashMap::default(); + self.field_cache_bytes = 0; + } + } } impl<'e, 'v> Default for WorkerCx<'e, 'v> { @@ -161,6 +191,21 @@ fn kind_of(view: &View, n: NodeIdx) -> NodeKind { } impl<'e, 'v> FileCx<'e, 'v> { + fn heap_bytes(&self) -> usize { + fn table(map: &FxHashMap) -> usize { + map.capacity() * (std::mem::size_of::<(K, V)>() + 8) + } + let extra_bindings = self.bindings.values().filter(|v| v.spilled()).map(|v| v.capacity() * std::mem::size_of::<&BindingRec>()).sum::(); + let extra_imports = self.import_by_local.values().filter(|v| v.spilled()).map(|v| v.capacity() * 4).sum::() + + self.imports_by_line.values().filter(|v| v.spilled()).map(|v| v.capacity() * 4).sum::(); + table(&self.bindings) + + table(&self.import_by_local) + + table(&self.imports_by_line) + + extra_bindings + + extra_imports + + self.imported_files.capacity() * (std::mem::size_of::() + 8) + + if self.wildcards.spilled() { self.wildcards.capacity() * 4 } else { 0 } + } pub fn new(e: &'e Engine<'v>, file: FileId) -> Self { let view = e.view; let fr = view.file(file); @@ -520,7 +565,7 @@ impl<'e, 'v> FileCx<'e, 'v> { } } } - wcx.field_cache.entry(t).or_default().push((name.into(), result)); + wcx.cache_field(t, name, result, self.e.budget.worker_cache_bytes()); result } @@ -1188,3 +1233,19 @@ impl<'e, 'v> FileCx<'e, 'v> { } } } + +#[cfg(test)] +mod cache_tests { + use super::*; + #[test] + fn optional_field_cache_releases_capacity_at_its_budget() { + let mut worker = WorkerCx::new(); + for i in 0..1000 { + worker.cache_field(NodeIdx(i / 3), &format!("field{i}"), Some(NodeIdx(i)), 1024); + assert!(worker.field_cache_bytes <= 1024); + } + let before = worker.field_cache_bytes; + worker.cache_field(NodeIdx(0), &"x".repeat(2048), None, 1024); + assert_eq!(worker.field_cache_bytes, before); + } +} diff --git a/crates/cg-resolve/src/synth/mod.rs b/crates/cg-resolve/src/synth/mod.rs index 1a0db73..1002b09 100644 --- a/crates/cg-resolve/src/synth/mod.rs +++ b/crates/cg-resolve/src/synth/mod.rs @@ -8,6 +8,7 @@ use crate::engine::Engine; use crate::resolve::{Accept, NameRes, WorkerCx}; use cg_core::{EdgeKind, FileId, Lang, NodeIdx, NodeKind, Provenance, NONE}; use cg_store::{outkind, EdgeBuild, View}; +use rayon::prelude::*; use rustc_hash::{FxHashMap, FxHashSet}; use sites::{PASS_EMITTER, PASS_JSX, PASS_REACT_RENDER, ROLE_EMIT, ROLE_ON}; use std::collections::BTreeMap; @@ -27,6 +28,7 @@ const MAX_GO_IMPLEMENTERS: usize = 24; /// Go method names that nearly every type has: an interface made only of these says nothing. const GENERIC_GO_METHODS: &[&str] = &["String", "Error", "Close", "Read", "Write", "Len", "Less", "Swap", "Reset", "Unwrap", "Is", "As", "Format", "Clone"]; +#[derive(Default)] struct Out { edges: Vec, emeta: Vec, @@ -34,6 +36,24 @@ struct Out { } impl Out { + fn append(&mut self, other: Out) { + let remap: Vec<_> = other + .emeta + .into_iter() + .map(|meta| { + *self.meta_ids.entry(meta.clone()).or_insert_with(|| { + self.emeta.push(meta); + (self.emeta.len() - 1) as u32 + }) + }) + .collect(); + self.edges.extend(other.edges.into_iter().map(|mut edge| { + if edge.meta != NONE { + edge.meta = remap[edge.meta as usize]; + } + edge + })); + } #[allow(clippy::too_many_arguments)] // one call site per synthesis pass; a builder would only add noise fn push( &mut self, @@ -191,7 +211,7 @@ fn go_implements(e: &Engine<'_>, out: &mut Out) -> Vec<(NodeIdx, NodeIdx)> { continue; } for c in cands { - if c != *iface && view.hot(c).file != view.hot(*iface).file || c != *iface { + if c != *iface { out.push(EdgeKind::Implements, c, *iface, view.hot(c).start_line, "go-implements", None, None, 10); pairs.push((c, *iface)); } @@ -241,33 +261,42 @@ fn interface_impl(e: &Engine<'_>, subs: &FxHashMap>, out: } /// `.emit('evt')` in a method -> the named handlers registered with `.on('evt', handler)`. -fn event_emitter<'e, 'v>(e: &'e Engine<'v>, out: &mut Out, wcx: &mut WorkerCx<'e, 'v>) { +fn event_emitter<'e, 'v>(e: &'e Engine<'v>, out: &mut Out, wcx: &mut WorkerCx<'e, 'v>, selected: Option<&FxHashSet>) { + if selected.is_some_and(|keys| keys.is_empty()) { + return; + } let view = e.view; #[derive(Default)] - struct Key { + struct Key<'a> { emitters: Vec, - handlers: Vec<(NodeIdx, String, u32, FileId)>, + handlers: Vec<(NodeIdx, &'a str, u32, FileId)>, } - let mut keys: BTreeMap = BTreeMap::new(); + let mut keys: BTreeMap<&str, Key<'_>> = BTreeMap::new(); for f in view.files() { for s in view.sites_of(f) { if s.pass != PASS_EMITTER { continue; } - let event = view.text_str(cg_core::StrId(s.key_a)).to_owned(); + let event = view.text_str(cg_core::StrId(s.key_a)); + if selected.is_some_and(|keys| !keys.contains(event)) { + continue; + } let k = keys.entry(event).or_default(); if s.role == ROLE_EMIT { k.emitters.push(s.node); } else if s.role == ROLE_ON { - k.handlers.push((s.node, view.text_str(s.payload).to_owned(), s.line, f)); + k.handlers.push((s.node, view.text_str(s.payload), s.line, f)); } } } - for (event, mut k) in keys { + for k in keys.values_mut() { k.emitters.sort(); k.emitters.dedup(); + } + let connect = |event: &str, k: &Key<'_>, wcx: &mut WorkerCx<'e, 'v>| { + let mut out = Out::default(); if k.emitters.is_empty() || k.emitters.len() > MAX_EMITTERS { - continue; + return out; } let mut targets: Vec<(NodeIdx, u32, FileId)> = Vec::new(); for (reg, handler, line, f) in &k.handlers { @@ -296,20 +325,41 @@ fn event_emitter<'e, 'v>(e: &'e Engine<'v>, out: &mut Out, wcx: &mut WorkerCx<'e targets.sort(); targets.dedup_by_key(|t| t.0); if targets.is_empty() || targets.len() > MAX_HANDLERS { - continue; + return out; } for em in &k.emitters { for (t, line, f) in &targets { - out.push(EdgeKind::Calls, *em, *t, 0, "event-emitter", Some(path_line(view, *f, *line)), Some(("event", &event)), 30); + out.push(EdgeKind::Calls, *em, *t, 0, "event-emitter", Some(path_line(view, *f, *line)), Some(("event", event)), 30); + } + } + out + }; + // Indexed collection and ordered merging preserve pass/key precedence regardless of + // scheduling. Bound in-flight link outputs to 256 keys; small edits stay sequential. + if keys.len() < 32 || rayon::current_num_threads() == 1 { + for (event, key) in &keys { + out.append(connect(event, key, wcx)); + } + } else { + let keys: Vec<_> = keys.into_iter().collect(); + let per_key = (MAX_EMITTERS * MAX_HANDLERS * (std::mem::size_of::() + 256)) as u64; + let keys_per_chunk = (e.budget.synth / 4 / per_key).clamp(1, 256) as usize; + for chunk in keys.chunks(keys_per_chunk) { + let parts: Vec<_> = chunk.par_iter().map_init(WorkerCx::new, |cx, (event, key)| connect(event, key, cx)).collect(); + for part in parts { + out.append(part); } } } } /// A React class component's method that calls `setState` re-renders the component. -fn react_render(e: &Engine<'_>, out: &mut Out) { +fn react_render(e: &Engine<'_>, out: &mut Out, selected: Option<&FxHashSet>) { let view = e.view; for f in view.files() { + if selected.is_some_and(|files| !files.contains(&f.0)) { + continue; + } let sites: Vec<_> = view.sites_of(f).iter().filter(|s| s.pass == PASS_REACT_RENDER).collect(); if sites.is_empty() { continue; @@ -331,9 +381,12 @@ fn react_render(e: &Engine<'_>, out: &mut Out) { } /// A component's JSX tag renders the child component it names. -fn jsx_render<'e, 'v>(e: &'e Engine<'v>, out: &mut Out, wcx: &mut WorkerCx<'e, 'v>) { +fn jsx_render<'e, 'v>(e: &'e Engine<'v>, out: &mut Out, wcx: &mut WorkerCx<'e, 'v>, selected: Option<&FxHashSet>) { let view = e.view; for f in view.files() { + if selected.is_some_and(|files| !files.contains(&f.0)) { + continue; + } let sites: Vec<_> = view.sites_of(f).iter().filter(|s| s.pass == PASS_JSX).collect(); if sites.is_empty() { continue; @@ -358,9 +411,74 @@ pub fn run_all(e: &Engine<'_>) -> SynthOutput { let go_pairs = go_implements(e, &mut out); let subs = subtypes(e, &go_pairs); interface_impl(e, &subs, &mut out); - event_emitter(e, &mut out, &mut wcx); - react_render(e, &mut out); - jsx_render(e, &mut out, &mut wcx); + event_emitter(e, &mut out, &mut wcx, None); + react_render(e, &mut out, None); + jsx_render(e, &mut out, &mut wcx, None); + SynthOutput { edges: out.edges, emeta: out.emeta } +} + +/// Reconnect only changed event keys / per-file render sites when the definition and type +/// tables did not change. Structural changes deliberately use the full pass set. +pub fn run_incremental(e: &Engine<'_>, previous: &View, changed: &FxHashSet, remap: &FxHashMap, structural: bool) -> SynthOutput { + if structural { + return run_all(e); + } + let mut keys = FxHashSet::default(); + for view in [previous, e.view] { + for f in view.files().filter(|f| changed.contains(&f.0)) { + for s in view.sites_of(f).iter().filter(|s| s.pass == PASS_EMITTER) { + keys.insert(view.text_str(cg_core::StrId(s.key_a)).to_owned()); + } + } + } + let mut out = Out { edges: Vec::new(), emeta: Vec::new(), meta_ids: FxHashMap::default() }; + // Unchanged keys keep their existing edges, including all provenance and registration + // locations. Ids of replaced nodes are remapped even when their meaning did not change. + for f in previous.files() { + for n in previous.nodes_in_file(f) { + for edge in previous.out_edges(NodeIdx(n)).filter(|edge| edge.class == outkind::CLASS_SYNTH) { + let meta = previous.edge_meta(edge.eid).unwrap_or("{}"); + let value: serde_json::Value = serde_json::from_str(meta).unwrap_or_default(); + let by = value.get("synthesizedBy").and_then(|v| v.as_str()).unwrap_or(""); + let recompute = match by { + "event-emitter" => value.get("event").and_then(|v| v.as_str()).is_some_and(|key| keys.contains(key)), + "react-render" | "jsx-render" => changed.contains(&f.0), + _ => false, + }; + if recompute { + continue; + } + let src = remap.get(&edge.src.0).copied().unwrap_or(edge.src.0); + let dst = remap.get(&edge.dst.0).copied().unwrap_or(edge.dst.0); + if !e.view.is_alive(NodeIdx(src)) || !e.view.is_alive(NodeIdx(dst)) { + continue; + } + let mid = *out.meta_ids.entry(meta.to_owned()).or_insert_with(|| { + out.emeta.push(meta.to_owned()); + (out.emeta.len() - 1) as u32 + }); + out.edges.push(EdgeBuild { + src, + dst, + kind: outkind::pack(edge.kind, edge.provenance, edge.class), + line: if changed.contains(&f.0) && matches!(by, "interface-impl" | "go-implements" | "go-method-contains") { + e.view.hot(NodeIdx(src)).start_line + } else { + edge.line.unwrap_or(NONE) + }, + col: edge.col.unwrap_or(NONE), + res: edge.res, + ref_idx: NONE, + meta: mid, + order: 0, + }); + } + } + } + let mut wcx = WorkerCx::new(); + event_emitter(e, &mut out, &mut wcx, Some(&keys)); + react_render(e, &mut out, Some(changed)); + jsx_render(e, &mut out, &mut wcx, Some(changed)); SynthOutput { edges: out.edges, emeta: out.emeta } } diff --git a/crates/cg-resolve/tests/resolve_e2e.rs b/crates/cg-resolve/tests/resolve_e2e.rs index 2a4bf9b..d33b3ab 100644 --- a/crates/cg-resolve/tests/resolve_e2e.rs +++ b/crates/cg-resolve/tests/resolve_e2e.rs @@ -20,6 +20,21 @@ fn has(edges: &[String], needle: &str) -> bool { edges.iter().any(|e| e == needle) } +#[test] +fn parallel_event_keys_and_resolution_are_byte_deterministic() { + let dir = tempfile::tempdir().unwrap(); + for i in 0..80 { + let source = format!("export function handler{i}() {{}}\nexport function fire{i}() {{ bus.emit('event{i}'); }}\nbus.on('event{i}', handler{i});\n"); + std::fs::write(dir.path().join(format!("file{i}.ts")), source).unwrap(); + } + let build = |threads| build_seg(dir.path(), &IndexOptions { threads: Some(threads), memory_mb: Some(2048), ..Default::default() }).unwrap(); + let sequential = build(1); + let parallel = build(4); + assert_eq!(sequential.seg.to_bytes(), parallel.seg.to_bytes(), "all partitions and metadata must be independent of scheduling"); + let view = View::new(Arc::new(parallel.seg)); + assert_eq!(dump::canonical(&view).iter().filter(|line| line.contains("event-emitter")).count(), 80); +} + #[test] fn typescript_imports_classes_and_receivers() { let (_d, view) = project(&[ diff --git a/crates/cg-store/src/build.rs b/crates/cg-store/src/build.rs index e7f6bf4..30fe87c 100644 --- a/crates/cg-store/src/build.rs +++ b/crates/cg-store/src/build.rs @@ -578,6 +578,7 @@ pub fn build_draft_at(batches: &[FileBatch], delta: Option<(&[FileId], BuildBase let (qname_post_off, qname_post) = postings(names_pool.len(), &hot, bb, |h| StrId(h.qname.0 - bb.names), &skip); let meta = SegMeta { + total_tokens: 0, kind: if is_delta { "delta" } else { "base" }.into(), format_version: brand::FORMAT_VERSION, engine_version: brand::ENGINE_VERSION, @@ -638,7 +639,7 @@ pub fn build_draft_at(batches: &[FileBatch], delta: Option<(&[FileId], BuildBase } /// CSR of node indexes per string id (ascending node order). -fn postings(n_strings: usize, hot: &[NodeHot], bb: BuildBase, key: impl Fn(&NodeHot) -> StrId, skip: &impl Fn(u8) -> bool) -> (Vec, Vec) { +pub(crate) fn postings(n_strings: usize, hot: &[NodeHot], bb: BuildBase, key: impl Fn(&NodeHot) -> StrId, skip: &impl Fn(u8) -> bool) -> (Vec, Vec) { let mut off = vec![0u32; n_strings + 1]; for h in hot { if !skip(h.kind) { diff --git a/crates/cg-store/src/compact.rs b/crates/cg-store/src/compact.rs new file mode 100644 index 0000000..550ad42 --- /dev/null +++ b/crates/cg-store/src/compact.rs @@ -0,0 +1,304 @@ +//! Repack a live base + delta view without reading source or running resolution. +//! Only reachable records and strings survive; all dense ids are reassigned in path order. + +use crate::build::{postings, BuildBase, Draft, EdgeBuild}; +use crate::intern::Interner; +use crate::{finish, outkind, status, FileFacts, FileNameRec, FinishInput, Part, Seg, SegParts, StrPool, View}; +use cg_core::{FileId, ListId, NodeIdx, NodeKind, ScopeId, StrId, NONE}; +use rustc_hash::FxHashMap; + +fn remap(map: &[u32], id: u32) -> u32 { + if id == NONE { + NONE + } else { + map[id as usize] + } +} + +fn text<'a>(view: &'a View, pool: &mut Interner<'a>, id: StrId) -> StrId { + if id.0 == NONE { + StrId::NONE + } else { + StrId(pool.intern(view.text_str(id))) + } +} + +/// Produce an ordinary base segment. This operation needs no project directory, parsers or +/// resolver and preserves reference outcomes, provenance, source locations and metadata. +pub fn compact<'a>(view: &'a View) -> Seg { + let mut files: Vec<_> = view.files().collect(); + files.sort_by(|a, b| view.file(*a).path().as_bytes().cmp(view.file(*b).path().as_bytes())); + // Edges-only files keep base import/export facts. Those advisory targets can still + // name a removed base file, including the last id; map them to NONE rather than panic. + let mut fm = vec![NONE; view.base.p.files.len().max(files.iter().map(|f| f.idx() + 1).max().unwrap_or(0))]; + let mut nm = vec![NONE; view.base.p.nodes_hot.len() + view.delta.as_ref().map_or(0, |d| d.seg.p.nodes_hot.len())]; + let mut rm = vec![NONE; view.n_refs()]; + let (mut nn, mut nr) = (0, 0); + for (i, &f) in files.iter().enumerate() { + fm[f.idx()] = i as u32; + for n in view.nodes_in_file(f) { + nm[n as usize] = nn; + nn += 1; + } + for r in view.ref_range(f) { + rm[r as usize] = nr; + nr += 1; + } + } + let mut p = SegParts::default(); + let (mut names, mut paths, mut texts) = (Interner::default(), Interner::default(), Interner::default()); + let (mut frs, mut hot, mut cold, mut refs) = (Vec::new(), Vec::new(), Vec::new(), Vec::new()); + let (mut imports, mut exports, mut scopes, mut bindings, mut sites, mut facts) = (Vec::new(), Vec::new(), Vec::new(), Vec::new(), Vec::new(), Vec::new()); + let (mut list_off, mut list_items) = (vec![0u32], Vec::new()); + let mut list_map: FxHashMap = FxHashMap::default(); + let mut copy_list = |id: ListId, texts: &mut Interner<'a>| -> ListId { + if id.0 == NONE { + return id; + } + *list_map.entry(id.0).or_insert_with(|| { + let new = ListId((list_off.len() - 1) as u32); + for s in view.list(id) { + list_items.push(texts.intern(s)); + } + list_off.push(list_items.len() as u32); + new + }) + }; + let (mut edges, mut emeta) = (Vec::new(), Vec::new()); + let mut emap = FxHashMap::default(); + let mut fnames = Vec::new(); + for &f in &files { + let file = FileId(fm[f.idx()]); + let rec = *view.file(f).rec; + let path = view.file(f).path(); + let base = path.rsplit('/').next().unwrap_or(path); + fnames.push(FileNameRec { name: StrId(names.intern(base)), file }); + if let Some((stem, _)) = base.rsplit_once('.') { + fnames.push(FileNameRec { name: StrId(names.intern(stem)), file }); + } + let mut new_rec = rec; + new_rec.path = StrId(paths.intern(path)); + new_rec.errors = text(view, &mut texts, rec.errors); + new_rec.node_start = NodeIdx(hot.len() as u32); + new_rec.ref_start = refs.len() as u32; + frs.push(new_rec); + for old in view.nodes_in_file(f) { + let n = NodeIdx(old); + let (mut h, mut c) = (*view.hot(n), *view.cold(n)); + h.file = file; + h.parent = NodeIdx(remap(&nm, h.parent.0)); + h.name = StrId(names.intern(view.name_str(h.name))); + h.qname = StrId(names.intern(view.name_str(h.qname))); + c.signature = text(view, &mut texts, c.signature); + c.docstring = text(view, &mut texts, c.docstring); + c.return_type = text(view, &mut texts, c.return_type); + c.extra = text(view, &mut texts, c.extra); + c.decorators = copy_list(c.decorators, &mut texts); + c.type_params = copy_list(c.type_params, &mut texts); + hot.push(h); + cold.push(c); + for e in view.out_edges(n) { + let meta = match view.edge_meta(e.eid) { + Some(s) => *emap.entry(s).or_insert_with(|| { + emeta.push(s.to_owned()); + (emeta.len() - 1) as u32 + }), + None => NONE, + }; + edges.push(EdgeBuild { + src: remap(&nm, old), + dst: remap(&nm, e.dst.0), + kind: outkind::pack(e.kind, e.provenance, e.class), + line: e.line.unwrap_or(NONE), + col: e.col.unwrap_or(NONE), + res: e.res, + ref_idx: e.ref_idx.map_or(NONE, |r| remap(&rm, r)), + meta, + order: 0, + }); + } + } + for old in view.refs_of_file(f) { + let mut r = *old; + r.from = NodeIdx(remap(&nm, r.from.0)); + if r.status == status::RESOLVED { + r.target = NodeIdx(remap(&nm, r.target.0)); + } + r.name = text(view, &mut texts, r.name); + r.tail = if r.tail.0 == NONE { StrId::NONE } else { StrId(names.intern(view.name_str(r.tail))) }; + r.candidates = copy_list(r.candidates, &mut texts); + refs.push(r); + } + let mut ff = FileFacts { + imp_start: imports.len() as u32, + imp_count: view.imports_of(f).len() as u32, + exp_start: exports.len() as u32, + exp_count: view.exports_of(f).len() as u32, + scope_start: scopes.len() as u32, + scope_count: view.scopes_of(f).len() as u32, + bind_start: bindings.len() as u32, + bind_count: view.bindings_of(f).len() as u32, + site_start: sites.len() as u32, + site_count: view.sites_of(f).len() as u32, + }; + let old_scope = view.scope_base(f); + let scope = |id: ScopeId| if id.0 == NONE { id } else { ScopeId(id.0 - old_scope + ff.scope_start) }; + for old in view.imports_of(f) { + let mut r = *old; + r.file = file; + r.target = FileId(remap(&fm, r.target.0)); + r.scope = scope(r.scope); + r.local = text(view, &mut texts, r.local); + r.imported = text(view, &mut texts, r.imported); + r.spec = text(view, &mut texts, r.spec); + imports.push(r); + } + for old in view.exports_of(f) { + let mut r = *old; + r.file = file; + r.node = NodeIdx(remap(&nm, r.node.0)); + r.target = FileId(remap(&fm, r.target.0)); + r.name = text(view, &mut texts, r.name); + r.from_spec = text(view, &mut texts, r.from_spec); + r.from_name = text(view, &mut texts, r.from_name); + exports.push(r); + } + for old in view.scopes_of(f) { + let mut r = *old; + r.file = file; + r.parent = scope(r.parent); + scopes.push(r); + } + for old in view.bindings_of(f) { + let mut r = *old; + r.scope = scope(r.scope); + r.node = NodeIdx(remap(&nm, r.node.0)); + r.name = text(view, &mut texts, r.name); + r.decl_type = text(view, &mut texts, r.decl_type); + r.init_ctor = text(view, &mut texts, r.init_ctor); + bindings.push(r); + } + for old in view.sites_of(f) { + let mut r = *old; + r.node = NodeIdx(remap(&nm, r.node.0)); + r.key_a = text(view, &mut texts, StrId(r.key_a)).0; + r.key_b = text(view, &mut texts, StrId(r.key_b)).0; + r.payload = text(view, &mut texts, r.payload); + sites.push(r); + } + // Scope ids are local to their owning segment; all are dense in the new base. + ff.scope_count = view.scopes_of(f).len() as u32; + facts.push(ff); + } + let (ns, nmap) = names.finalize(); + let (ps, pmap) = paths.finalize(); + let (ts, tmap) = texts.finalize(); + let rt = |id: &mut StrId| id.0 = remap(&tmap, id.0); + for f in &mut frs { + f.path.0 = remap(&pmap, f.path.0); + rt(&mut f.errors); + } + for h in &mut hot { + h.name.0 = remap(&nmap, h.name.0); + h.qname.0 = remap(&nmap, h.qname.0); + } + for c in &mut cold { + rt(&mut c.signature); + rt(&mut c.docstring); + rt(&mut c.return_type); + rt(&mut c.extra); + } + for r in &mut refs { + rt(&mut r.name); + r.tail.0 = remap(&nmap, r.tail.0); + } + for r in &mut imports { + rt(&mut r.local); + rt(&mut r.imported); + rt(&mut r.spec); + } + for r in &mut exports { + rt(&mut r.name); + rt(&mut r.from_spec); + rt(&mut r.from_name); + } + for r in &mut bindings { + rt(&mut r.name); + rt(&mut r.decl_type); + rt(&mut r.init_ctor); + } + for r in &mut sites { + r.key_a = remap(&tmap, r.key_a); + r.key_b = remap(&tmap, r.key_b); + rt(&mut r.payload); + } + for r in &mut list_items { + *r = remap(&tmap, *r); + } + for r in &mut fnames { + r.name.0 = remap(&nmap, r.name.0); + } + fnames.sort_unstable_by_key(|r| (r.name.0, r.file.0)); + fnames.dedup(); + let names = StrPool::from_sorted_unique(&ns); + let paths = StrPool::from_sorted_unique(&ps); + let texts = StrPool::from_sorted_unique(&ts); + let skip = |kind| !NodeKind::from_u8(kind).is_some_and(|k| k.is_definition()); + let (no, np) = postings(names.len(), &hot, BuildBase::default(), |h| h.name, &skip); + let (qo, qp) = postings(names.len(), &hot, BuildBase::default(), |h| h.qname, &skip); + p.names_off = names.off; + p.names_data = names.data; + p.paths_off = paths.off; + p.paths_data = paths.data; + p.text_off = texts.off; + p.text_data = texts.data; + p.files = Part::owned(frs); + p.nodes_hot = Part::owned(hot); + p.nodes_cold = Part::owned(cold); + p.refs = Part::owned(refs); + p.imports = Part::owned(imports); + p.exports = Part::owned(exports); + p.scopes = Part::owned(scopes); + p.bindings = Part::owned(bindings); + p.sites = Part::owned(sites); + p.file_facts = Part::owned(facts); + p.lists_off = Part::owned(list_off); + p.lists = Part::owned(list_items); + p.name_post_off = Part::owned(no); + p.name_post = Part::owned(np); + p.qname_post_off = Part::owned(qo); + p.qname_post = Part::owned(qp); + p.fname_idx = Part::owned(fnames); + let mut meta = view.base.meta.clone(); + meta.langs = files.iter().map(|f| view.file(*f).lang().as_str().to_owned()).collect(); + meta.langs.sort(); + meta.langs.dedup(); + meta.kind = "base".into(); + meta.base_gen = 0; + meta.n_files = files.len() as u32; + meta.n_nodes = nn; + meta.n_refs = nr; + finish(Draft { seg: Seg::from_parts(meta, p), edges, emeta }, FinishInput::default()) +} + +#[cfg(test)] +mod tests { + use super::*; + use cg_core::Lang; + use cg_extract::{extract_bytes, ExtractConfig}; + + #[test] + fn spilled_and_resident_extraction_produce_identical_segments() { + let dir = tempfile::tempdir().unwrap(); + let cfg = ExtractConfig::new(dir.path()); + let mut batches = vec![ + extract_bytes("lib.ts", b"export class Service { run(): number { return 1; } }\n", 7, Lang::TypeScript, &cfg), + extract_bytes("app.ts", b"import { Service } from './lib';\nexport function start() { new Service().run(); }\n", 8, Lang::TypeScript, &cfg), + ]; + batches.sort_by(|a, b| a.rel.cmp(&b.rel)); + let build = |b: &[cg_extract::FileBatch]| finish(crate::build_draft(b, &crate::ProjectMeta::default()), FinishInput::default()).to_bytes(); + let resident = build(&batches); + assert!(cg_extract::spill::spill(&mut batches, dir.path()).unwrap() > 0); + assert_eq!(resident, build(&batches)); + } +} diff --git a/crates/cg-store/src/delta.rs b/crates/cg-store/src/delta.rs index d4bce2b..8b80c49 100644 --- a/crates/cg-store/src/delta.rs +++ b/crates/cg-store/src/delta.rs @@ -206,7 +206,7 @@ pub fn assemble_delta(base: &Seg, parts: DeltaParts) -> Seg { seg.p.tri_keys = Part::owned(ni.tri_keys); seg.p.tri_off = Part::owned(ni.tri_off); seg.p.tri_post = Part::owned(ni.tri_post); - let slot: FxHashMap = seg.p.delta_files.chunks_exact(3).enumerate().map(|(i, t)| (t[0], i)).collect(); + let slot: FxHashMap = seg.p.delta_files.as_chunks::<3>().0.iter().enumerate().map(|(i, t)| (t[0], i)).collect(); let ti = index::build_token_index(&TokenInputs { hot: &seg.p.nodes_hot, cold: &seg.p.nodes_cold, @@ -223,6 +223,7 @@ pub fn assemble_delta(base: &Seg, parts: DeltaParts) -> Seg { seg.p.tokens_fst = Part::owned(ti.fst); seg.p.token_post_off = Part::owned(ti.post_off); seg.p.token_post = Part::owned(ti.post); + seg.p.token_df = Part::owned(ti.df); seg.p.node_norm = Part::owned(ti.node_norm); } diff --git a/crates/cg-store/src/edges.rs b/crates/cg-store/src/edges.rs index 2af2b8c..e363081 100644 --- a/crates/cg-store/src/edges.rs +++ b/crates/cg-store/src/edges.rs @@ -42,9 +42,67 @@ fn cmp(a: &EdgeBuild, b: &EdgeBuild) -> Ordering { .then_with(|| a.kind.cmp(&b.kind)) } +fn identity(e: &EdgeBuild) -> (u32, u8, u32, u32, u32) { + (e.src, e.kind & 15, e.dst, e.line, e.col) +} + +/// Stable LSD radix sorting of the integer identity, followed by comparison sorting only +/// inside duplicate groups to preserve class / pass precedence. Constant bytes are skipped. +fn sort_edges(edges: &mut Vec) { + let scratch_bytes = edges.len().saturating_mul(std::mem::size_of::()); + if edges.len() < 32_768 || scratch_bytes as u64 > cg_core::budget::MemBudget::detect().extract / 2 { + edges.par_sort_unstable_by(cmp); + return; + } + let mut scratch = vec![edges[0]; edges.len()]; + let key = |e: &EdgeBuild, field| match field { + 0 => e.col, + 1 => e.line, + 2 => e.dst, + 3 => (e.kind & 15) as u32, + _ => e.src, + }; + for field in 0..5 { + let first = key(&edges[0], field); + let varied = edges.iter().fold(0, |v, e| v | (key(e, field) ^ first)); + for shift in (0..32).step_by(8) { + if varied & (255 << shift) == 0 { + continue; + } + let mut counts = [0usize; 256]; + for e in edges.iter() { + counts[((key(e, field) >> shift) & 255) as usize] += 1; + } + let mut sum = 0; + for count in &mut counts { + let n = *count; + *count = sum; + sum += n; + } + for e in edges.iter() { + let slot = &mut counts[((key(e, field) >> shift) & 255) as usize]; + scratch[*slot] = *e; + *slot += 1; + } + std::mem::swap(edges, &mut scratch); + } + } + let mut start = 0; + while start < edges.len() { + let mut end = start + 1; + while end < edges.len() && identity(&edges[start]) == identity(&edges[end]) { + end += 1; + } + if end - start > 1 { + edges[start..end].sort_unstable_by(cmp); + } + start = end; + } +} + /// Sort and deduplicate `edges` in place and build the CSR arrays for `n_nodes` nodes. pub fn assemble(edges: &mut Vec, n_nodes: usize) -> EdgeAsm { - edges.par_sort_unstable_by(cmp); + sort_edges(edges); edges.dedup_by(|b, a| a.src == b.src && a.dst == b.dst && (a.kind & 0x0F) == (b.kind & 0x0F) && a.line == b.line && a.col == b.col); let m = edges.len(); let mut out_off = vec![0u32; n_nodes + 1]; @@ -103,7 +161,7 @@ pub struct SparseAsm { /// Like [`assemble`], but the CSR is indexed by position in the sorted list of source nodes /// (`d_src_nodes`) and, for the reverse side, of destination nodes (`d_dst_nodes`). pub fn assemble_sparse(edges: &mut Vec) -> SparseAsm { - edges.par_sort_unstable_by(cmp); + sort_edges(edges); edges.dedup_by(|b, a| a.src == b.src && a.dst == b.dst && (a.kind & 0x0F) == (b.kind & 0x0F) && a.line == b.line && a.col == b.col); let m = edges.len(); let mut src_nodes: Vec = Vec::new(); @@ -186,6 +244,42 @@ mod tests { use super::*; use cg_core::{EdgeKind, Provenance, NONE}; + #[test] + fn radix_order_matches_comparison_order_and_sparse_csr() { + let mut state = 0x0000_5eed_f00d_u64; + let mut edges = Vec::with_capacity(40_004); + for i in 0..40_000 { + state ^= state << 13; + state ^= state >> 7; + state ^= state << 17; + let key = state as u32; + let mut edge = e(key % 503, (key >> 9) % 997, EdgeKind::Calls, (i % 4) as u8, key >> 20); + edge.col = key & 127; + edge.meta = i; + edge.order = (i % 5) as u16; + edge.ref_idx = i; + edges.push(edge); + } + for class in 0..4 { + let mut edge = e(502, 996, EdgeKind::Contains, class, NONE); + edge.col = NONE; + edges.push(edge); + } + let mut expected = edges.clone(); + expected.sort_unstable_by(cmp); + sort_edges(&mut edges); + let fields = |e: &EdgeBuild| (identity(e), e.kind, e.res, e.ref_idx, e.meta, e.order); + assert_eq!(edges.iter().map(fields).collect::>(), expected.iter().map(fields).collect::>()); + let mut dense_edges = edges.clone(); + let dense = assemble(&mut dense_edges, 997); + let sparse = assemble_sparse(&mut edges); + assert_eq!(dense.out_dst, sparse.asm.out_dst); + assert_eq!(dense.out_kind, sparse.asm.out_kind); + for (pos, &node) in sparse.src_nodes.iter().enumerate() { + assert_eq!(dense.out_off[node as usize + 1] - dense.out_off[node as usize], sparse.asm.out_off[pos + 1] - sparse.asm.out_off[pos]); + } + } + fn e(src: u32, dst: u32, kind: EdgeKind, class: u8, line: u32) -> EdgeBuild { EdgeBuild { src, dst, kind: outkind::pack(kind, Provenance::TreeSitter, class), line, col: 0, res: 0, ref_idx: NONE, meta: NONE, order: 0 } } diff --git a/crates/cg-store/src/finish.rs b/crates/cg-store/src/finish.rs index fd4f2a6..624b116 100644 --- a/crates/cg-store/src/finish.rs +++ b/crates/cg-store/src/finish.rs @@ -131,7 +131,9 @@ pub fn finish(draft: Draft, input: FinishInput) -> Seg { seg.p.tokens_fst = Part::owned(ti.fst); seg.p.token_post_off = Part::owned(ti.post_off); seg.p.token_post = Part::owned(ti.post); + seg.p.token_df = Part::owned(ti.df); seg.p.node_norm = Part::owned(ti.node_norm); + seg.meta.total_tokens = seg.p.node_norm.iter().map(|n| *n as u64).sum(); seg.p.filestats = Part::owned(file_stats(&seg)); seg.meta.n_edges = seg.p.out_dst.len() as u32; diff --git a/crates/cg-store/src/index.rs b/crates/cg-store/src/index.rs index 90b8f09..649f0be 100644 --- a/crates/cg-store/src/index.rs +++ b/crates/cg-store/src/index.rs @@ -105,7 +105,8 @@ pub fn build_name_indexes(names: &StrPool, has_post: impl Fn(usize) -> bool + Sy pub struct TokenIndex { pub fst: Vec, pub post_off: Vec, - pub post: Vec, + pub post: Vec, + pub df: Vec, pub node_norm: Vec, } @@ -270,13 +271,112 @@ pub fn build_token_index(inp: &TokenInputs<'_>) -> TokenIndex { post[fill[r] as usize] = PostEntry { node: e.node, mask: e.mask, tf: e.tf, _p: 0 }; fill[r] += 1; } - TokenIndex { fst, post_off, post, node_norm: norm } + let df: Vec = post_off.windows(2).map(|w| w[1] - w[0]).collect(); + let mut encoded = Vec::with_capacity(post.len() * 3); + let mut offsets = Vec::with_capacity(post_off.len()); + offsets.push(0); + for range in post_off.windows(2) { + encode_postings(&post[range[0] as usize..range[1] as usize], &mut encoded); + offsets.push(u32::try_from(encoded.len()).expect("token posting partition exceeds 4 GiB")); + } + TokenIndex { fst, post_off: offsets, post: encoded, df, node_norm: norm } +} + +fn encode_postings(entries: &[PostEntry], out: &mut Vec) { + let mut previous = 0; + for entry in entries { + let mut delta = entry.node - previous; + previous = entry.node; + while delta >= 128 { + out.push(delta as u8 | 0x80); + delta >>= 7; + } + out.extend_from_slice(&[delta as u8, entry.mask, entry.tf]); + } +} + +/// Allocation-free decoder for one sorted, delta-varint encoded posting list. +pub struct TokenPostings<'a> { + bytes: &'a [u8], + previous: u32, +} + +impl<'a> TokenPostings<'a> { + pub fn new(bytes: &'a [u8]) -> Self { + Self { bytes, previous: 0 } + } +} + +impl Iterator for TokenPostings<'_> { + type Item = PostEntry; + fn next(&mut self) -> Option { + let result = self.decode(); + if result.is_none() { + self.bytes = &[]; + } + result + } +} + +impl std::iter::FusedIterator for TokenPostings<'_> {} + +impl TokenPostings<'_> { + fn decode(&mut self) -> Option { + let mut delta = 0u32; + for shift in (0..35).step_by(7) { + let (&byte, rest) = self.bytes.split_first()?; + self.bytes = rest; + if shift == 28 && byte > 15 { + self.bytes = &[]; + return None; + } + delta |= ((byte & 127) as u32) << shift; + if byte & 128 == 0 { + let node = self.previous.checked_add(delta)?; + let entry = PostEntry { node, mask: *self.bytes.first()?, tf: *self.bytes.get(1)?, _p: 0 }; + self.bytes = &self.bytes[2..]; + self.previous = node; + return Some(entry); + } + } + self.bytes = &[]; + None + } } #[cfg(test)] mod tests { use super::*; + #[test] + fn compressed_postings_cover_varint_boundaries_without_losing_fields() { + let expected: Vec<_> = + [0, 127, 255, 16_639, u32::MAX].into_iter().enumerate().map(|(i, node)| PostEntry { node, mask: 1 << i, tf: i as u8 + 1, _p: 0 }).collect(); + let mut bytes = Vec::new(); + encode_postings(&expected, &mut bytes); + let decoded: Vec<_> = TokenPostings::new(&bytes).collect(); + let fields = |p: &PostEntry| (p.node, p.mask, p.tf); + assert_eq!(decoded.iter().map(fields).collect::>(), expected.iter().map(fields).collect::>()); + assert_eq!(&bytes[..3], &[0, 1, 1]); + assert!(bytes.len() < expected.len() * std::mem::size_of::()); + } + + #[test] + fn malformed_postings_terminate_without_overflow_or_partial_records() { + for bytes in [&[128][..], &[0, 1], &[255, 255, 255, 255, 16, 1, 1], &[255, 255, 255, 255, 255, 1, 1]] { + let mut decoder = TokenPostings::new(bytes); + assert!(decoder.next().is_none()); + assert!(decoder.next().is_none()); + } + let mut bytes = Vec::new(); + encode_postings(&[PostEntry { node: u32::MAX, mask: 1, tf: 1, _p: 0 }], &mut bytes); + bytes.extend_from_slice(&[1, 1, 1]); + let mut decoder = TokenPostings::new(&bytes); + assert_eq!(decoder.next().unwrap().node, u32::MAX); + assert!(decoder.next().is_none()); + assert!(decoder.next().is_none()); + } + #[test] fn name_indexes_find_by_segment_and_trigram() { let names = StrPool::from_sorted_unique(&["Alpha", "getUserName", "render", "renderStaticScene"]); diff --git a/crates/cg-store/src/lib.rs b/crates/cg-store/src/lib.rs index 535bb33..dc9d11f 100644 --- a/crates/cg-store/src/lib.rs +++ b/crates/cg-store/src/lib.rs @@ -1,6 +1,7 @@ //! cg-store: the immutable snapshot format and everything that reads or writes it. pub mod build; +pub mod compact; pub mod delta; pub mod dump; pub mod edges; diff --git a/crates/cg-store/src/seg.rs b/crates/cg-store/src/seg.rs index af91890..ed7bcf5 100644 --- a/crates/cg-store/src/seg.rs +++ b/crates/cg-store/src/seg.rs @@ -92,7 +92,7 @@ seg_parts! { tri_post: u32 = 36, tokens_fst: u8 = 37, token_post_off: u32 = 38, - token_post: PostEntry = 39, + token_post: u8 = 39, node_norm: u16 = 40, filestats: FileStat = 41, refs: RefRec = 42, @@ -105,6 +105,7 @@ seg_parts! { config_digests: u8 = 49, fname_idx: FileNameRec = 50, tail_idx: TailRec = 51, + token_df: u32 = 52, // ---- delta layers (design doc 5.6) ---- dead_nodes: u64 = 60, emeta_off: u32 = 62, @@ -137,6 +138,9 @@ pub struct SegMeta { pub n_nodes: u32, pub n_edges: u32, pub n_refs: u32, + /// Sum of BM25 document lengths, computed once while building the snapshot. + #[serde(default)] + pub total_tokens: u64, /// For a delta: the generation of the base it applies to. pub base_gen: u64, } @@ -261,6 +265,17 @@ impl Seg { out } + /// Exact encoded size without hashing partitions or allocating a serialized copy. + pub fn encoded_len(&self) -> u64 { + let entries = self.p.entries(); + let align = |n: usize| (n + 7) & !7; + let mut size = align(HEADER_SIZE + entries.len() * DIR_ENTRY_SIZE); + for (id, _, bytes, _) in entries { + size = align(size) + if id == 1 { serde_json::to_vec(&self.meta).expect("meta json").len() } else { bytes.len() }; + } + align(size) as u64 + } + pub fn write_into(&self, w: &mut W) -> std::io::Result { let mut parts = self.p.entries(); // Partition 1 (meta) is always regenerated from `self.meta`. @@ -370,6 +385,9 @@ mod tests { std::fs::write(&path, &bytes).unwrap(); let back = Seg::open(&path).unwrap(); assert!(back.verify().is_err()); + // Immutable mappings must be closed before truncating their file; Windows also + // enforces this at the OS boundary. + drop(back); std::fs::write(&path, &bytes[..n - 8]).unwrap(); assert!(Seg::open(&path).is_err()); std::fs::write(&path, b"nope").unwrap(); diff --git a/crates/cg-store/src/view.rs b/crates/cg-store/src/view.rs index 6739623..29a9510 100644 --- a/crates/cg-store/src/view.rs +++ b/crates/cg-store/src/view.rs @@ -219,7 +219,7 @@ impl View { } let b = &self.base; let mut file_pos = FxHashMap::default(); - for (i, tri) in delta.p.delta_files.chunks_exact(3).enumerate() { + for (i, tri) in delta.p.delta_files.as_chunks::<3>().0.iter().enumerate() { file_pos.insert(tri[0], i); } let idx = DeltaIdx { @@ -243,6 +243,7 @@ impl View { // ---------------------------------------------------------------- strings pub fn name_str(&self, id: StrId) -> &str { + cg_core::cancel::check(); match self.d() { Some(d) if id.0 != NONE && id.0 >= d.base_names => d.seg.names.get(StrId(id.0 - d.base_names)), _ => self.base.names.get(id), @@ -289,7 +290,7 @@ impl View { // ------------------------------------------------------------------ files pub fn n_files(&self) -> usize { - self.base.p.files.len() + self.d().map_or(0, |d| d.seg.p.delta_files.chunks_exact(3).filter(|t| t[1] == DeltaState::New as u32).count()) + self.base.p.files.len() + self.d().map_or(0, |d| d.seg.p.delta_files.as_chunks::<3>().0.iter().filter(|t| t[1] == DeltaState::New as u32).count()) } fn delta_file(&self, f: FileId) -> Option<(usize, DeltaState)> { @@ -312,6 +313,7 @@ impl View { } pub fn file(&self, f: FileId) -> FileRef<'_> { + cg_core::cancel::check(); let rec = match self.delta_file(f) { Some((pos, _)) => &self.d().unwrap().seg.p.files[pos], None => &self.base.p.files[f.idx()], @@ -324,7 +326,7 @@ impl View { let nb = self.base.p.files.len() as u32; let added: Vec = match self.d() { Some(d) => { - let mut v: Vec = d.seg.p.delta_files.chunks_exact(3).filter(|t| t[1] == DeltaState::New as u32).map(|t| t[0]).collect(); + let mut v: Vec = d.seg.p.delta_files.as_chunks::<3>().0.iter().filter(|t| t[1] == DeltaState::New as u32).map(|t| t[0]).collect(); v.sort_unstable(); v } @@ -338,7 +340,7 @@ impl View { // The delta's own paths: replaced / added files. if let Some(local) = d.seg.paths.find(p.as_bytes()) { let global = StrId(local.0 + d.base_paths); - for (i, tri) in d.seg.p.delta_files.chunks_exact(3).enumerate() { + for (i, tri) in d.seg.p.delta_files.as_chunks::<3>().0.iter().enumerate() { if d.seg.p.files[i].path == global && tri[1] != DeltaState::Removed as u32 { return Some(FileId(tri[0])); } @@ -364,6 +366,7 @@ impl View { #[inline] pub fn hot(&self, n: NodeIdx) -> &NodeHot { + cg_core::cancel::check(); let i = n.idx(); match self.d() { Some(d) if i >= d.n_base_nodes as usize => &d.seg.p.nodes_hot[i - d.n_base_nodes as usize], @@ -453,6 +456,7 @@ impl View { /// Outgoing edges of `n`. pub fn out_edges(&self, n: NodeIdx) -> OutEdges<'_> { + cg_core::cancel::check(); let hot = self.hot(n); if let Some(d) = self.d() { let in_delta_nodes = n.0 >= d.n_base_nodes; diff --git a/crates/cg-sync/Cargo.toml b/crates/cg-sync/Cargo.toml index 85d9df9..044c9ef 100644 --- a/crates/cg-sync/Cargo.toml +++ b/crates/cg-sync/Cargo.toml @@ -18,6 +18,7 @@ tracing = { workspace = true } ignore = { workspace = true } [dev-dependencies] +serde_json = { workspace = true } tempfile = { workspace = true } cg-query = { workspace = true } proptest = { workspace = true } diff --git a/crates/cg-sync/src/lib.rs b/crates/cg-sync/src/lib.rs index 3114837..d2f3f14 100644 --- a/crates/cg-sync/src/lib.rs +++ b/crates/cg-sync/src/lib.rs @@ -5,7 +5,7 @@ pub mod detect; pub mod sync; pub use detect::{diff_view, Diff}; -pub use sync::{sync, sync_locked, SyncOptions, SyncReport}; +pub use sync::{compact_if_due, sync, sync_locked, SyncOptions, SyncReport}; pub mod watch; pub use watch::{spawn_watcher, WatchHandle}; diff --git a/crates/cg-sync/src/sync.rs b/crates/cg-sync/src/sync.rs index 1f56d78..0b7f071 100644 --- a/crates/cg-sync/src/sync.rs +++ b/crates/cg-sync/src/sync.rs @@ -16,7 +16,7 @@ use crate::detect::{diff_view, Diff}; use cg_core::budget::MemBudget; use cg_core::config::ProjectConfig; use cg_core::{brand, CgError, EdgeKind, FileId, NodeIdx, RefKind, Result, StrId, NONE}; -use cg_extract::{extract_stream, fileflags, ExtractConfig, FileBatch}; +use cg_extract::{fileflags, try_extract_stream, ExtractConfig, FileBatch}; use cg_resolve::driver::{make_edge, round_of, update_of, FNREF_META}; use cg_resolve::{hooks_for, index_with_guard, resolve_refs, Engine, IndexOptions, Outcome, ProjectInfo, Round_}; use cg_scan::{scan, ScanOptions}; @@ -26,7 +26,7 @@ use cg_store::{ }; use rustc_hash::{FxHashMap, FxHashSet}; use std::collections::{BTreeMap, BTreeSet}; -use std::path::Path; +use std::path::{Path, PathBuf}; use std::sync::Arc; use std::time::{Duration, Instant}; @@ -35,11 +35,16 @@ pub struct SyncOptions { /// Rebuild the base instead of writing a delta. pub force_full: bool, pub index: IndexOptions, + /// Exact watcher paths. `None` performs a full scan; directory/config/burst events + /// conservatively fall back to one. Periodic full scans recover dropped events. + pub paths: Option>, } #[derive(Clone, Debug, Default)] pub struct SyncReport { pub up_to_date: bool, + /// A base + delta snapshot was repacked without extracting or resolving source. + pub compacted: bool, /// Why the base was rebuilt instead of writing a delta. pub full_rebuild: Option, pub added: usize, @@ -96,6 +101,107 @@ const MAX_AFFECTED_FILE_PCT: usize = 20; /// Merge when the delta covers this much of the project, or after this many syncs (5.7). const MERGE_FILE_PCT: usize = 5; const MERGE_AFTER_SYNCS: u32 = 200; +const MERGE_BYTES: u64 = 64 << 20; +const MERGE_IDLE_MS: u64 = 10 * 60 * 1000; +const MERGE_QUIET_MS: u64 = 30 * 1000; + +fn merge_due(view: &View, manifest: &cg_store::Manifest) -> bool { + let Some(delta) = &view.delta else { + return false; + }; + let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap_or_default().as_millis() as u64; + let idle = now.saturating_sub(manifest.updated_ms); + let files = view.base.p.files.len(); + idle >= MERGE_QUIET_MS + && ((files >= 40 && delta.seg.p.delta_files.len() / 3 * 100 > files * MERGE_FILE_PCT) + || manifest.syncs_since_merge >= MERGE_AFTER_SYNCS + || delta.seg.file_len >= MERGE_BYTES + || idle >= MERGE_IDLE_MS) +} + +fn compact_snapshot(store: &Store, guard: &WriteGuard, view: &View, rep: &mut SyncReport, ph: &mut Phase, budget: MemBudget) -> Result<()> { + let pool = cg_resolve::index::indexing_pool(budget, true)?; + let seg = pool.install(|| cg_store::compact::compact(view)); + ph.lap("compact"); + store.publish_base(&seg, guard)?; + ph.lap("write"); + rep.compacted = true; + Ok(()) +} + +/// Idle maintenance, called by the elected watcher. This never scans, stats or extracts +/// source files. An active writer wins immediately; maintenance retries on the next tick. +pub fn compact_if_due(root: &Path) -> Result { + let store = Store::at(root); + let mut rep = SyncReport { up_to_date: true, ..Default::default() }; + let mut phase = Phase::new(); + let guard = match store.write_lock(Duration::ZERO) { + Ok(guard) => guard, + Err(CgError::Lock(_)) => return Ok(rep), + Err(e) => return Err(e), + }; + let snapshot = store.open_snapshot()?; + if merge_due(&snapshot.view, &snapshot.manifest) { + compact_snapshot(&store, &guard, &snapshot.view, &mut rep, &mut phase, MemBudget::detect())?; + rep.timings = phase.timings; + } + Ok(rep) +} + +fn event_paths(root: &Path, view: &View, paths: Option<&[PathBuf]>) -> Option> { + let paths = paths?; + if paths.len() > 500 { + return None; + } + let mut rels = Vec::new(); + for p in paths { + let p = if p.is_absolute() { p.strip_prefix(root).ok()?.to_path_buf() } else { p.clone() }; + if p.components().any(|c| !matches!(c, std::path::Component::Normal(_))) { + return None; + } + let rel = p.to_str()?.replace('\\', "/"); + if rel.is_empty() || crate::watch::is_config_name(rel.rsplit('/').next()?) || root.join(&p).is_dir() { + return None; + } + // A removed directory is no longer identifiable with stat. + let prefix = format!("{rel}/"); + if view.files().any(|f| view.file(f).path().starts_with(&prefix)) { + return None; + } + rels.push(rel); + } + rels.sort(); + rels.dedup(); + Some(rels) +} + +fn scan_events(root: &Path, config: &ProjectConfig, view: &View, paths: &[String]) -> Option> { + let rules = cg_scan::IgnoreRules::new(root, config); + let mut entries: BTreeMap = view + .files() + .map(|f| { + let file = view.file(f); + let e = cg_scan::FileEntry { rel: file.path().to_owned(), lang: file.lang(), size: file.rec.size as u64, mtime_ns: file.rec.mtime_ns }; + (e.rel.clone(), e) + }) + .collect(); + for rel in paths { + match cg_scan::stat_entry(&rules, config, rel) { + Some(e) => { + entries.insert(rel.clone(), e); + } + None => { + // A tracked-but-ignored file can be included by the full scanner's git + // supplement; use that scanner rather than silently dropping the file. + if entries.contains_key(rel) && root.join(rel).is_file() && rules.is_ignored(rel, false) { + return None; + } + entries.remove(rel); + } + } + } + Some(entries.into_values().collect()) +} /// Bring the index of `root` up to date. pub fn sync(root: &Path, opts: &SyncOptions) -> Result { @@ -232,6 +338,7 @@ fn hierarchy_keys(view: &View, pairs: impl Iterator) pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOptions) -> Result { let mut rep = SyncReport::default(); let mut ph = Phase::new(); + let budget = MemBudget::configured(opts.index.memory_mb, opts.index.threads); let snap = match store.open_snapshot() { Ok(s) => s, @@ -248,7 +355,11 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp } let config = ProjectConfig::load(root); - let project = ProjectInfo::load(root, &config); + let paths = event_paths(root, &snap.view, opts.paths.as_deref()); + let project = match &paths { + Some(_) => ProjectInfo::load_known(root, &base.meta.config_digests.keys().cloned().collect::>()), + None => ProjectInfo::load(root, &config), + }; if base.meta.config_digest != config.digest() { return full_rebuild(store, guard, root, opts, rep, "project configuration changed".into()); } @@ -259,16 +370,19 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp // ---- 1. diff ---- let mut sopts = ScanOptions::new(root); sopts.config = config.clone(); - let scanned = scan(&sopts); + let entries = paths.as_ref().and_then(|p| scan_events(root, &config, &snap.view, p)).unwrap_or_else(|| scan(&sopts).files); ph.lap("scan"); - let current = diff_view(&snap.view, root, &scanned.files); + let current = diff_view(&snap.view, root, &entries); if current.is_empty() { + if merge_due(&snap.view, &manifest) { + compact_snapshot(store, guard, &snap.view, &mut rep, &mut ph, budget)?; + } rep.up_to_date = true; rep.timings = ph.timings; return Ok(rep); } let base_view = View::new(base.clone()).with_root(root.to_path_buf()); - let d: Diff = diff_view(&base_view, root, &scanned.files); + let d: Diff = diff_view(&base_view, root, &entries); ph.lap("diff"); if d.is_empty() { // the tree equals the base again @@ -282,21 +396,9 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp rep.removed = d.removed.len(); rep.touched = d.touched.len(); let n_base_files = base.p.files.len(); - if d.changed() * 100 > n_base_files * MERGE_FILE_PCT.max(1) && n_base_files >= 40 { - let why = format!("{} changed files exceed {MERGE_FILE_PCT}% of the project (merge)", d.changed()); - return full_rebuild(store, guard, root, opts, rep, why); - } - if manifest.syncs_since_merge >= MERGE_AFTER_SYNCS { - return full_rebuild(store, guard, root, opts, rep, format!("{MERGE_AFTER_SYNCS} syncs since the last merge")); - } // ---- 2. re-extract the changed files ---- - let mut budget = MemBudget::detect(); - if let Some(t) = opts.index.threads { - budget.threads = t.max(1); - } - let pool = - rayon::ThreadPoolBuilder::new().num_threads(budget.threads).stack_size(16 << 20).build().map_err(|e| CgError::other(format!("thread pool: {e}")))?; + let pool = cg_resolve::index::indexing_pool(budget, opts.index.low_priority)?; let replaced_ids: FxHashMap<&str, FileId> = d.replaced.iter().map(|(f, e)| (e.rel.as_str(), *f)).collect(); let mut entries: Vec<_> = d.replaced.iter().map(|x| x.1.clone()).chain(d.added.iter().cloned()).collect(); entries.sort_by(|a, b| a.rel.as_bytes().cmp(b.rel.as_bytes())); @@ -305,17 +407,25 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp xcfg.batch_bytes = budget.extract_batch_bytes(); xcfg.overrides = config.extensions.clone(); xcfg.low_priority = opts.index.low_priority; - xcfg.hooks = hooks_for(&project); + xcfg.set_hooks(hooks_for(&project)); let deprio = cg_resolve::index::deprioritize_matcher(root, &config); let mut extracted: Vec> = (0..entries.len()).map(|_| None).collect(); - extract_stream(&entries, &xcfg, |i, b| { + let mut resident = 0u64; + let spill_dir = root.join(brand::DATA_DIR).join("tmp"); + try_extract_stream(&entries, &xcfg, |i, b| { if let Some(mut b) = b { if deprio.as_ref().is_some_and(|g| g.matched_path_or_any_parents(&entries[i].rel, false).is_ignore()) { b.flags |= fileflags::DEPRIORITIZED; } + resident += cg_extract::spill::resident_bytes(&b); extracted[i] = Some(b); + if resident >= budget.extract / 2 { + cg_extract::spill::spill(extracted.iter_mut().filter_map(Option::as_mut), &spill_dir)?; + resident = 0; + } } - }); + Ok::<_, std::io::Error>(()) + })?; ph.lap("extract"); let mut batches: Vec = Vec::new(); @@ -348,7 +458,7 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp let r = &base.p.files[f.idx()]; dead_nodes.extend(r.node_start.0..r.node_start.0 + r.node_count); } - let c_files: FxHashSet = replaced.iter().chain(removed.iter()).map(|f| f.0).collect(); + let c_files: FxHashSet = ids.iter().chain(removed.iter()).map(|f| f.0).collect(); // ---- provisional delta -> resolution engine over the new state ---- let bb = base_offsets(&base); @@ -380,7 +490,7 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp let v1_seg = assemble_delta(&base, provisional(&draft, false)); let view1 = base_view.with_delta(Arc::new(v1_seg))?; ph.lap("draft"); - let mut engine = Engine::build(&view1, &project); + let mut engine = pool.install(|| Engine::build_with_budget(&view1, &project, budget)); ph.lap("tables"); // ---- 3. affected references ---- @@ -394,11 +504,13 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp } // definition / export differences of replaced, new and removed files let mut d_names: BTreeSet = BTreeSet::new(); + let mut definitions_changed = false; let mut export_changed: Vec = Vec::new(); let mut remap: FxHashMap = FxHashMap::default(); for (i, f) in ids.iter().enumerate() { let old = if states[i] == DeltaState::Replaced { defs_of(&base_view, *f) } else { BTreeMap::new() }; let new = defs_of(&view1, *f); + definitions_changed |= old != new; for (k, details) in &new { if (old.get(k) != Some(details)) && !old.contains_key(k) { d_names.insert(k.name.clone()); @@ -438,8 +550,13 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp for g in &live { let mut srcs: FxHashSet = FxHashSet::default(); for t in &engine.imp_targets[g.idx()] { - if let cg_resolve::ImportTarget::File(x) = t { - srcs.insert(x.0); + match t { + cg_resolve::ImportTarget::Pkg(pkg) => srcs.extend(engine.modules.pkgs[pkg.0 as usize].files.iter().map(|file| file.0)), + _ => { + if let Some(file) = engine.modules.target_file(*t) { + srcs.insert(file.0); + } + } } } for x in &engine.stars[g.idx()] { @@ -512,10 +629,7 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp // an import now resolving to another file (or to none) changes what the file binds let facts = base.p.file_facts[fi]; for (i, old) in base.p.imports[facts.imp_start as usize..(facts.imp_start + facts.imp_count) as usize].iter().enumerate() { - let new_t = match engine.imp_targets[fi].get(i) { - Some(cg_resolve::ImportTarget::File(x)) => Some(view1.file(*x).path()), - _ => None, - }; + let new_t = engine.imp_targets[fi].get(i).and_then(|t| engine.modules.target_file(*t)).map(|file| view1.file(file).path()); let old_t = old.target.opt().map(|t| base.paths.get(base.p.files[t.idx()].path)); if new_t != old_t { file_hit = true; @@ -643,7 +757,15 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp ph.lap("resolve"); // ---- synthesised edges: recomputed over the new state, compared with the base ---- - let synth = cg_resolve::synth::run_all(&engine); + let synth = pool.install(|| { + cg_resolve::synth::run_incremental( + &engine, + &base_view, + &c_files, + &remap, + hierarchy_changed || definitions_changed || !export_changed.is_empty() || hit_imports > 0 || !removed.is_empty(), + ) + }); let mut synth_by_file: FxHashMap> = FxHashMap::default(); for (i, e) in synth.edges.iter().enumerate() { synth_by_file.entry(view1.hot(NodeIdx(e.src)).file.0).or_default().push(i); @@ -765,15 +887,8 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp } let ref_updates: Vec = (n_base_refs..n_base_refs + n_draft_refs).map(|id| update_of(&outcomes.get(&id).copied().unwrap_or(Outcome::PENDING))).collect(); - let import_targets: Vec = ids - .iter() - .flat_map(|f| { - engine.imp_targets[f.idx()].iter().map(|t| match t { - cg_resolve::ImportTarget::File(tf) => tf.0, - _ => NONE, - }) - }) - .collect(); + let import_targets: Vec = + ids.iter().flat_map(|f| engine.imp_targets[f.idx()].iter().map(|t| engine.modules.target_file(*t).map_or(NONE, |file| file.0))).collect(); let export_targets: Vec = ids.iter().flat_map(|f| view1.exports_of(*f).iter().map(|ex| engine.export_target(*f, ex)).collect::>()).collect(); let mut edges_only: Vec = Vec::new(); @@ -820,7 +935,7 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp line: e.line.unwrap_or(NONE), col: e.col.unwrap_or(NONE), res: e.res, - ref_idx: e.ref_idx.map_or(NONE, &final_ref), + ref_idx: e.ref_idx.map_or(NONE, final_ref), meta, order: 0, }); @@ -894,6 +1009,8 @@ pub fn sync_locked(store: &Store, guard: &WriteGuard, root: &Path, opts: &SyncOp let mut delta = assemble_delta(&base, parts); delta.meta.n_edges = base.meta.n_edges - dropped_edges + delta.p.out_dst.len() as u32; ph.lap("assemble"); + // Publishing a source edit never waits for compaction. The quiet-period maintenance + // repacks this snapshot later, under the same writer lock. store.publish_delta(&delta, guard)?; ph.lap("write"); rep.timings = ph.timings; diff --git a/crates/cg-sync/src/watch.rs b/crates/cg-sync/src/watch.rs index 41da001..c369f1b 100644 --- a/crates/cg-sync/src/watch.rs +++ b/crates/cg-sync/src/watch.rs @@ -1,8 +1,8 @@ //! File watching (design doc 9.1). One process per project holds `watch.lock` and runs the //! watcher; the others poll the lock and take over within a few seconds if the holder dies. //! Events are filtered by the indexing rules, debounced adaptively (300 ms of quiet for up to two -//! files, 2 s for more), and answered with one sync. A change burst of more than 500 files is just -//! a sync as well: sync always compares the whole tree against the index. +//! files, 2 s for more), then synced by path. A burst of more than 500 files, a config change +//! or a lost event triggers a full scan instead. use crate::sync::{sync, SyncOptions}; use cg_core::brand; @@ -27,6 +27,27 @@ pub struct WatchHandle { } impl WatchHandle { + /// Stop all projects against one shared deadline, so eight watched projects do not + /// multiply shutdown latency. In-progress atomic writes may finish or be abandoned. + pub fn stop_all(mut handles: Vec, wait: Duration) { + for h in &handles { + h.stop.store(true, Ordering::SeqCst); + } + let deadline = Instant::now() + wait; + loop { + for h in &mut handles { + if h.thread.as_ref().is_some_and(|t| t.is_finished()) { + if let Some(t) = h.thread.take() { + let _ = t.join(); + } + } + } + if handles.iter().all(|h| h.thread.is_none()) || Instant::now() >= deadline { + break; + } + std::thread::sleep(Duration::from_millis(10)); + } + } /// Ask the watcher to stop and wait (up to ~1 s) for it. pub fn stop(mut self) { self.stop.store(true, Ordering::SeqCst); @@ -48,7 +69,7 @@ impl Drop for WatchHandle { } } -fn is_config_name(name: &str) -> bool { +pub(crate) fn is_config_name(name: &str) -> bool { matches!( name, ".gitignore" @@ -122,8 +143,6 @@ fn watch_loop(root: PathBuf, opts: SyncOptions, stop: Arc) { } }; tracing::debug!("watching {}", root.display()); - // catch up with whatever changed while nobody watched - run_sync(&root, &opts); run_watch(&root, &opts, &stop); drop(guard); } @@ -138,9 +157,13 @@ fn run_sync(root: &Path, opts: &SyncOptions) { } fn run_watch(root: &Path, opts: &SyncOptions, stop: &AtomicBool) { - let (tx, rx) = mpsc::channel::>(); + let (tx, rx) = mpsc::sync_channel::>(1024); + let overflow = Arc::new(AtomicBool::new(false)); + let over = overflow.clone(); let watcher = notify::recommended_watcher(move |res| { - let _ = tx.send(res); + if tx.try_send(res).is_err() { + over.store(true, Ordering::Relaxed); + } }); let mut watcher = match watcher { Ok(w) => Some(w), @@ -156,39 +179,63 @@ fn run_watch(root: &Path, opts: &SyncOptions, stop: &AtomicBool) { degraded = true; } } + // Register first, then catch up: edits made while the initial scan runs must + // remain queued rather than falling into a scan/watch registration gap. + run_sync(root, opts); let mut filter = Filter::new(root); let mut dirty: HashSet = HashSet::new(); + let mut full_scan = false; let mut last_event = Instant::now(); + let mut burst_started = None; + let mut last_maintenance = Instant::now(); let mut last_scan = Instant::now(); let store = Store::at(root); while !stop.load(Ordering::SeqCst) { + if overflow.swap(false, Ordering::Relaxed) { + full_scan = true; + burst_started.get_or_insert_with(Instant::now); + } match rx.recv_timeout(Duration::from_millis(50)) { Ok(Ok(ev)) => { - if matches!(ev.kind, EventKind::Access(_)) { - continue; + if ev.need_rescan() { + full_scan = true; + burst_started.get_or_insert_with(Instant::now); } - for p in ev.paths { - if p.file_name().and_then(|n| n.to_str()).is_some_and(is_config_name) { - filter = Filter::new(root); - } - if filter.relevant(&p) { - dirty.insert(p); - last_event = Instant::now(); + if !matches!(ev.kind, EventKind::Access(_)) { + for p in ev.paths { + if p.file_name().and_then(|n| n.to_str()).is_some_and(is_config_name) { + filter = Filter::new(root); + } + if filter.relevant(&p) { + burst_started.get_or_insert_with(Instant::now); + if dirty.len() < 500 { + dirty.insert(p); + } else { + full_scan = true; + } + last_event = Instant::now(); + } } } } Ok(Err(e)) => { tracing::warn!("watcher error: {e}"); degraded = true; + full_scan = true; + burst_started.get_or_insert_with(Instant::now); } Err(mpsc::RecvTimeoutError::Timeout) => {} Err(mpsc::RecvTimeoutError::Disconnected) => break, } - if !dirty.is_empty() { + if !dirty.is_empty() || full_scan { let quiet = if dirty.len() <= 2 { QUIET_SMALL } else { QUIET_LARGE }; - if last_event.elapsed() >= quiet { + if last_event.elapsed() >= quiet || burst_started.is_some_and(|t| t.elapsed() >= Duration::from_secs(5)) { + let mut local = opts.clone(); + local.paths = if full_scan { None } else { Some(dirty.drain().collect()) }; dirty.clear(); - run_sync(root, opts); + full_scan = false; + burst_started = None; + run_sync(root, &local); last_scan = Instant::now(); } } else if degraded && last_scan.elapsed() >= FALLBACK_SCAN.min(Duration::from_secs(5)) { @@ -199,6 +246,14 @@ fn run_watch(root: &Path, opts: &SyncOptions, stop: &AtomicBool) { run_sync(root, opts); last_scan = Instant::now(); } + if dirty.is_empty() && !full_scan && last_event.elapsed() >= Duration::from_secs(30) && last_maintenance.elapsed() >= Duration::from_secs(30) { + if let Ok(report) = crate::compact_if_due(root) { + if report.compacted { + tracing::info!("compacted idle snapshot"); + } + } + last_maintenance = Instant::now(); + } // the index may have been removed (`cgr uninit`): stop watching if store.read_manifest().ok().flatten().is_none() { break; diff --git a/crates/cg-sync/tests/incremental.rs b/crates/cg-sync/tests/incremental.rs index 6f05237..abe7593 100644 --- a/crates/cg-sync/tests/incremental.rs +++ b/crates/cg-sync/tests/incremental.rs @@ -280,6 +280,41 @@ fn run_sequence(seed: u64, steps: usize) { type Edit = Box; +#[test] +fn watcher_paths_match_full_after_edits_renames_and_directory_events() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + for (i, (f, c)) in FILES.iter().enumerate() { + write(root, f, c, i as u64); + } + index_project(root, &opts()).unwrap(); + let edits: Vec<(&str, Edit, Vec<&str>)> = vec![ + ("edit", replace_in("ts/service.ts", "return 1", "return 22"), vec!["ts/service.ts"]), + ("add", put("ts/fresh.ts", "import { helper } from './service';\nexport const fresh = helper();\n"), vec!["ts/fresh.ts"]), + ("rename old", delete("ts/fresh.ts"), vec!["ts/fresh.ts", "ts/renamed.ts"]), + ("rename new", put("ts/renamed.ts", "export function fresh() {}\n"), vec!["ts/renamed.ts"]), + ("register", put("bus/reg.ts", "import { bus } from './bus';\nimport { onTick } from './handler';\nbus.on('tock', onTick);\n"), vec!["bus/reg.ts"]), + ("emit", replace_in("bus/bus.ts", "'tick'", "'tock'"), vec!["bus/bus.ts"]), + ("directory", put("folder/child/new.ts", "export function nested() {}\n"), vec!["folder"]), + ("config", put("package.json", "{\"name\":\"fixture\",\"dependencies\":{\"express\":\"4\"}}\n"), vec!["package.json"]), + ]; + for (step, (name, edit, paths)) in edits.into_iter().enumerate() { + edit(root, 900 + step as u64); + let paths = paths.into_iter().map(|p| root.join(p)).collect(); + sync(root, &SyncOptions { index: opts(), paths: Some(paths), ..Default::default() }).unwrap(); + assert_eq!(canonical_of(root), full_canonical(root), "event {name}"); + } + fs::remove_dir_all(root.join("folder")).unwrap(); + sync(root, &SyncOptions { index: opts(), paths: Some(vec![root.join("folder")]), ..Default::default() }).unwrap(); + assert_eq!(canonical_of(root), full_canonical(root), "removed directory"); + // An absent event is recovered by the periodic / explicit full scan. + write(root, "ts/unreported.ts", "export function recovered() {}\n", 999); + sync(root, &SyncOptions { index: opts(), paths: Some(vec![]), ..Default::default() }).unwrap(); + assert!(!canonical_of(root).iter().any(|s| s.contains("recovered"))); + sync(root, &SyncOptions { index: opts(), ..Default::default() }).unwrap(); + assert_eq!(canonical_of(root), full_canonical(root)); +} + fn replace_in(rel: &'static str, from: &'static str, to: &'static str) -> Edit { Box::new(move |root, step| { let cur = fs::read_to_string(root.join(rel)).unwrap(); @@ -387,6 +422,8 @@ fn targeted_edits_match_full_rebuild() { ("py remove later", delete("py/pkg/later.py")), ("go define Later", put("go/a/later.go", "package a\n\nfunc Later() {}\n")), ("go remove Later", delete("go/a/later.go")), + ("go package representative changes", put("go/a/00.go", "package a\n\nfunc Zero() {}\n")), + ("go package representative restored", delete("go/a/00.go")), ("java define Later", put("java/com/ex/Later.java", "package com.ex;\n\npublic class Later {\n public void go() {}\n}\n")), ("java remove Later", delete("java/com/ex/Later.java")), ("rs define later", replace_in("rs/src/util.rs", "fn inner() {}", "fn inner() {}\npub fn later() {}")), @@ -413,6 +450,13 @@ fn targeted_edits_match_full_rebuild() { ("emitter: handler file back", original("bus/handler.ts")), ("emitter: second handler", put("bus/h2.ts", "import { bus } from './bus';\nfunction second() {}\nbus.on('tick', second);\n")), ("emitter: second handler removed", delete("bus/h2.ts")), + ( + "emitter: registration without a new definition", + put("bus/reg.ts", "import { bus } from './bus';\nimport { onTick } from './handler';\nbus.on('tock', onTick);\n"), + ), + ("emitter: activate new registration", replace_in("bus/bus.ts", "'tick'", "'tock'")), + ("emitter: remove registration", delete("bus/reg.ts")), + ("emitter: restore event", original("bus/bus.ts")), ("java: remove Square", delete("java/com/ex/Square.java")), ("java: Square back", original("java/com/ex/Square.java")), ("java: Square stops implementing", replace_in("java/com/ex/Square.java", " implements Shape", "")), @@ -488,6 +532,87 @@ fn up_to_date_tree_writes_nothing() { assert_eq!(Store::at(root).read_manifest().unwrap().unwrap().generation, m2.generation); } +#[test] +fn compaction_preserves_live_graph_and_future_sync_without_source_access() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + for (i, (f, c)) in FILES.iter().enumerate() { + write(root, f, c, i as u64); + } + write(root, "zz.ts", "export function lastFile() {}\n", 100); + write(root, "ts/tail-consumer.ts", "import { lastFile } from '../zz';\nexport function tailConsumer() { lastFile(); }\n", 101); + index_project(root, &opts()).unwrap(); + write(root, "ts/util.ts", "export function util() { return 5; }\nexport function added() { return util(); }\n", 999); + fs::remove_file(root.join("zz.ts")).unwrap(); + let so = SyncOptions { index: opts(), ..Default::default() }; + sync(root, &so).unwrap(); + let store = Store::at(root); + let snap = store.open_snapshot().unwrap(); + assert!(snap.view.delta.is_some(), "compaction must exercise a delta, including a removed last-file dependency"); + let before = dump::canonical(&snap.view); + // This view deliberately has no project/source root. Compaction still has everything + // it needs in the mapped snapshot, including all facts for later syncs. + let mut offline = cg_store::View::new(snap.view.base.clone()); + if let Some(delta) = &snap.view.delta { + offline = offline.with_delta(delta.seg.clone()).unwrap(); + } + let seg = cg_store::compact::compact(&offline); + assert_eq!(seg.encoded_len(), seg.to_bytes().len() as u64); + let packed = cg_store::View::new(std::sync::Arc::new(seg)); + assert_eq!(before, dump::canonical(&packed)); + let again = cg_store::compact::compact(&packed); + assert_eq!(packed.base.to_bytes(), again.to_bytes(), "compaction is deterministic and idempotent"); + let guard = store.write_lock(Duration::from_secs(1)).unwrap(); + store.publish_base(&again, &guard).unwrap(); + drop(guard); + assert_eq!(before, dump::canonical(&snap.view), "readers keep the previous complete generation during publication"); + write(root, "ts/new.ts", "import { util } from './util';\nexport function main() { return util(); }\n", 1001); + write(root, "zz.ts", "export function lastFile() {}\n", 1002); + sync(root, &so).unwrap(); + assert_eq!(canonical_of(root), full_canonical(root)); + let added = store.open_snapshot().unwrap(); + let packed = cg_store::View::new(std::sync::Arc::new(cg_store::compact::compact(&added.view))); + assert_eq!(dump::canonical(&added.view), dump::canonical(&packed), "new-file ids also survive compaction"); +} + +#[test] +fn idle_compaction_waits_for_quiet_and_never_needs_the_source_tree() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + for (i, (f, c)) in FILES.iter().enumerate() { + write(root, f, c, i as u64); + } + index_project(root, &opts()).unwrap(); + write(root, "ts/util.ts", &format!("// comment\n{}", FILES.iter().find(|f| f.0 == "ts/util.ts").unwrap().1), 1000); + sync(root, &SyncOptions { index: opts(), ..Default::default() }).unwrap(); + let store = Store::at(root); + assert!(store.open_snapshot().unwrap().view.delta.is_some()); + let before = canonical_of(root); + let mut manifest = store.read_manifest().unwrap().unwrap(); + manifest.syncs_since_merge = 200; + fs::write(root.join(".cgr/manifest"), serde_json::to_vec(&manifest).unwrap()).unwrap(); + assert!(!cg_sync::compact_if_due(root).unwrap().compacted, "a save must not trigger an immediate merge"); + manifest.updated_ms = manifest.updated_ms.saturating_sub(60_000); + fs::write(root.join(".cgr/manifest"), serde_json::to_vec(&manifest).unwrap()).unwrap(); + // Delete every source directory. Re-extraction / re-resolution would change the graph. + for entry in fs::read_dir(root).unwrap().flatten() { + if entry.file_name() == ".cgr" { + continue; + } + if entry.path().is_dir() { + fs::remove_dir_all(entry.path()).unwrap(); + } else { + fs::remove_file(entry.path()).unwrap(); + } + } + let guard = store.write_lock(Duration::from_secs(1)).unwrap(); + assert!(!cg_sync::compact_if_due(root).unwrap().compacted, "maintenance must yield to an active writer"); + drop(guard); + assert!(cg_sync::compact_if_due(root).unwrap().compacted); + assert!(store.open_snapshot().unwrap().view.delta.is_none()); + assert_eq!(before, canonical_of(root)); +} + #[allow(dead_code)] fn _p(_: PathBuf) {} diff --git a/crates/cgr/Cargo.toml b/crates/cgr/Cargo.toml index 48b8a50..a43db01 100644 --- a/crates/cgr/Cargo.toml +++ b/crates/cgr/Cargo.toml @@ -34,3 +34,6 @@ regex = { workspace = true } tracing = { workspace = true } tracing-subscriber = { workspace = true } mimalloc = { workspace = true } + +[dev-dependencies] +tempfile = { workspace = true } diff --git a/crates/cgr/src/cmds.rs b/crates/cgr/src/cmds.rs index 573ccc0..35e6932 100644 --- a/crates/cgr/src/cmds.rs +++ b/crates/cgr/src/cmds.rs @@ -55,6 +55,7 @@ pub fn index(path: Option, _force: bool, quiet: bool, timings: bool, re "files": report.files, "nodes": report.nodes, "edges": report.edges, "refs": report.refs, "resolved": report.resolved_refs, "totalMs": report.total().as_secs_f64() * 1e3, "peakRssMb": report.peak_rss_mb, "threads": report.threads, "sourceMb": report.source_bytes / (1 << 20), "segmentMb": report.segment_bytes / (1 << 20), "phasesMs": phases, + "spilledKernelBytes": report.spilled_kernel_bytes, "budgetMb": report.budget_mb, }) ); return Ok(0); @@ -430,11 +431,15 @@ pub fn serve(base: Option, http: Option, token: Option, } })); let w = watchers.clone(); + server.set_close_hook(std::sync::Arc::new(move |root: &Path| { + // Dropping requests a stop without holding the project-cache lock or delaying + // the next query while an index write finishes. + w.lock().unwrap_or_else(|e| e.into_inner()).remove(root); + })); + let w = watchers.clone(); server.on_shutdown(Box::new(move || { let handles: Vec<_> = w.lock().unwrap_or_else(|e| e.into_inner()).drain().map(|(_, h)| h).collect(); - for h in handles { - h.stop(); - } + cg_sync::WatchHandle::stop_all(handles, std::time::Duration::from_millis(600)); })); // watch the default project from the start let start = std::env::var_os("CGR_WATCH_ROOT").map(PathBuf::from).or_else(|| cfg_path.clone()).or_else(|| std::env::current_dir().ok()); diff --git a/crates/cgr/src/hook.rs b/crates/cgr/src/hook.rs index 8b0b7bb..4d16e9f 100644 --- a/crates/cgr/src/hook.rs +++ b/crates/cgr/src/hook.rs @@ -74,30 +74,53 @@ fn is_machine_message(p: &str) -> bool { /// Nearest index at or above `cwd`; for a monorepo root without one, a sub-project below it. fn find_project(cwd: &Path, prompt_words: &[String]) -> Option<(Store, bool)> { + cg_core::cancel::check_now(); if let Some(s) = Store::locate(cwd) { return Some((s, false)); } + // Discovery is a convenience for a monorepo, never a search of the entire machine. + if cwd.parent().is_none() || std::env::var_os("HOME").is_some_and(|p| cwd == Path::new(&p)) { + return None; + } + let deadline = std::time::Instant::now() + std::time::Duration::from_millis(200); + let mut visited = 0; let mut found: Vec = Vec::new(); let mut frontier = vec![cwd.to_path_buf()]; - for _ in 0..3 { + 'discover: for _ in 0..3 { let mut next = Vec::new(); for d in frontier { + cg_core::cancel::check_now(); + if visited >= 128 || std::time::Instant::now() >= deadline { + break; + } + visited += 1; let Ok(rd) = std::fs::read_dir(&d) else { continue }; - for e in rd.flatten() { + let mut entries: Vec<_> = rd.take(2048).filter_map(Result::ok).collect(); + entries.sort_by_key(|e| e.file_name()); + for e in entries { + if std::time::Instant::now() >= deadline { + break; + } let p = e.path(); let name = e.file_name().to_string_lossy().into_owned(); - if !p.is_dir() || name.starts_with('.') || matches!(name.as_str(), "node_modules" | "target" | "vendor" | "dist" | "build") { + if !e.file_type().ok().is_some_and(|t| t.is_dir()) + || name.starts_with('.') + || matches!(name.as_str(), "node_modules" | "target" | "vendor" | "dist" | "build") + { continue; } if p.join(brand::DATA_DIR).join("manifest").is_file() { found.push(p); - } else { + if found.len() >= 8 { + break 'discover; + } + } else if next.len() < 128 { next.push(p); } } } frontier = next; - if found.len() >= 8 { + if found.len() >= 8 || visited >= 128 || std::time::Instant::now() >= deadline { break; } } @@ -177,7 +200,8 @@ pub fn run() -> i32 { let Ok(v) = serde_json::from_str::(&input) else { return 0 }; let Some(prompt) = v.get("prompt").and_then(|p| p.as_str()).map(str::to_owned) else { return 0 }; let cwd = v.get("cwd").and_then(|c| c.as_str()).map(PathBuf::from).or_else(|| std::env::current_dir().ok()).unwrap_or_default(); - let out = std::panic::catch_unwind(|| compute(&prompt, &cwd)).ok().flatten(); + let deadline = cg_core::cancel::Cancellation::new(std::time::Duration::from_millis(450)); + let out = std::panic::catch_unwind(|| deadline.run(|| compute(&prompt, &cwd))).ok().flatten(); if let Some(text) = out { use std::io::Write; let mut so = std::io::stdout().lock(); @@ -191,6 +215,27 @@ pub fn run() -> i32 { mod tests { use super::*; + #[test] + fn discovery_skips_filesystem_root_and_directory_symlinks() { + #[cfg(unix)] + assert!(find_project(Path::new("/"), &[]).is_none()); + let dir = tempfile::tempdir().unwrap(); + let indexed = tempfile::tempdir().unwrap(); + std::fs::create_dir_all(indexed.path().join(".cgr")).unwrap(); + std::fs::write(indexed.path().join(".cgr/manifest"), "{}").unwrap(); + #[cfg(unix)] + { + std::os::unix::fs::symlink(indexed.path(), dir.path().join("linked")).unwrap(); + assert!(find_project(dir.path(), &[]).is_none()); + } + let child = dir.path().join("child"); + std::fs::create_dir_all(child.join(".cgr")).unwrap(); + std::fs::write(child.join(".cgr/manifest"), "{}").unwrap(); + let (store, nested) = find_project(dir.path(), &[]).unwrap(); + assert!(nested); + assert_eq!(store.dir, child.join(".cgr")); + } + #[test] fn system_reminders_are_removed() { assert_eq!(strip_system_reminders("a secret b"), "a b"); diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 33911d9..fde5ebd 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -38,7 +38,12 @@ format. Besides node and file records, the segment holds: provenance, class: extracted / resolved / synthesised / framework); * the resolver's *facts* (imports, exports, scopes, bindings, per-file reference ranges and their outcomes) so incremental sync can re-resolve without re-parsing unchanged files; -* search indexes: FSTs for name / lower-cased name / path segments, a trigram index, a BM25 token index. +* search indexes: FSTs for name / lower-cased name / path segments, a trigram index, a BM25 token + index with delta-varint posting lists, document frequencies and persisted global document length. + +The current format is version 2. Version-1 segments require `cgr index` after upgrading. +Large edge sets use an adaptive radix/comparison sorter; duplicate identities keep the original +class and synthesis-pass precedence. `.cgr/manifest` names the current base and delta; a new generation is published by writing the segment, fsyncing, then switching the manifest (`manifest.prev` is kept). Readers take a shared @@ -54,7 +59,9 @@ the rest, which may use the hierarchy produced by round 1 — over a frozen *eng export closure, per-package indexes). Per reference the strategies run in a fixed order (framework, lexical scope, import binding, receiver type, qualified name, same package, global name) and the first answer wins; ambiguity produces no edge ("better unresolved than wrong"). -Each outcome records status, strategy and confidence; unresolved references stay queryable +Full builds write outcomes directly into preallocated, disjoint per-file slots. Optional worker +caches have byte limits derived from the memory budget, and table construction uses the same +configured pool as resolution. Each outcome records status, strategy and confidence; unresolved references stay queryable (`cgr debug unresolved`). Results do not depend on thread count or scheduling. Route extraction (Express, Flask, FastAPI, Django, Gin, Spring, Axum, Next.js) adds `route` @@ -62,11 +69,13 @@ nodes with handler references. Synthesis passes add edges that only exist at run interface → implementation, Go structural interfaces, event emitter `emit` → `on` handlers, React `setState` → `render`, JSX parent → child component — each tagged with the pass that made it, capped per pass so a hub cannot explode, and computed from per-file *sites* persisted in the -snapshot so the link step needs no source. +snapshot so the link step needs no source. One shared Aho-Corasick trigger scan selects extract +hooks. Event keys link in parallel in bounded chunks; incremental non-structural edits reconnect +changed event keys and changed React/JSX files. Structural changes conservatively run all passes. ## Incremental sync (`cg-sync`) -`delta = f(base, tree)`: a delta layer never stacks on another delta. A sync scans, compares with +`delta = f(base, tree)`: a delta layer never stacks on another delta. A sync compares with the snapshot (mtime/size first, then content hash), re-extracts the changed files, and works out which base references an edit can change: @@ -78,17 +87,30 @@ which base references an edit can change: so a line inserted above a class is not a hierarchy change); * files whose synthesised edges changed. -Unchanged files whose references are affected are rewritten as *edges-only* entries. Too much -change (more than 30 % of references, 20 % of files, 5 % of files changed, 200 syncs since the -last merge, or a changed config file / engine version) falls back to a full rebuild, which is -also the merge. `cargo test -p cg-sync` generates random edit sequences over a multi-language +Unchanged files whose references are affected are rewritten as *edges-only* entries. Broad +reference/file invalidation or changed configuration/engine versions can fall back to a full +rebuild. Compaction is a separate pure data rewrite of `View(base + delta)`, preserving facts, +reference outcomes, provenance and metadata without parsing source. After at least 30 seconds +of quiet, >5% affected files (bases of at least 40 files), 64 MiB, 200 syncs or 10 idle minutes +make a delta eligible for compaction. The watcher performs maintenance without a source scan +and yields to an active writer. Readers keep the previous complete snapshot. + +`cargo test -p cg-sync` generates random edit sequences over a multi-language fixture and requires `canonical(incremental) == canonical(full rebuild)` after every step; disabling any rule makes a specific scenario fail (the rules were sabotage-checked). `cgr debug diff` runs the same comparison on a real project, `cgr bench sync --verify` does it after a random edit run. One watcher per project is elected through `watch.lock` (released by the OS when a process dies, -so another session takes over); events are debounced and fed to the same sync. +so another session takes over). A bounded queue feeds debounced paths to sync, avoiding a whole +filesystem walk for normal events. Config/directory changes, rescan flags, errors or lost/burst +events fall back to a full scan. Periodic scanning recovers missing events. The inventory is +still assembled in memory and earlier delta files may be re-extracted against the same base. + +Kernel output buffers accumulated during extraction can spill to shared immutable temporary +chunks, then be read through the same borrowed ABI views. Source text is not retained there. +Facts and graph/resolution tables still grow with repository size: `CGR_MEMORY_MB` is a tuning +budget, not a process RSS limit. ## Query and MCP (`cg-query`, `cg-mcp`) @@ -96,10 +118,18 @@ so another session takes over); events are debounced and fed to the same sync. graph, orders a *flow* section first when the query names endpoints, lists the blast radius, and renders line-numbered source windows under a character budget (13k / 18k / 24k by project size, hard cap min(1.5×, 25k)). Strings, section titles and notices are the upstream ones with the -product name substituted. `cg-mcp` is a small hand-written JSON-RPC server: a fixed 4-thread -pool, a 30 s watchdog that answers with a success-shaped note, project resolution by client roots +product name substituted. Source windows use a bounded owned text buffer and lazy line offsets; +mutable working-tree source is reread for each call rather than mmap'd or cached across calls. +`cg-mcp` is a small hand-written JSON-RPC server: a fixed 4-thread pool with a 16-job admission +queue, a 30 s deadline with cooperative cancellation, project resolution by client roots → `--path` → `projectPath`, a per-session history for optional de-duplication, and the "index -lags writes" staleness banner based on `stat` of the files in the answer. +lags writes" staleness banner based on `stat` of the files in the answer. Project LRU eviction +also releases watcher resources, and repeated old-format/corrupt-index requests share one +background rebuild. EOF drains short calls briefly, cancels remaining work and stops watchers +against a shared shutdown deadline. Filesystem blocking calls are not forcibly interruptible. + +See [performance implementation](PERFORMANCE-IMPLEMENTATION.md) for the design mapping, +validation evidence, benchmark preflight result and work that remains. ## Installer (`cg-install`) diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 07f5ac3..dac3a04 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -1,5 +1,12 @@ # Local performance comparison +**Revision scope:** these measurements describe `fcc59af`, before the subsequent performance +implementation update. New compaction, event sync, spill, compression and service changes are +documented in the bilingual [implementation record](PERFORMANCE-IMPLEMENTATION.md). A new +guarded measurement was refused at preflight; a final repeat still had only 5,889 MiB available +memory against the required 6,144 MiB (swap used 977 MiB). +The historical results below have not been relabeled as measurements of the new revision. + Measured on **2026-11-01** against the upstream CodeGraph source. The tables below use only the final measurement round, with AC power checked before and after every command. Earlier exploratory rounds, during which the machine changed from battery to AC power, are excluded. diff --git a/docs/PERFORMANCE-IMPLEMENTATION.md b/docs/PERFORMANCE-IMPLEMENTATION.md new file mode 100644 index 0000000..fecf75e --- /dev/null +++ b/docs/PERFORMANCE-IMPLEMENTATION.md @@ -0,0 +1,183 @@ +# Performance implementation / 性能优化实现记录 + +[English](#english) · [简体中文](#简体中文) + +## English + +This update implements the next engine performance work from the technical design. It follows +the design's correctness rule: an incremental graph must match a fresh full build. The previous +upstream comparison in [BENCHMARKS.md](BENCHMARKS.md) measured `fcc59af`; it does **not** measure +the changes described here. + +### Implemented changes + +| Design area | Implementation and effect on work performed | +|---|---| +| §3.3 resource control | macOS memory detection uses Mach statistics; Windows uses `GlobalMemoryStatusEx`. Explicit thread counts remain capped by the memory-derived worker limit. Project/module table construction now uses the configured pool, rather than the global Rayon pool. Index workers use Linux nice, macOS utility QoS or Windows below-normal priority. | +| §4.6 extraction spill | Accumulated kernel buffers reaching half the extraction share are written as an immutable temporary chunk in `.cgr/tmp/` and mapped back. One mapping/file backs a chunk of source files. All five ABI buffers remain byte-identical; owned facts remain resident. I/O failure stops the pipeline before the next batch and leaves its existing buffers intact. | +| §7.2 fused scanning | Hook trigger literals are compiled once into a shared Aho-Corasick automaton. Source is scanned once to select applicable hooks, including overlapping/shared literals. Hook order and output numbering remain deterministic. | +| §6.4 resolution | Full-build workers write outcomes directly into disjoint preallocated file ranges. The former per-file vectors, flattened round vectors and subsequent copies are removed. Optional field and file-context caches have budget-derived byte limits; cache eviction recomputes answers rather than discarding candidates. | +| §7.2 synthesis | Event keys can link in parallel, with budget-derived chunks of at most 256 keys and deterministic merging. Incremental event links reconnect old/new changed keys; React/JSX links reconnect changed files. Definition, export, import-target or hierarchy changes use a conservative full synthesis fallback. | +| §5.4 BM25 storage | Posting lists store node-ID differences as varints plus field masks and term frequencies. A separate document-frequency partition preserves BM25 statistics. Total document length is persisted, avoiding an all-node sum on query startup. Decoding is allocation-free and rejects truncated/overflowing entries. | +| §5.5 edge assembly | Large edge sets can use stable integer radix sorting of `(src, kind, dst, line, col)`, skipping constant bytes. Duplicate groups retain the original class/pass/metadata precedence. Small sets or insufficient scratch memory use comparison sorting. Both dense and sparse CSR builders use the same ordering. | +| §5.7 compaction | Compaction repacks `View(base + delta)` into a new base without source access, extraction or resolution. It preserves live facts, reference outcomes, provenance, positions and metadata, and rebuilds dense IDs/search indexes. Readers retain their previous complete generation. | +| §5.7 maintenance | Saves publish a delta first. After at least 30 seconds without a published change, maintenance can compact at >5% affected files (bases of at least 40 files), 64 MiB, 200 syncs, or 10 minutes of idle time. The watcher checks maintenance every 30 seconds and yields immediately to an existing writer. | +| §9.1–9.3 event sync | Ordinary file events update the file inventory by path without walking/statting every source file. Configuration/directory events, >500 dirty paths, watcher errors, rescan flags and queue overflow fall back to a full scan. A 1,024-event queue and 500-path set are bounded; sustained events flush within five seconds. Periodic scans recover missing events. | +| §4.6 / §8.3 source windows | `SourceView` keeps one bounded owned UTF-8 text buffer and lazy newline offsets instead of allocating a string per line. Files are reread for each call. Mutable editor files are not mapped; immutable snapshots/spills are. CRLF and unterminated final lines retain the previous line semantics. | +| §10 service execution | Four query workers have a 16-job admission queue. Saturation returns a readable busy result while control requests remain responsive. Thirty-second deadlines include queue time; cancellation notifications and hot-path checkpoints stop cooperative work. Queued cancelled work does not open a project. One background rebuild is admitted at a time; LRU eviction also releases its watcher resource. | +| §10.5 / §11.2 lifecycle | EOF gives final short calls 200 ms, then requests cancellation and stops watchers against one shared 600 ms deadline. Hook discovery skips an unindexed filesystem/home root, avoids directory symlinks, and caps depth/directories/entries/results. Discovery has a 200 ms deadline and the compute path a cooperative 450 ms deadline. The smoke helper preserves MCP stdin on macOS Bash 3.2 and cleans up its timer. | + +### Compatibility + +The snapshot **format is now version 2** because token posting records changed. Rebuild an old +index with `cgr index /path/to/project`. A format-1 segment is rejected by the format guard; +it is not reinterpreted as compressed data. Source files and agent configuration are unaffected. +The MCP background rebuild path coalesces repeated requests for an unreadable/old-format index. +The engine revision is also 2: package/module representatives are ordered by source path rather +than appended delta IDs. Package and declaration imports persist their file dependencies, so +adding/removing a package's representative invalidates its consumers correctly. Compaction +also handles advisory dependencies that still name the last removed base file. + +### Validation and measurement status + +Local regression checks cover byte-identical 1-thread/4-thread snapshots, resident/spilled ABI +buffers and segments, radix/comparison ordering and CSR equivalence, malformed posting lists, +native compaction without a source root and after source removal, old-reader survival, future +incremental sync, exact watcher paths, directory/config events, dropped-event recovery, queue +saturation, queued/running cancellation, rebuild admission and project LRU cleanup. Random and +targeted multi-language edit sequences compare canonical graphs after every edit. +The final randomized check uses the CI depth of six sequences of 100 edits, including package +membership changes; targeted cases explicitly add and remove a lexically earlier Go file. + +The final macOS run passed **137 tests, with one ignored**, plus the locked release build, +Clippy with `-D warnings`, formatting and the complete smoke script under macOS Bash 3.2. +Four independent real-project copies passed partition checksum verification and canonical +full-build equivalence after indexing, adding a function, syncing and restoring the source: + +| Project | Files | Nodes | Edges | Initial / edited / restored graph equals full build | +|---|---:|---:|---:|---| +| Express | 142 | 866 | 2,527 | Pass / Pass / Pass | +| Gin | 99 | 2,436 | 7,921 | Pass / Pass / Pass | +| Tokio | 814 | 14,460 | 36,511 | Pass / Pass / Pass | +| Django | 2,984 | 62,660 | 185,506 | Pass / Pass / Pass | + +For all four projects, file/node/edge counts, sorted non-containment edge dumps and unresolved +reference dumps also match the retained pre-update binary. This checks regression against our +own prior engine; it does not establish upstream oracle parity. A separate 64-file fixture +verified automatic watcher delta publication, MCP refresh to the added function, full-build +equivalence and exit within a two-second functional deadline after stdin closed. An actual +format-1 index was rejected by the new reader and rebuilt into a verified format-2 index. +These are correctness checks, not performance samples. Code revision `f82506e` also passed +all nine [hosted CI jobs](https://github.com/NianJiuZst/CodeGrap-Rust/actions/runs/41286327676): +Linux/macOS/Windows unit/integration tests and end-to-end smoke, lint, kernel reproducibility +and a static musl build. Windows passed 137 tests with one ignored. Local/hosted checks use +Rust 1.99; the retained before/after release binaries use the same local Rust 1.94.1 toolchain. + +The guarded performance run was **refused before starting**. The initial preflight reported +4,233 MiB available memory, 1,559 MiB swap and 40,800 MiB free disk. A repeat after the final +release build at that stage reported 5,889 MiB available memory, 977 MiB swap and 39,027 MiB free disk; +available memory still fell below the required 6,144 MiB (swap must be below 1,024 MiB and free +disk at least 20,480 MiB). +Protection thresholds were not relaxed. No new speedup percentage, latency target or large-repo +RSS guarantee is claimed for this update. The earlier **6.40×** result remains historical. + +To collect a new baseline, use separate corpus/output directories for old and new binaries, +the same release profile and thread/memory settings, and the protected wrapper. For example: + +```sh +scripts/bench-local.sh --tier restricted --cgr /path/to/cgr-before \ + --repos-dir /path/to/corpus-before --kinds index,sync,explore \ + --runs 3 --edits 200 --out /path/to/results-before --history /path/to/history-before \ + express gin tokio django +scripts/bench-local.sh --tier restricted --cgr /path/to/cgr-after \ + --repos-dir /path/to/corpus-after --kinds index,sync,explore \ + --runs 3 --edits 200 --out /path/to/results-after --history /path/to/history-after \ + express gin tokio django +``` + +Compare graph coverage and returned content alongside medians, P50/P95/P99, segment sizes and +process-tree RSS. Use the standard tier as a separate series. Keep raw one-off results outside +the source checkout. + +### Remaining boundaries + +- `CGR_MEMORY_MB` is a tuning budget. Kernel spill and cache limits do not impose an OS RSS + limit; facts, graph builders and shared resolution tables still scale with repository size. +- Path-driven sync avoids filesystem-wide probing, but still assembles an in-memory inventory. + Its delta is computed against the base, so earlier files in the current delta can be extracted + again. Hierarchy changes still conservatively widen round-two resolution. +- Changed keys reduce linking work for implemented event/render rules. A persistent inverse + index for every synthesis pass and finer definition/hierarchy invalidation remain future work. + Missing callback/registry/Gin/Spring passes and framework coverage parity are separate work. +- Compaction uses the writer lock. Background compaction with delta rebasing is the design's M6 + work; it is not part of this update. +- Deadlines are cooperative. Blocking filesystem operations cannot be forcibly interrupted by + a safe Rust thread cancellation mechanism. Source text has no cross-request LRU, preserving + the requirement to read current disk contents on every call. + +## 简体中文 + +本轮依据技术设计落地引擎性能优化,并继续以“增量结果等于全量重建”为正确性约束。 +[历史 benchmark](BENCHMARKS.md) 测量的是 `fcc59af`,**不包含本轮新增优化**。 + +### 已落地的优化 + +| 设计章节 | 实现与减少的工作量 | +|---|---| +| §3.3 资源控制 | macOS 直接读取 Mach 内存统计,Windows 使用原生内存 API;显式线程数也受内存预算约束。模块/项目表进入配置的线程池。索引线程分别使用 Linux nice、macOS utility QoS、Windows 低于正常优先级。 | +| §4.6 提取降级 | 累计 kernel 缓冲达到提取份额一半后,按批写入 `.cgr/tmp/` 并通过不可变映射读取,释放堆缓冲。一批文件共享一个映射,五类 ABI 缓冲保持逐字节一致;facts 仍驻留内存。写入失败会在下一批开始前终止。 | +| §7.2 融合扫描 | 将所有 hook 的触发字面量一次编译为共享 Aho-Corasick 自动机,一次扫描选择需运行的 hook,保留重叠匹配、执行顺序与节点编号。 | +| §6.4 解析结果槽 | 全量解析直接写入互不重叠的预分配结果区,消除每文件结果、每轮展平数组及再次复制。字段缓存和文件上下文缓存受预算约束;淘汰只导致重算,不截断候选集。 | +| §7.2 合成连接 | 事件键按预算分批并行连接,每批最多 256 键,按固定顺序合并。普通增量只重连变化事件键和变化文件的 React/JSX 站点;定义、导出、导入目标、继承结构变化时保守回退。 | +| §5.4 BM25 存储 | 节点 ID 差值 + varint 压缩,保留字段掩码和词频;独立保存文档频率。总文档长度持久化,查询启动不再遍历所有节点求和。解码不分配数组,处理截断和溢出记录。 | +| §5.5 边组装 | 大规模边可按整数唯一键使用稳定基数排序,跳过不变化的字节;同键边继续遵守原有来源、pass 和元数据优先级。小数据或临时空间不足时使用比较排序。 | +| §5.7 纯数据合并 | 直接从 base + delta 重写基础段,保留存活 facts、引用结果、来源、位置与元数据,重建密集 ID 和搜索索引。合并不读取源码,不调用提取器或解析器,旧读者继续使用原快照。 | +| §5.7 静默维护 | 保存时先发布增量;至少静默 30 秒后,按受影响文件比例、64 MiB、200 次同步或空闲 10 分钟触发合并。文件比例阈值为 >5%,适用于基础段至少 40 文件的项目。维护遇到已有写者会立即让出。 | +| §9 事件同步 | 普通事件按具体路径更新文件清单;目录/配置变化、事件丢失、重扫标记或溢出时完整扫描。事件队列上限 1,024,脏路径上限 500,持续事件最多积累五秒;定期扫描兜底漏事件。 | +| §4.6 / §8.3 源码窗口 | 单份有界源码文本 + 延迟生成的行偏移,替代逐行字符串分配。每次调用重读磁盘;可被编辑器原地修改的源码使用自有缓冲,不采用 mmap,保留 CRLF 和末行语义。 | +| §10 MCP 执行 | 四个查询线程 + 16 个排队任务;满载立即返回可读的繁忙说明,控制请求继续响应。排队计入 30 秒期限;取消通知和检查点让运行任务协作退出。后台重建合并为单个任务,项目 LRU 淘汰也释放监听资源。 | +| §10.5 / §11.2 生命周期 | EOF 给短请求 200 ms 收尾,然后取消剩余查询,所有监听器共享 600 ms 停止期限。提示钩子限制发现范围、数量和时间,跳过未索引根目录及目录符号链接。Smoke 脚本兼容 macOS Bash 3.2,保留 MCP stdin 并清理计时器。 | + +### 兼容与验证 + +倒排记录变更使快照格式升级为 **version 2**。升级后运行 `cgr index /path/to/project` 重建旧索引; +格式检查会拒绝 version 1,避免按新结构误读。源码和 Agent 配置不受格式升级影响。 +引擎修订也升级为 2:包/模块代表文件按源码路径排序,不依赖 delta 追加的 ID;包与声明导入持久化 +对应文件依赖,增删代表文件时会正确使消费者失效。合并也能处理仍指向最后一个已删除基础文件的辅助依赖。 + +最终 macOS 回归为 **137 项通过、1 项忽略**,覆盖线程确定性、临时映射前后字节一致、压缩边界、排序与 +双向 CSR、无源码合并、旧读者和后续同步、事件路径、目录/配置变化、漏事件恢复、排队与执行取消、 +重建去重及 LRU 资源释放。随机和定向编辑序列在每一步对照全量图谱。锁定依赖的 release 编译、 +Clippy `-D warnings`、格式检查和 macOS Bash 3.2 下的完整 smoke 检查均已通过。 +最终随机检查采用 CI 的 **6 组 × 100 步编辑**,定向用例还覆盖增删排序靠前的 Go 包成员。 + +Express(142 文件)、Gin(99)、Tokio(814)、Django(2,984)的独立副本均通过分区校验; +初始索引、添加函数并同步、恢复源码并同步这三个状态,都与全量重建的 canonical 图谱一致。 +四个项目的文件/节点/边数量、排序后的非 contains 边输出、未解析引用输出,也与保留的优化前二进制一致。 +这是对自身旧引擎的回归检查,不代表与上游 oracle 的覆盖度已完全对齐。 + +额外的 64 文件项目验证了监听器自动发布 delta、MCP 读到新增函数、结果等于全量,以及关闭输入后在 +2 秒功能期限内退出。真实的旧格式索引已验证会被新读者拒绝,并能通过 `index` 重建为校验通过的格式 2。 +这些是正确性检查,不作为性能样本;macOS 结果也不等于 Linux/Windows 运行验证或远程 CI 已通过。 + +代码版本 `f82506e` 的[最终远程 CI](https://github.com/NianJiuZst/CodeGrap-Rust/actions/runs/41286327676) +已全部通过,共 9 个任务:Linux/macOS/Windows 单元/集成测试与端到端 smoke、格式和 Clippy、 +内核来源复现及静态 musl 构建。Windows 同样为 137 通过、1 忽略。最终本机/远程检查使用 Rust 1.99; +保留的新旧 release 二进制均使用本机 Rust 1.94.1 构建。 + +正式性能复测在启动前被保护脚本拒绝。首次读数为可用内存 **4,233 MiB**、Swap **1,559 MiB**、 +空闲磁盘 **40,800 MiB**;当阶段 release 编译后重试,读数为 **5,889 MiB / 977 MiB / 39,027 MiB**。 +重试仍未达到可用内存至少 **6,144 MiB** 的要求;Swap 必须低于 **1,024 MiB**,空闲磁盘至少 **20,480 MiB**。 +保护阈值保持原值。因此本轮没有新增可发布的加速百分比、延迟达标结论或大仓库 RSS 保证;历史 +**6.40×** 仍只属于原被测版本。资源条件满足后,按上方命令在独立目录逐一测旧、新二进制,保留相同的 +源码、release 配置与线程/内存档位,同时检查覆盖度和返回内容,再比较中位数、分位数、段大小与 RSS。 + +### 仍需继续推进的边界 + +- 内存预算仍不是操作系统 RSS 硬限制;facts、图谱构建数组和共享解析表仍随项目规模增长。 +- 路径事件避免全仓库文件系统探测,但仍构造内存清单;当前 delta 相对 base 计算,之前变化过的文件 + 可能再次提取。继承关系变化仍可能扩大第二轮解析范围。 +- 已实现规则的事件/渲染键可以局部重连;全部 pass 的持久化反向键索引、更细的定义/继承失效分析,以及 + 缺失的 callback、registry、Gin、Spring 合成规则和框架覆盖度对齐仍需后续工作。 +- 当前合并仍持有写锁;后台合并与 delta 变基属于设计 M6。 +- 取消是协作式的,不能强制中断阻塞文件系统调用。源码不使用跨请求 LRU,以保证每次调用读取当前内容。 diff --git a/scripts/smoke.sh b/scripts/smoke.sh index 50a82f9..0ee833d 100755 --- a/scripts/smoke.sh +++ b/scripts/smoke.sh @@ -21,9 +21,16 @@ expect() { # expect # run a command with a time limit, without relying on GNU `timeout` (absent on macOS runners) run_limited() { # run_limited local limit="$1"; shift - "$@" & + # Background jobs otherwise inherit /dev/null as stdin in a non-interactive shell. + "$@" <&0 & local pid=$! - ( sleep "$limit"; kill -9 "$pid" 2>/dev/null ) & + ( + sleep "$limit" & + local sleeper=$! + trap 'kill "$sleeper" 2>/dev/null || true; exit 0' TERM INT + wait "$sleeper" + kill -9 "$pid" 2>/dev/null + ) /dev/null 2>&1 & local killer=$! local rc=0 wait "$pid" || rc=$? @@ -125,7 +132,7 @@ out="$(printf '{"prompt":"how does greet call format?","cwd":"%s"}' "$DIR" | "$B echo "(hook output: ${#out} bytes)" # broken input and a directory without an index must not fail either printf 'not json' | "$BIN" prompt-hook >/dev/null || fail "prompt-hook must exit 0 on garbage" -( cd / && printf '{"prompt":"x","cwd":"/"}' | "$BIN" prompt-hook >/dev/null ) || fail "prompt-hook must exit 0 without an index" +( cd / && printf '{"prompt":"x","cwd":"/"}' | run_limited 3 "$BIN" prompt-hook >/dev/null ) || fail "prompt-hook must exit 0 without an index" echo "== export" if "$BIN" export --sqlite "$DIR/graph.sqlite" >/dev/null 2>&1; then