diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index ffde0e6..65dfe23 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,3 +1,5 @@ +# Pull request + ## What changed @@ -15,4 +17,4 @@ - [ ] New user-facing strings are in **both** `en-US.json` and `zh-CN.json` - [ ] New errors carry a stable code and a user-facing hint - [ ] No search logic was added to a front end — it belongs in `search-daemon` -- [ ] Nothing on the search path reaches the network \ No newline at end of file +- [ ] Nothing on the search path reaches the network diff --git a/.markdownlint.json b/.markdownlint.json new file mode 100644 index 0000000..ea87d5f --- /dev/null +++ b/.markdownlint.json @@ -0,0 +1,8 @@ +{ + "default": true, + "MD013": false, + "MD033": false, + "MD024": { + "siblings_only": true + } +} diff --git a/CHANGELOG.md b/CHANGELOG.md index f6f1a9f..970d152 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,142 +1,55 @@ # Changelog -All notable changes to this project are documented here. The format follows -[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project -adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +User-visible release history. v1.0.0 is a preparation target, not a published +version. Earlier detailed engineering notes remain in the documentation archive. -## [0.2.0] - 2026-10-01 +## v1.0.0 -A v0.2 engineering pass over the v0.1 MVP. Nothing that worked in 0.1 was -removed: the CLI, the MCP server, the desktop shell, the bilingual UI, the Local -Search DSL, the ranking crate and the `SearchService` API all keep their -behaviour. +**Unreleased draft.** This section consolidates the intended first stable release +scope, including capabilities already shipped in v0.2.0; it does not imply that +they were all introduced after that release. ### Added -- **Per-volume journal state** (`windows_files::journal`). `JournalRegistry` - keys every NTFS volume by a stable `VolumeId` — a volume GUID, never a drive - letter — and tracks its own journal id, cursor, first/next USN and sync - status. A journal that is recreated, that rolls over, or that moves backwards - marks **only that volume** stale and schedules a rebuild. -- **Event-driven index workers** (`windows_files::worker`). One `VolumeWorker` - per volume tails its own USN journal with a `Timeout`/`BytesToWaitFor` read, - so the thread blocks in the kernel instead of polling. An `IndexSupervisor` - owns the threads, tracks per-volume lifecycle (`starting`, `running`, - `backing-off`, `failed`, `stopped`) and backs off exponentially on failure. -- **A privileged index service** (`lce-index-service.exe`) plus - `index-protocol` and `index-client`. The desktop, CLI and MCP server stay - unprivileged; the service owns the MFT and the journals and answers typed, - versioned requests over a Named Pipe whose DACL is built from an explicit SDDL - string (SYSTEM, Administrators and the installing user — never Everyone or - Anonymous). `FILE_FLAG_FIRST_PIPE_INSTANCE` defeats name squatting and - `PIPE_REJECT_REMOTE_CLIENTS` refuses remote callers at the pipe itself. - `install`, `uninstall`, `start`, `stop`, `status` and `--console` are - supported. -- **Search accelerators** (`windows_files::accelerators`): an extension index, a - prefix table and a trigram index over normalised **full paths**, with - high-frequency trigrams dropped and a compaction policy driven by mutation - drift. -- **A query planner** (`windows_files::planner`) that picks - `extension` / `prefix` / `trigram` / `intersection` / `linear` per query and - reports what it did. `localsearch --explain` prints the plan, the candidate - counts and the elapsed time. -- **A unified mutation model** (`IndexMutation`) so USN parsing and index - application are separate steps, applied as one batch under a short write lock. -- **`prefix:` in the DSL**: the one predicate a prefix table can answer exactly. -- **Differential tests**: accelerated search must equal the pre-v0.2 linear scan - over a randomised corpus, after mutations, and when the candidate cap - truncates. This is what caught two real bugs during development (trigrams - indexed names but not paths; a live insert could resurrect a dropped trigram - and silently lose matches). +- User guides for query planning, indexing, providers, security and development. +- A release-oriented capability map that distinguishes Stable, Experimental and Future. +- Reproducible million-record performance evidence with complete query results and limitations. ### Changed -- The persisted index schema is now **version 3** and carries the per-volume - journal registry plus the accelerators. A v0.1 or v0.2 cache is discarded and - rebuilt rather than migrated — guessing at an older cursor is how a stale - index happens. -- MessagePack is now written with **named** fields. Positional encoding plus - `skip_serializing_if` shifted every field after a skipped `Option` and - corrupted the read. -- The file provider holds its store and accelerators behind one `RwLock`, and - applies journal batches under that lock only after all disk I/O and parsing is - done. Cache persistence happens outside the lock. -- Process detail resolution is chunked across up to eight threads. +- Installation and usage documentation now leads with local Windows discovery and explainable queries. +- Development history and raw measurement captures are preserved under `docs/archive`. +- Compatibility statements describe the implemented CLI/query workflows without promising universal desktop or service deployment support. -### Fixed +### Performance -- A single `Option` could not describe a machine with more than - one NTFS volume; one disk's cursor could overwrite another's. +- Existing selective planning retains useful indexed conditions when other terms require verification. +- Existing early posting pruning reduces peak construction working set in the recorded million-record probe; retained accelerator bytes are unchanged. +- Common-query fallback and workload/hardware limits remain explicit; this documentation revision introduces no runtime performance change. ### Security -- Threat model documented for the privileged boundary, the pipe ACL, malformed - clients, protocol downgrade, stale indexes and service crashes. It states - plainly that MFT enumeration can reveal file *names* the calling user could - not otherwise list, and why that is inherent to the technique. -- The protocol has no `Execute`, `Shell` or `Command` message, and a test - enumerates the variants so adding one is a visible diff. +- Clarified offline metadata indexing, local cache sensitivity and the optional service's MFT filename-visibility boundary. +- Corrected README license identification to MPL-2.0, matching LICENSE and Cargo metadata. +- No changes to permissions, protocols, persistence, application logic or tests. -### Known limitations +## v0.2.0 — 2026-10-13 -- The accelerators roughly **double** the index footprint (measured: +60% at - 100 000 entries), which pushes a one million entry index past the project's - 100 MB goal. The prefix table is the largest single contributor. -- Common-only queries (`readme`, `report`) and fuzzy queries still fall back to - a linear scan, because their trigrams are too common to narrow anything. - A `path:` filter also forces a scan. These are unchanged from v0.1 in cost. -- A `Changes` subscription reports volume-level journal advances, not a - per-file event stream. -- The MFT backend still cannot report file sizes. +Published Windows release with extension/prefix/trigram accelerators, +cost-aware candidate planning, `prefix:` queries, per-volume MFT/USN state, +optional privileged indexing service and differential/index consistency tests. +Long-prefix and combined-filter correctness is verified against linear search. +Peak construction memory is lower in the paired probe; broad queries can still +scan. Existing schema-3 caches and explicit service permissions remain the +documented boundaries. See [benchmarks](docs/BENCHMARKS.md) and +[indexing](docs/INDEXING.md) for limitations. -[0.2.0]: https://github.com/styayur/local-context-engine/releases/tag/v0.2.0 -## [0.1.0] - 2026-10-01 +## v0.1.0 — 2026-10-01 -The first runnable MVP. Everything below works today; nothing is a placeholder. +Initial Windows preview: local query DSL, bilingual rule-based compilation, +deterministic ranking, file/application/process/service/window providers, +CLI, stdio MCP and desktop shell. Optional usage history and explicit process +termination confirmation were included. -### Added - -- **Rust workspace** with hard module boundaries: `search-core`, `query-dsl`, - `ranking`, five `windows-*` providers, `search-daemon`, `search-cli` and - `search-mcp`. -- **Unified entity model** (`LocalEntity`) with six variants — file, directory, - process, application, service, window — and a single `SearchResult` shape. -- **Local Search DSL**: `type:`, `ext:`, `path:`, `name:`, `drive:`, - `modified:`, `created:`, `size:`, `state:`, `pid:`, `user:`, `visible:`, - `sort:` and `limit:`, with graceful fallback to plain text. -- **Rule-based natural language compiler** for English and Simplified Chinese. - `最近的 PDF` and `recent PDFs` compile to the same query, as do - `正在运行的 python` and `running python processes`. -- **Deterministic heuristic ranking** with named, configurable weights, fuzzy - subsequence matching and a bounded local usage history. -- **Filesystem index** with two interchangeable backends: a directory scan that - needs no privileges and records sizes, and an NTFS MFT + USN change journal - backend that needs elevation but enumerates a volume in one pass and updates - incrementally. -- **Compact index storage**: a string arena plus 48-byte records, so a one - million entry index costs roughly 70 MB rather than several hundred. -- **Atomically persisted index cache** in `%LOCALAPPDATA%\LocalContextEngine`. -- **Live providers** for processes (Toolhelp32 + PSAPI, resolved in parallel), - services (`EnumServicesStatusEx`), top-level windows (`EnumWindows`) and - applications (App Paths, Start Menu, `PATH`, WindowsApps). -- **CLI** `localsearch.exe` with human and `--json` output, `--explain`, - `--index-status`, `--rebuild-index`, `--update-index`, `--providers`, - `--list-usage` and `--reset-usage`. -- **MCP server** `localsearch-mcp.exe` over stdio with eleven tools, including - `search_system`, the five scoped search tools, `compile_query`, - `index_status`, `dsl_reference`, `record_selection` and `terminate_process`. -- **Tauri 2 desktop shell** with a React + TypeScript + Tailwind interface: - keyboard-first navigation, type filters, match highlighting, per-entity - context menus, a confirmation dialog for the one destructive action, and - full zh-CN/en-US localisation that switches without a restart. -- **290 tests** across the workspace plus workspace-level integration - tests, benchmarks, and the `docs/` set below. - -### Security - -- File search is read-only. -- Process termination is gated behind an explicit confirmation on both the UI - and the Rust side. -- No network access anywhere in the codebase. - -[0.1.0]: https://github.com/styayur/local-context-engine/releases/tag/v0.1.0 \ No newline at end of file +Detailed historical notes are retained in the +[original changelog](docs/archive/development-notes/changelog-before-v1.md). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fe32a12..37d2709 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,12 +23,12 @@ cargo test # Desktop shell cd apps/desktop -npm install +npm ci npm run tauri:dev ``` You need the MSVC toolchain (Visual Studio Build Tools with "Desktop development -with C++") plus Node 20 or newer. The desktop shell also needs WebView2, which +with C++") plus Node 22 (the CI version). The desktop shell also needs WebView2, which ships with Windows 11 and is installed by the NSIS bundle otherwise. ## Before you open a pull request @@ -74,4 +74,4 @@ belongs in the bundle; nothing is hard-coded in a component. ## Commit messages Conventional commits, please: `feat:`, `fix:`, `docs:`, `refactor:`, `test:`, -`chore:`. The release workflow generates its notes from them. \ No newline at end of file +`chore:`. The release workflow generates its notes from them. diff --git a/README.md b/README.md index 9130dd4..2edcba4 100644 --- a/README.md +++ b/README.md @@ -1,359 +1,135 @@ -
- 简体中文 | English -
- - - - - # Local Context Engine -Offline-first search over your own machine. - -Natural language in → deterministic query → Rust search core → results. - -[Download](https://github.com/styayur/local-context-engine/releases/latest) · [Architecture](docs/ARCHITECTURE.md) · [DSL](docs/DSL.md) · [MCP](docs/MCP.md) · [Benchmarks](docs/BENCHMARKS.md) +**Local-first context discovery engine for Windows.** Find files, applications, +processes, services and windows through one explainable local query. [![CI](https://github.com/styayur/local-context-engine/actions/workflows/ci.yml/badge.svg)](https://github.com/styayur/local-context-engine/actions/workflows/ci.yml) -[![release](https://img.shields.io/github/v/release/styayur/local-context-engine?label=release&color=orange)](https://github.com/styayur/local-context-engine/releases) -[![license: MPL-2.0](https://img.shields.io/badge/license-MPL--2.0-blue)](LICENSE) - +[![License: MPL-2.0](https://img.shields.io/badge/license-MPL--2.0-blue)](LICENSE) +[简体中文](README.zh-CN.md) · [Download v0.2.0](https://github.com/styayur/local-context-engine/releases/tag/v0.2.0) · [Documentation](docs/README.md) ---- +Current published release: **v0.2.0**. The v1.0.0 notes are an **unreleased +candidate**, not an available binary or a new compatibility guarantee. -What is this? --------------- +## Why Local Context Engine -Local Context Engine is a local search tool for Windows: one input box that -finds your **files and folders**, **applications**, **running processes**, -**services** and **windows** at the same time, and opens them. +A filename, an installed application and a running process can refer to the same +work. LCE puts those local sources behind one query interface, while keeping the +query, candidate selection and ranking inspectable. File search indexes names, +paths and metadata rather than file contents. Normal search needs no account, +model service or network connection. -Type `vscode` and you get the application, the running process and any matching -file in one list. Type `正在运行的 python` or `running python processes` and you -get the processes. Type `最近的 pdf` or `recent PDFs` and you get today's PDFs, -newest first. +## Features and maturity -It is built from three ideas: +| Status | Capability | Boundary | +|---|---|---| +| Stable | Query DSL and English/Chinese rule-based query compilation | Explicit filters; no model-generated retrieval | +| Stable | Query Planner with extension, prefix and full-path trigram accelerators | Exact verification remains authoritative; broad queries can scan | +| Stable | File/directory, app, process, service and window providers | Availability and metadata depend on Windows permissions | +| Stable | CLI, local stdio MCP, ranking explanations and index health | Same query **and snapshot/settings/usage state** give deterministic ordering | +| Experimental | Optional privileged NTFS MFT/USN service and desktop acceptance across machines | Implemented and tested, but require deployment-specific validation; see security and roadmap | +| Future | Full-text, semantic retrieval and model-backed query compilation | Not installed, enabled or run by default; no current implementation promise | -1. **Search must be deterministic.** The same query returns the same list in the - same order, every time. No model, no embeddings, no vector database sits on - the hot path. -2. **Understanding is a compiler problem.** Natural language is *compiled* into - a structured query — `最近的 pdf` becomes `type:file ext:pdf - sort:modified-desc` — and then a Rust engine answers it. -3. **Offline is the default, not a mode.** There is no account, no telemetry and - no network code in the repository. +Stable means implemented workflows with regression coverage. It does not mean +all hardware, filesystems or desktop environments have been certified. -Why it exists -------------- - -Desktop search on Windows is either a full-text indexer that rebuilds for -minutes, or a launcher that cannot see anything except shortcuts. Meanwhile the -"AI" answer to search is usually "embed everything and hope" — which is slow, -non-deterministic, unauditable, and quietly sends your file names somewhere. - -This project takes the opposite position: - -> AI, if it appears at all, is a **query compiler**. Retrieval is a -> deterministic local operation. - -How it differs from a traditional desktop search ------------------------------------------------- - -| | Traditional indexer | Launcher | Local Context Engine | -|-----------------------------------|---------------------|------------|-----------------------------------------------| -| Time to first useful result | minutes of indexing | instant | instant for live data, one scan for files | -| Indexes file **contents** | yes | no | no — names, paths and metadata only | -| Sees processes/services/windows | rarely | no | yes, all four | -| Query language | one text box | one text box | text box **and** a documented DSL **and** MCP | -| Deterministic ranking | usually | usually | always, with `--explain` to show the arithmetic | -| AI on the search path | sometimes | sometimes | never | -| Network required | often | no | never | -| Memory footprint | hundreds of MB | tens of MB | ~70 MB for a one million entry index | - -Architecture ------------- +## Architecture ```text - User / AI - │ - ▼ - Query Compiler ← the only place an AI could ever plug in - │ - ▼ - SearchQuery - │ - ▼ - Search Core - │ - ┌───────────┼───────────┐ - ▼ ▼ ▼ - Files Processes Apps - │ │ │ - MFT/USN Win32 Registry +User + ↓ +CLI / Desktop / MCP + ↓ +Search Core (via SearchService) + ↓ +Providers ── live Windows snapshots + ↓ file-provider branch +Query Planner → Accelerators → candidate verification + ↓ +Ranking + deterministic ordering + explanations ``` -The workspace is split so that "where does this belong?" never needs a meeting: +The planner lives inside the file provider. Accelerators narrow candidates; +providers own operating-system access. The interfaces, data flow and failure +boundaries are described in [Architecture](docs/ARCHITECTURE.md). -```text -local-context-engine/ -├─ apps/desktop/ Tauri 2 + React + TypeScript shell -├─ crates/ -│ ├─ search-core/ entity/query/result model, engine, fuzzy matcher -│ ├─ query-dsl/ the DSL parser and the rule-based NL compiler -│ ├─ ranking/ heuristic scoring + bounded local usage memory -│ ├─ windows-files/ directory-scan and NTFS MFT + USN index backends -│ ├─ windows-processes/ live process snapshots -│ ├─ windows-apps/ App Paths, Start Menu, PATH, WindowsApps -│ ├─ windows-services/ read-only service snapshots -│ ├─ windows-windows/ top-level window snapshots -│ ├─ search-daemon/ the composition root every front end shares -│ ├─ search-cli/ localsearch.exe -│ ├─ search-mcp/ localsearch-mcp.exe (MCP over stdio) -│ └─ benchmarks/ Criterion benchmarks for 10k / 100k / 1M entries -├─ docs/ -└─ tests/ workspace-level integration tests -``` +## Performance -Full detail — including why `AI != Search Engine`, the provider boundary and the -memory arithmetic — is in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). +A paired synthetic **one-million-record** probe measured the following on +Windows with Rust 1.98.1. These are local observations, not latency guarantees. -Local Search DSL ----------------- +| Measurement | Before planner optimization | After | +|---|---:|---:| +| Accelerator construction | 11.93 s | 8.91 s | +| Construction process peak working set | 1005.25 MB | 258.98 MB | +| `ext:pdf report`, mean / P95 | 89.36 / 99.74 ms | 20.95 / 23.14 ms | +| Common `report`, mean / P95 | 8.34 / 9.61 ms | 8.47 / 9.87 ms | +| Estimated retained accelerator bytes | 75.80 MB | 75.80 MB | -```text -chrome -type:file rust -type:file ext:rs -type:file ext:pdf modified:<24h -type:process python -type:process name:node -type:app vscode -type:service state:running -type:window github -path:projects ext:toml -drive:C ext:exe sort:size-desc -path:"My Projects" ext:toml -size:>10mb sort:size-desc limit:20 -``` - -Keys: `type`, `ext`, `path`, `name`, `drive`, `modified`, `created`, `size`, -`state`, `pid`, `user`, `visible`, `sort`, `limit`. Parsing never fails — an -unrecognised key stays free text, so `localsearch "C:\Users\me"` is a search -rather than a syntax error. - -The full reference, including the time and size operators, is in -[docs/DSL.md](docs/DSL.md). - -MCP ---- - -`localsearch-mcp.exe` exposes the same core to any MCP client over stdio: - -```jsonc -{ - "mcpServers": { - "localsearch": { "command": "C:\\path\\to\\localsearch-mcp.exe" } - } -} -``` +The memory reduction concerns **peak construction working set**, not a reduction +in retained index size. All eight full ordered comparisons matched the linear +reference. See [Benchmarks](docs/BENCHMARKS.md) for every query, hardware provenance, +sample counts, candidate limits, raw results and limitations. -| tool | what it does | -|---------------------|--------------------------------------------------| -| `search_system` | search every domain at once | -| `search_files` | files and folders | -| `search_processes` | running processes | -| `search_apps` | installed applications | -| `search_services` | Windows services | -| `search_windows` | open top-level windows | -| `compile_query` | show how an input compiles, without searching | -| `index_status` | backend, entry counts, memory, provider health | -| `dsl_reference` | the key table and examples | -| `record_selection` | boost a result next time | -| `terminate_process` | destructive; requires `confirm: true` | +## Installation -See [docs/MCP.md](docs/MCP.md). +Download the Windows x64 portable archive or desktop installer from +[v0.2.0](https://github.com/styayur/local-context-engine/releases/tag/v0.2.0). +Verify the download against `SHA256SUMS.txt`. Extract the portable archive and run +`localsearch.exe`; add its directory to `PATH` if you want the commands below. +The desktop requires WebView2. A successful build is not a substitute for a +working desktop smoke test on your Windows installation. -CLI ---- +To build the CLI and MCP server from source: -```bash -localsearch chrome -localsearch "type:file ext:rs" -localsearch "正在运行的 python" -localsearch --type process node -localsearch --json vscode -localsearch --explain "昨天修改的 rust 文件" -localsearch --index-status -localsearch --rebuild-index -localsearch --update-index -localsearch --providers -localsearch --list-usage -localsearch --reset-usage -``` - -Exit codes are part of the contract: `0` results found, `1` valid query with no -matches, `2` the command could not be completed. `--json` writes a single JSON -document to stdout; diagnostics always go to stderr. - -Desktop -------- - -```bash -cd apps/desktop -npm install -npm run tauri:dev # development, hot reload -npm run tauri:build # NSIS installer in src-tauri/target/release/bundle -``` - -* Autofocused search box, keyboard-first: `↑`/`↓` navigate, `Enter` opens, - `Shift+Enter` opens the context menu, `Tab` cycles the type filters, `Esc` - clears or closes, `Ctrl+,` opens settings. -* Type filter chips with live counts, match highlighting, per-entity context - menus, and a status bar showing the compile result and the elapsed time. -* **Simplified Chinese and English**, switchable without a restart. Every - string — errors, empty states, tooltips, context menus — lives in - `src/i18n/{zh-CN,en-US}.json`; nothing is hard-coded in a component. -* Settings for language, theme, result limit, fuzzy matching, usage ranking, - indexed drives, index backend, the global shortcut, and a full index panel. -* The only destructive action, ending a process, is behind a confirmation - dialog whose default button is Cancel — and the Rust side refuses the call - without an explicit `confirmed: true`. - -Build ------ - -Requirements: Rust (stable, MSVC toolchain), Visual Studio Build Tools with -"Desktop development with C++", Node 20+, and WebView2 (already present on -Windows 11). - -```bash -git clone https://github.com/styayur/local-context-engine +```powershell +git clone https://github.com/styayur/local-context-engine.git cd local-context-engine - -# CLI + MCP server -cargo build --release - -# Desktop shell + installer -cd apps/desktop -npm install -npm run tauri:build +git checkout v0.2.0 +cargo build --release --locked +.\target\release\localsearch.exe --help ``` -The binaries land in `target/release/`: +Use a current stable Rust MSVC toolchain and Visual Studio C++ Build Tools on +Windows. [Development](docs/DEVELOPMENT.md) covers desktop prerequisites and CI. +An unprivileged directory scan is available without installing the optional +index service. Read the [MFT visibility boundary](docs/SECURITY.md#mft-and-permissions) +before choosing that service on a shared machine. -```text -target/release/localsearch.exe -target/release/localsearch-mcp.exe -target/release/lce-index-service.exe -``` - -### Optional: the privileged index service (v0.2) - -The desktop, CLI and MCP server never need administrator rights. If you want the -fast NTFS MFT backend, install the small service once and the front ends will use -it automatically: +## Usage ```powershell -# from an elevated prompt -.\target\release\lce-index-service.exe install -.\target\release\lce-index-service.exe start -localsearch --service-status -``` - -The service owns the MFT and the USN journals and answers typed, versioned -requests over a Named Pipe whose ACL is built explicitly (SYSTEM, Administrators -and you — never Everyone or Anonymous). Without it, everything still works on the -unprivileged directory-scan backend. See [SECURITY.md](SECURITY.md) for the -boundary and for the one disclosure it implies: MFT enumeration can see file -*names* an unprivileged directory walk could not. - -Development ------------ - -```bash -cargo fmt --all -- --check -cargo clippy --all-targets --all-features -- -D warnings -cargo test - -cd apps/desktop -npm run lint -npm run typecheck -npm run build -``` - -Benchmarks: - -```bash -cargo bench -p lce-benchmarks --bench file_search -cargo bench -p lce-benchmarks --bench ranking +localsearch --providers +localsearch --backend scan "type:file ext:pdf" +localsearch --explain "ext:pdf report" +localsearch "prefix:read" +localsearch "type:process python" +localsearch "type:service state:running" +localsearch --index-status ``` -Layout, conventions and how to add a provider or a translation are covered in -[CONTRIBUTING.md](CONTRIBUTING.md). - -Security --------- - -* File search is **read-only**. -* No `exec`, no `eval`, no shell string built from a query. -* Persisted state is a MessagePack index and two JSON files this project wrote; - no untrusted payload is ever deserialised as code. -* Paths are treated as opaque Unicode strings and are tested with spaces, - Simplified Chinese and very long names. Junctions and symlinks are never - followed. -* Process termination requires confirmation on both sides of the boundary. -* Services are strictly read-only in this version. - -Full threat model: [SECURITY.md](SECURITY.md). - -Privacy -------- - -> **Your local index never leaves your machine.** - -* No telemetry, no analytics, no crash reporting, no update check. -* No account, no login, no cloud service. -* Query history is optional, bounded, stored locally in - `%LOCALAPPDATA%\LocalContextEngine\usage.json`, and removable from Settings or - with `localsearch --reset-usage`. -* There is no HTTP client in the dependency tree of the search core. - -Benchmarks ----------- - -Measured with Criterion on a synthetic corpus, not estimated. Full table, -machine specification, method and caveats: [docs/BENCHMARKS.md](docs/BENCHMARKS.md). - -| what | 1 000 000 indexed entries | -|---|---:| -| query compilation (`最近的 pdf`, `recent PDFs`, plain text, DSL) | 1.4 – 11.6 µs | -| file search that fills the candidate budget (the common typing case) | **~8 ms** | -| file search that must visit every record | **~62 ms** | -| the same, with fuzzy matching on every record | **~76 ms** | -| ranking 4 000 candidates | **~2.8 ms** | -| full hot path: walk the index, collect, rank | **~10 ms** | -| building the in-memory index | 316 ms (3.2 M entries/s) | -| memory for one million entries | ≈ 69 MB | +Search commands return `0` for results, `1` for a valid query with no matches, +and `2` for an operation that could not complete. Relative time filters and live +providers naturally change with the clock and system state. Use the +[Query Engine](docs/QUERY_ENGINE.md), [DSL reference](docs/DSL.md) and +[MCP guide](docs/MCP.md) for filtering, JSON output and local agent integration. -The candidate budget (`CANDIDATE_CAP = 4 000`) is why a one million entry index -does not cost fifty times a ten thousand entry one. The rows that do grow are -the ones that genuinely have to touch every record. +## Security and privacy -Roadmap -------- +Search does not upload data or index file contents. Cached paths, metadata, +settings and optional usage history remain sensitive local data. Activating a +result opens it through Windows; process termination requires explicit +confirmation. The optional elevated service can reveal filenames that a normal +directory walk cannot list. It does not grant permission to read their contents. +See [Security](docs/SECURITY.md) and [vulnerability reporting](SECURITY.md). -* MSIX/Appx discovery through `AppModel\Repository` rather than Start Menu links. -* File sizes from the MFT by parsing the `$DATA` attribute. -* Event-driven index updates instead of replaying the journal on demand. -* An optional frameless palette window. -* An optional AI query compiler behind the existing `QueryCompiler` trait — - off by default, and never on the search path. +## Development -Licence -------- +Read [Development](docs/DEVELOPMENT.md) and [Contributing](CONTRIBUTING.md) for +checks and extension points. [Roadmap](docs/ROADMAP.md) separates release gates +from future work; [Changelog](CHANGELOG.md) describes user-visible changes. -[MIT](LICENSE). Not affiliated with, derived from, or containing any code, -assets or UI of Listary or any other launcher. The only thing borrowed is the -public idea that a local index should be fast. \ No newline at end of file +Repository license: [MPL-2.0](LICENSE). See [license notes](THIRD_PARTY_LICENSES.md) +for the remaining desktop package metadata inconsistency before v1.0.0. +LCE runs independently and has no runtime or +Rust crate dependency on CodeForge Engine. diff --git a/README.zh-CN.md b/README.zh-CN.md index 9865a1d..996f808 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -1,292 +1,89 @@ -
- 简体中文 | English -
- - - - - # Local Context Engine -离线优先的本地系统搜索。 - -自然语言输入 → 确定性查询 → Rust 搜索内核 → 结果。 - -[下载](https://github.com/styayur/local-context-engine/releases/latest) · [架构](docs/ARCHITECTURE.md) · [查询语法](docs/DSL.md) · [MCP](docs/MCP.md) · [性能](docs/BENCHMARKS.md) - -[![CI](https://github.com/styayur/local-context-engine/actions/workflows/ci.yml/badge.svg)](https://github.com/styayur/local-context-engine/actions/workflows/ci.yml) -[![release](https://img.shields.io/github/v/release/styayur/local-context-engine?label=release&color=orange)](https://github.com/styayur/local-context-engine/releases) -[![license: MPL-2.0](https://img.shields.io/badge/license-MPL--2.0-blue)](LICENSE) - - - ---- - -## 这是什么 - -Local Context Engine 是一个 Windows 本地搜索工具：一个输入框，同时搜索**文件与文件夹**、**应用**、**正在运行的进程**、**服务**和**窗口**，并且可以直接打开它们。 - -输入 `vscode`，你在同一个列表里看到应用、进程和匹配的文件。输入 `正在运行的 python` 或 `running python processes`，得到 Python 进程。输入 `最近的 pdf` 或 `recent PDFs`，得到按修改时间排序的 PDF。 - -它建立在三个判断之上： +**面向 Windows 的本地优先上下文发现引擎。** 用一套可解释查询查找文件、应用、进程、服务和窗口。 -1. **检索必须确定性。** 同一个查询，每次都返回同样的结果、同样的顺序。热路径上不出现模型、embedding 或向量数据库。 -2. **理解是编译问题。** 自然语言被**编译**成结构化查询——`最近的 pdf` 变成 `type:file ext:pdf sort:modified-desc`——然后交给 Rust 内核执行。 -3. **离线是默认状态，而不是一种模式。** 没有账号、没有遥测，仓库里没有一行网络代码。 +[English](README.md) · [下载 v0.2.0](https://github.com/styayur/local-context-engine/releases/tag/v0.2.0) · [文档目录](docs/README.md) -## 为什么做这个 +当前已发布版本为 **v0.2.0**。v1.0.0 仍是未发布候选说明，不代表已有对应安装包或新的兼容性承诺。 -Windows 上的桌面搜索要么是全量内容索引器（建索引要几分钟），要么是只能看见快捷方式的启动器。而所谓“AI 搜索”的常见答案通常是“把所有东西都做 embedding 再碰运气”——慢、不确定、无法解释，而且悄悄把文件名发到了别处。 +## 解决什么问题 -这个项目选择相反的路线： +同一项工作可能对应文件、应用和运行中的进程。LCE 将这些本地来源放进同一个查询入口，并展示候选选择与排序依据。文件索引保存名称、路径和元数据，不读取全文用于检索；日常搜索不需要账户、模型服务或网络。 -> 如果一定要有 AI，它是**查询编译器**。检索始终是确定性的本地操作。 +## 能力与成熟度 -## 与传统桌面搜索的区别 +| 状态 | 能力 | 边界 | +|---|---|---| +| Stable | Query DSL、中英文规则式查询编译 | 检索不依赖模型 | +| Stable | Query Planner、扩展名/prefix/trigram 索引 | 候选仍需验证；宽泛条件可能扫描索引 | +| Stable | 文件、应用、进程、服务、窗口 Providers | 元数据受 Windows 权限与后端能力限制 | +| Stable | CLI、本地 stdio MCP、解释与健康状态 | 查询、快照、设置、使用记录相同才具有确定性排序 | +| Experimental | 可选的特权 MFT/USN 服务、跨环境桌面验收 | 已实现并有测试，部署环境仍需验证 | +| Future | 全文、语义检索、模型查询编译器 | 不属于默认能力，也不承诺交付时间 | -| | 传统索引器 | 启动器 | Local Context Engine | -|-----------------------------|-----------------|-------------|--------------------------------------------| -| 首次可用时间 | 建索引数分钟 | 即时 | 实时数据即时可用，文件索引一次扫描 | -| 索引文件**内容** | 是 | 否 | 否——只索引名称、路径与元数据 | -| 覆盖进程 / 服务 / 窗口 | 罕见 | 否 | 四类全部覆盖 | -| 查询方式 | 单一输入框 | 单一输入框 | 输入框 + 公开的 DSL + MCP 接口 | -| 排序确定性 | 通常不确定 | 通常不确定 | 始终确定，`--explain` 可给出评分依据 | -| AI 在检索热路径上 | 有时 | 有时 | 从不 | -| 是否需要网络 | 经常需要 | 不需要 | 从不需要 | -| 内存占用 | 数百 MB | 数十 MB | 一百万个条目约 70 MB | +Stable 表示已有实现和回归覆盖，不表示通过了所有硬件、文件系统或桌面环境的认证。 ## 架构 ```text - User / AI - │ - ▼ - Query Compiler ← 未来唯一允许接入 AI 的位置 - │ - ▼ - SearchQuery - │ - ▼ - Search Core - │ - ┌───────────┼───────────┐ - ▼ ▼ ▼ - Files Processes Apps - │ │ │ - MFT/USN Win32 Registry +用户 → CLI / Desktop / MCP → Search Core → Providers + ↓ 文件分支 + Query Planner → Accelerators + ↓ + 候选验证 → 排序与解释 ``` -工作区的划分让“这段代码该放哪”不再需要讨论： +Planner 位于文件 Provider 内部；其他 Provider 读取各自的系统快照。详见[架构](docs/ARCHITECTURE.md)。 -```text -local-context-engine/ -├─ apps/desktop/ Tauri 2 + React + TypeScript 桌面外壳 -├─ crates/ -│ ├─ search-core/ 实体 / 查询 / 结果模型、搜索引擎、模糊匹配 -│ ├─ query-dsl/ DSL 解析器与规则式自然语言编译器 -│ ├─ ranking/ 启发式评分 + 有界的本地使用记录 -│ ├─ windows-files/ 目录扫描与 NTFS MFT + USN 两种索引后端 -│ ├─ windows-processes/ 实时进程快照 -│ ├─ windows-apps/ App Paths、开始菜单、PATH、WindowsApps -│ ├─ windows-services/ 只读的服务快照 -│ ├─ windows-windows/ 顶层窗口快照 -│ ├─ search-daemon/ 三个前端共用的组合根 -│ ├─ search-cli/ localsearch.exe -│ ├─ search-mcp/ localsearch-mcp.exe（stdio 上的 MCP） -│ └─ benchmarks/ 10k / 100k / 100 万条目的 Criterion 基准 -├─ docs/ -└─ tests/ 工作区级集成测试 -``` +## 性能 -完整说明（包括为什么 `AI != Search Engine`、Provider 边界、内存账目）见 -[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)。 +Windows、Rust 1.98.1、百万条合成数据的成对测试： -## 查询语法 +| 指标 | 优化前 | 优化后 | +|---|---:|---:| +| 加速索引构建 | 11.93 秒 | 8.91 秒 | +| 构建进程峰值工作集 | 1005.25 MB | 258.98 MB | +| `ext:pdf report` 平均 / P95 | 89.36 / 99.74 ms | 20.95 / 23.14 ms | +| 常见词 `report` 平均 / P95 | 8.34 / 9.61 ms | 8.47 / 9.87 ms | +| 估算保留加速索引 | 75.80 MB | 75.80 MB | -```text -chrome -type:file rust -type:file ext:rs -type:file ext:pdf modified:<24h -type:process python -type:process name:node -type:app vscode -type:service state:running -type:window github -path:projects ext:toml -drive:C ext:exe sort:size-desc -path:"My Projects" ext:toml -size:>10mb sort:size-desc limit:20 -``` - -可用键：`type`、`ext`、`path`、`name`、`drive`、`modified`、`created`、`size`、 -`state`、`pid`、`user`、`visible`、`sort`、`limit`。解析**永不失败**——无法识别的键会作为普通文本参与搜索，所以 -`localsearch "C:\Users\me"` 是一次搜索而不是语法错误。 +内存收益是构建峰值下降，不能解读为保留索引变小。八组完整有序结果与线性参考一致。完整方法、环境信息缺口、样本数和非加速案例见[性能报告](docs/BENCHMARKS.md)。 -完整的时间与体积运算符说明见 [docs/DSL.md](docs/DSL.md)。 +## 安装 -## MCP +从 [v0.2.0 Release](https://github.com/styayur/local-context-engine/releases/tag/v0.2.0) 下载 Windows x64 便携包或安装程序，对照 `SHA256SUMS.txt` 校验。解压后运行 `localsearch.exe`，可将目录加入 PATH。桌面需要 WebView2，应在实际机器上做启动与搜索验收。 -`localsearch-mcp.exe` 通过 stdio 把同一个内核暴露给任何 MCP 客户端： +源码构建使用当前 stable Rust MSVC 与 Visual Studio C++ Build Tools： -```jsonc -{ - "mcpServers": { - "localsearch": { "command": "C:\\path\\to\\localsearch-mcp.exe" } - } -} +```powershell +git clone https://github.com/styayur/local-context-engine.git +cd local-context-engine +git checkout v0.2.0 +cargo build --release --locked +.\target\release\localsearch.exe --help ``` -| 工具 | 作用 | -|---------------------|-------------------------------------| -| `search_system` | 跨全部数据源搜索 | -| `search_files` | 文件与文件夹 | -| `search_processes` | 正在运行的进程 | -| `search_apps` | 已安装的应用 | -| `search_services` | Windows 服务 | -| `search_windows` | 打开的顶层窗口 | -| `compile_query` | 只展示编译结果，不执行搜索 | -| `index_status` | 后端、条目数、内存、各数据源状态 | -| `dsl_reference` | 键表与示例 | -| `record_selection` | 记住选择，下次提升排序 | -| `terminate_process` | 危险操作，必须传 `confirm: true` | +无需安装特权服务即可使用普通用户目录扫描。共享机器启用 MFT 前，请先阅读[权限边界](docs/SECURITY.md#mft-and-permissions)。 -详见 [docs/MCP.md](docs/MCP.md)。 +## 使用 -## 命令行 - -```bash -localsearch chrome -localsearch "type:file ext:rs" -localsearch "正在运行的 python" -localsearch --type process node -localsearch --json vscode -localsearch --explain "昨天修改的 rust 文件" -localsearch --index-status -localsearch --rebuild-index -localsearch --update-index +```powershell localsearch --providers -localsearch --list-usage -localsearch --reset-usage -``` - -退出码是契约的一部分：`0` 有结果，`1` 查询合法但无匹配，`2` 命令未能完成。 -`--json` 会把单个 JSON 文档写到 stdout，诊断信息始终走 stderr。 - -## 桌面端 - -```bash -cd apps/desktop -npm install -npm run tauri:dev # 开发模式，热重载 -npm run tauri:build # 生成 NSIS 安装包 +localsearch --backend scan "type:file ext:pdf" +localsearch --explain "ext:pdf report" +localsearch "prefix:read" +localsearch "type:process python" +localsearch --index-status ``` -* 搜索框自动聚焦，键盘优先：`↑`/`↓` 导航，`Enter` 打开，`Shift+Enter` 打开右键菜单， - `Tab` 切换类型筛选，`Esc` 清空或关闭，`Ctrl+,` 打开设置。 -* 类型筛选标签带实时计数、匹配高亮、按实体类型区分的右键菜单，状态栏显示编译结果与耗时。 -* **简体中文与 English**，切换无需重启。所有文案——错误、空状态、提示、右键菜单——都在 - `src/i18n/{zh-CN,en-US}.json` 中，组件里没有任何硬编码文本。 -* 设置涵盖语言、主题、结果上限、模糊匹配、使用频率加权、索引磁盘、索引后端、全局快捷键，以及完整的索引面板。 -* 唯一的破坏性操作“结束进程”需要确认，对话框默认焦点在“取消”；Rust 侧在没有显式 - `confirmed: true` 时同样会拒绝执行。 +查询退出码：`0` 有结果，`1` 查询有效但无匹配，`2` 操作无法完成。相对时间、实时快照和使用记录变化会影响结果。详见[查询引擎](docs/QUERY_ENGINE.md)、[DSL](docs/DSL.md) 与 [MCP](docs/MCP.md)。 -## 构建 +## 安全与隐私 -需要：Rust（stable，MSVC 工具链）、安装 “使用 C++ 的桌面开发” 工作负载的 Visual Studio Build Tools、Node 20+，以及 WebView2（Windows 11 自带）。 - -```bash -git clone https://github.com/styayur/local-context-engine -cd local-context-engine - -# 命令行与 MCP 服务 -cargo build --release - -# 桌面端与安装包 -cd apps/desktop -npm install -npm run tauri:build -``` - -产物位于 `target/release/`： - -```text -target/release/localsearch.exe -target/release/localsearch-mcp.exe -``` +默认检索不上传数据、不索引全文。路径缓存、元数据和使用记录仍是敏感本地数据。打开结果由 Windows 处理；结束进程需要确认。可选特权服务可能暴露普通目录遍历无法列出的文件名，但不会授予内容读取权限。详见[安全模型](docs/SECURITY.md)与[漏洞报告](SECURITY.md)。 ## 开发 -```bash -cargo fmt --all -- --check -cargo clippy --all-targets --all-features -- -D warnings -cargo test - -cd apps/desktop -npm run lint -npm run typecheck -npm run build -``` - -基准测试： - -```bash -cargo bench -p lce-benchmarks --bench file_search -cargo bench -p lce-benchmarks --bench ranking -``` - -目录约定、如何新增数据源、如何新增语言，见 [CONTRIBUTING.md](CONTRIBUTING.md)。 - -## 安全 - -* 文件搜索是**只读**的。 -* 没有 `exec`、没有 `eval`、不会用查询拼出 shell 命令。 -* 持久化状态只有本项目自己写的 MessagePack 索引和两个 JSON 文件，不会把不受信任的数据当作可执行内容反序列化。 -* 路径一律按不透明 Unicode 字符串处理，并针对空格、简体中文、超长路径编写了测试；从不跟随 junction 与符号链接。 -* 结束进程在边界两侧都需要确认。 -* 本版本对服务严格只读。 - -完整威胁模型见 [SECURITY.md](SECURITY.md)。 - -## 隐私 - -> **你的本地索引永远不会离开这台机器。** - -* 没有遥测、没有分析、没有崩溃上报、没有更新检查。 -* 不需要账号、不需要登录、没有云服务。 -* 查询历史可选、有上限、保存在本地 - `%LOCALAPPDATA%\LocalContextEngine\usage.json`，可在设置里或通过 - `localsearch --reset-usage` 清空。 -* 搜索内核的依赖树中没有 HTTP 客户端。 - -## 性能 - -下面都是 Criterion 在合成数据集上的实测值，不是估算。完整表格、测试机器、方法与注意事项见 -[docs/BENCHMARKS.md](docs/BENCHMARKS.md)。 - -| 项目 | 索引 100 万条目时 | -|---|---:| -| 查询编译（`最近的 pdf`、`recent PDFs`、纯文本、DSL） | 1.4 – 11.6 µs | -| 命中候选上限的常见输入场景 | **约 8 ms** | -| 必须遍历全部记录的搜索 | **约 62 ms** | -| 同上，且对每条记录做模糊匹配 | **约 76 ms** | -| 对 4 000 个候选排序打分 | **约 2.8 ms** | -| 完整热路径：遍历索引 → 收集候选 → 排序 | **约 10 ms** | -| 构建内存索引 | 316 ms（约 320 万条目/秒） | -| 100 万条目的内存占用 | 约 69 MB | - -候选数量上限（`CANDIDATE_CAP = 4 000`）是 100 万条目索引不会比 1 万条目慢五十倍的原因； -真正会随规模增长的是那些必须遍历全部记录的情况。 - -## 路线图 - -* 通过 `AppModel\Repository` 支持 MSIX/Appx 应用发现，而不仅依赖开始菜单快捷方式。 -* 解析 `$DATA` 属性，让 MFT 后端也能给出文件大小。 -* 由事件驱动索引更新，而不是按需重放变更日志。 -* 可选的无边框命令面板窗口。 -* 可选的 AI 查询编译器，实现同一个 `QueryCompiler` trait——默认关闭，且永远不在检索热路径上。 - -## 许可 +[开发指南](docs/DEVELOPMENT.md)说明构建和检查；[贡献指南](CONTRIBUTING.md)说明扩展原则；[路线](docs/ROADMAP.md)区分发布门槛与未来能力；[变更记录](CHANGELOG.md)面向用户描述变化。 -[MIT](LICENSE)。与 Listary 及其它启动器**没有任何关联**，也不包含其代码、资源或界面。 -本项目只借鉴了“本地索引应当足够快”这一公开的设计思路。 \ No newline at end of file +仓库许可证为 [MPL-2.0](LICENSE)。桌面 package 元数据仍有待对齐的许可证差异，见[许可证说明](THIRD_PARTY_LICENSES.md)。本仓独立运行，不依赖 CodeForge Engine 的运行时或 Rust crate。 diff --git a/SECURITY.md b/SECURITY.md index acc1d03..5a9c94f 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,243 +1,15 @@ -# Security +# Security policy -## Threat model +For runtime data, permissions, MFT filename visibility, process actions and the +optional privileged service, read [Security and privacy](docs/SECURITY.md). -Local Context Engine reads from the operating system and, on request, opens or -reveals the things it finds. It is not a sandbox and it does not try to be one. +## Reporting vulnerabilities -What it does: +Use [GitHub private vulnerability reporting](https://github.com/styayur/local-context-engine/security/advisories/new) +when available. Include the affected version, Windows build, backend and a minimal +reproduction without private paths or credentials. Do not put sensitive +reproductions in a public issue. If private reporting is unavailable, open a +non-sensitive issue asking the maintainer for a private contact channel. -* Reads the NTFS master file table or walks directories to build a **name** index. -* Reads live process, window, service and Start Menu snapshots. -* Opens files, folders and applications **only** when you activate a result. -* Ends a process **only** after you accept a confirmation dialog, and the Rust - side refuses the call unless it was explicitly confirmed. - -What it never does: - -* It never executes shell commands derived from a query. -* It never evaluates query text. -* It never deserialises an executable payload. Persisted state is a - MessagePack index file and two JSON files this project wrote. -* It never opens a socket. There is no telemetry, no analytics and no update - check. -* It never uploads a file list, a query, a query history or a process list. - -## Path handling - -Paths are treated as opaque UTF-16/UTF-8 strings throughout: - -* Spaces, Simplified Chinese, emoji and other Unicode are covered by tests. -* Length is not assumed. The index stores file names in a string arena and - reconstructs paths on demand, so a very long path is a `String`, not a - fixed-size buffer. -* Junction points and symbolic links are never followed, so a directory scan - cannot loop forever. -* Destructive operations are restricted to a single API - (`actions::terminate_process`) whose signature requires `confirmed = true`. - -## Untrusted input - -The MCP server accepts JSON-RPC from a local client and treats every field as -untrusted: - -* Tool arguments are parsed with explicit type checks; a wrong type is an error, - never a coercion. -* Unknown tools, unknown filter names and unknown entity types are rejected. -* Query text is never interpolated into SQL, a shell or a path; it only ever - reaches the matcher. - -## Permissions - -The index backends are honest about what they need: - -| backend | requirement | -|-----------|----------------------------------------------------| -| `scan` | none beyond the user's own read access | -| `mft-usn` | administrator rights to open `\\.\C:` | - -The default is `auto`, which uses `mft-usn` when it is available and otherwise -falls back to `scan`. A denied volume produces a translated, actionable message -in the UI and a full error in the developer log — never a bare `OsError(5)`. - -## Reporting a vulnerability - -Open a private security advisory on GitHub rather than a public issue. Please -include the version, the Windows build, and the smallest reproduction you can -manage. ---- - -# Threat model: the privileged index service (v0.2) - -Version 0.2 moves the operations that need administrator rights out of the -desktop, CLI and MCP processes and into one small service. That is a smaller -attack surface, but it is a trust boundary, and boundaries deserve to be written -down rather than assumed. - -## What runs elevated, and what does not - -```text -Desktop ─┐ -CLI ─────┼── ordinary user, no UAC prompt, no admin token -MCP ─────┘ - │ - │ Named Pipe \\.\pipe\localsearch-index-v1 - ▼ -lce-index-service.exe ── LocalSystem / Administrator - ├── open \\.\C: and enumerate the MFT - ├── read and tail the USN change journals - └── answer index queries -``` - -The service exists because opening `\\.\C:` for `FILE_READ_DATA` requires -elevation. Nothing else does: opening files, revealing folders and ending -processes all stay in the unprivileged processes, where they belong. - -## What the service can be asked to do - -Exactly five things, and the list is the whole protocol: - -| request | effect | -|------------------|---------------------------------------------------------| -| `Hello` | handshake: protocol version, service version, volumes | -| `Search` | run a query against the service's index | -| `IndexStatus` | report backend, entry counts, per-volume sync state | -| `RebuildVolume` | re-enumerate the MFT for one volume (or all) | -| `Changes` | poll for journal advances since a cursor | - -There is no `Execute`, no `Shell`, no `Command`, no `WriteFile` and no `Open`. -This is not a policy that a reviewer has to check line by line: a message that -cannot be constructed cannot be handled. `crates/index-protocol` has a test that -enumerates the variants so that adding a dangerous one is a visible diff. - -## Raw MFT visibility: the honest statement - -**MFT enumeration can reveal file names that the calling user could not discover -by walking the directory tree.** - -The master file table does not apply per-directory ACLs. A volume-wide -enumeration returns every name on the volume, including names inside directories -the user cannot open. This is inherent to the technique — the same is true of -`Everything`, of `fsutil usn readjournal`, and of any MFT-based indexer — and it -is *not* something this project can fix by being careful. - -The consequences, stated plainly: - -* An index built by the service contains names the user may not be permitted to - read. Search results can therefore surface a path whose contents will still be - denied on open, because the open goes through the normal ACL check. -* The service does not read file *contents*, and it never returns file - contents. It returns names, paths and metadata. -* On a machine where more than one person has an account, that is a real - disclosure of *names*. If that is unacceptable for a given machine, do not - install the service: the unprivileged directory-scan backend indexes exactly - what the user can already see. - -## Pipe access control - -The DACL is built from an SDDL string, not assembled by hand: - -```text -D:(A;;GA;;;SY)(A;;GA;;;BA)(A;;GRGW;;;) -``` - -* `SY` — LocalSystem, which owns the service. -* `BA` — the built-in Administrators group, which may manage it. -* the installing user's SID — read and write, enough to ask a question. - -Deliberately absent: `WD` (Everyone) and `AN` (Anonymous). A pipe created with a -default security descriptor grants both, and that is the single most common -mistake in this kind of design. - -Additional hardening, all in `crates/index-service/src/pipe.rs`: - -* `FILE_FLAG_FIRST_PIPE_INSTANCE` — if another process already owns the name, - creating the pipe **fails** instead of silently sharing. This defeats pipe - name squatting, where a hostile local process creates the pipe first and - harvests whatever the front ends send it. -* `PIPE_REJECT_REMOTE_CLIENTS` — remote clients are refused by the pipe itself, - not merely by the ACL, so the boundary does not depend on the DACL being - perfect. -* Byte-mode pipes with an explicit length prefix, so a half-written message is a - detectable truncated frame rather than a silently short read. - -## Malformed client input - -Every field that crosses the pipe is treated as hostile: - -| input | handling | -|-----------------------------|------------------------------------------------------| -| bad frame magic | `BadMagic`, connection dropped | -| unknown protocol version | `VersionMismatch`, connection dropped | -| frame larger than 8 MiB | `TooLarge`, **rejected before the body is allocated** | -| truncated frame | `Truncated`, connection dropped | -| malformed MessagePack body | `Malformed`, connection dropped | -| unknown request variant | decode failure, connection dropped | -| unknown entity type in `Search` | ignored; the type filter simply does not match it | -| impossible `limit` | clamped to the service's own maximum (2 000 hits) | - -The service never unwraps a client-controlled value and never indexes a buffer -with a client-supplied length: the frame reader checks the announced length -against its own constant before allocating. - -## Protocol downgrade - -A downgrade is refused in two places: - -1. **Transport.** Every frame carries its version, and a mismatch is an error - before the body is even parsed. -2. **Application.** `Hello` carries the client's version and the service answers - `protocol-mismatch` if they disagree, so a client that somehow negotiated the - wrong version at the transport layer still cannot proceed. - -There is no "compatibility mode" and no best-effort decode. If the revisions -differ, the front ends fall back to their own in-process provider and the user is -told the service needs updating. - -## A stale index is a correctness failure, not a performance one - -The service must never answer from an index it cannot prove is current. The -per-volume journal registry makes that decision explicit: - -| condition | status | what the caller sees | -|----------------------------------------|---------------------|------------------------------------| -| cursor is inside the journal window | `healthy` | results presented as current | -| journal was recreated | `stale` | volume marked degraded, rebuild | -| cursor fell behind `FirstUsn` (rollover) | `stale` | volume marked degraded, rebuild | -| journal moved backwards | `stale` | volume marked degraded, rebuild | -| volume handle cannot be opened | `permission-denied` | volume marked degraded, no rebuild | -| volume is not mounted | `offline` | volume marked degraded | - -The UI and `--index-status` surface these states. A degraded volume is reported -as degraded; it is never silently presented as up to date. - -## Service crash and volume loss - -* A crash of `lce-index-service.exe` cannot crash the desktop, the CLI or the - MCP server: they hold a pipe, and a closed pipe is an I/O error that triggers - the in-process fallback. -* One volume failing does not stop another. Each volume has its own worker, - status and cursor; the supervisor reports per-volume health. -* A worker that cannot read its journal backs off exponentially to 30 seconds - instead of spinning, and marks only its own volume degraded. -* Shutdown is a flag plus a bounded wait, so the service stops promptly even - while a journal read is in flight. - -## Authentication of the caller - -The DACL answers "may this account connect". It does **not** distinguish two -processes belonging to the same user, and this build does not attempt to: the -service serves the interactive user that installed it, and nothing more -sensitive than an index query is exposed. There is no admin-only operation -reachable over the pipe — `RebuildVolume` is the most expensive one, and a -rebuild is a read of the volume the user can already see. - -## What is still out of scope - -* The service is not a sandbox for a compromised same-user process. Such a - process can already do everything the service can. -* There is no signature verification of the client. On a machine where the - user's own account is hostile to itself, this design adds nothing. -* The `Changes` feed is volume-granular (it reports journal advances), not a - per-file event stream. It is honest about that rather than pretending to - deliver events it does not have. \ No newline at end of file +The current published release is v0.2.0. v1.0.0 remains an unreleased candidate; +no maintenance lifetime or third-party security certification is implied. diff --git a/THIRD_PARTY_LICENSES.md b/THIRD_PARTY_LICENSES.md new file mode 100644 index 0000000..cc92d81 --- /dev/null +++ b/THIRD_PARTY_LICENSES.md @@ -0,0 +1,20 @@ +# Third-party licenses + +Local Context Engine is [MPL-2.0](LICENSE) licensed, matching the Rust workspace +metadata. The [license text](https://www.mozilla.org/en-US/MPL/2.0/) governs the +project; an earlier README incorrectly identified it as MIT. + +An unresolved packaging inconsistency remains: the desktop package manifest and +its root lockfile entry still say MIT. No separate desktop LICENSE was found. +Resolve and review that metadata against the intended license before v1.0.0; +this documentation change does not silently relicense files or alter manifests. + +Cargo.lock and the desktop package lockfile record the dependency inventory. +Dependencies retain their own licenses. This document is not a complete +installer notice bundle: review the exact linked/shipped components, upstream +notices and source obligations before distributing a release artifact. + +Renovate and zvec-grep were design references, not framework dependencies. No +upstream implementation was copied or bundled in the planner upgrade. There +is no runtime or Rust crate dependency on CodeForge Engine; its existing CI +Regression Gate is an external development check. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 7a57480..95a1254 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,366 +1,52 @@ # Architecture -## The one-sentence version +LCE separates query meaning, operating-system access, candidate planning and +ranking. The same SearchService composes these parts for CLI, Desktop and MCP. -Natural language goes in, a deterministic query comes out, and a Rust search -core answers it from local data — no model, no network and no database on the -hot path. - -## The pipeline - -```mermaid -flowchart TD - U["User / AI"] --> QC["Query Compiler
DSL + rule-based NL"] - QC --> SQ["SearchQuery"] - SQ --> SC["Search Core"] - - SC --> F["Files"] - SC --> P["Processes"] - SC --> A["Apps"] - SC --> S["Services"] - SC --> W["Windows"] - - F --> MFT["MFT / USN"] - F --> SCN["Directory scan"] - P --> TH["Toolhelp32 + PSAPI"] - A --> REG["Registry + Start Menu + PATH"] - S --> SCM["EnumServicesStatusEx"] - W --> EW["EnumWindows"] - - MFT --> R["Ranking"] - SCN --> R - TH --> R - REG --> R - SCM --> R - EW --> R - R --> RES["SearchResponse"] - RES --> CLI["localsearch.exe"] - RES --> MCP["MCP server"] - RES --> UI["Tauri desktop"] -``` - -## Why `AI != Search Engine` - -The tempting design is to hand the whole question to a model: embed the query, -embed the file names, ask a vector database for the nearest few. It fails for -the thing a desktop search has to be good at. - -* **It is not deterministic.** The same query has to return the same list in the - same order, or the muscle memory you build after a week of use is worthless. -* **It is not instant.** A local embedding pass plus a vector lookup is tens to - hundreds of milliseconds before you have even started ranking. -* **It is not auditable.** "Why is this file above that one?" has to have an - answer a person can read. A cosine similarity is not an answer. -* **It is not private by construction.** The moment a model is on the query - path, the question "could this call out to the network?" is back. - -So the model, if there is one, goes **here**: - -```text -User ──▶ Query Compiler ──▶ SearchQuery ──▶ Search Core ──▶ Results - ▲ - │ - an AI may replace this box -``` - -The compiler's whole job is translation: "最近的 pdf" becomes -`type:file ext:pdf sort:modified-desc`. Retrieval is untouched by it. Ship the -MVP with a rule-based compiler (this repository does), and a future provider can -swap in a model behind the same `QueryCompiler` trait without the engine -noticing. - -## Crate boundaries - -```text -apps/desktop ────────────────────────────────┐ -crates/search-cli ─────┐ │ -crates/search-mcp ─────┼──▶ crates/search-daemon ──▶ crates/search-core - │ │ ▲ - │ ├─▶ crates/query-dsl ┘ - │ ├─▶ crates/ranking ┘ - │ └─▶ crates/windows-{files,processes,apps,services,windows} - │ - └── all three call exactly one `SearchService` -``` - -| crate | responsibility | must not know about | -|---------------------|---------------------------------------------------------|------------------------------| -| `search-core` | entity/query/result model, engine, fuzzy matcher | Win32, Tauri, CLI | -| `query-dsl` | DSL parser, rule-based NL compiler | the OS, ranking | -| `ranking` | heuristic scoring, usage memory | providers, presentation | -| `windows-files` | index backends, persistence | other domains | -| `windows-processes` | process snapshots | other domains | -| `windows-apps` | application discovery | other domains | -| `windows-services` | service snapshots (read-only) | other domains | -| `windows-windows` | window snapshots | other domains | -| `search-daemon` | composition root, settings, actions | UI frameworks | -| `search-cli` | `localsearch.exe` | search logic | -| `search-mcp` | MCP/JSON-RPC transport, tool schemas | search logic | -| `apps/desktop` | Tauri shell, React UI, i18n | search logic | - -The rule that keeps this honest: **if two front ends would need the same -decision, that decision belongs in `search-daemon` or below it.** Ranking, -filtering, compiling, opening results and terminating processes each exist -exactly once. - -## The `EntityProvider` boundary - -```rust -pub trait EntityProvider: Send + Sync + Debug { - fn name(&self) -> &'static str; - fn scope(&self) -> SnapshotScope; - fn entity_types(&self) -> &'static [EntityType]; - fn collect(&self, query: &SearchQuery) -> Result>; - fn stats(&self) -> ProviderStats; - fn refresh(&self) -> Result<()>; -} -``` - -A provider is the only place that knows how to read one domain out of the -operating system. It returns normalised entities and nothing else — no ranking, -no formatting, no presentation concerns. Adding a domain means adding one -provider; it does not mean touching the engine, the CLI, the MCP server or the -UI's search logic. - -## The ranker boundary - -```rust -pub trait Ranker: Send + Sync + Debug { - fn score(&self, query: &SearchQuery, entity: &LocalEntity) -> Option; -} -``` - -`Some` means "this answers the text"; `None` drops the candidate. The bundled -`HeuristicRanker` sums named, weighted components -(`exact_match` 100, `prefix_match` 60, `filename_match` 40, `token_match` 30, -`path_match` 20, `fuzzy_match` 10, `usage_frequency` 8, `recency` 5, plus a -per-type preference), which is what makes `--explain` able to show its work. - -## File indexing +## Data flow ```text - ┌──────────────────────────────┐ - \\.\C: ─────▶ │ mft::build │ ─┐ - FSCTL_ENUM_USN_DATA │ initials index │ │ - └──────────────────────────────┘ │ - ├──▶ store::FileStore ──▶ search - ┌──────────────────────────────┐ │ - directories ─▶ │ scan::build_store │ ─┘ - └──────────────────────────────┘ +Input → query-dsl → SearchQuery → SearchService / Search Core + ├─ file provider → planner → accelerators + │ ↓ ↓ + │ authoritative candidate verification + └─ app / process / service / window snapshots + ↓ + ranking → stable tie-breaking → results ``` -| backend | needs admin | file sizes | warm start | -|-----------|-------------|------------|----------------------------------| -| `scan` | no | yes | loads the persisted cache | -| `mft-usn` | yes | no | replays the USN change journal | - -`auto` prefers `mft-usn` when the volume handle can be opened and silently falls -back to `scan`, so the provider always works. See -[`crates/windows-files/src/mft.rs`](../crates/windows-files/src/mft.rs) for why -the MFT path cannot report sizes without per-file NTFS attribute parsing. +A candidate index is a conservative filter, not the definition of a match. +Unindexable conjuncts remain in verification. The same query, snapshot, settings +and usage state produce the same ordered results; live state and relative time +are explicit inputs. Result and candidate budgets bound work, so a capped result +is not a claim that every matching record was returned. -### Memory +## Responsibilities -Entries live in a string arena plus fixed-size records rather than one -allocation-heavy struct per file: - -```text -1_000_000 rows × 48 B = 48 MB -names + directories ≈ 20 MB ------------------------------------- - ≈ 68 MB -``` - -`store::FileStore::memory_bytes()` reports the real figure in Settings → Index, -so the README never has to guess. - -### Persistence - -`PersistedIndex` is MessagePack written to a temporary file and then renamed, so -an interrupted save can never leave a half-written cache. A version mismatch is -not an error: the cache is ignored and rebuilt. - -## Live snapshots - -Processes, windows and services are never persisted. A snapshot of "now" is the -only thing that makes sense for them, and a stale one is worse than none. - -| domain | API | caching | -|-----------|----------------------------------|--------------------------------------| -| processes | `CreateToolhelp32Snapshot` + PSAPI | 250 ms, resolved in parallel | -| windows | `EnumWindows` | 250 ms | -| services | `EnumServicesStatusEx` | 5 s, config resolved only where needed | -| apps | registry + Start Menu + `PATH` | 5 min (installed apps rarely change) | - -## Error model - -```text -SearchError IndexError PlatformError PermissionError - \ | | / - +----------+------+------+-------------+ - │ - LceError - │ - ┌──────────────────┼──────────────────┐ - │ │ │ - code() string hint() (English) Display (developer) - │ - i18n key `error.` -``` - -The UI never renders `OsError(5)`. It renders -`error.volume-access-denied` → "无法访问该卷。可能需要管理员权限。", and the -original error goes to the log. - -## Threading - -* Providers are `Send + Sync`; the engine fans out serially but each provider is - cheap once warm. -* `FileProvider::rebuild` is the only genuinely long operation. It runs on a - Tauri blocking thread from the UI and as a plain call from the CLI. -* Process detail resolution is chunked across up to eight OS threads, which is - what turns ~30 ms of serial syscalls into single digits. -* There is no polling loop anywhere. Idle CPU is zero because idle means "no - thread is doing anything". - -## Roadmap - -| area | next step | -|------------------------|------------------------------------------------------------------| -| MSIX/Appx discovery | read `AppModel\Repository` instead of relying on Start Menu links | -| MFT file sizes | parse the `$DATA` attribute from `FSCTL_GET_NTFS_FILE_RECORD` | -| Event-driven file index| subscribe to the USN journal instead of replaying it on demand | -| Frameless palette mode | optional chromeless window with a custom drag handle | -| AI query compiler | a second `QueryCompiler` implementation, opt-in and off by default| ---- - -# v0.2: per-volume state, workers, the privileged boundary and accelerators - -## Per-volume journal state - -A single `Option` cannot describe a machine with more than one -NTFS volume. `C:` and `D:` have different journal identifiers, different -cursors, and roll over independently. - -```rust -pub struct JournalRegistry { - pub schema_version: u32, - pub volumes: BTreeMap, // keyed by VolumeId -} - -pub struct VolumeJournalState { - pub identity: VolumeIdentity, // volume GUID + serial + mount points - pub journal_id: u64, - pub first_usn: i64, - pub next_usn: i64, - pub cursor_usn: i64, - pub status: VolumeSyncStatus, - pub has_cursor: bool, - pub pending_rebuild: Option, - // ... -} -``` - -The key is a **`VolumeId`**, never a drive letter: `GetVolumeNameForVolumeMountPoint` -gives `\\?\Volume{...}\`, which survives a re-lettering, and the serial number is -the fallback when no GUID exists. `reconcile_identity` recognises "same disk, -new mount point" and keeps the cursor; `observe` is the single place that decides -incremental versus rebuild. - -| condition | decision | +| Module | Owns | |---|---| -| no cursor yet | `Rebuild { NewVolume }` | -| journal id changed | `Rebuild { JournalRecreated }` | -| `cursor < first_usn` | `Rebuild { CursorBeforeFirstUsn }` (rollover) | -| `cursor > next_usn` | `Rebuild { CursorAheadOfJournal }` | -| `cursor == next_usn` | `UpToDate` | -| otherwise | `Incremental { from_usn }` | - -One volume failing, rolling over or being offline never affects another. - -## Event-driven workers - -```text - IndexSupervisor - ├── VolumeWorker(C:) ── FSCTL_READ_USN_JOURNAL (blocking) - ├── VolumeWorker(D:) - └── VolumeWorker(...) -``` - -Each worker calls `FSCTL_READ_USN_JOURNAL` with a `Timeout` and a -`BytesToWaitFor`, so the kernel holds the call open until the journal moves. -There is no polling loop and idle CPU stays at zero. The supervisor tracks a -lifecycle per volume (`starting` → `running` → `backing-off` → `failed` → -`stopped`) and backs off exponentially to 30 seconds, marking **only that -volume** degraded. - -Journal reads and index application are separate types: - -```text - USN_RECORD_V2 --parse--> Vec --apply--> FileStore - (mft.rs) (mutation.rs) (store.rs) -``` - -`read_journal_batches` performs all disk I/O and parsing with **no lock held**; -`apply_journal_batch` takes the write lock only to apply a batch; persistence -happens after the lock is released. - -## The privileged boundary - -```text -Desktop ─┐ -CLI ─────┼── ordinary user -MCP ─────┘ - │ \\.\pipe\localsearch-index-v1 (byte mode, length-prefixed MessagePack) - ▼ -lce-index-service.exe ── elevated - ├── raw NTFS MFT - └── USN journals -``` - -The service's entire vocabulary is five requests: `Hello`, `Search`, -`IndexStatus`, `RebuildVolume`, `Changes`. There is no `Execute`, `Shell` or -`Command`, and a test enumerates the variants so adding one shows up in review. -Open, reveal and terminate-process stay in the unprivileged processes. - -Framing is `u32 length | u32 magic | u32 version | body`, with an 8 MiB ceiling -checked **before** the body is allocated. A wrong magic, an unknown version, a -truncated frame or a malformed body is a clean error; the connection is dropped -and the front end falls back to its in-process provider. - -## Search accelerators and the query planner - -```text -ext:rs -> ExtensionIndex (extension -> sorted RecordId postings) -prefix:vsco -> PrefixIndex (sorted 24-byte keys, binary search) -*substring* -> TrigramIndex (path trigrams, intersected, then verified) -anything else -> LinearFallback (the v0.1 scan) -``` - -Three rules make this safe: - -1. **Accelerators only propose.** Every candidate is verified against the real - string in the `FileStore`, which stays the single source of truth. -2. **The trigram index covers full paths, not names.** A token matches a record - when it appears in the name *or* the path, so indexing names alone would not - be a superset — the differential tests found exactly that bug. -3. **A dropped trigram stays dropped.** High-frequency trigrams are pruned, and - a live insert must never resurrect one: a one-element posting list for a - trigram that appears in half the corpus silently loses matches. - -`planner::plan` returns the chosen plan, the sources it consulted and how many -candidates each produced. `localsearch --explain` prints them; the accelerated -and linear paths are asserted equal by differential tests over a randomised -corpus, after mutations, and under the candidate cap. - -## Concurrency - -| structure | protection | held for | -|---|---|---| -| `FileStore` + `SearchAccelerators` | one `RwLock` | reads: a whole query; writes: one mutation batch | -| `JournalRegistry` | inside the same `RwLock` | as above | -| index cache file | none | written outside every lock | -| provider caches (processes, services, windows, apps) | `Mutex` + TTL | a snapshot copy | - -The rule is that no disk I/O, no `DeviceIoControl` and no serialisation happens -while the index lock is held. \ No newline at end of file +| `search-core` | Entity, query, result, matcher and provider/ranker interfaces | +| `query-dsl` | Explicit DSL and rule-based English/Chinese compilation | +| `ranking` | Named score components and local usage weighting | +| `windows-files` | Store, planner, accelerators, scan/MFT/USN, persistence | +| Other `windows-*` providers | Domain-specific Windows snapshots | +| `search-daemon` | Composition, settings and user actions; not a required background daemon | +| `index-protocol`, `index-client`, `index-service` | Optional privileged process boundary | +| CLI, MCP and Desktop | Presentation and transport | + +See [Query Engine](QUERY_ENGINE.md), [Indexing](INDEXING.md) and +[Providers](PROVIDERS.md) for the implemented interfaces and constraints. + +## Reliability boundaries + +Per-volume identity and journal cursors prevent one volume's failure from +invalidating another. Store and accelerators mutate together under a write lock; +journal I/O and parsing occur outside that critical section. Persisted state is +versioned, and unsupported caches are rebuilt. Health/degraded states are visible +to callers rather than being represented as proof of freshness. + +The optional index service answers typed requests over a local named pipe. +Search is read-only; activation and confirmed process termination remain in the +user process. [Security](SECURITY.md) documents what that boundary does and does +not protect. Full-text or semantic providers are [future work](ROADMAP.md), not +part of the current search path. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index ae010a6..c9aa70b 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -1,274 +1,93 @@ # Benchmarks -Every number on this page was measured on the machine described below with the -commands shown at the bottom. Nothing here is estimated, projected or copied -from a different workload. +This report uses the existing paired million-record planner probe. It measures +in-memory file candidate selection and accelerator construction, not end-to-end +desktop responsiveness, filesystem enumeration or live provider latency. -## Measured on +## Method and environment -| | | +| Item | Recorded value | |---|---| -| CPU | AMD Ryzen 7 255 H (8 cores / 16 threads, Zen 4c class) | -| RAM | 22.8 GB | -| Disk | KY3000-512GB, NVMe SSD | -| OS | Windows 11 Home (build 26200) | -| Toolchain | `rustc 1.98.1`, `stable-x86_64-pc-windows-gnu` | -| Profile | `[profile.bench]` — `opt-level = 3`, `lto = "thin"`, `codegen-units = 1` | -| Corpus | synthetic, deterministic, seed `0x123456789ABCDEF0` | -| Harness | Criterion 0.5, 20 samples, 6 s measurement, 1 s warm-up | - -The corpus is generated by `lce_benchmarks::synthetic_store`: realistic project -tree names drawn from a fixed word list, roughly one directory per six entries, -files in the 0–50 MB range with plausible timestamps. See -[`crates/benchmarks/src/lib.rs`](../crates/benchmarks/src/lib.rs). - -## File index search - -`file_search/*` measures `windows_files::collect_candidates` — the exact -function `FileProvider::collect` calls. Times are Criterion's median estimate. - -| benchmark | 10 000 | 100 000 | 1 000 000 | -|---|---:|---:|---:| -| `exact` ("readme") | 1.03 ms | 8.11 ms | 8.29 ms | -| `prefix` ("re") | 1.32 ms | 3.63 ms | 3.75 ms | -| `substring` ("port") | 871 µs | 7.54 ms | 7.79 ms | -| `rare_substring` ("zzqxjw") | 580 µs | 6.11 ms | 63.4 ms | -| `fuzzy` ("rpt") | 724 µs | 7.46 ms | 76.1 ms | -| `filters` (`ext:pdf` + `drive:C` + `size:>1mb`) | 544 µs | 5.92 ms | 62.4 ms | -| `path_filter` (`path:docs`) | 889 µs | 9.99 ms | – | -| `sort` (`sort:size-desc`) | 857 µs | 7.58 ms | 7.46 ms | - -### Why some rows barely change between 100 000 and 1 000 000 - -That is not a measurement mistake, it is the candidate cap doing its job. - -`collect_candidates` stops after `CANDIDATE_CAP = 4 000` matches. Queries like -`readme`, `re` and `port` match thousands of entries in the synthetic corpus, so -the scan terminates early once it has enough work for the ranker. The cost stops -growing with the index. - -The rows that **do** grow are the interesting ones: - -* `rare_substring` and `filters` match almost nothing, so they must visit every - record. **A full scan of one million entries costs 62–63 ms.** -* `fuzzy` additionally runs the subsequence matcher on every candidate and - costs **76 ms** for one million entries. - -So the honest summary is: - -| case | 1 000 000 entries | -|---|---| -| enough matches to hit the cap (the common typing case) | **~8 ms** | -| full scan required (rare term, or an uncommon filter) | **~62 ms** | -| full scan plus fuzzy matching on every record | **~76 ms** | - -## Ranking - -`ranking/*` measures `HeuristicRanker::score` over the candidate set the file -index produced. The set is capped at 4 000, so this cost is bounded regardless -of index size. - -| benchmark | 10 000 | 100 000 | 1 000 000 | +| Platform/toolchain | Windows, Rust 1.98.1, release build | +| Dataset | 1,000,000 synthetic records; seed `0x1234_5678_9ABC_DEF0` | +| Timing | 3 warmups and 40 samples per query; arithmetic mean and nearest-rank P95 | +| Timed result cap | 2,000 matches; not an exhaustive full-result latency measurement | +| Correctness | Separate uncapped comparison of every result ID in record order with the linear reference | +| Memory | Windows process peak working set; retained accelerator bytes are a separate internal estimate | +| Provenance | October 2026 paired local captures linked below | + +The paired capture does **not** record the CPU, RAM, disk or full target triple. +Those missing fields limit reproducibility and must be recorded in a new +release-quality run. The older Criterion report records a Ryzen 7 255 H, +22.8 GB RAM, KY3000-512GB NVMe, Windows 11 build 26200 and GNU toolchain, but it +is a different harness/workload and must not supply assumed hardware metadata +for this pair. Its original report remains in the archive. + +## All query results + +Times are milliseconds; zero-result rows are explicit so a fast negative lookup +is not mistaken for materializing a large result set. + +| Query | Results | Before mean / P95 | After mean / P95 | |---|---:|---:|---:| -| `heuristic` (name only) | 346 µs | 3.06 ms | 2.75 ms | -| `with_usage` (usage boost enabled) | 532 µs | 3.68 ms | – | - -The three numbers per row are close because the *candidate count* is what -matters, not the index size: all three corpora produce 4 000 candidates. - -## End to end - -`end_to_end/query` is the full hot path: walk the index, collect candidates, -rank them. - -| entries | median | -|---|---:| -| 10 000 | 1.28 ms | -| 100 000 | 10.59 ms | -| 1 000 000 | 10.40 ms | - -## Query compilation - -The compiler runs before every search. It is microseconds, which is why it can -sit on the keystroke path without a second thought. - -| input | median | -|---|---:| -| `query_compiler/dsl` (`type:file ext:rs modified:<24h sort:size-desc`) | 1.37 µs | -| `query_compiler/plain` (`vscode`) | 10.6 µs | -| `query_compiler/nl-process` (`正在运行的 python`) | 10.7 µs | -| `query_compiler/nl-zh` (`最近的 pdf`) | 11.2 µs | -| `query_compiler/nl-en` (`recent PDFs`) | 11.6 µs | - -Plain text is slower than DSL because the natural language rules are attempted -before the input falls through to a plain keyword query. - -## Index construction - -`file_store/load` measures building the in-memory store from the corpus, which -is what the directory-scan backend does after `read_dir`: - -| entries | median | throughput | +| `zzqxjw-nothing-matches` | 0 | 106.4053 / 114.7028 | 0.0022 / 0.0025 | +| `ext:pdf re` | 10,445 | 30.0422 / 34.3426 | 7.4280 / 9.3880 | +| `ext:pdf report` | 3,441 | 89.3596 / 99.7413 | 20.9528 / 23.1421 | +| `ext:pdf path:projects` | 0 | 267.6831 / 286.4835 | 0.0019 / 0.0020 | +| `prefix:design report` | 0 | 87.8492 / 93.3898 | 19.8860 / 21.8997 | +| `design-99999` | 0 | 2.7867 / 3.6018 | 2.0751 / 2.2992 | +| `report` | 50,076 | 8.3357 / 9.6114 | 8.4723 / 9.8673 | +| `prefix:read` | 49,998 | 4.0103 / 4.8976 | 3.3955 / 3.7737 | + +All eight uncapped ordered comparisons passed. Common `report` slightly worsened +in this run; no speedup is claimed for that query. These are local observations, +not confidence intervals or portable latency guarantees. + +## Construction and resources + +| Measurement | Before | After | |---|---:|---:| -| 10 000 | 2.88 ms | 3.5 M entries/s | -| 100 000 | 30.5 ms | 3.3 M entries/s | -| 1 000 000 | 316 ms | 3.2 M entries/s | - -`file_store/extension_lookup/1000000` walks every record and reads its -extension: **10.7 ms**. `file_store/stats/100000` computes file/directory counts -and the memory estimate: **103 µs**. - -## Memory - -Measured with `FileStore::memory_bytes()` on a fully populated store, not -estimated: - -```text -1_000_000 entries - records : 1_000_000 × 48 B = 48.0 MB - arena : 21.4 MB (names, UTF-8) - ------------------------------------ - total : ≈ 69 MB -``` - -That is what the desktop UI reports in Settings → Index. - -## Reproducing - -```bash -cargo bench -p lce-benchmarks -``` - -Individual suites, or a subset of the corpus sizes: - -```bash -cargo bench -p lce-benchmarks --bench file_search -cargo bench -p lce-benchmarks --bench ranking -cargo bench -p lce-benchmarks --bench file_search -- 100000 -``` - -Criterion writes an HTML report to `target/criterion/report/index.html`. - -## Caveats - -* These numbers describe the **search core**, not the Windows APIs. A fresh - process table, service list or application catalogue is not in these tables; - see the status bar in the app, or `localsearch --providers`, for live figures - on your own machine. -* The directory-scan backend additionally pays `read_dir` and a `stat` per - entry; the MFT backend does not. Both then produce the same in-memory store, - which is what is measured here. -* Numbers come from one machine. Run the benchmarks yourself before drawing - conclusions about yours; `docs/BENCHMARKS.md` is a snapshot, not a promise. -* The first iteration of any benchmark pays page faults on a cold corpus. The - harness warms up for a second per benchmark to damp that. ---- - -# v0.2: accelerators and the query planner - -The numbers above are the **before** picture: every query walked the whole -index. v0.2 adds an extension index, a prefix table and a trigram index, and a -planner that picks between them. - -Measured on the same machine and toolchain as the table above, against the same -synthetic corpus, with `cargo bench -p lce-benchmarks --bench accelerators` -(Criterion, 10 samples, 3 s measurement). - -## The headline: selective queries no longer scan - -| query (100 000 entries) | plan | v0.1 linear | v0.2 | change | -|---|---|---:|---:|---:| -| `design-99999` (rare substring) | **trigram** | 4.48 ms | **31.6 µs** | ~140× faster | -| `design-99999` (10 000 entries) | **trigram** | 430 µs | **1.77 µs** | ~240× faster | -| `prefix:read` | **prefix** | 4.48 ms¹ | 2.94 ms | ~1.5× faster | -| `ext:pdf` | **extension** | 4.48 ms¹ | 2.60 ms | ~1.7× faster | - -¹ The linear column for the prefix and extension rows is the same full-scan -`rare_substring` measurement, because in v0.1 every predicate was evaluated by -walking the index. The v0.2 rows include the planner run itself. - -## What did **not** get faster - -This is the part that matters as much as the win. These queries still plan as -`linear`, and they cost what they cost in v0.1 — no regression, no -improvement: - -| query (100 000 entries) | plan | v0.2 median | -|---|---|---:| -| `readme` (exact) | linear | 9.39 ms | -| `report` (common substring) | linear | 8.70 ms | -| `rpt` (fuzzy) | linear | 8.37 ms | -| `report` + `ext:pdf` (combined) | linear | 7.29 ms | -| `zzqxjw-nothing-matches` (negative) | linear | 4.48 ms | - -Why: - -* **Common trigrams are dropped.** `readme` and `report` consist of trigrams - that appear in a large fraction of the corpus. Indexing them would cost - memory and narrow nothing, so they are pruned; a query whose every trigram was - pruned has no candidate source and falls back to the scan. -* **Three characters is the floor.** Fuzzy matching runs against short needles - where a trigram index has nothing to offer, and the scan is the honest answer. -* **`combined` inherits the fallback.** If any one token of a multi-token query - cannot be accelerated, the whole query takes the linear path. Intersecting the - extension postings with a scan would be a worthwhile follow-up. - -## What the accelerators cost - -| measurement | 10 000 | 100 000 | -|---|---:|---:| -| build (`accelerated/build`) | 35.9 ms | 568 ms | -| planning only (`accelerated/planning`) | 1.67 µs | 1.42 µs | - -The build cost is the price of indexing every record's **full path**: ~57 bytes -per record at these sizes, so ~58 trigram insertions per record. It is paid once -and persisted; the planner itself is microseconds. - -Memory, measured with `SearchAccelerators::stats()` on a 100 000 entry corpus: - -```text -100 000 entries - FileStore 9.96 MB - SearchAccelerators 5.95 MB (+60%) - ------------------------------------ - total 15.91 MB - - distinct trigrams 1 223 - trigrams dropped 147 - trigram postings 579 441 - prefix table entries 100 001 (100 000 records + the volume root) - distinct extensions 1 -``` - -The prefix table is the largest single contributor at 32 bytes per record; the -trigram postings are ~5.8 per record. - -**This is a real cost, not a rounding error.** Extrapolating linearly, a one -million entry index lands around 125 MB rather than the ~69 MB measured in v0.1, -which is past the project's 100 MB goal. The v0.2 run was made at 10 000 and -100 000 entries; the million-entry case is available with -`LCE_BENCH_1M=1 cargo bench -p lce-benchmarks --bench accelerators`, and it is -not run by default because building the accelerators for a million paths takes -roughly six seconds and the suite would take an hour. - -Planned mitigations, in the order they are worth doing: - -1. Drop the prefix table when trigrams are available — a prefix of three or more - characters is already answerable as a trigram intersection, and the table is - 32 bytes per record. -2. Delta-encode the posting lists, which are sorted and dense. -3. Shrink `PREFIX_KEY_LEN` from 24 bytes to 12, halving the table. - -## Reproducing - -```bash -# the "before" measurement -cargo bench -p lce-benchmarks --bench file_search - -# the "after" measurement +| Accelerator construction, ms | 11,934.3107 | 8,908.7187 | +| Construction process peak working set, bytes | 1,005,248,512 | 258,981,888 | +| Estimated retained accelerator bytes | 75,796,186 | 75,796,186 | +| Standalone probe binary, bytes | 289,792 | 292,352 | +| Separately measured CLI binary, bytes | 2,346,496 | 2,349,568 | + +Early pruning reduces transient construction memory; retained accelerator size +did not decrease. Working set is the Windows resident-memory analogue, not a +portable RSS/heap measurement. The retained estimate excludes the FileStore and +other process memory. The old approximately 69 MB FileStore figure was an +internal accounting estimate, not measured process RSS. Cargo lockfile inventory +remains 492 packages (475 external); the optimization added no dependencies. + +## Reproduce and verify + +```powershell +cargo run --release --locked -p lce-benchmarks --example planner_probe -- 1000000 cargo bench -p lce-benchmarks --bench accelerators - -# include the 1 000 000 entry corpus (slow) -LCE_BENCH_1M=1 cargo bench -p lce-benchmarks --bench accelerators +$env:LCE_BENCH_1M = "1" +cargo bench -p lce-benchmarks --bench accelerators +Remove-Item Env:LCE_BENCH_1M +cargo test -p windows-files ``` -See [October 2026 planner measurements](UPGRADE-2026-10.md) for the paired million-record probe, raw measurements and limitations. + +The [probe source](../crates/benchmarks/examples/planner_probe.rs) specifies the +queries, cap and equality oracle. Record commit, target, toolchain, hardware, +OS, background load and process-memory sampling method for new measurements. +Criterion results use a different statistical method and must be reported +separately. Differential/index tests also cover mutation, persistence, long +prefixes, Unicode and capped order; a synthetic probe alone is not coverage of +all real-world names, permissions or journal failures. + +## Raw evidence and limitations + +- [Before CSV](archive/benchmark-history/planner-20261012/planner-before.csv) and [after CSV](archive/benchmark-history/planner-20261012/planner-after.csv). +- [Before process capture](archive/benchmark-history/planner-20261012/planner-before-process.json) and [after process capture](archive/benchmark-history/planner-20261012/planner-after-process.json). +- [Before build output](archive/benchmark-history/planner-20261012/planner-before-build.txt) and [after build output](archive/benchmark-history/planner-20261012/planner-after-build.txt). +- [Original Criterion history](archive/benchmark-history/benchmarks-before-v1.md), retained as historical evidence, not a current comparison. + +The historical prefix/extension comparison reused a different rare-substring +baseline; it cannot support a like-for-like speedup claim. Historical memory +extrapolations are not measured million-file RSS. Real disk scans, ACL behavior, +MFT startup, USN updates and desktop rendering need separate deployment tests. +No new benchmark was run for this documentation-only revision. diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md new file mode 100644 index 0000000..a426918 --- /dev/null +++ b/docs/DEVELOPMENT.md @@ -0,0 +1,46 @@ +# Development + +## Environment + +Use Windows x64, current stable Rust MSVC, Visual Studio C++ Build Tools, Node 22 +(the CI version) and npm. Desktop requires WebView2. Cargo declares Rust 1.82, +but current CI tests stable, not that minimum; minimum-toolchain compatibility +must be checked before a v1.0 promise. Linux is not a supported runtime backend. + +## Build and existing gates + +```powershell +cargo build --release --locked +cargo fmt --all -- --check +cargo clippy --all-targets --all-features -- -D warnings +cargo test +cd apps/desktop +npm ci +npm run lint +npm run typecheck +npm run build +npm run tauri:build +``` + +Workspace output is under `target/release/`, including CLI, MCP and optional +index-service binaries. Installer output is under `target/release/bundle/`. +Default Cargo members keep the desktop dependency stack out of ordinary tests; +the existing desktop CI checks its crate separately. CI also runs CodeForge +Regression Gate. That external **development check** is not a runtime/crate +dependency between the projects and does not relax search tests. + +## Documentation and contribution + +Keep English and Chinese README capability/version statements aligned. Run the +[documentation checks](DOCUMENTATION_CHECKS.md) and verify CLI examples against +`--help`. Release descriptions must distinguish Stable, Experimental and Future. +Follow [Contributing](../CONTRIBUTING.md) for provider and translation changes. +Historical architecture, delivery notes and raw benchmark captures are retained +in the [archive](archive/README.md); they are not current installation guidance. + +## Release readiness + +[Release preparation](RELEASE_PREPARATION.md) lists remaining gates. The draft +[release notes](releases/v1.0.0.md) do not change Cargo/package versions or create +a tag. Desktop build success alone does not establish successful rendering on a +real Windows machine. diff --git a/docs/DOCUMENTATION_CHECKS.md b/docs/DOCUMENTATION_CHECKS.md new file mode 100644 index 0000000..24de796 --- /dev/null +++ b/docs/DOCUMENTATION_CHECKS.md @@ -0,0 +1,43 @@ +# Documentation checks + +Documentation-only changes must preserve runtime source, manifests, lockfiles, +workflows, protocol/data formats, fixtures and tests. + +## Markdown + +Run a pinned standalone checker without adding it to package manifests: + +```bash +npx --yes --package markdownlint-cli@0.45.0 markdownlint "*.md" "docs/**/*.md" ".github/**/*.md" --ignore "docs/archive/**" --disable MD013 MD033 +``` + +The initial tool invocation may download its npm package into the local tool +cache; it is a development check, not a runtime dependency. Line length and inline +HTML are allowed. Legacy archives retain historical formatting and are excluded +from style normalization; they remain included in local link/asset checks. + +## Links, examples and rendering + +Check every tracked Markdown inline/reference link and local image, including +archive paths, exact case on case-sensitive hosts, and heading fragments. +Check external README/docs links and badge/image endpoints; distinguish a +permission-gated reporting page or rate limit from a confirmed missing page. +Do not substitute a successful homepage response for a broken linked resource. + +Validate CLI examples and prerequisites against the current CLI help and source. +Keep source-only commands distinct from published packages. Do not run examples +that install services, apply changes or execute untrusted project commands just +to validate formatting. Check license/version statements against LICENSE, +manifests and published release metadata. + +Render both README tables/diagrams and code blocks using GitHub-flavored Markdown +at desktop and narrow widths. Inspect existing screenshots and captions; a +welcome-screen image is not proof of a successful analysis or platform smoke. +Use a fresh reader to check installation, maturity, evidence and security claims. + +## Existing CI + +Run the checks in [Development](DEVELOPMENT.md) and the original PR workflows, +including CodeForge Regression Gate. Do not change a baseline, fixture, test or +core logic to make a documentation-only check pass. Record unresolved failures +and release blockers in [Release preparation](RELEASE_PREPARATION.md). diff --git a/docs/DOCUMENTATION_MIGRATION.md b/docs/DOCUMENTATION_MIGRATION.md new file mode 100644 index 0000000..b7758e2 --- /dev/null +++ b/docs/DOCUMENTATION_MIGRATION.md @@ -0,0 +1,77 @@ +# Documentation migration inventory + +This inventory records documentation organization, not runtime changes. + +## Canonical structure + +- [ARCHITECTURE.md](ARCHITECTURE.md) +- [QUERY_ENGINE.md](QUERY_ENGINE.md) +- [INDEXING.md](INDEXING.md) +- [BENCHMARKS.md](BENCHMARKS.md) +- [SECURITY.md](SECURITY.md) +- [PROVIDERS.md](PROVIDERS.md) +- [DEVELOPMENT.md](DEVELOPMENT.md) +- [ROADMAP.md](ROADMAP.md) + +## Moved files + +| Previous path | Current path | +|---|---| +| `docs/UPGRADE-2026-10.md` | `docs/archive/upgrade-2026-10.md` | +| `docs/README_EVIDENCE.md` | `docs/archive/development-notes/readme-evidence-2026-10-08.md` | +| `docs/measurements/planner-after-build.txt` | `docs/archive/benchmark-history/planner-20261012/planner-after-build.txt` | +| `docs/measurements/planner-after-process.json` | `docs/archive/benchmark-history/planner-20261012/planner-after-process.json` | +| `docs/measurements/planner-after.csv` | `docs/archive/benchmark-history/planner-20261012/planner-after.csv` | +| `docs/measurements/planner-before-build.txt` | `docs/archive/benchmark-history/planner-20261012/planner-before-build.txt` | +| `docs/measurements/planner-before-process.json` | `docs/archive/benchmark-history/planner-20261012/planner-before-process.json` | +| `docs/measurements/planner-before.csv` | `docs/archive/benchmark-history/planner-20261012/planner-before.csv` | + +## Added files and preserved snapshots + +- `.markdownlint.json` +- `THIRD_PARTY_LICENSES.md` +- `docs/DEVELOPMENT.md` +- `docs/DOCUMENTATION_CHECKS.md` +- `docs/INDEXING.md` +- `docs/PROVIDERS.md` +- `docs/QUERY_ENGINE.md` +- `docs/README.md` +- `docs/RELEASE_PREPARATION.md` +- `docs/ROADMAP.md` +- `docs/SECURITY.md` +- `docs/archive/README.md` +- `docs/archive/benchmark-history/benchmarks-before-v1.md` +- `docs/archive/development-notes/architecture-before-v1.md` +- `docs/archive/development-notes/changelog-before-v1.md` +- `docs/archive/development-notes/readme-before-v1.md` +- `docs/archive/development-notes/readme-zh-before-v1.md` +- `docs/archive/development-notes/security-before-v1.md` +- `docs/archive/development-notes/upgrade-delivery-2026-10.md` +- `docs/releases/v1.0.0.md` +- `docs/DOCUMENTATION_MIGRATION.md` (this inventory). + +## Rewritten entry points + +README, CHANGELOG and canonical guides now describe user workflows and current +limits. Earlier README/architecture/security/changelog content remains in the +archive. Existing supporting references remain available. Relative links were +updated to follow moved files. + +## Removed from current narrative + +- PR-by-PR delivery logs and phase plans moved out of the user entry points. +- Unsupported performance comparisons and absolute safety/privacy claims were corrected. +- Obsolete safe-fix/version statements were replaced with current limitations. + +No historical report, raw measurement, screenshot, core source, fixture or test +was intentionally deleted. File paths retired by moves are listed above. +Raw moved CSV/JSON/text captures were compared with their original Git contents +and retain identical content (allowing repository newline normalization). + +## Verification scope + +Markdown lint covers current guides and templates; archives retain historical +formatting. Link checks include archives and image targets. README GFM renders +were checked at desktop and narrow widths. Existing Rust tests/format/Clippy +were run without modifying tests or baseline. Original PR CI and Regression Gate +remain required. See [release preparation](RELEASE_PREPARATION.md) for blockers. diff --git a/docs/DSL.md b/docs/DSL.md index 1f47352..9909bae 100644 --- a/docs/DSL.md +++ b/docs/DSL.md @@ -9,7 +9,7 @@ natural language, and so that whatever compiles a query has one obvious target. ```text query := token* token := key ":" value | word -key := type | ext | path | name | drive | modified | created | size +key := type | ext | path | name | prefix | drive | modified | created | size | state | pid | user | visible | sort | limit value := quoted | bare ``` @@ -26,6 +26,7 @@ than a syntax error. | `ext` | `rs`, `.rs`, `pdf,toml` | extension, without or with the leading dot | | `path` | any substring, quote when it has spaces | full-path substring | | `name` | any substring | entity-name substring | +| `prefix` | any name prefix | full prefix verified after candidate lookup | | `drive` | a letter | restrict to one volume | | `modified` | `<24h`, `>7d`, `>=2024-01-01`, `<=2024-01-01` | last-write time | | `created` | same operators as `modified` | creation time | @@ -93,4 +94,4 @@ DSL, the stage that produced it and the rules that fired. Text matching itself requires every query token to match, against the entity name first and then against its path. That is why `type:file ext:rs modified:<24h` matches nothing when no Rust file changed today -rather than quietly dropping the predicate. \ No newline at end of file +rather than quietly dropping the predicate. diff --git a/docs/INDEXING.md b/docs/INDEXING.md new file mode 100644 index 0000000..802d19d --- /dev/null +++ b/docs/INDEXING.md @@ -0,0 +1,66 @@ +# Indexing + +## Stored data + +The file store keeps names and metadata and reconstructs full paths from parent +relationships. Accelerators retain extension postings, prefix keys and selected +full-path trigrams. File contents, embeddings and a vector database are not part +of this index. Settings and optional bounded usage data are also stored locally; +protect the cache as sensitive metadata. See [Security](SECURITY.md). + +## Backends + +On first run, an empty `indexedDrives` setting selects every fixed local volume, +with a default ceiling of 1,000,000 entries including directories. Permission +failures and that ceiling can make the index incomplete. Settings live in +`%LOCALAPPDATA%\LocalContextEngine\settings.json`, alongside local cache/usage +data. To restrict the local index, set `indexedDrives` to drive letters such as +`["C"]` in the existing settings file (preserving its other fields), restart the +client and rebuild the index. Desktop settings expose backend and entry-limit +controls; drive selection is configured in the JSON file. +`path:` filters query results; they do not restrict collection. The CLI has no +`--root` option. Optional service-owned indexing has its own deployment scope; +client settings must not be treated as a service permission boundary. + +| Backend | Behavior | Limitations | +|---|---|---| +| Directory scan | Walk readable directories under the current account | Filesystem I/O cost; inaccessible paths omitted; links/junctions not followed | +| NTFS MFT/USN | Enumerate names and replay journal mutations | Elevated volume access; NTFS-specific; raw metadata may expose names beyond normal directory ACL visibility | +| Optional index service | Own privileged volume handles and per-volume workers | Experimental deployment boundary; not installed by a normal CLI build | + +`auto`, `scan` and `mft-usn` are backend choices. Availability and privilege +influence selection; inspect `--index-status` rather than assuming MFT was used. +MFT file-size metadata is limited; a size-dependent query is not equivalent to +reading file contents or stat-ing every file. + +## Journal consistency + +Volume identity uses a volume GUID with serial/mount metadata, not only a drive +letter. Each volume has its own journal ID, window and cursor. A missing cursor, +recreated journal or out-of-window cursor requires a rebuild. Offline or denied +volumes are reported separately. Workers use bounded journal reads and backoff; +mutation batches update the store and accelerators together. `Changes` reports +volume journal advances, not a complete per-file event API. + +The cache uses versioned MessagePack with named fields (current schema 3). +Incompatible state is rebuilt rather than guessed at. A stale/degraded state is +visible health information, not a guarantee of current results. + +## Resource use + +Common trigram postings are pruned during construction; dropped-trigram state +survives updates and persistence. Prefix capacity is compacted, and path caching +retains directory strings. These reduce transient allocations without a new +compressed posting format. Retained index estimates and process working set +measure different things; [Benchmarks](BENCHMARKS.md) reports both. + +```powershell +localsearch --index-status +localsearch --rebuild-index +localsearch --update-index +localsearch --service-status +``` + +Rebuild/update modify local index state, not source files. Follow the +[service security boundary](SECURITY.md#mft-and-permissions) before installing +an elevated service. diff --git a/docs/MCP.md b/docs/MCP.md index d39e068..afbe054 100644 --- a/docs/MCP.md +++ b/docs/MCP.md @@ -1,7 +1,8 @@ # MCP server -`localsearch-mcp.exe` speaks the Model Context Protocol over stdio, so an agent -can ask this machine a question without any network access at all. +`localsearch-mcp.exe` speaks the Model Context Protocol over local stdio. The +server's search needs no network access. Your MCP client or model may forward +results to another service; its network and privacy settings are separate. ## Running it @@ -92,4 +93,4 @@ Response (abridged): tool error. Neither can panic the server. * `terminate_process` exists because the spec asks for a complete action surface, and it is deliberately awkward: the caller has to pass - `confirm: true`, and the Rust side refuses everything else. \ No newline at end of file + `confirm: true`, and the Rust side refuses everything else. diff --git a/docs/PROVIDERS.md b/docs/PROVIDERS.md new file mode 100644 index 0000000..d27fe05 --- /dev/null +++ b/docs/PROVIDERS.md @@ -0,0 +1,32 @@ +# Providers + +Providers collect normalized entities; the shared ranker scores them. They do +not contain UI logic. The actual interface is +[EntityProvider](../crates/search-core/src/provider.rs); consult source before +implementing it rather than copying an obsolete trait declaration. + +| Provider | Source | Important limits | +|---|---|---| +| Files/directories | Directory scan or NTFS MFT/USN index | Metadata only; health, permission and backend limits apply | +| Applications | App Paths, Start Menu, PATH and WindowsApps discovery | Discovery is not a complete package inventory | +| Processes | Live Windows process snapshots | Protected processes and optional account lookup restrict metadata | +| Services | Windows service snapshots | Read-only inspection; no service-management command | +| Windows | Top-level window snapshots | Visibility/ownership filters; not accessibility-tree or browser-tab search | + +```powershell +localsearch --providers +localsearch "type:app vscode" +localsearch "type:process python" +localsearch "type:service state:running" +localsearch "type:window github" +``` + +A snapshot may change immediately after a query. Failed or unavailable providers +report health; an empty result does not establish that the operating system has +no such object. Opening a result and confirmed process termination are separate +user actions, described in [Security](SECURITY.md). + +To extend a domain, implement the existing provider boundary and register it in +SearchService. Keep filtering/ranking/presentation responsibilities separate and +add deterministic fixture/error tests. The repository has no provider dependency +on another project. Full-text and semantic providers remain future extensions. diff --git a/docs/QUERY_ENGINE.md b/docs/QUERY_ENGINE.md new file mode 100644 index 0000000..89f34eb --- /dev/null +++ b/docs/QUERY_ENGINE.md @@ -0,0 +1,49 @@ +# Query engine + +## Query meaning + +Explicit DSL filters, rule-based English/Chinese input and plain text compile +into SearchQuery. Recognized DSL takes precedence; unknown key-like text remains +search text. This is not general natural-language understanding. See the +[DSL reference](DSL.md) for supported syntax and units. + +## Candidate planning + +The file provider offers extension postings, a sorted prefix table and trigrams +over normalized full paths. The planner orders available sources by cardinality, +chooses a selective seed and intersects useful sources. A broad additional source +may be skipped when verifying the smaller seed costs less. Short tokens and +common trigrams do not discard a useful indexed condition in the same query. + +The current candidate-planning budget is 250,000 IDs; a source more than eight +times the current seed can be skipped. These are implementation cost heuristics, +not DSL semantics or stable API promises. No selective source means linear +fallback. Missing trigrams can establish an empty candidate set; deliberately +pruned common trigrams cannot. Prefix keys retain 24 bytes; longer queries still +verify the full prefix. Accelerated IDs are sorted and deduplicated before the +same authoritative match/filter logic used by the reference path. + +## Ranking and explanation + +Ranking uses named heuristic components for name, prefix, tokens, path, fuzzy, +recency and optional usage. `--explain` exposes compilation and plan/ranking +information; `--providers` and `--index-status` expose provider health. + +```powershell +localsearch --explain "ext:pdf report" +localsearch --explain "prefix:read" +localsearch --json "type:process python" +``` + +Different snapshots, clocks, usage history or fuzzy settings can change results. +Stable ordering applies to identical inputs. Broad/common-only or fuzzy queries +may still scan; see [Benchmarks](BENCHMARKS.md). Completeness beyond a requested +result/candidate budget is not promised. + +## Correctness checks + +Tests compare accelerated and linear results, including order and caps; they +cover absent/common trigrams, combined conditions, Unicode and long prefixes. +Mutation/persistence tests protect accelerator consistency. Performance changes +must preserve these checks; an incomplete index can lose matches and is not safe +merely because candidates are later verified. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..7a05ce8 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,17 @@ +# Documentation + +Start with the [project overview](../README.md) or [简体中文](../README.zh-CN.md). + +| Guide | Question answered | +|---|---| +| [Architecture](ARCHITECTURE.md) | How are responsibilities and failure boundaries separated? | +| [Query Engine](QUERY_ENGINE.md) | How are candidates planned and verified? | +| [DSL](DSL.md) | Which filters and query forms are supported? | +| [Indexing](INDEXING.md) | What is stored, refreshed and rebuilt? | +| [Providers](PROVIDERS.md) | Which Windows objects can be discovered? | +| [Benchmarks](BENCHMARKS.md) | What was measured, and what was not? | +| [Security](SECURITY.md) | What can leave the process or cross a permission boundary? | +| [MCP](MCP.md) | How can a local client query the engine? | +| [Development](DEVELOPMENT.md) | How do I build, test and contribute? | +| [Roadmap](ROADMAP.md) | What is Stable, Experimental or Future? | +| [Release preparation](RELEASE_PREPARATION.md) | What blocks the v1.0.0 draft from publication? | diff --git a/docs/RELEASE_PREPARATION.md b/docs/RELEASE_PREPARATION.md new file mode 100644 index 0000000..6a4bd0c --- /dev/null +++ b/docs/RELEASE_PREPARATION.md @@ -0,0 +1,33 @@ +# v1.0.0 release preparation + +**Status: draft, not ready to tag solely on documentation approval.** Runtime, +frontend and desktop package versions remain 0.2.0. This documentation change +does not create a compatibility guarantee, tag or release binary. + +## Prepared in this documentation stage + +- User-facing README, canonical guides and Stable/Experimental/Future labels. +- Evidence-preserving history archive and honest performance methodology. +- [Changelog](../CHANGELOG.md) and [user-facing release draft](releases/v1.0.0.md). +- [Documentation verification procedure](DOCUMENTATION_CHECKS.md). + +## Required before publication + +- Merge the reviewed documentation PR only after its original CI and Regression Gate are green. +- Align Cargo/workspace, frontend and Tauri package metadata to 1.0.0 in a separately reviewed release change, including generated lockfile version entries as needed. +- Validate CLI/protocol/cache compatibility and state the supported platform/toolchain policy. +- Build final artifacts from the exact reviewed tag candidate; verify installation, uninstall, version output and published checksums. +- Review license notices against final dependency inventories and bundled artifacts; optional external tools are not licensed by this project. + +- Validate Rust 1.82 or revise the declared minimum in a separate code/release change. +- Resolve desktop package/lockfile root license metadata (MIT) against repository LICENSE and Cargo metadata (MPL-2.0); no separate desktop LICENSE was found. +- Repeat the million-record probe with complete hardware/target metadata; report all queries and process memory separately from index estimates. +- Validate desktop rendering on a clean Windows/WebView2 installation; historical notes include a blank-window observation. +- Exercise optional service installation/uninstallation, named-pipe ACLs, multi-volume journal recovery and filename visibility under the intended user accounts. + +## Release handling + +The GitHub draft uses the proposed `v1.0.0` name but remains unpublished. Do not +push that tag or publish the draft until all release gates above have recorded +evidence. The draft contains no fabricated binary/validation claim. Passing a +documentation PR does not resolve packaging, MSRV or platform-acceptance work. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md new file mode 100644 index 0000000..77af43c --- /dev/null +++ b/docs/ROADMAP.md @@ -0,0 +1,33 @@ +# Roadmap + +## Stable + +CLI and local MCP discovery, explicit DSL, rule-based query compilation, +explainable ranking, indexed file candidate planning, unprivileged scan and live +Windows providers have implementations and regression coverage. Stability here +is scoped to those workflows and supported environments, not a v1 API promise. + +## Experimental + +The optional privileged MFT/USN service and Desktop need deployment acceptance +across the intended Windows/filesystem/WebView2 matrix. MFT filename exposure, +missing file-size metadata, per-volume journal health and shared-user access +must remain documented. The changes feed is volume-level, not per-file. + +## v1.0 release gates + +- Align Cargo, desktop package and installer versions in a separate release change. +- Validate the declared Rust minimum or document a newly tested minimum. +- Run existing CI, Regression Gate, package checksums and fresh-install smoke tests. +- Capture real desktop search behavior: an earlier environment showed a blank + WebView; a successful installer build does not close that observation. +- Validate service installation, ACLs, fallback, journal rollover and uninstall + on representative machines before describing the service as stable. + +## Future + +Evaluate compressed postings only when retained memory justifies decoding, +mutation and cache-format costs. Improve metadata completeness and application +package discovery. Any full-text, semantic or model-based compiler extension +must be explicit opt-in and preserve the filename DSL, permission checks and +default offline path. These are directions, not scheduled capabilities. diff --git a/docs/SECURITY.md b/docs/SECURITY.md new file mode 100644 index 0000000..f645915 --- /dev/null +++ b/docs/SECURITY.md @@ -0,0 +1,46 @@ +# Security and privacy + +## Local data and network boundary + +Default search runs locally, with no account, telemetry, source upload or model +call. It indexes names, paths and metadata, not file contents. Settings, index +caches and optional bounded selection history are local data; filenames alone +can disclose sensitive information. Usage history can be cleared with +`localsearch --reset-usage`. MCP communicates over local stdio, but the MCP +client controls where it forwards returned results. Opening an application or +file may start software with its own network behavior. Build/package downloads +are separate from offline runtime search. + +## MFT and permissions + +Ordinary directory scanning uses the current user's access rights. Raw NTFS MFT +enumeration requires elevated volume access and **can disclose names inside +folders the searching user cannot normally list**. Opening file contents still +goes through Windows access checks. If this metadata exposure is unacceptable, +use unprivileged `scan` and do not install the index service. + +The optional service owns raw handles and journal workers; CLI/Desktop/MCP can +remain ordinary-user processes. Service installation/start/stop are explicit +administrative actions. The local named pipe has an explicit DACL for SYSTEM, +Administrators and the installing user's SID, rejects remote clients and uses +first-instance creation. A permitted same-user process can also connect; this +is not caller attestation or an OS sandbox. + +The protocol has five requests: Hello, Search, IndexStatus, RebuildVolume and +Changes. It has no shell, execute or file-content request. Frames have magic and +version checks and an 8 MiB allocation ceiling. Search limits are bounded. A +protocol mismatch or service failure is surfaced and the user-side provider can +fall back; confirm the actual backend and health after failure. + +## Actions and untrusted input + +Query text is data, not a shell command. MCP arguments and pipe frames are +validated. Directory scans do not follow junctions/symlinks. File opening and +folder reveal are explicit actions. Process termination requires confirmation +at the interface and Rust action boundary; it can still lose unsaved work. +Services are inspected, not controlled, by the search provider. + +A raw index is not a per-result authorization system. Protect local cache files +and review service deployment on shared machines. The project does not claim a +complete independent security audit. Report vulnerabilities using the root +[security policy](../SECURITY.md). diff --git a/docs/archive/README.md b/docs/archive/README.md new file mode 100644 index 0000000..e1c59ca --- /dev/null +++ b/docs/archive/README.md @@ -0,0 +1,18 @@ +# Documentation archive + +Historical material is retained for provenance and technical evidence. It can +describe obsolete behavior, earlier plans or claims corrected by current docs. +Start with the [documentation index](../README.md) for supported usage. + +| Material | Location | +|---|---| +| October 2026 audit and staged implementation | [Upgrade record](upgrade-2026-10.md) | +| Original README, architecture, security and changelog | [Development notes](development-notes) | +| Cross-project delivery report, copied without deleting the original | [Delivery evidence](development-notes/upgrade-delivery-2026-10.md) | +| Earlier reports and raw samples | [Benchmark history](benchmark-history) | + +Original benchmark CSV/JSON/text data and screenshots are preserved. Relative +Markdown links were adjusted for moved files. Historical statements are not +silently rewritten into new evidence; current guides explain corrected claims. +The delivery report mentions both projects as historical context, not a runtime +or crate dependency of Local Context Engine. diff --git a/docs/archive/benchmark-history/benchmarks-before-v1.md b/docs/archive/benchmark-history/benchmarks-before-v1.md new file mode 100644 index 0000000..4a508d3 --- /dev/null +++ b/docs/archive/benchmark-history/benchmarks-before-v1.md @@ -0,0 +1,276 @@ +> Historical snapshot. Claims and versions on this page are not current guidance. + +# Benchmarks + +Every number on this page was measured on the machine described below with the +commands shown at the bottom. Nothing here is estimated, projected or copied +from a different workload. + +## Measured on + +| | | +|---|---| +| CPU | AMD Ryzen 7 255 H (8 cores / 16 threads, Zen 4c class) | +| RAM | 22.8 GB | +| Disk | KY3000-512GB, NVMe SSD | +| OS | Windows 11 Home (build 26200) | +| Toolchain | `rustc 1.98.1`, `stable-x86_64-pc-windows-gnu` | +| Profile | `[profile.bench]` — `opt-level = 3`, `lto = "thin"`, `codegen-units = 1` | +| Corpus | synthetic, deterministic, seed `0x123456789ABCDEF0` | +| Harness | Criterion 0.5, 20 samples, 6 s measurement, 1 s warm-up | + +The corpus is generated by `lce_benchmarks::synthetic_store`: realistic project +tree names drawn from a fixed word list, roughly one directory per six entries, +files in the 0–50 MB range with plausible timestamps. See +[`crates/benchmarks/src/lib.rs`](../../../crates/benchmarks/src/lib.rs). + +## File index search + +`file_search/*` measures `windows_files::collect_candidates` — the exact +function `FileProvider::collect` calls. Times are Criterion's median estimate. + +| benchmark | 10 000 | 100 000 | 1 000 000 | +|---|---:|---:|---:| +| `exact` ("readme") | 1.03 ms | 8.11 ms | 8.29 ms | +| `prefix` ("re") | 1.32 ms | 3.63 ms | 3.75 ms | +| `substring` ("port") | 871 µs | 7.54 ms | 7.79 ms | +| `rare_substring` ("zzqxjw") | 580 µs | 6.11 ms | 63.4 ms | +| `fuzzy` ("rpt") | 724 µs | 7.46 ms | 76.1 ms | +| `filters` (`ext:pdf` + `drive:C` + `size:>1mb`) | 544 µs | 5.92 ms | 62.4 ms | +| `path_filter` (`path:docs`) | 889 µs | 9.99 ms | – | +| `sort` (`sort:size-desc`) | 857 µs | 7.58 ms | 7.46 ms | + +### Why some rows barely change between 100 000 and 1 000 000 + +That is not a measurement mistake, it is the candidate cap doing its job. + +`collect_candidates` stops after `CANDIDATE_CAP = 4 000` matches. Queries like +`readme`, `re` and `port` match thousands of entries in the synthetic corpus, so +the scan terminates early once it has enough work for the ranker. The cost stops +growing with the index. + +The rows that **do** grow are the interesting ones: + +* `rare_substring` and `filters` match almost nothing, so they must visit every + record. **A full scan of one million entries costs 62–63 ms.** +* `fuzzy` additionally runs the subsequence matcher on every candidate and + costs **76 ms** for one million entries. + +So the honest summary is: + +| case | 1 000 000 entries | +|---|---| +| enough matches to hit the cap (the common typing case) | **~8 ms** | +| full scan required (rare term, or an uncommon filter) | **~62 ms** | +| full scan plus fuzzy matching on every record | **~76 ms** | + +## Ranking + +`ranking/*` measures `HeuristicRanker::score` over the candidate set the file +index produced. The set is capped at 4 000, so this cost is bounded regardless +of index size. + +| benchmark | 10 000 | 100 000 | 1 000 000 | +|---|---:|---:|---:| +| `heuristic` (name only) | 346 µs | 3.06 ms | 2.75 ms | +| `with_usage` (usage boost enabled) | 532 µs | 3.68 ms | – | + +The three numbers per row are close because the *candidate count* is what +matters, not the index size: all three corpora produce 4 000 candidates. + +## End to end + +`end_to_end/query` is the full hot path: walk the index, collect candidates, +rank them. + +| entries | median | +|---|---:| +| 10 000 | 1.28 ms | +| 100 000 | 10.59 ms | +| 1 000 000 | 10.40 ms | + +## Query compilation + +The compiler runs before every search. It is microseconds, which is why it can +sit on the keystroke path without a second thought. + +| input | median | +|---|---:| +| `query_compiler/dsl` (`type:file ext:rs modified:<24h sort:size-desc`) | 1.37 µs | +| `query_compiler/plain` (`vscode`) | 10.6 µs | +| `query_compiler/nl-process` (`正在运行的 python`) | 10.7 µs | +| `query_compiler/nl-zh` (`最近的 pdf`) | 11.2 µs | +| `query_compiler/nl-en` (`recent PDFs`) | 11.6 µs | + +Plain text is slower than DSL because the natural language rules are attempted +before the input falls through to a plain keyword query. + +## Index construction + +`file_store/load` measures building the in-memory store from the corpus, which +is what the directory-scan backend does after `read_dir`: + +| entries | median | throughput | +|---|---:|---:| +| 10 000 | 2.88 ms | 3.5 M entries/s | +| 100 000 | 30.5 ms | 3.3 M entries/s | +| 1 000 000 | 316 ms | 3.2 M entries/s | + +`file_store/extension_lookup/1000000` walks every record and reads its +extension: **10.7 ms**. `file_store/stats/100000` computes file/directory counts +and the memory estimate: **103 µs**. + +## Memory + +Measured with `FileStore::memory_bytes()` on a fully populated store, not +estimated: + +```text +1_000_000 entries + records : 1_000_000 × 48 B = 48.0 MB + arena : 21.4 MB (names, UTF-8) + ------------------------------------ + total : ≈ 69 MB +``` + +That is what the desktop UI reports in Settings → Index. + +## Reproducing + +```bash +cargo bench -p lce-benchmarks +``` + +Individual suites, or a subset of the corpus sizes: + +```bash +cargo bench -p lce-benchmarks --bench file_search +cargo bench -p lce-benchmarks --bench ranking +cargo bench -p lce-benchmarks --bench file_search -- 100000 +``` + +Criterion writes an HTML report to `target/criterion/report/index.html`. + +## Caveats + +* These numbers describe the **search core**, not the Windows APIs. A fresh + process table, service list or application catalogue is not in these tables; + see the status bar in the app, or `localsearch --providers`, for live figures + on your own machine. +* The directory-scan backend additionally pays `read_dir` and a `stat` per + entry; the MFT backend does not. Both then produce the same in-memory store, + which is what is measured here. +* Numbers come from one machine. Run the benchmarks yourself before drawing + conclusions about yours; `docs/BENCHMARKS.md` is a snapshot, not a promise. +* The first iteration of any benchmark pays page faults on a cold corpus. The + harness warms up for a second per benchmark to damp that. +--- + +# v0.2: accelerators and the query planner + +The numbers above are the **before** picture: every query walked the whole +index. v0.2 adds an extension index, a prefix table and a trigram index, and a +planner that picks between them. + +Measured on the same machine and toolchain as the table above, against the same +synthetic corpus, with `cargo bench -p lce-benchmarks --bench accelerators` +(Criterion, 10 samples, 3 s measurement). + +## The headline: selective queries no longer scan + +| query (100 000 entries) | plan | v0.1 linear | v0.2 | change | +|---|---|---:|---:|---:| +| `design-99999` (rare substring) | **trigram** | 4.48 ms | **31.6 µs** | ~140× faster | +| `design-99999` (10 000 entries) | **trigram** | 430 µs | **1.77 µs** | ~240× faster | +| `prefix:read` | **prefix** | 4.48 ms¹ | 2.94 ms | ~1.5× faster | +| `ext:pdf` | **extension** | 4.48 ms¹ | 2.60 ms | ~1.7× faster | + +¹ The linear column for the prefix and extension rows is the same full-scan +`rare_substring` measurement, because in v0.1 every predicate was evaluated by +walking the index. The v0.2 rows include the planner run itself. + +## What did **not** get faster + +This is the part that matters as much as the win. These queries still plan as +`linear`, and they cost what they cost in v0.1 — no regression, no +improvement: + +| query (100 000 entries) | plan | v0.2 median | +|---|---|---:| +| `readme` (exact) | linear | 9.39 ms | +| `report` (common substring) | linear | 8.70 ms | +| `rpt` (fuzzy) | linear | 8.37 ms | +| `report` + `ext:pdf` (combined) | linear | 7.29 ms | +| `zzqxjw-nothing-matches` (negative) | linear | 4.48 ms | + +Why: + +* **Common trigrams are dropped.** `readme` and `report` consist of trigrams + that appear in a large fraction of the corpus. Indexing them would cost + memory and narrow nothing, so they are pruned; a query whose every trigram was + pruned has no candidate source and falls back to the scan. +* **Three characters is the floor.** Fuzzy matching runs against short needles + where a trigram index has nothing to offer, and the scan is the honest answer. +* **`combined` inherits the fallback.** If any one token of a multi-token query + cannot be accelerated, the whole query takes the linear path. Intersecting the + extension postings with a scan would be a worthwhile follow-up. + +## What the accelerators cost + +| measurement | 10 000 | 100 000 | +|---|---:|---:| +| build (`accelerated/build`) | 35.9 ms | 568 ms | +| planning only (`accelerated/planning`) | 1.67 µs | 1.42 µs | + +The build cost is the price of indexing every record's **full path**: ~57 bytes +per record at these sizes, so ~58 trigram insertions per record. It is paid once +and persisted; the planner itself is microseconds. + +Memory, measured with `SearchAccelerators::stats()` on a 100 000 entry corpus: + +```text +100 000 entries + FileStore 9.96 MB + SearchAccelerators 5.95 MB (+60%) + ------------------------------------ + total 15.91 MB + + distinct trigrams 1 223 + trigrams dropped 147 + trigram postings 579 441 + prefix table entries 100 001 (100 000 records + the volume root) + distinct extensions 1 +``` + +The prefix table is the largest single contributor at 32 bytes per record; the +trigram postings are ~5.8 per record. + +**This is a real cost, not a rounding error.** Extrapolating linearly, a one +million entry index lands around 125 MB rather than the ~69 MB measured in v0.1, +which is past the project's 100 MB goal. The v0.2 run was made at 10 000 and +100 000 entries; the million-entry case is available with +`LCE_BENCH_1M=1 cargo bench -p lce-benchmarks --bench accelerators`, and it is +not run by default because building the accelerators for a million paths takes +roughly six seconds and the suite would take an hour. + +Planned mitigations, in the order they are worth doing: + +1. Drop the prefix table when trigrams are available — a prefix of three or more + characters is already answerable as a trigram intersection, and the table is + 32 bytes per record. +2. Delta-encode the posting lists, which are sorted and dense. +3. Shrink `PREFIX_KEY_LEN` from 24 bytes to 12, halving the table. + +## Reproducing + +```bash +# the "before" measurement +cargo bench -p lce-benchmarks --bench file_search + +# the "after" measurement +cargo bench -p lce-benchmarks --bench accelerators + +# include the 1 000 000 entry corpus (slow) +LCE_BENCH_1M=1 cargo bench -p lce-benchmarks --bench accelerators +``` +See [October 2026 planner measurements](../upgrade-2026-10.md) for the paired million-record probe, raw measurements and limitations. diff --git a/docs/measurements/planner-after-build.txt b/docs/archive/benchmark-history/planner-20261012/planner-after-build.txt similarity index 100% rename from docs/measurements/planner-after-build.txt rename to docs/archive/benchmark-history/planner-20261012/planner-after-build.txt diff --git a/docs/measurements/planner-after-process.json b/docs/archive/benchmark-history/planner-20261012/planner-after-process.json similarity index 100% rename from docs/measurements/planner-after-process.json rename to docs/archive/benchmark-history/planner-20261012/planner-after-process.json diff --git a/docs/measurements/planner-after.csv b/docs/archive/benchmark-history/planner-20261012/planner-after.csv similarity index 100% rename from docs/measurements/planner-after.csv rename to docs/archive/benchmark-history/planner-20261012/planner-after.csv diff --git a/docs/measurements/planner-before-build.txt b/docs/archive/benchmark-history/planner-20261012/planner-before-build.txt similarity index 100% rename from docs/measurements/planner-before-build.txt rename to docs/archive/benchmark-history/planner-20261012/planner-before-build.txt diff --git a/docs/measurements/planner-before-process.json b/docs/archive/benchmark-history/planner-20261012/planner-before-process.json similarity index 100% rename from docs/measurements/planner-before-process.json rename to docs/archive/benchmark-history/planner-20261012/planner-before-process.json diff --git a/docs/measurements/planner-before.csv b/docs/archive/benchmark-history/planner-20261012/planner-before.csv similarity index 100% rename from docs/measurements/planner-before.csv rename to docs/archive/benchmark-history/planner-20261012/planner-before.csv diff --git a/docs/archive/development-notes/architecture-before-v1.md b/docs/archive/development-notes/architecture-before-v1.md new file mode 100644 index 0000000..0d054fc --- /dev/null +++ b/docs/archive/development-notes/architecture-before-v1.md @@ -0,0 +1,368 @@ +> Historical snapshot. Claims and versions on this page are not current guidance. + +# Architecture + +## The one-sentence version + +Natural language goes in, a deterministic query comes out, and a Rust search +core answers it from local data — no model, no network and no database on the +hot path. + +## The pipeline + +```mermaid +flowchart TD + U["User / AI"] --> QC["Query Compiler
DSL + rule-based NL"] + QC --> SQ["SearchQuery"] + SQ --> SC["Search Core"] + + SC --> F["Files"] + SC --> P["Processes"] + SC --> A["Apps"] + SC --> S["Services"] + SC --> W["Windows"] + + F --> MFT["MFT / USN"] + F --> SCN["Directory scan"] + P --> TH["Toolhelp32 + PSAPI"] + A --> REG["Registry + Start Menu + PATH"] + S --> SCM["EnumServicesStatusEx"] + W --> EW["EnumWindows"] + + MFT --> R["Ranking"] + SCN --> R + TH --> R + REG --> R + SCM --> R + EW --> R + R --> RES["SearchResponse"] + RES --> CLI["localsearch.exe"] + RES --> MCP["MCP server"] + RES --> UI["Tauri desktop"] +``` + +## Why `AI != Search Engine` + +The tempting design is to hand the whole question to a model: embed the query, +embed the file names, ask a vector database for the nearest few. It fails for +the thing a desktop search has to be good at. + +* **It is not deterministic.** The same query has to return the same list in the + same order, or the muscle memory you build after a week of use is worthless. +* **It is not instant.** A local embedding pass plus a vector lookup is tens to + hundreds of milliseconds before you have even started ranking. +* **It is not auditable.** "Why is this file above that one?" has to have an + answer a person can read. A cosine similarity is not an answer. +* **It is not private by construction.** The moment a model is on the query + path, the question "could this call out to the network?" is back. + +So the model, if there is one, goes **here**: + +```text +User ──▶ Query Compiler ──▶ SearchQuery ──▶ Search Core ──▶ Results + ▲ + │ + an AI may replace this box +``` + +The compiler's whole job is translation: "最近的 pdf" becomes +`type:file ext:pdf sort:modified-desc`. Retrieval is untouched by it. Ship the +MVP with a rule-based compiler (this repository does), and a future provider can +swap in a model behind the same `QueryCompiler` trait without the engine +noticing. + +## Crate boundaries + +```text +apps/desktop ────────────────────────────────┐ +crates/search-cli ─────┐ │ +crates/search-mcp ─────┼──▶ crates/search-daemon ──▶ crates/search-core + │ │ ▲ + │ ├─▶ crates/query-dsl ┘ + │ ├─▶ crates/ranking ┘ + │ └─▶ crates/windows-{files,processes,apps,services,windows} + │ + └── all three call exactly one `SearchService` +``` + +| crate | responsibility | must not know about | +|---------------------|---------------------------------------------------------|------------------------------| +| `search-core` | entity/query/result model, engine, fuzzy matcher | Win32, Tauri, CLI | +| `query-dsl` | DSL parser, rule-based NL compiler | the OS, ranking | +| `ranking` | heuristic scoring, usage memory | providers, presentation | +| `windows-files` | index backends, persistence | other domains | +| `windows-processes` | process snapshots | other domains | +| `windows-apps` | application discovery | other domains | +| `windows-services` | service snapshots (read-only) | other domains | +| `windows-windows` | window snapshots | other domains | +| `search-daemon` | composition root, settings, actions | UI frameworks | +| `search-cli` | `localsearch.exe` | search logic | +| `search-mcp` | MCP/JSON-RPC transport, tool schemas | search logic | +| `apps/desktop` | Tauri shell, React UI, i18n | search logic | + +The rule that keeps this honest: **if two front ends would need the same +decision, that decision belongs in `search-daemon` or below it.** Ranking, +filtering, compiling, opening results and terminating processes each exist +exactly once. + +## The `EntityProvider` boundary + +```rust +pub trait EntityProvider: Send + Sync + Debug { + fn name(&self) -> &'static str; + fn scope(&self) -> SnapshotScope; + fn entity_types(&self) -> &'static [EntityType]; + fn collect(&self, query: &SearchQuery) -> Result>; + fn stats(&self) -> ProviderStats; + fn refresh(&self) -> Result<()>; +} +``` + +A provider is the only place that knows how to read one domain out of the +operating system. It returns normalised entities and nothing else — no ranking, +no formatting, no presentation concerns. Adding a domain means adding one +provider; it does not mean touching the engine, the CLI, the MCP server or the +UI's search logic. + +## The ranker boundary + +```rust +pub trait Ranker: Send + Sync + Debug { + fn score(&self, query: &SearchQuery, entity: &LocalEntity) -> Option; +} +``` + +`Some` means "this answers the text"; `None` drops the candidate. The bundled +`HeuristicRanker` sums named, weighted components +(`exact_match` 100, `prefix_match` 60, `filename_match` 40, `token_match` 30, +`path_match` 20, `fuzzy_match` 10, `usage_frequency` 8, `recency` 5, plus a +per-type preference), which is what makes `--explain` able to show its work. + +## File indexing + +```text + ┌──────────────────────────────┐ + \\.\C: ─────▶ │ mft::build │ ─┐ + FSCTL_ENUM_USN_DATA │ initials index │ │ + └──────────────────────────────┘ │ + ├──▶ store::FileStore ──▶ search + ┌──────────────────────────────┐ │ + directories ─▶ │ scan::build_store │ ─┘ + └──────────────────────────────┘ +``` + +| backend | needs admin | file sizes | warm start | +|-----------|-------------|------------|----------------------------------| +| `scan` | no | yes | loads the persisted cache | +| `mft-usn` | yes | no | replays the USN change journal | + +`auto` prefers `mft-usn` when the volume handle can be opened and silently falls +back to `scan`, so the provider always works. See +[`crates/windows-files/src/mft.rs`](../../../crates/windows-files/src/mft.rs) for why +the MFT path cannot report sizes without per-file NTFS attribute parsing. + +### Memory + +Entries live in a string arena plus fixed-size records rather than one +allocation-heavy struct per file: + +```text +1_000_000 rows × 48 B = 48 MB +names + directories ≈ 20 MB +------------------------------------ + ≈ 68 MB +``` + +`store::FileStore::memory_bytes()` reports the real figure in Settings → Index, +so the README never has to guess. + +### Persistence + +`PersistedIndex` is MessagePack written to a temporary file and then renamed, so +an interrupted save can never leave a half-written cache. A version mismatch is +not an error: the cache is ignored and rebuilt. + +## Live snapshots + +Processes, windows and services are never persisted. A snapshot of "now" is the +only thing that makes sense for them, and a stale one is worse than none. + +| domain | API | caching | +|-----------|----------------------------------|--------------------------------------| +| processes | `CreateToolhelp32Snapshot` + PSAPI | 250 ms, resolved in parallel | +| windows | `EnumWindows` | 250 ms | +| services | `EnumServicesStatusEx` | 5 s, config resolved only where needed | +| apps | registry + Start Menu + `PATH` | 5 min (installed apps rarely change) | + +## Error model + +```text +SearchError IndexError PlatformError PermissionError + \ | | / + +----------+------+------+-------------+ + │ + LceError + │ + ┌──────────────────┼──────────────────┐ + │ │ │ + code() string hint() (English) Display (developer) + │ + i18n key `error.` +``` + +The UI never renders `OsError(5)`. It renders +`error.volume-access-denied` → "无法访问该卷。可能需要管理员权限。", and the +original error goes to the log. + +## Threading + +* Providers are `Send + Sync`; the engine fans out serially but each provider is + cheap once warm. +* `FileProvider::rebuild` is the only genuinely long operation. It runs on a + Tauri blocking thread from the UI and as a plain call from the CLI. +* Process detail resolution is chunked across up to eight OS threads, which is + what turns ~30 ms of serial syscalls into single digits. +* There is no polling loop anywhere. Idle CPU is zero because idle means "no + thread is doing anything". + +## Roadmap + +| area | next step | +|------------------------|------------------------------------------------------------------| +| MSIX/Appx discovery | read `AppModel\Repository` instead of relying on Start Menu links | +| MFT file sizes | parse the `$DATA` attribute from `FSCTL_GET_NTFS_FILE_RECORD` | +| Event-driven file index| subscribe to the USN journal instead of replaying it on demand | +| Frameless palette mode | optional chromeless window with a custom drag handle | +| AI query compiler | a second `QueryCompiler` implementation, opt-in and off by default| +--- + +# v0.2: per-volume state, workers, the privileged boundary and accelerators + +## Per-volume journal state + +A single `Option` cannot describe a machine with more than one +NTFS volume. `C:` and `D:` have different journal identifiers, different +cursors, and roll over independently. + +```rust +pub struct JournalRegistry { + pub schema_version: u32, + pub volumes: BTreeMap, // keyed by VolumeId +} + +pub struct VolumeJournalState { + pub identity: VolumeIdentity, // volume GUID + serial + mount points + pub journal_id: u64, + pub first_usn: i64, + pub next_usn: i64, + pub cursor_usn: i64, + pub status: VolumeSyncStatus, + pub has_cursor: bool, + pub pending_rebuild: Option, + // ... +} +``` + +The key is a **`VolumeId`**, never a drive letter: `GetVolumeNameForVolumeMountPoint` +gives `\\?\Volume{...}\`, which survives a re-lettering, and the serial number is +the fallback when no GUID exists. `reconcile_identity` recognises "same disk, +new mount point" and keeps the cursor; `observe` is the single place that decides +incremental versus rebuild. + +| condition | decision | +|---|---| +| no cursor yet | `Rebuild { NewVolume }` | +| journal id changed | `Rebuild { JournalRecreated }` | +| `cursor < first_usn` | `Rebuild { CursorBeforeFirstUsn }` (rollover) | +| `cursor > next_usn` | `Rebuild { CursorAheadOfJournal }` | +| `cursor == next_usn` | `UpToDate` | +| otherwise | `Incremental { from_usn }` | + +One volume failing, rolling over or being offline never affects another. + +## Event-driven workers + +```text + IndexSupervisor + ├── VolumeWorker(C:) ── FSCTL_READ_USN_JOURNAL (blocking) + ├── VolumeWorker(D:) + └── VolumeWorker(...) +``` + +Each worker calls `FSCTL_READ_USN_JOURNAL` with a `Timeout` and a +`BytesToWaitFor`, so the kernel holds the call open until the journal moves. +There is no polling loop and idle CPU stays at zero. The supervisor tracks a +lifecycle per volume (`starting` → `running` → `backing-off` → `failed` → +`stopped`) and backs off exponentially to 30 seconds, marking **only that +volume** degraded. + +Journal reads and index application are separate types: + +```text + USN_RECORD_V2 --parse--> Vec --apply--> FileStore + (mft.rs) (mutation.rs) (store.rs) +``` + +`read_journal_batches` performs all disk I/O and parsing with **no lock held**; +`apply_journal_batch` takes the write lock only to apply a batch; persistence +happens after the lock is released. + +## The privileged boundary + +```text +Desktop ─┐ +CLI ─────┼── ordinary user +MCP ─────┘ + │ \\.\pipe\localsearch-index-v1 (byte mode, length-prefixed MessagePack) + ▼ +lce-index-service.exe ── elevated + ├── raw NTFS MFT + └── USN journals +``` + +The service's entire vocabulary is five requests: `Hello`, `Search`, +`IndexStatus`, `RebuildVolume`, `Changes`. There is no `Execute`, `Shell` or +`Command`, and a test enumerates the variants so adding one shows up in review. +Open, reveal and terminate-process stay in the unprivileged processes. + +Framing is `u32 length | u32 magic | u32 version | body`, with an 8 MiB ceiling +checked **before** the body is allocated. A wrong magic, an unknown version, a +truncated frame or a malformed body is a clean error; the connection is dropped +and the front end falls back to its in-process provider. + +## Search accelerators and the query planner + +```text +ext:rs -> ExtensionIndex (extension -> sorted RecordId postings) +prefix:vsco -> PrefixIndex (sorted 24-byte keys, binary search) +*substring* -> TrigramIndex (path trigrams, intersected, then verified) +anything else -> LinearFallback (the v0.1 scan) +``` + +Three rules make this safe: + +1. **Accelerators only propose.** Every candidate is verified against the real + string in the `FileStore`, which stays the single source of truth. +2. **The trigram index covers full paths, not names.** A token matches a record + when it appears in the name *or* the path, so indexing names alone would not + be a superset — the differential tests found exactly that bug. +3. **A dropped trigram stays dropped.** High-frequency trigrams are pruned, and + a live insert must never resurrect one: a one-element posting list for a + trigram that appears in half the corpus silently loses matches. + +`planner::plan` returns the chosen plan, the sources it consulted and how many +candidates each produced. `localsearch --explain` prints them; the accelerated +and linear paths are asserted equal by differential tests over a randomised +corpus, after mutations, and under the candidate cap. + +## Concurrency + +| structure | protection | held for | +|---|---|---| +| `FileStore` + `SearchAccelerators` | one `RwLock` | reads: a whole query; writes: one mutation batch | +| `JournalRegistry` | inside the same `RwLock` | as above | +| index cache file | none | written outside every lock | +| provider caches (processes, services, windows, apps) | `Mutex` + TTL | a snapshot copy | + +The rule is that no disk I/O, no `DeviceIoControl` and no serialisation happens +while the index lock is held. \ No newline at end of file diff --git a/docs/archive/development-notes/changelog-before-v1.md b/docs/archive/development-notes/changelog-before-v1.md new file mode 100644 index 0000000..08b3a0c --- /dev/null +++ b/docs/archive/development-notes/changelog-before-v1.md @@ -0,0 +1,144 @@ +> Historical snapshot. Claims and versions on this page are not current guidance. + +# Changelog + +All notable changes to this project are documented here. The format follows +[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project +adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [0.2.0] - 2026-10-01 + +A v0.2 engineering pass over the v0.1 MVP. Nothing that worked in 0.1 was +removed: the CLI, the MCP server, the desktop shell, the bilingual UI, the Local +Search DSL, the ranking crate and the `SearchService` API all keep their +behaviour. + +### Added + +- **Per-volume journal state** (`windows_files::journal`). `JournalRegistry` + keys every NTFS volume by a stable `VolumeId` — a volume GUID, never a drive + letter — and tracks its own journal id, cursor, first/next USN and sync + status. A journal that is recreated, that rolls over, or that moves backwards + marks **only that volume** stale and schedules a rebuild. +- **Event-driven index workers** (`windows_files::worker`). One `VolumeWorker` + per volume tails its own USN journal with a `Timeout`/`BytesToWaitFor` read, + so the thread blocks in the kernel instead of polling. An `IndexSupervisor` + owns the threads, tracks per-volume lifecycle (`starting`, `running`, + `backing-off`, `failed`, `stopped`) and backs off exponentially on failure. +- **A privileged index service** (`lce-index-service.exe`) plus + `index-protocol` and `index-client`. The desktop, CLI and MCP server stay + unprivileged; the service owns the MFT and the journals and answers typed, + versioned requests over a Named Pipe whose DACL is built from an explicit SDDL + string (SYSTEM, Administrators and the installing user — never Everyone or + Anonymous). `FILE_FLAG_FIRST_PIPE_INSTANCE` defeats name squatting and + `PIPE_REJECT_REMOTE_CLIENTS` refuses remote callers at the pipe itself. + `install`, `uninstall`, `start`, `stop`, `status` and `--console` are + supported. +- **Search accelerators** (`windows_files::accelerators`): an extension index, a + prefix table and a trigram index over normalised **full paths**, with + high-frequency trigrams dropped and a compaction policy driven by mutation + drift. +- **A query planner** (`windows_files::planner`) that picks + `extension` / `prefix` / `trigram` / `intersection` / `linear` per query and + reports what it did. `localsearch --explain` prints the plan, the candidate + counts and the elapsed time. +- **A unified mutation model** (`IndexMutation`) so USN parsing and index + application are separate steps, applied as one batch under a short write lock. +- **`prefix:` in the DSL**: the one predicate a prefix table can answer exactly. +- **Differential tests**: accelerated search must equal the pre-v0.2 linear scan + over a randomised corpus, after mutations, and when the candidate cap + truncates. This is what caught two real bugs during development (trigrams + indexed names but not paths; a live insert could resurrect a dropped trigram + and silently lose matches). + +### Changed + +- The persisted index schema is now **version 3** and carries the per-volume + journal registry plus the accelerators. A v0.1 or v0.2 cache is discarded and + rebuilt rather than migrated — guessing at an older cursor is how a stale + index happens. +- MessagePack is now written with **named** fields. Positional encoding plus + `skip_serializing_if` shifted every field after a skipped `Option` and + corrupted the read. +- The file provider holds its store and accelerators behind one `RwLock`, and + applies journal batches under that lock only after all disk I/O and parsing is + done. Cache persistence happens outside the lock. +- Process detail resolution is chunked across up to eight threads. + +### Fixed + +- A single `Option` could not describe a machine with more than + one NTFS volume; one disk's cursor could overwrite another's. + +### Security + +- Threat model documented for the privileged boundary, the pipe ACL, malformed + clients, protocol downgrade, stale indexes and service crashes. It states + plainly that MFT enumeration can reveal file *names* the calling user could + not otherwise list, and why that is inherent to the technique. +- The protocol has no `Execute`, `Shell` or `Command` message, and a test + enumerates the variants so adding one is a visible diff. + +### Known limitations + +- The accelerators roughly **double** the index footprint (measured: +60% at + 100 000 entries), which pushes a one million entry index past the project's + 100 MB goal. The prefix table is the largest single contributor. +- Common-only queries (`readme`, `report`) and fuzzy queries still fall back to + a linear scan, because their trigrams are too common to narrow anything. + A `path:` filter also forces a scan. These are unchanged from v0.1 in cost. +- A `Changes` subscription reports volume-level journal advances, not a + per-file event stream. +- The MFT backend still cannot report file sizes. + +[0.2.0]: https://github.com/styayur/local-context-engine/releases/tag/v0.2.0 +## [0.1.0] - 2026-10-01 + +The first runnable MVP. Everything below works today; nothing is a placeholder. + +### Added + +- **Rust workspace** with hard module boundaries: `search-core`, `query-dsl`, + `ranking`, five `windows-*` providers, `search-daemon`, `search-cli` and + `search-mcp`. +- **Unified entity model** (`LocalEntity`) with six variants — file, directory, + process, application, service, window — and a single `SearchResult` shape. +- **Local Search DSL**: `type:`, `ext:`, `path:`, `name:`, `drive:`, + `modified:`, `created:`, `size:`, `state:`, `pid:`, `user:`, `visible:`, + `sort:` and `limit:`, with graceful fallback to plain text. +- **Rule-based natural language compiler** for English and Simplified Chinese. + `最近的 PDF` and `recent PDFs` compile to the same query, as do + `正在运行的 python` and `running python processes`. +- **Deterministic heuristic ranking** with named, configurable weights, fuzzy + subsequence matching and a bounded local usage history. +- **Filesystem index** with two interchangeable backends: a directory scan that + needs no privileges and records sizes, and an NTFS MFT + USN change journal + backend that needs elevation but enumerates a volume in one pass and updates + incrementally. +- **Compact index storage**: a string arena plus 48-byte records, so a one + million entry index costs roughly 70 MB rather than several hundred. +- **Atomically persisted index cache** in `%LOCALAPPDATA%\LocalContextEngine`. +- **Live providers** for processes (Toolhelp32 + PSAPI, resolved in parallel), + services (`EnumServicesStatusEx`), top-level windows (`EnumWindows`) and + applications (App Paths, Start Menu, `PATH`, WindowsApps). +- **CLI** `localsearch.exe` with human and `--json` output, `--explain`, + `--index-status`, `--rebuild-index`, `--update-index`, `--providers`, + `--list-usage` and `--reset-usage`. +- **MCP server** `localsearch-mcp.exe` over stdio with eleven tools, including + `search_system`, the five scoped search tools, `compile_query`, + `index_status`, `dsl_reference`, `record_selection` and `terminate_process`. +- **Tauri 2 desktop shell** with a React + TypeScript + Tailwind interface: + keyboard-first navigation, type filters, match highlighting, per-entity + context menus, a confirmation dialog for the one destructive action, and + full zh-CN/en-US localisation that switches without a restart. +- **290 tests** across the workspace plus workspace-level integration + tests, benchmarks, and the `docs/` set below. + +### Security + +- File search is read-only. +- Process termination is gated behind an explicit confirmation on both the UI + and the Rust side. +- No network access anywhere in the codebase. + +[0.1.0]: https://github.com/styayur/local-context-engine/releases/tag/v0.1.0 \ No newline at end of file diff --git a/docs/archive/development-notes/readme-before-v1.md b/docs/archive/development-notes/readme-before-v1.md new file mode 100644 index 0000000..46bd1b9 --- /dev/null +++ b/docs/archive/development-notes/readme-before-v1.md @@ -0,0 +1,361 @@ +> Historical snapshot. Claims and versions on this page are not current guidance. + +
+ 简体中文 | English +
+ + + + + +# Local Context Engine + +Offline-first search over your own machine. + +Natural language in → deterministic query → Rust search core → results. + +[Download](https://github.com/styayur/local-context-engine/releases/latest) · [Architecture](../../ARCHITECTURE.md) · [DSL](../../DSL.md) · [MCP](../../MCP.md) · [Benchmarks](../../BENCHMARKS.md) + +[![CI](https://github.com/styayur/local-context-engine/actions/workflows/ci.yml/badge.svg)](https://github.com/styayur/local-context-engine/actions/workflows/ci.yml) +[![release](https://img.shields.io/github/v/release/styayur/local-context-engine?label=release&color=orange)](https://github.com/styayur/local-context-engine/releases) +[![license: MPL-2.0](https://img.shields.io/badge/license-MPL--2.0-blue)](../../../LICENSE) + + + +--- + +What is this? +-------------- + +Local Context Engine is a local search tool for Windows: one input box that +finds your **files and folders**, **applications**, **running processes**, +**services** and **windows** at the same time, and opens them. + +Type `vscode` and you get the application, the running process and any matching +file in one list. Type `正在运行的 python` or `running python processes` and you +get the processes. Type `最近的 pdf` or `recent PDFs` and you get today's PDFs, +newest first. + +It is built from three ideas: + +1. **Search must be deterministic.** The same query returns the same list in the + same order, every time. No model, no embeddings, no vector database sits on + the hot path. +2. **Understanding is a compiler problem.** Natural language is *compiled* into + a structured query — `最近的 pdf` becomes `type:file ext:pdf + sort:modified-desc` — and then a Rust engine answers it. +3. **Offline is the default, not a mode.** There is no account, no telemetry and + no network code in the repository. + +Why it exists +------------- + +Desktop search on Windows is either a full-text indexer that rebuilds for +minutes, or a launcher that cannot see anything except shortcuts. Meanwhile the +"AI" answer to search is usually "embed everything and hope" — which is slow, +non-deterministic, unauditable, and quietly sends your file names somewhere. + +This project takes the opposite position: + +> AI, if it appears at all, is a **query compiler**. Retrieval is a +> deterministic local operation. + +How it differs from a traditional desktop search +------------------------------------------------ + +| | Traditional indexer | Launcher | Local Context Engine | +|-----------------------------------|---------------------|------------|-----------------------------------------------| +| Time to first useful result | minutes of indexing | instant | instant for live data, one scan for files | +| Indexes file **contents** | yes | no | no — names, paths and metadata only | +| Sees processes/services/windows | rarely | no | yes, all four | +| Query language | one text box | one text box | text box **and** a documented DSL **and** MCP | +| Deterministic ranking | usually | usually | always, with `--explain` to show the arithmetic | +| AI on the search path | sometimes | sometimes | never | +| Network required | often | no | never | +| Memory footprint | hundreds of MB | tens of MB | ~70 MB for a one million entry index | + +Architecture +------------ + +```text + User / AI + │ + ▼ + Query Compiler ← the only place an AI could ever plug in + │ + ▼ + SearchQuery + │ + ▼ + Search Core + │ + ┌───────────┼───────────┐ + ▼ ▼ ▼ + Files Processes Apps + │ │ │ + MFT/USN Win32 Registry +``` + +The workspace is split so that "where does this belong?" never needs a meeting: + +```text +local-context-engine/ +├─ apps/desktop/ Tauri 2 + React + TypeScript shell +├─ crates/ +│ ├─ search-core/ entity/query/result model, engine, fuzzy matcher +│ ├─ query-dsl/ the DSL parser and the rule-based NL compiler +│ ├─ ranking/ heuristic scoring + bounded local usage memory +│ ├─ windows-files/ directory-scan and NTFS MFT + USN index backends +│ ├─ windows-processes/ live process snapshots +│ ├─ windows-apps/ App Paths, Start Menu, PATH, WindowsApps +│ ├─ windows-services/ read-only service snapshots +│ ├─ windows-windows/ top-level window snapshots +│ ├─ search-daemon/ the composition root every front end shares +│ ├─ search-cli/ localsearch.exe +│ ├─ search-mcp/ localsearch-mcp.exe (MCP over stdio) +│ └─ benchmarks/ Criterion benchmarks for 10k / 100k / 1M entries +├─ docs/ +└─ tests/ workspace-level integration tests +``` + +Full detail — including why `AI != Search Engine`, the provider boundary and the +memory arithmetic — is in [docs/ARCHITECTURE.md](../../ARCHITECTURE.md). + +Local Search DSL +---------------- + +```text +chrome +type:file rust +type:file ext:rs +type:file ext:pdf modified:<24h +type:process python +type:process name:node +type:app vscode +type:service state:running +type:window github +path:projects ext:toml +drive:C ext:exe sort:size-desc +path:"My Projects" ext:toml +size:>10mb sort:size-desc limit:20 +``` + +Keys: `type`, `ext`, `path`, `name`, `drive`, `modified`, `created`, `size`, +`state`, `pid`, `user`, `visible`, `sort`, `limit`. Parsing never fails — an +unrecognised key stays free text, so `localsearch "C:\Users\me"` is a search +rather than a syntax error. + +The full reference, including the time and size operators, is in +[docs/DSL.md](../../DSL.md). + +MCP +--- + +`localsearch-mcp.exe` exposes the same core to any MCP client over stdio: + +```jsonc +{ + "mcpServers": { + "localsearch": { "command": "C:\\path\\to\\localsearch-mcp.exe" } + } +} +``` + +| tool | what it does | +|---------------------|--------------------------------------------------| +| `search_system` | search every domain at once | +| `search_files` | files and folders | +| `search_processes` | running processes | +| `search_apps` | installed applications | +| `search_services` | Windows services | +| `search_windows` | open top-level windows | +| `compile_query` | show how an input compiles, without searching | +| `index_status` | backend, entry counts, memory, provider health | +| `dsl_reference` | the key table and examples | +| `record_selection` | boost a result next time | +| `terminate_process` | destructive; requires `confirm: true` | + +See [docs/MCP.md](../../MCP.md). + +CLI +--- + +```bash +localsearch chrome +localsearch "type:file ext:rs" +localsearch "正在运行的 python" +localsearch --type process node +localsearch --json vscode +localsearch --explain "昨天修改的 rust 文件" +localsearch --index-status +localsearch --rebuild-index +localsearch --update-index +localsearch --providers +localsearch --list-usage +localsearch --reset-usage +``` + +Exit codes are part of the contract: `0` results found, `1` valid query with no +matches, `2` the command could not be completed. `--json` writes a single JSON +document to stdout; diagnostics always go to stderr. + +Desktop +------- + +```bash +cd apps/desktop +npm install +npm run tauri:dev # development, hot reload +npm run tauri:build # NSIS installer in src-tauri/target/release/bundle +``` + +* Autofocused search box, keyboard-first: `↑`/`↓` navigate, `Enter` opens, + `Shift+Enter` opens the context menu, `Tab` cycles the type filters, `Esc` + clears or closes, `Ctrl+,` opens settings. +* Type filter chips with live counts, match highlighting, per-entity context + menus, and a status bar showing the compile result and the elapsed time. +* **Simplified Chinese and English**, switchable without a restart. Every + string — errors, empty states, tooltips, context menus — lives in + `src/i18n/{zh-CN,en-US}.json`; nothing is hard-coded in a component. +* Settings for language, theme, result limit, fuzzy matching, usage ranking, + indexed drives, index backend, the global shortcut, and a full index panel. +* The only destructive action, ending a process, is behind a confirmation + dialog whose default button is Cancel — and the Rust side refuses the call + without an explicit `confirmed: true`. + +Build +----- + +Requirements: Rust (stable, MSVC toolchain), Visual Studio Build Tools with +"Desktop development with C++", Node 20+, and WebView2 (already present on +Windows 11). + +```bash +git clone https://github.com/styayur/local-context-engine +cd local-context-engine + +# CLI + MCP server +cargo build --release + +# Desktop shell + installer +cd apps/desktop +npm install +npm run tauri:build +``` + +The binaries land in `target/release/`: + +```text +target/release/localsearch.exe +target/release/localsearch-mcp.exe +target/release/lce-index-service.exe +``` + +### Optional: the privileged index service (v0.2) + +The desktop, CLI and MCP server never need administrator rights. If you want the +fast NTFS MFT backend, install the small service once and the front ends will use +it automatically: + +```powershell +# from an elevated prompt +.\target\release\lce-index-service.exe install +.\target\release\lce-index-service.exe start +localsearch --service-status +``` + +The service owns the MFT and the USN journals and answers typed, versioned +requests over a Named Pipe whose ACL is built explicitly (SYSTEM, Administrators +and you — never Everyone or Anonymous). Without it, everything still works on the +unprivileged directory-scan backend. See [SECURITY.md](../../../SECURITY.md) for the +boundary and for the one disclosure it implies: MFT enumeration can see file +*names* an unprivileged directory walk could not. + +Development +----------- + +```bash +cargo fmt --all -- --check +cargo clippy --all-targets --all-features -- -D warnings +cargo test + +cd apps/desktop +npm run lint +npm run typecheck +npm run build +``` + +Benchmarks: + +```bash +cargo bench -p lce-benchmarks --bench file_search +cargo bench -p lce-benchmarks --bench ranking +``` + +Layout, conventions and how to add a provider or a translation are covered in +[CONTRIBUTING.md](../../../CONTRIBUTING.md). + +Security +-------- + +* File search is **read-only**. +* No `exec`, no `eval`, no shell string built from a query. +* Persisted state is a MessagePack index and two JSON files this project wrote; + no untrusted payload is ever deserialised as code. +* Paths are treated as opaque Unicode strings and are tested with spaces, + Simplified Chinese and very long names. Junctions and symlinks are never + followed. +* Process termination requires confirmation on both sides of the boundary. +* Services are strictly read-only in this version. + +Full threat model: [SECURITY.md](../../../SECURITY.md). + +Privacy +------- + +> **Your local index never leaves your machine.** + +* No telemetry, no analytics, no crash reporting, no update check. +* No account, no login, no cloud service. +* Query history is optional, bounded, stored locally in + `%LOCALAPPDATA%\LocalContextEngine\usage.json`, and removable from Settings or + with `localsearch --reset-usage`. +* There is no HTTP client in the dependency tree of the search core. + +Benchmarks +---------- + +Measured with Criterion on a synthetic corpus, not estimated. Full table, +machine specification, method and caveats: [docs/BENCHMARKS.md](../../BENCHMARKS.md). + +| what | 1 000 000 indexed entries | +|---|---:| +| query compilation (`最近的 pdf`, `recent PDFs`, plain text, DSL) | 1.4 – 11.6 µs | +| file search that fills the candidate budget (the common typing case) | **~8 ms** | +| file search that must visit every record | **~62 ms** | +| the same, with fuzzy matching on every record | **~76 ms** | +| ranking 4 000 candidates | **~2.8 ms** | +| full hot path: walk the index, collect, rank | **~10 ms** | +| building the in-memory index | 316 ms (3.2 M entries/s) | +| memory for one million entries | ≈ 69 MB | + +The candidate budget (`CANDIDATE_CAP = 4 000`) is why a one million entry index +does not cost fifty times a ten thousand entry one. The rows that do grow are +the ones that genuinely have to touch every record. + +Roadmap +------- + +* MSIX/Appx discovery through `AppModel\Repository` rather than Start Menu links. +* File sizes from the MFT by parsing the `$DATA` attribute. +* Event-driven index updates instead of replaying the journal on demand. +* An optional frameless palette window. +* An optional AI query compiler behind the existing `QueryCompiler` trait — + off by default, and never on the search path. + +Licence +------- + +[MIT](../../../LICENSE). Not affiliated with, derived from, or containing any code, +assets or UI of Listary or any other launcher. The only thing borrowed is the +public idea that a local index should be fast. \ No newline at end of file diff --git a/docs/README_EVIDENCE.md b/docs/archive/development-notes/readme-evidence-2026-10-08.md similarity index 100% rename from docs/README_EVIDENCE.md rename to docs/archive/development-notes/readme-evidence-2026-10-08.md diff --git a/docs/archive/development-notes/readme-zh-before-v1.md b/docs/archive/development-notes/readme-zh-before-v1.md new file mode 100644 index 0000000..2921d4d --- /dev/null +++ b/docs/archive/development-notes/readme-zh-before-v1.md @@ -0,0 +1,294 @@ +> Historical snapshot. Claims and versions on this page are not current guidance. + +
+ 简体中文 | English +
+ + + + + +# Local Context Engine + +离线优先的本地系统搜索。 + +自然语言输入 → 确定性查询 → Rust 搜索内核 → 结果。 + +[下载](https://github.com/styayur/local-context-engine/releases/latest) · [架构](../../ARCHITECTURE.md) · [查询语法](../../DSL.md) · [MCP](../../MCP.md) · [性能](../../BENCHMARKS.md) + +[![CI](https://github.com/styayur/local-context-engine/actions/workflows/ci.yml/badge.svg)](https://github.com/styayur/local-context-engine/actions/workflows/ci.yml) +[![release](https://img.shields.io/github/v/release/styayur/local-context-engine?label=release&color=orange)](https://github.com/styayur/local-context-engine/releases) +[![license: MPL-2.0](https://img.shields.io/badge/license-MPL--2.0-blue)](../../../LICENSE) + + + +--- + +## 这是什么 + +Local Context Engine 是一个 Windows 本地搜索工具：一个输入框，同时搜索**文件与文件夹**、**应用**、**正在运行的进程**、**服务**和**窗口**，并且可以直接打开它们。 + +输入 `vscode`，你在同一个列表里看到应用、进程和匹配的文件。输入 `正在运行的 python` 或 `running python processes`，得到 Python 进程。输入 `最近的 pdf` 或 `recent PDFs`，得到按修改时间排序的 PDF。 + +它建立在三个判断之上： + +1. **检索必须确定性。** 同一个查询，每次都返回同样的结果、同样的顺序。热路径上不出现模型、embedding 或向量数据库。 +2. **理解是编译问题。** 自然语言被**编译**成结构化查询——`最近的 pdf` 变成 `type:file ext:pdf sort:modified-desc`——然后交给 Rust 内核执行。 +3. **离线是默认状态，而不是一种模式。** 没有账号、没有遥测，仓库里没有一行网络代码。 + +## 为什么做这个 + +Windows 上的桌面搜索要么是全量内容索引器（建索引要几分钟），要么是只能看见快捷方式的启动器。而所谓“AI 搜索”的常见答案通常是“把所有东西都做 embedding 再碰运气”——慢、不确定、无法解释，而且悄悄把文件名发到了别处。 + +这个项目选择相反的路线： + +> 如果一定要有 AI，它是**查询编译器**。检索始终是确定性的本地操作。 + +## 与传统桌面搜索的区别 + +| | 传统索引器 | 启动器 | Local Context Engine | +|-----------------------------|-----------------|-------------|--------------------------------------------| +| 首次可用时间 | 建索引数分钟 | 即时 | 实时数据即时可用，文件索引一次扫描 | +| 索引文件**内容** | 是 | 否 | 否——只索引名称、路径与元数据 | +| 覆盖进程 / 服务 / 窗口 | 罕见 | 否 | 四类全部覆盖 | +| 查询方式 | 单一输入框 | 单一输入框 | 输入框 + 公开的 DSL + MCP 接口 | +| 排序确定性 | 通常不确定 | 通常不确定 | 始终确定，`--explain` 可给出评分依据 | +| AI 在检索热路径上 | 有时 | 有时 | 从不 | +| 是否需要网络 | 经常需要 | 不需要 | 从不需要 | +| 内存占用 | 数百 MB | 数十 MB | 一百万个条目约 70 MB | + +## 架构 + +```text + User / AI + │ + ▼ + Query Compiler ← 未来唯一允许接入 AI 的位置 + │ + ▼ + SearchQuery + │ + ▼ + Search Core + │ + ┌───────────┼───────────┐ + ▼ ▼ ▼ + Files Processes Apps + │ │ │ + MFT/USN Win32 Registry +``` + +工作区的划分让“这段代码该放哪”不再需要讨论： + +```text +local-context-engine/ +├─ apps/desktop/ Tauri 2 + React + TypeScript 桌面外壳 +├─ crates/ +│ ├─ search-core/ 实体 / 查询 / 结果模型、搜索引擎、模糊匹配 +│ ├─ query-dsl/ DSL 解析器与规则式自然语言编译器 +│ ├─ ranking/ 启发式评分 + 有界的本地使用记录 +│ ├─ windows-files/ 目录扫描与 NTFS MFT + USN 两种索引后端 +│ ├─ windows-processes/ 实时进程快照 +│ ├─ windows-apps/ App Paths、开始菜单、PATH、WindowsApps +│ ├─ windows-services/ 只读的服务快照 +│ ├─ windows-windows/ 顶层窗口快照 +│ ├─ search-daemon/ 三个前端共用的组合根 +│ ├─ search-cli/ localsearch.exe +│ ├─ search-mcp/ localsearch-mcp.exe（stdio 上的 MCP） +│ └─ benchmarks/ 10k / 100k / 100 万条目的 Criterion 基准 +├─ docs/ +└─ tests/ 工作区级集成测试 +``` + +完整说明（包括为什么 `AI != Search Engine`、Provider 边界、内存账目）见 +[docs/ARCHITECTURE.md](../../ARCHITECTURE.md)。 + +## 查询语法 + +```text +chrome +type:file rust +type:file ext:rs +type:file ext:pdf modified:<24h +type:process python +type:process name:node +type:app vscode +type:service state:running +type:window github +path:projects ext:toml +drive:C ext:exe sort:size-desc +path:"My Projects" ext:toml +size:>10mb sort:size-desc limit:20 +``` + +可用键：`type`、`ext`、`path`、`name`、`drive`、`modified`、`created`、`size`、 +`state`、`pid`、`user`、`visible`、`sort`、`limit`。解析**永不失败**——无法识别的键会作为普通文本参与搜索，所以 +`localsearch "C:\Users\me"` 是一次搜索而不是语法错误。 + +完整的时间与体积运算符说明见 [docs/DSL.md](../../DSL.md)。 + +## MCP + +`localsearch-mcp.exe` 通过 stdio 把同一个内核暴露给任何 MCP 客户端： + +```jsonc +{ + "mcpServers": { + "localsearch": { "command": "C:\\path\\to\\localsearch-mcp.exe" } + } +} +``` + +| 工具 | 作用 | +|---------------------|-------------------------------------| +| `search_system` | 跨全部数据源搜索 | +| `search_files` | 文件与文件夹 | +| `search_processes` | 正在运行的进程 | +| `search_apps` | 已安装的应用 | +| `search_services` | Windows 服务 | +| `search_windows` | 打开的顶层窗口 | +| `compile_query` | 只展示编译结果，不执行搜索 | +| `index_status` | 后端、条目数、内存、各数据源状态 | +| `dsl_reference` | 键表与示例 | +| `record_selection` | 记住选择，下次提升排序 | +| `terminate_process` | 危险操作，必须传 `confirm: true` | + +详见 [docs/MCP.md](../../MCP.md)。 + +## 命令行 + +```bash +localsearch chrome +localsearch "type:file ext:rs" +localsearch "正在运行的 python" +localsearch --type process node +localsearch --json vscode +localsearch --explain "昨天修改的 rust 文件" +localsearch --index-status +localsearch --rebuild-index +localsearch --update-index +localsearch --providers +localsearch --list-usage +localsearch --reset-usage +``` + +退出码是契约的一部分：`0` 有结果，`1` 查询合法但无匹配，`2` 命令未能完成。 +`--json` 会把单个 JSON 文档写到 stdout，诊断信息始终走 stderr。 + +## 桌面端 + +```bash +cd apps/desktop +npm install +npm run tauri:dev # 开发模式，热重载 +npm run tauri:build # 生成 NSIS 安装包 +``` + +* 搜索框自动聚焦，键盘优先：`↑`/`↓` 导航，`Enter` 打开，`Shift+Enter` 打开右键菜单， + `Tab` 切换类型筛选，`Esc` 清空或关闭，`Ctrl+,` 打开设置。 +* 类型筛选标签带实时计数、匹配高亮、按实体类型区分的右键菜单，状态栏显示编译结果与耗时。 +* **简体中文与 English**，切换无需重启。所有文案——错误、空状态、提示、右键菜单——都在 + `src/i18n/{zh-CN,en-US}.json` 中，组件里没有任何硬编码文本。 +* 设置涵盖语言、主题、结果上限、模糊匹配、使用频率加权、索引磁盘、索引后端、全局快捷键，以及完整的索引面板。 +* 唯一的破坏性操作“结束进程”需要确认，对话框默认焦点在“取消”；Rust 侧在没有显式 + `confirmed: true` 时同样会拒绝执行。 + +## 构建 + +需要：Rust（stable，MSVC 工具链）、安装 “使用 C++ 的桌面开发” 工作负载的 Visual Studio Build Tools、Node 20+，以及 WebView2（Windows 11 自带）。 + +```bash +git clone https://github.com/styayur/local-context-engine +cd local-context-engine + +# 命令行与 MCP 服务 +cargo build --release + +# 桌面端与安装包 +cd apps/desktop +npm install +npm run tauri:build +``` + +产物位于 `target/release/`： + +```text +target/release/localsearch.exe +target/release/localsearch-mcp.exe +``` + +## 开发 + +```bash +cargo fmt --all -- --check +cargo clippy --all-targets --all-features -- -D warnings +cargo test + +cd apps/desktop +npm run lint +npm run typecheck +npm run build +``` + +基准测试： + +```bash +cargo bench -p lce-benchmarks --bench file_search +cargo bench -p lce-benchmarks --bench ranking +``` + +目录约定、如何新增数据源、如何新增语言，见 [CONTRIBUTING.md](../../../CONTRIBUTING.md)。 + +## 安全 + +* 文件搜索是**只读**的。 +* 没有 `exec`、没有 `eval`、不会用查询拼出 shell 命令。 +* 持久化状态只有本项目自己写的 MessagePack 索引和两个 JSON 文件，不会把不受信任的数据当作可执行内容反序列化。 +* 路径一律按不透明 Unicode 字符串处理，并针对空格、简体中文、超长路径编写了测试；从不跟随 junction 与符号链接。 +* 结束进程在边界两侧都需要确认。 +* 本版本对服务严格只读。 + +完整威胁模型见 [SECURITY.md](../../../SECURITY.md)。 + +## 隐私 + +> **你的本地索引永远不会离开这台机器。** + +* 没有遥测、没有分析、没有崩溃上报、没有更新检查。 +* 不需要账号、不需要登录、没有云服务。 +* 查询历史可选、有上限、保存在本地 + `%LOCALAPPDATA%\LocalContextEngine\usage.json`，可在设置里或通过 + `localsearch --reset-usage` 清空。 +* 搜索内核的依赖树中没有 HTTP 客户端。 + +## 性能 + +下面都是 Criterion 在合成数据集上的实测值，不是估算。完整表格、测试机器、方法与注意事项见 +[docs/BENCHMARKS.md](../../BENCHMARKS.md)。 + +| 项目 | 索引 100 万条目时 | +|---|---:| +| 查询编译（`最近的 pdf`、`recent PDFs`、纯文本、DSL） | 1.4 – 11.6 µs | +| 命中候选上限的常见输入场景 | **约 8 ms** | +| 必须遍历全部记录的搜索 | **约 62 ms** | +| 同上，且对每条记录做模糊匹配 | **约 76 ms** | +| 对 4 000 个候选排序打分 | **约 2.8 ms** | +| 完整热路径：遍历索引 → 收集候选 → 排序 | **约 10 ms** | +| 构建内存索引 | 316 ms（约 320 万条目/秒） | +| 100 万条目的内存占用 | 约 69 MB | + +候选数量上限（`CANDIDATE_CAP = 4 000`）是 100 万条目索引不会比 1 万条目慢五十倍的原因； +真正会随规模增长的是那些必须遍历全部记录的情况。 + +## 路线图 + +* 通过 `AppModel\Repository` 支持 MSIX/Appx 应用发现，而不仅依赖开始菜单快捷方式。 +* 解析 `$DATA` 属性，让 MFT 后端也能给出文件大小。 +* 由事件驱动索引更新，而不是按需重放变更日志。 +* 可选的无边框命令面板窗口。 +* 可选的 AI 查询编译器，实现同一个 `QueryCompiler` trait——默认关闭，且永远不在检索热路径上。 + +## 许可 + +[MIT](../../../LICENSE)。与 Listary 及其它启动器**没有任何关联**，也不包含其代码、资源或界面。 +本项目只借鉴了“本地索引应当足够快”这一公开的设计思路。 \ No newline at end of file diff --git a/docs/archive/development-notes/security-before-v1.md b/docs/archive/development-notes/security-before-v1.md new file mode 100644 index 0000000..44f7ebc --- /dev/null +++ b/docs/archive/development-notes/security-before-v1.md @@ -0,0 +1,245 @@ +> Historical snapshot. Claims and versions on this page are not current guidance. + +# Security + +## Threat model + +Local Context Engine reads from the operating system and, on request, opens or +reveals the things it finds. It is not a sandbox and it does not try to be one. + +What it does: + +* Reads the NTFS master file table or walks directories to build a **name** index. +* Reads live process, window, service and Start Menu snapshots. +* Opens files, folders and applications **only** when you activate a result. +* Ends a process **only** after you accept a confirmation dialog, and the Rust + side refuses the call unless it was explicitly confirmed. + +What it never does: + +* It never executes shell commands derived from a query. +* It never evaluates query text. +* It never deserialises an executable payload. Persisted state is a + MessagePack index file and two JSON files this project wrote. +* It never opens a socket. There is no telemetry, no analytics and no update + check. +* It never uploads a file list, a query, a query history or a process list. + +## Path handling + +Paths are treated as opaque UTF-16/UTF-8 strings throughout: + +* Spaces, Simplified Chinese, emoji and other Unicode are covered by tests. +* Length is not assumed. The index stores file names in a string arena and + reconstructs paths on demand, so a very long path is a `String`, not a + fixed-size buffer. +* Junction points and symbolic links are never followed, so a directory scan + cannot loop forever. +* Destructive operations are restricted to a single API + (`actions::terminate_process`) whose signature requires `confirmed = true`. + +## Untrusted input + +The MCP server accepts JSON-RPC from a local client and treats every field as +untrusted: + +* Tool arguments are parsed with explicit type checks; a wrong type is an error, + never a coercion. +* Unknown tools, unknown filter names and unknown entity types are rejected. +* Query text is never interpolated into SQL, a shell or a path; it only ever + reaches the matcher. + +## Permissions + +The index backends are honest about what they need: + +| backend | requirement | +|-----------|----------------------------------------------------| +| `scan` | none beyond the user's own read access | +| `mft-usn` | administrator rights to open `\\.\C:` | + +The default is `auto`, which uses `mft-usn` when it is available and otherwise +falls back to `scan`. A denied volume produces a translated, actionable message +in the UI and a full error in the developer log — never a bare `OsError(5)`. + +## Reporting a vulnerability + +Open a private security advisory on GitHub rather than a public issue. Please +include the version, the Windows build, and the smallest reproduction you can +manage. +--- + +# Threat model: the privileged index service (v0.2) + +Version 0.2 moves the operations that need administrator rights out of the +desktop, CLI and MCP processes and into one small service. That is a smaller +attack surface, but it is a trust boundary, and boundaries deserve to be written +down rather than assumed. + +## What runs elevated, and what does not + +```text +Desktop ─┐ +CLI ─────┼── ordinary user, no UAC prompt, no admin token +MCP ─────┘ + │ + │ Named Pipe \\.\pipe\localsearch-index-v1 + ▼ +lce-index-service.exe ── LocalSystem / Administrator + ├── open \\.\C: and enumerate the MFT + ├── read and tail the USN change journals + └── answer index queries +``` + +The service exists because opening `\\.\C:` for `FILE_READ_DATA` requires +elevation. Nothing else does: opening files, revealing folders and ending +processes all stay in the unprivileged processes, where they belong. + +## What the service can be asked to do + +Exactly five things, and the list is the whole protocol: + +| request | effect | +|------------------|---------------------------------------------------------| +| `Hello` | handshake: protocol version, service version, volumes | +| `Search` | run a query against the service's index | +| `IndexStatus` | report backend, entry counts, per-volume sync state | +| `RebuildVolume` | re-enumerate the MFT for one volume (or all) | +| `Changes` | poll for journal advances since a cursor | + +There is no `Execute`, no `Shell`, no `Command`, no `WriteFile` and no `Open`. +This is not a policy that a reviewer has to check line by line: a message that +cannot be constructed cannot be handled. `crates/index-protocol` has a test that +enumerates the variants so that adding a dangerous one is a visible diff. + +## Raw MFT visibility: the honest statement + +**MFT enumeration can reveal file names that the calling user could not discover +by walking the directory tree.** + +The master file table does not apply per-directory ACLs. A volume-wide +enumeration returns every name on the volume, including names inside directories +the user cannot open. This is inherent to the technique — the same is true of +`Everything`, of `fsutil usn readjournal`, and of any MFT-based indexer — and it +is *not* something this project can fix by being careful. + +The consequences, stated plainly: + +* An index built by the service contains names the user may not be permitted to + read. Search results can therefore surface a path whose contents will still be + denied on open, because the open goes through the normal ACL check. +* The service does not read file *contents*, and it never returns file + contents. It returns names, paths and metadata. +* On a machine where more than one person has an account, that is a real + disclosure of *names*. If that is unacceptable for a given machine, do not + install the service: the unprivileged directory-scan backend indexes exactly + what the user can already see. + +## Pipe access control + +The DACL is built from an SDDL string, not assembled by hand: + +```text +D:(A;;GA;;;SY)(A;;GA;;;BA)(A;;GRGW;;;) +``` + +* `SY` — LocalSystem, which owns the service. +* `BA` — the built-in Administrators group, which may manage it. +* the installing user's SID — read and write, enough to ask a question. + +Deliberately absent: `WD` (Everyone) and `AN` (Anonymous). A pipe created with a +default security descriptor grants both, and that is the single most common +mistake in this kind of design. + +Additional hardening, all in `crates/index-service/src/pipe.rs`: + +* `FILE_FLAG_FIRST_PIPE_INSTANCE` — if another process already owns the name, + creating the pipe **fails** instead of silently sharing. This defeats pipe + name squatting, where a hostile local process creates the pipe first and + harvests whatever the front ends send it. +* `PIPE_REJECT_REMOTE_CLIENTS` — remote clients are refused by the pipe itself, + not merely by the ACL, so the boundary does not depend on the DACL being + perfect. +* Byte-mode pipes with an explicit length prefix, so a half-written message is a + detectable truncated frame rather than a silently short read. + +## Malformed client input + +Every field that crosses the pipe is treated as hostile: + +| input | handling | +|-----------------------------|------------------------------------------------------| +| bad frame magic | `BadMagic`, connection dropped | +| unknown protocol version | `VersionMismatch`, connection dropped | +| frame larger than 8 MiB | `TooLarge`, **rejected before the body is allocated** | +| truncated frame | `Truncated`, connection dropped | +| malformed MessagePack body | `Malformed`, connection dropped | +| unknown request variant | decode failure, connection dropped | +| unknown entity type in `Search` | ignored; the type filter simply does not match it | +| impossible `limit` | clamped to the service's own maximum (2 000 hits) | + +The service never unwraps a client-controlled value and never indexes a buffer +with a client-supplied length: the frame reader checks the announced length +against its own constant before allocating. + +## Protocol downgrade + +A downgrade is refused in two places: + +1. **Transport.** Every frame carries its version, and a mismatch is an error + before the body is even parsed. +2. **Application.** `Hello` carries the client's version and the service answers + `protocol-mismatch` if they disagree, so a client that somehow negotiated the + wrong version at the transport layer still cannot proceed. + +There is no "compatibility mode" and no best-effort decode. If the revisions +differ, the front ends fall back to their own in-process provider and the user is +told the service needs updating. + +## A stale index is a correctness failure, not a performance one + +The service must never answer from an index it cannot prove is current. The +per-volume journal registry makes that decision explicit: + +| condition | status | what the caller sees | +|----------------------------------------|---------------------|------------------------------------| +| cursor is inside the journal window | `healthy` | results presented as current | +| journal was recreated | `stale` | volume marked degraded, rebuild | +| cursor fell behind `FirstUsn` (rollover) | `stale` | volume marked degraded, rebuild | +| journal moved backwards | `stale` | volume marked degraded, rebuild | +| volume handle cannot be opened | `permission-denied` | volume marked degraded, no rebuild | +| volume is not mounted | `offline` | volume marked degraded | + +The UI and `--index-status` surface these states. A degraded volume is reported +as degraded; it is never silently presented as up to date. + +## Service crash and volume loss + +* A crash of `lce-index-service.exe` cannot crash the desktop, the CLI or the + MCP server: they hold a pipe, and a closed pipe is an I/O error that triggers + the in-process fallback. +* One volume failing does not stop another. Each volume has its own worker, + status and cursor; the supervisor reports per-volume health. +* A worker that cannot read its journal backs off exponentially to 30 seconds + instead of spinning, and marks only its own volume degraded. +* Shutdown is a flag plus a bounded wait, so the service stops promptly even + while a journal read is in flight. + +## Authentication of the caller + +The DACL answers "may this account connect". It does **not** distinguish two +processes belonging to the same user, and this build does not attempt to: the +service serves the interactive user that installed it, and nothing more +sensitive than an index query is exposed. There is no admin-only operation +reachable over the pipe — `RebuildVolume` is the most expensive one, and a +rebuild is a read of the volume the user can already see. + +## What is still out of scope + +* The service is not a sandbox for a compromised same-user process. Such a + process can already do everything the service can. +* There is no signature verification of the client. On a machine where the + user's own account is hostile to itself, this design adds nothing. +* The `Changes` feed is volume-granular (it reports journal advances), not a + per-file event stream. It is honest about that rather than pretending to + deliver events it does not have. \ No newline at end of file diff --git a/docs/archive/development-notes/upgrade-delivery-2026-10.md b/docs/archive/development-notes/upgrade-delivery-2026-10.md new file mode 100644 index 0000000..53e0379 --- /dev/null +++ b/docs/archive/development-notes/upgrade-delivery-2026-10.md @@ -0,0 +1,74 @@ +# 双仓轻量架构升级交付记录（2026-10-13） + +本轮保持两个仓库独立；无跨仓 crate、统一核心、数据库、常驻服务或新 GUI。 +复用既有 Provider、Tree-sitter、Git、Verification、SARIF、Fleet 和 tracing。 + +## 变更与架构差异 + +| 仓库 / 阶段 | 之前 | 本轮改变 | PR | +|---|---|---|---| +| LCE 查询与索引 | 短词/路径条件可触发整表扫描；常见 postings 在构建时大量分配 | 逐条件选择候选源、按基数规划；提前丢弃高频 postings；借用候选切片；仅保留目录路径缓存；修复长 prefix 漏检 | https://github.com/styayur/local-context-engine/pull/19 | +| CodeForge Fleet | 部分非通过状态可跨过必需门禁；任务/PR 身份不稳定 | 必需门禁要求明确 Passed；结构化 Task ID、确定性分支、PR 查询失败时拒绝新建；任务与证据 tracing 关联 | https://github.com/styayur/codeforge-engine/pull/21 | +| CodeForge 上下文 | 无统一按需源码关系提取 | 有界、只读 Tree-sitter 符号/候选调用图，Git 变更反向影响遍历、测试提示、带位置与内容哈希的源码片段 | https://github.com/styayur/codeforge-engine/pull/22 | +| CodeForge 外部分析 | SARIF 字段存在非标准 snake_case；无显式路径证据入口 | 标准 SARIF 兼容旧格式；分析路径关联；显式本地 CodeQL Adapter；限制输入/输出/超时；修复继承管道无限等待 | https://github.com/styayur/codeforge-engine/pull/24 | +| CodeForge 修复精度 | 长度规则可将整个复合条件误替换 | 仅匹配直接 AST 操作数；未解析类型的建议不获得 safe-fix 权限；补充幂等性与精确范围测试 | https://github.com/styayur/codeforge-engine/pull/25 | +| CodeForge 发布准备 | Rust/桌面 0.2.1 | 0.2.2 版本对齐及发布说明 | https://github.com/styayur/codeforge-engine/pull/23 | + +## 性能与正确性 + +LCE：Windows release、固定 seed、100 万记录；每组 3 次预热、40 个样本；限额 2000 的计时与完整有序差分分开。 + +| 指标 | 升级前 | 升级后 | +|---|---:|---:| +| 索引构建 | 11934.31 ms | 8908.72 ms | +| 构建进程峰值工作集 | 1,005,248,512 B | 258,981,888 B | +| 估算保留索引 | 75,796,186 B | 75,796,186 B | +| `ext:pdf re` 平均 / P95 | 30.042 / 34.343 ms | 7.428 / 9.388 ms | +| `ext:pdf report` 平均 / P95 | 89.360 / 99.741 ms | 20.953 / 23.142 ms | +| `prefix:design report` 平均 / P95 | 87.849 / 93.390 ms | 19.886 / 21.900 ms | +| 常见词 `report` 平均 / P95 | 8.336 / 9.611 ms | 8.472 / 9.867 ms | +| 实际 localsearch.exe | 2,346,496 B | 2,349,568 B | +| 完整有序差分 | — | 8/8 一致 | + +峰值内存下降约 74.2%，构建时间下降约 25.4%；保留索引估算不变。常见词回退没有明显加速，不作加速承诺。 +LCE Cargo.lock 共 492 个包，其中外部包 475 个，数量不变。 + +CodeForge：默认 `scan fixtures/python --json`，3 次预热、30 个样本。原始基线平均/P95 为 3239.81/4373.10 ms；相邻复测基线为 2745.89/3530.67 ms；功能阶段为 2699.55/3208.55 ms。路径工具发现和本机负载影响较大，不归因为架构带来的加速。输出摘要一致。 +实际 CLI 从 15,139,328 B 到 15,563,264 B，增加 423,936 B（2.80%）。六轮交替测量未复现显著工作集回归；完整读数见 benchmarks/UPGRADE.md。Cargo.lock 共 516 个包，其中外部包 501 个，未增加 Rust 依赖。前端 source-map-js 从 1.2.1 修补至 1.2.2，无新增包。 + +## 验证与边界 + +本地运行原始基线测试、修改后的 workspace 测试/clippy、百万数据差分、索引变更/持久化一致性、Unicode/长 prefix/限额排序测试。CodeForge 补充稳定任务与 PR、必需门禁状态矩阵、图歧义/循环/预算/过期源码、SARIF 标准及旧格式、不可信修复、未知 URI base、失败进程、畸形输出及继承管道测试。真实 CLI 冒烟完成。 + +每个 PR 继续运行原有 CI、CodeForge Regression Gate 与安全检查；失败不合并，也未放宽 Baseline。Release 仅针对检查通过的提交发布。 + +证据分级：调用关系和测试定位是启发式；源码片段是语法提取；SARIF 是外部静态分析；编译/测试需要实际命令成功；上述任一项都不自动构成语义等价证明。现有事务、回滚、Baseline、权限和 MFT/USN 边界保留。 + +CodeQL 未安装，使用编译出的本地替身验证协议及故障路径，未声称完成真实 CodeQL 查询集兼容验证。适配器不是 OS 沙箱：不会自动安装或默认运行；用户显式运行的 CLI 可能修改数据库/联网；超时终止直接子进程且限制管道等待，不保证回收全部后代。 +图不解析类型、导入、动态分派、宏和被删除符号；partial/歧义显式输出；相关测试建议不代替完整 Verification。 + +## 许可证和后续路线 + +只移植思想和接口，未复制框架源码。Renovate/Repowise 的 AGPL 边界、zvec-grep 的 Apache-2.0、jevgrep/Jeview 的 MIT 均已核查。CodeQL CLI 使用和分发限制独立于查询源码许可证；不捆绑 CLI 或查询包，由用户提供符合资格的本地安装。 + +后续按实测收益推进：真实项目 CodeQL 兼容矩阵与进程树隔离；模块/导入解析改善关系图精度；仅在保留 postings 成为内存主因时评估 delta/varint；全文/语义通过显式启用的 Provider 扩展，继续避开默认热路径。 + +## 合并与发布状态 + +LCE #19，CodeForge #21/#22/#23/#24/#25 均在各自的所有 CI、Regression Gate 及适用安全检查通过后合并。两仓主工作区已快进至 main 且无未提交源码。 + +- LCE main：1bb29900a36880ae26dd91af73274741ca3292d0；v0.2.0 指向该提交。 +- CodeForge main：cb390cee1e2cb8baa5b4d73daeb773e1e97b582d。 +- CodeForge v0.2.2：4a0c96e83ad08dcd7b89f6d56f1620e5b9efd256；包含 Fleet、安全规则与版本对齐，保留 prerelease 标记。#22/#24 的可选能力在 main，未纳入本次补丁包。 +- LCE 打包：https://github.com/styayur/local-context-engine/actions/runs/38396083419 +- CodeForge 打包：https://github.com/styayur/codeforge-engine/actions/runs/38396195845 + +合并后的 CodeForge main 已再次通过完整 workspace 测试及 all-targets/all-features clippy，并完成 release 构建和 context/import-sarif CLI 冒烟。最终 main 的 CLI 为 15,565,312 B（相对基线 +425,984 B，约 +2.81%）；前表的阶段测量为 15,563,264 B，差异来自最后的规则与版本集成。 + +发布已完成，发布流水线与最终 main CI 均成功： + +- [LCE v0.2.0](https://github.com/styayur/local-context-engine/releases/tag/v0.2.0)：Windows 便携包、安装程序及 SHA256SUMS。两份二进制产物均已实际下载并验证 SHA-256。 +- [CodeForge v0.2.2 prerelease](https://github.com/styayur/codeforge-engine/releases/tag/v0.2.2)：Windows EXE/MSI、Linux AppImage/DEB 及 checksums。四份产物的清单摘要均与 GitHub 上传摘要一致；已修正清单文件名为实际下载名称，并复核发布后的清单摘要。 + +CodeForge 现有打包流程对带空格文件名生成清单，而 GitHub 上传时会转换成点号；本次 Release 已修正，后续应把文件名规范化移入打包流程，避免再次人工修正。 +两仓主工作区均为 main，无未提交改动。 diff --git a/docs/UPGRADE-2026-10.md b/docs/archive/upgrade-2026-10.md similarity index 100% rename from docs/UPGRADE-2026-10.md rename to docs/archive/upgrade-2026-10.md diff --git a/docs/releases/v1.0.0.md b/docs/releases/v1.0.0.md new file mode 100644 index 0000000..cf6c1e5 --- /dev/null +++ b/docs/releases/v1.0.0.md @@ -0,0 +1,44 @@ +# Local Context Engine v1.0.0 + +**Release draft — unreleased.** Available binaries remain v0.2.0. No v1.0.0 +artifact, platform certification or new compatibility guarantee is announced. + +## Why it exists + +Local Context Engine brings files, applications, processes, services and windows +into one explainable local query on Windows. It indexes file names, paths and +metadata without uploading source or requiring a cloud account or model. + +## Capabilities + +Stable workflows include the query DSL, English/Chinese rule-based compilation, +CLI/MCP access, deterministic ranking for fixed inputs and provider snapshots, +and selective extension/prefix/trigram planning with authoritative verification. +Index health and explanations help distinguish an empty result from unavailable +or stale data. Optional NTFS MFT/USN service deployment and desktop acceptance +across machines remain Experimental. Full-text and semantic retrieval are Future. + +## Validation and performance + +The existing million-record probe matches every ordered result to the linear +reference for all eight queries. Recorded accelerator construction time changed +from 11.93 s to 8.91 s; construction process peak working set changed from +1005.25 MB to 258.98 MB. Retained accelerator estimates stayed at 75.80 MB. +Common `report` latency did not improve. These Windows/Rust 1.98.1 synthetic +observations are not end-to-end or universal latency promises, and hardware +provenance is incomplete. Regression tests cover mutations, persistence and +query correctness; final v1.0.0 artifacts still require release validation. + +## Security and limitations + +Broad/short/fuzzy queries can still scan. Live providers depend on permissions +and changing system state. The optional elevated MFT service can expose file +names that an ordinary directory walk cannot list; normal file-content access +checks remain in force. Local indexes and usage data are sensitive. Desktop +build success alone does not establish working rendering on every machine. + +The project remains independent, offline-first and MPL-2.0 licensed. No new +runtime dependency, core logic or protocol change is part of this documentation +preparation. Desktop license metadata alignment, version alignment, +minimum-toolchain validation, complete benchmark +environment capture and final installer/service smoke tests remain release gates.