diff --git a/COOKBOOK.md b/COOKBOOK.md index 9394ed5..89a6116 100644 --- a/COOKBOOK.md +++ b/COOKBOOK.md @@ -3498,6 +3498,104 @@ hprscript -p TODO -name t -file-where 'count(t) >= 3 AND churn(30) > 2' -llm -gl hprscript -p TODO -c -order-by score -glob '**/*.go' ``` +### 34.8 "Which functions are involved, and how heavily?" — scope rollup + +**Problem:** Per-match output answers "which lines"; `-hotspots` answers "which files". The first question of an investigation usually sits in between: which *functions*, and how many hits in each — ten hits inside one function is one fact, not ten. `-rollup` prints one line per innermost enclosing scope (range, kind, name, hit count with per-pattern breakdown) plus the scope's first matched line as a representative, then joins the patterns per file in a co-occurrence footer. + +**Input:** Source tree. + +```bash +hprscript -p 'sha1' -name weak_hash -p 'rand\.Intn' -name weak_rand -rollup -glob '**/*.go' +# → crypto/sign.go +# → (top level) — 2 hits (weak_hash×2) +# → 3: import "crypto/sha1" +# → 6-11 func SignToken — 2 hits (weak_hash×2) +# → 7: h := sha1.New() +# → crypto/token.go +# → 5-8 func Nonce — 1 hit (weak_rand×1) +# → 6: n := rand.Intn(1 << 20) +# → --- files: weak_hash 1, weak_rand 1; both: 0 --- +``` + +Matches outside any detected scope — including whole files in languages without a scope pack — group under `(top level)`, so a config file or log collapses to one row per file. Drill into a specific function afterwards with `hprscript expand` ([recipe 34.11](#3411-the-search--expand-loop--verified-refs-that-survive-edits)), `-in-scope`, or `-elide`. + +### 34.9 Result blocks that explain themselves — query legend, absence, and correlation + +**Problem:** An agent (or teammate) reading a saved result block hours later sees matches for patterns named `p0`/`p1` with no record of what was asked, and a pattern that matched *nothing* simply vanishes — even when "no hits" was the finding. `-desc` attaches a human meaning to each pattern and turns on a query-legend header; the `-llm`/`-elide`/`-rollup` modes always name zero-match patterns in a footer and report the per-file pattern overlap. + +**Input:** Source tree; output destined to be read out of context. + +```bash +hprscript -p 'sha1' -name weak_hash -desc 'weak hash algorithm usage' \ + -p 'rc4|des\.' -name weak_cipher -desc 'legacy cipher usage' -llm crypto/sign.go +# → query: 2 patterns over crypto/sign.go +# → weak_hash — weak hash algorithm usage +# → weak_cipher — legacy cipher usage +# → crypto/sign.go +# → 3: [weak_hash] import "crypto/sha1" [string] +# → 5: [weak_hash] // SignToken still uses sha1 — TODO migrate [comment] +# → 7: [weak_hash] h := sha1.New() +# → 9: [weak_hash] s := "sha1 legacy" [string] +# → --- no matches: weak_cipher (1 of 2 patterns) --- +``` + +The zero-match and co-occurrence counts cover the whole scan (post-filter), not just displayed lines, so they stay truthful under `-sample`/`-hotspots`; when `-limit` or `-max-output-bytes` stopped the scan early the footer says `(scan stopped early)`. Absence of overlap is stated too: `both: 0` is a finding, not silence. + +### 34.10 Is that hit code, or a comment, string, or import? — role tags + +**Problem:** Half the triage effort after a search is discarding hits that only *mention* the term — in a comment, inside a string literal, on an import line. hprscript classifies every hit lexically by default: `-llm` appends bracket tags, JSONL grows a `"role"` field (omitted for plain code), `-format` gets `$ROLE`, and with `-scope` active a hit on the signature line itself reads `[def …]` rather than `[in …]`. + +**Input:** Source tree in a recognized language (scope-pack languages plus Python, shell, Ruby, YAML, TOML). + +```bash +hprscript -p 'sha1' -p 'import' -p 'func Sign\w+' -llm -scope auto crypto/sign.go +# → crypto/sign.go +# → 3: [p1] import "crypto/sha1" [import] +# → 3: [p0] import "crypto/sha1" [string] +# → 5: [p0] // SignToken still uses sha1 — TODO migrate [comment] +# → 6: [p2] func SignToken(b []byte) string { [def func SignToken] +# → 7: [p0] h := sha1.New() [in func SignToken] +# → 9: [p0] s := "sha1 legacy" [string] [in func SignToken] + +# JSONL: filter comment-only mentions out mechanically +hprscript -p 'TODO' -glob '**/*.go' | jq -c 'select(.role != "comment")' +``` + +Precedence is position-accurate: the `sha1` inside the quoted import path is `[string]`, while the `import` keyword on the same line is `[import]`. Classification is lexical, computed lazily only for files that produce output; `-no-roles` turns it off. + +### 34.11 The search → expand loop — verified refs that survive edits + +**Problem:** An agent finds a hit, works on something else, then needs the surrounding function — but the file may have changed since, and a saved `file:line` silently points at the wrong code. `-refs` stamps each hit's line number with a hash of the line's content; `hprscript expand` turns any `file:line[@hash]` ref into the full enclosing scope, verifying the hash first. + +**Input:** Source tree, across multiple agent turns with edits in between. + +```bash +# 1. Search once with -refs — every hit line carries a content hash. +hprscript -p 'sha1\.New' -llm -refs crypto/sign.go +# → crypto/sign.go +# → 7@9679f2: h := sha1.New() + +# 2. Any time later: expand refs (batch several in one call) into full scopes. +hprscript expand crypto/sign.go:7@9679f2 +# → crypto/sign.go:6-11 func SignToken +# → func SignToken(b []byte) string { +# → h := sha1.New() +# → h.Write(b) +# → s := "sha1 legacy" +# → return s +# → } + +# 3. A line was inserted above in the meantime? The hash no longer matches +# line 7, so expand recovers the line by content and says so: +# → crypto/sign.go:7-12 func SignToken (ref line moved: 7 → 8) + +# 4. The line is gone entirely? Stale is reported in-band, exit code 3 — +# expand never silently renders the wrong code: +# → crypto/sign.go:7@9679f2: stale — line 7 changed and no line with this hash exists +``` + +A ref without `@hash` still works (no verification, plain line lookup), and a ref outside any detected scope falls back to a numbered `-C`-sized context window with the ref line marked `>`. Exit codes: 0 all refs expanded, 3 at least one stale, 2 bad ref syntax or unreadable file. + --- ## See also diff --git a/HPRSCRIPT.md b/HPRSCRIPT.md index f4bdc73..8d050a5 100644 --- a/HPRSCRIPT.md +++ b/HPRSCRIPT.md @@ -2396,7 +2396,7 @@ hprscript -p 'TODO' -llm -limit 1 -glob '**/*.go' - **Quick human eyeballing.** The grouped layout reads like `grep -n` output rather than JSON Lines — easier to skim. - **You don't need byte offsets / capture groups.** Pattern IDs still show up (as `[]` tags when multiple patterns are active), but if a downstream tool needs `from`/`to` offsets or the `extracted` map, stick with `-j`. -`-llm` is mutually exclusive with the other output modes (`-j`, `-f`, `-c`, `-o`, `-format`, `-absent`, `-elide`). +`-llm` is mutually exclusive with the other output modes (`-j`, `-f`, `-c`, `-o`, `-format`, `-absent`, `-elide`, `-rollup`). --- diff --git a/README.md b/README.md index bdffe36..10ccea1 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ It is a single self-contained binary with no runtime dependencies beyond the pla | Multi-pass workflows in one process (collect → resolve) | ❌ | ✅ via [phases](HPRSCRIPT.md#phases) | | Per-file aggregation (counts, ranking, grouping) in one process | ❌ | ✅ via [scripts](HPRSCRIPT.md#script-mode--s---script) | | Files **missing** a pattern | `grep -L` | `-absent` (also works inside scripts) | +| Output an agent can interpret cold (role tags, query legend, absence/overlap footers) | ❌ | ✅ [LLM-facing modes](HPRSCRIPT.md#llm-output-mode) | | Pattern compile cost scales with N patterns | linear | constant — patterns share one DFA | If you find yourself piping `grep` into `grep`, running ripgrep in a loop over a list of patterns, or writing throwaway Python to aggregate match counts per file, those are the workloads `hprscript` is designed for. @@ -66,6 +67,9 @@ Default per-match record: - **Script mode (JSON DSL).** Variables, lifecycle hooks, sub-pattern matching, conditionals, grouping, ranking, and multi-phase scans — all in one invocation. See [Script mode](HPRSCRIPT.md#script-mode--s---script). - **`-pi` per-pattern case-insensitivity.** Mix case-sensitive and case-insensitive patterns in the same scan. - **`-absent` mode.** Find files where a pattern is *not* found (like `grep -L`, but also works inside scripts). +- **LLM-facing output modes.** `-llm` compact per-match text, `-elide` folded scope excerpts, `-rollup` one line per function with per-pattern counts — framed by an optional query legend (`-desc`) and automatic no-match / co-occurrence footers. See [below](#self-describing-output-for-llm-agents). +- **Per-match role tags.** Every hit self-classifies as definition, comment, string, or import — `[def func X]`/`[comment]`/`[string]`/`[import]` in `-llm`, a `"role"` field in JSONL, `$ROLE` in `-format`. See [Per-match role tags](HPRSCRIPT.md#per-match-role-tags). +- **Search → expand loop.** `-refs` stamps each hit as `file:line@hash`; `hprscript expand` later renders the full enclosing function — recovering moved lines by content and refusing stale ones instead of showing the wrong code. See [Stable refs & expand](HPRSCRIPT.md#stable-refs--expand--refs--hprscript-expand). - **Unicode by default.** UTF-8 mode is on; `-pi` folds across scripts (`CAFÉ` ↔ `café`, `ПРИВЕТ` ↔ `привет`). See [UTF-8 / Unicode](HPRSCRIPT.md#utf-8--unicode-support). - **grep-compatible output modes:** `-f` (file list), `-c` (per-file counts), `-o` (matched text only), `-format` (custom template), `-A`/`-B`/`-C` (context lines). - **Single static binary** — no runtime dependencies beyond `libc`/`libm`/`libpthread`. @@ -147,6 +151,39 @@ grouping/aggregation, ordering, strict resource limits, and adaptive derived-pattern stages. Compatible sets share one matcher/traversal; equality joins use hash indexes. See [Declarative query mode](HPRSCRIPT.md#declarative-query-mode-hprscript-query). +## Self-describing output for LLM agents + +Search results are often read far from the command that produced them — by an +agent several turns later, or by a different agent entirely. The LLM-facing +modes (`-llm`, `-elide`, `-rollup`) make each result block carry its own +interpretation: a query legend says what each pattern means, role tags classify +every hit, and footers state absence and correlation explicitly. + +```bash +hprscript -p 'sha1' -name weak_hash -desc 'weak hash algorithm usage' \ + -p 'rand\.Intn' -name weak_rand -rollup -refs -glob '**/*.go' +# query: 2 patterns over **/*.go +# weak_hash — weak hash algorithm usage +# weak_rand — /rand\.Intn/ +# crypto/sign.go +# (top level) — 2 hits (weak_hash×2) +# 3@6db2fd: import "crypto/sha1" +# 6-11 func SignToken — 2 hits (weak_hash×2) +# 7@9679f2: h := sha1.New() +# crypto/token.go +# 5-8 func Nonce — 1 hit (weak_rand×1) +# 6@bc19e4: n := rand.Intn(1 << 20) +# --- files: weak_hash 1, weak_rand 1; both: 0 --- + +# Later — possibly after edits — turn any ref into the full function, verified: +hprscript expand crypto/sign.go:7@9679f2 +``` + +- **Role tags** classify every hit lexically (`def` / `comment` / `string` / `import`) in `-llm`, JSONL (`"role"`), and `-format` (`$ROLE`) — [reference](HPRSCRIPT.md#per-match-role-tags). +- **`-rollup`** prints one line per enclosing function with per-pattern hit counts — "where is this concentrated?" answered in a handful of tokens — [reference](HPRSCRIPT.md#scope-rollup--rollup). +- **Query legend and footers.** `-desc` opens the block with each pattern's meaning; patterns that matched nothing are named explicitly (absence is evidence: `--- no matches: weak_cipher (1 of 2 patterns) ---`), and a co-occurrence footer reports per-pattern file counts plus their overlap — [reference](HPRSCRIPT.md#query-header-and-result-footers). +- **Stable refs.** `hprscript expand file:line@hash` recovers moved lines by content and reports vanished ones as stale (exit 3) rather than rendering the wrong code — [reference](HPRSCRIPT.md#stable-refs--expand--refs--hprscript-expand). + --- ## Install @@ -227,7 +264,7 @@ make test `hprscript` ships two portable agent skills: -- [`skills/hprscript-search/SKILL.md`](skills/hprscript-search/SKILL.md) teaches read-only quick search, investigation, query, ranking, and context packing. +- [`skills/hprscript-search/SKILL.md`](skills/hprscript-search/SKILL.md) teaches read-only quick search, investigation, query, ranking, context packing, scope surveys (`-rollup`), and the search → expand loop. - [`skills/hprscript-edit/SKILL.md`](skills/hprscript-edit/SKILL.md) teaches persistent edit plans, guarded application, and the boundary with localized semantic patching. Each is a single Markdown file with YAML frontmatter and focused instructions. Install either skill independently or install both for the complete workflow. They drive the CLI binary directly, carry inline cheat sheets, and point at [`HPRSCRIPT.md`](HPRSCRIPT.md) / [`COOKBOOK.md`](COOKBOOK.md) for depth. diff --git a/skills/hprscript-edit/SKILL.md b/skills/hprscript-edit/SKILL.md index 49db363..9cb38c2 100644 --- a/skills/hprscript-edit/SKILL.md +++ b/skills/hprscript-edit/SKILL.md @@ -31,6 +31,8 @@ The plan binds review to exact paths, file identities and hashes, byte ranges, o - Narrow with `-glob`, `-exclude`, `-git-changed`, `-git-staged`, `-git-range`, `-git-added-lines`, `-lines`, `-in-scope`, `-near`, `-far`, `-same-scope`, `-not-same-scope`, or `-file-where`. - Prefer a stable scope name over line numbers when code may move. - Run `hprscript -list-scopes -llm ` before a scope or function-body replacement. +- Read the exact current body with `hprscript expand :` before replacing a scope or block; a `file:line@hash` ref from a `-refs` search is verified first and reports stale (exit 3) instead of rendering outdated code. +- Search with `-llm` first when a pattern could hit non-code text: role tags mark `[comment]`, `[string]`, and `[import]` sites the edit would rewrite; narrow the pattern or targeting before planning. - Mark relation-only patterns with `-ref`; otherwise their matches also become edit sites. ## Choose the edited span diff --git a/skills/hprscript-search/SKILL.md b/skills/hprscript-search/SKILL.md index 81b3f23..25c6615 100644 --- a/skills/hprscript-search/SKILL.md +++ b/skills/hprscript-search/SKILL.md @@ -13,7 +13,7 @@ Invoke the binary through Bash as `hprscript`. Use one call per reasoning stage - Put distinguishable terms in separate `-p` or `-pi` flags so every hit retains its pattern ID. - Prefer `-llm` when reading results, `-f` for paths, `-c` for counts, and `-limit N` for existence checks. In `-llm`/`-elide`/`-rollup` output, patterns with zero matches are named in a trailing `--- no matches: … ---` footer — treat that as explicit evidence of absence, qualified with "scan stopped early" when a limit cut the scan. With ≥2 matching patterns a `--- files: … ---` footer gives per-pattern file counts and the overlap (`both:`/`multi-pattern:`) — read the correlation from there instead of joining by hand. -- Trust per-match role tags instead of re-deriving them: `[comment]`/`[string]`/`[import]` in `-llm` (a `role` field in JSONL, `$ROLE` in `-format`) classify each hit lexically, and with `-scope` active a hit on a signature line reads `[def func X]` while body hits read `[in func X]`. Untagged hits in a recognized language are plain code. +- Trust per-match role tags instead of re-deriving them: `[comment]`/`[string]`/`[import]` in `-llm` (a `role` field in JSONL, `$ROLE` in `-format`) classify each hit lexically, and with `-scope` active a hit on a signature line reads `[def func X]` while body hits read `[in func X]`. Untagged hits in a recognized language are plain code; `-no-roles` disables tagging. - Use an absolute path or glob when the effective cwd is uncertain. Inspect the first emitted path and stop if it escapes the intended tree. - Add `-summary -require-complete` when a broad sweep must be exhaustive. Do not present a partial scan as complete. - Restructure unsupported lookarounds or backreferences, or express the relationship with `query` or script phases. Do not fall back to grep or rg. @@ -38,7 +38,7 @@ Invoke the binary through Bash as `hprscript`. Use one call per reasoning stage | Compact scope excerpts | `-elide` | | Cross-run chunk deduplication | `-elide` or `-budget` with `-seen ` | -Cheapness ladder: `-f` / `-absent` / `-c` are cheaper than `-o` / `-llm`, which are cheaper than default JSONL. +Cheapness ladder: `-f` / `-absent` / `-c` are cheaper than `-rollup`, which is cheaper than `-o` / `-llm`, which are cheaper than default JSONL. ## Quick search @@ -52,7 +52,7 @@ hprscript -ident 'parse config' -glob '**/*.go' - `-p` / `-pi`: case-sensitive / case-insensitive regex, repeatable. - `-F` / `-Fi`: literal fixed strings. - `-name `: name the preceding pattern for output, relations, and `-file-where`. -- `-desc `: describe the preceding pattern; `-llm`/`-elide` output then opens with a query legend, keeping the result block self-describing for later readers. +- `-desc `: describe the preceding pattern; `-llm`/`-elide`/`-rollup` output then opens with a query legend, keeping the result block self-describing for later readers. - `-patterns-from `: load a JSONL rule pack (entries may carry a `description`). - `-ident ''`: find identifier variants such as `parseConfig`, `parse_config`, and `ConfigParser`. - `-w`: wrap all patterns in word boundaries; use inline `\b` for per-pattern control. @@ -95,14 +95,17 @@ hprscript -p '\bLock\(\)' -name lock -p '\bUnlock\(\)' -name unlock \ Use `-near A:B:K`, `-far A:B:K`, `-same-scope A:B`, and `-not-same-scope A:B`. Multiple relations combine with AND. -## Rank and pack context +## Survey, rank, and pack context ```bash +hprscript -p 'AuthMiddleware' -p 'validateToken' -rollup -glob '**/*.go' hprscript -p 'AuthMiddleware' -p 'validateToken' -hotspots 5 -llm -glob '**/*.go' hprscript -p 'AuthMiddleware' -p 'validateToken' -budget 8000 -glob '**/*.go' hprscript -p 'AuthMiddleware' -elide -seen /tmp/auth-review.seen -glob '**/*.go' ``` +- `-rollup`: one line per innermost enclosing scope — line range, kind, name, hit count with a per-pattern breakdown, and the scope's first matched line; matches outside any scope group under `(top level)`. Use it as the first call of an investigation, then drill into specific scopes with `expand`, `-in-scope`, or `-elide`. +- `-refs`: stamp `-llm`/`-rollup` line numbers as `line@hash` so a later `expand` verifies content before rendering. - `-hotspots N`: rank files and expose dense match windows. - `-budget N`: rank and render full or compact chunks, with an explicit dropped-items footer. - `-elide`: show scope signatures and matched lines while folding untouched interiors.