From 57f5644799c96d2f74b933917019f3a994bcd484 Mon Sep 17 00:00:00 2001 From: Pinkhas Nisanov Date: Thu, 13 Aug 2026 20:27:51 +0300 Subject: [PATCH 1/4] Add -llm query legend (-desc) and zero-match footer Two LLM-facing output additions for -llm/-elide modes: - A pattern that ends the scan with zero kept matches is now named in a trailing '--- no matches: ... ---' footer instead of vanishing silently; qualified with '(scan stopped early)' when -limit or -max-output-bytes cut the scan. Counted over the whole scan, not the displayed subset, so -sample/-hotspots stay truthful. - New -desc postfix modifier (and 'description' field in -patterns-from rules): when any pattern carries one, output opens with an echo of the interpreted query plus a one-line legend per pattern, making each result block self-describing. Input tokens are shared with -explain-plan so the two renderings cannot drift. Co-Authored-By: Claude Fable 5 --- HPRSCRIPT.md | 47 +++++++++++++++-- skills/hprscript-search/SKILL.md | 5 +- src/cli.cpp | 37 +++++++++++-- src/cli.hpp | 4 ++ src/common.hpp | 4 ++ src/output.cpp | 37 +++++++++++++ src/output.hpp | 25 +++++++++ src/pipeline.cpp | 2 + src/runner.cpp | 91 ++++++++++++++++++++++++++++---- tests/run.sh | 60 +++++++++++++++++++++ 10 files changed, 291 insertions(+), 21 deletions(-) diff --git a/HPRSCRIPT.md b/HPRSCRIPT.md index cfe76a4..89627f8 100644 --- a/HPRSCRIPT.md +++ b/HPRSCRIPT.md @@ -102,7 +102,8 @@ simultaneous. | `-pi ` | Case-insensitive search pattern (HS `CASELESS`; folds Unicode by default; repeatable, mixable with `-p`) | | `-F ` / `-Fi ` | Fixed-string pattern (case-sensitive / -insensitive) — matched literally, no regex interpretation. Repeatable, mixable with `-p`/`-pi`. | | `-name ` | Name the preceding `-p`/`-pi`/`-F`/`-Fi`/`-ident`: the id (`[A-Za-z_]\w*`) replaces the auto `p`/`ident` in `pat`, `$PAT_ID`, `-llm` tags, relations, and `-file-where`. | -| `-patterns-from ` | Load additional patterns from a JSONL rule file — one `{"id","regexp"\|"literal","case_insensitive","word_boundary","utf8","ref"}` object per line, `#` comments allowed. Repeatable. See [Fixed strings & pattern files](#fixed-strings--pattern-files--f--fi--patterns-from). | +| `-desc ` | Describe the preceding pattern in free text. When any pattern has one, `-llm`/`-elide` output opens with a [query legend](#query-header-and-no-matches-footer) so each result block is self-describing. | +| `-patterns-from ` | Load additional patterns from a JSONL rule file — one `{"id","regexp"\|"literal","description","case_insensitive","word_boundary","utf8","ref"}` object per line, `#` comments allowed. Repeatable. See [Fixed strings & pattern files](#fixed-strings--pattern-files--f--fi--patterns-from). | | `-ident ''` | Match identifiers whose subtokens include ALL given space-separated terms, regardless of casing/separator (`parseConfig` ~ `parse_config`). Repeatable = OR. See [Identifier matching](#identifier-matching--ident). | | `-file-where ` | Per-file predicate: pattern ids, plus `count(pat) > n` / `churn(days) > n` / `lang == name` conditions (`'err AND NOT recovery'`, `'churn(30) > 2'`). See [Per-file conditions](#per-file-conditions--file-where). | | `-order-by ` | Sort `-f`/`-c` output by `score`/`count`/`path` instead of walk order. See [Sorting file-grouped output](#sorting-file-grouped-output--order-by). | @@ -366,6 +367,20 @@ Names must be identifiers (`[A-Za-z_][A-Za-z0-9_]*`) and unique — a collision with another name or with a different pattern's auto id (`p0`, `p1`, …) is rejected at startup. +`-desc ` is the same kind of postfix modifier, but for meaning instead +of identity: free text explaining what the pattern is *for*. It surfaces as a +one-time query legend at the top of `-llm`/`-elide` output (see +[Query header and no-matches footer](#query-header-and-no-matches-footer)), +so a reader of the results — human or LLM — doesn't have to reverse-engineer +intent from the regex: + +```bash +hprscript \ + -p 'md5|sha1' -name weak_hash -desc 'weak hash algorithm usage' \ + -p 'rand\.Intn' -name weak_rand -desc 'non-crypto RNG in security context' \ + -llm -glob '**/*.go' +``` + --- ## Fixed strings & pattern files (`-F` / `-Fi` / `-patterns-from`) @@ -393,9 +408,11 @@ hprscript -patterns-from ioc-pack.jsonl -C 1 -glob '**/*.log' ``` Each entry takes exactly one of `regexp` or `literal`, plus optional `id` -(same rules as `-name`), `case_insensitive`, `word_boundary`, and `utf8` -(the latter two default to the global `-w` / `-no-utf8` flags). Unknown -fields are rejected with the file and line number. `-patterns-from` is +(same rules as `-name`), `description` (free text, same role as `-desc` — +shown in the `-llm`/`-elide` query legend), `case_insensitive`, +`word_boundary`, and `utf8` (the latter two default to the global `-w` / +`-no-utf8` flags). Unknown fields are rejected with the file and line +number. `-patterns-from` is repeatable, appends to any `-p`/`-F` patterns, and cannot be combined with `-s`/`-script` (scripts declare their own patterns). @@ -2268,6 +2285,28 @@ Layout: one **file header** per file (deduped — never repeated), then each mat When `-limit` or `-max-output-bytes` truncates the output, a final `--- limit reached: ... ---` (or `--- output-byte budget reached ... ---`) footer line is emitted so the reader knows the result was cut, not finished. +### Query header and no-matches footer + +Two additions make each `-llm`/`-elide` result block self-describing — useful when the output is read later (or by a different agent) than the command that produced it: + +**Query header.** When any pattern carries a description (`-desc`, or `description` in a `-patterns-from` rule file), the output opens with an echo of the interpreted query and a one-line legend per pattern — described patterns show their description, the rest fall back to their regexp so the legend is complete on its own: + +``` +query: 2 patterns over src, **/*.go + weak_hash — weak hash algorithm usage + p1 — /rand\.Intn/ +``` + +No `-desc` anywhere → no header, so existing output is unchanged unless you opt in. + +**No-matches footer.** A batched pattern that matched nothing would otherwise just vanish from the output — and absence is often the finding (`no callers of X` can be the whole answer). Any pattern that ends the scan with zero matches (after `-in-scope`/relations/`-file-where` filtering) is named explicitly in a trailing footer: + +``` +--- no matches: weak_rand (1 of 2 patterns) --- +``` + +The counts cover the whole scan, not just the displayed subset, so the footer stays truthful under `-sample`/`-hotspots`. When `-limit` or `-max-output-bytes` stopped the scan early the claim only holds for the scanned prefix, and the footer says so: `--- no matches (scan stopped early): … ---`. Exit codes are unaffected (still 1 when nothing matched at all). + ### Examples ```bash diff --git a/skills/hprscript-search/SKILL.md b/skills/hprscript-search/SKILL.md index c638030..6073dcc 100644 --- a/skills/hprscript-search/SKILL.md +++ b/skills/hprscript-search/SKILL.md @@ -12,7 +12,7 @@ Invoke the binary through Bash as `hprscript`. Use one call per reasoning stage ## Non-negotiable defaults - Put distinguishable terms in separate `-p` or `-pi` flags so every hit retains its pattern ID. -- Prefer `-llm` when reading results, `-f` for paths, `-c` for counts, and `-limit N` for existence checks. +- Prefer `-llm` when reading results, `-f` for paths, `-c` for counts, and `-limit N` for existence checks. In `-llm`/`-elide` output, patterns with zero matches are named in a trailing `--- no matches: … ---` footer — treat that as explicit evidence of absence, qualified with "scan stopped early" when a limit cut the scan. - Use an absolute path or glob when the effective cwd is uncertain. Inspect the first emitted path and stop if it escapes the intended tree. - Add `-summary -require-complete` when a broad sweep must be exhaustive. Do not present a partial scan as complete. - Restructure unsupported lookarounds or backreferences, or express the relationship with `query` or script phases. Do not fall back to grep or rg. @@ -49,7 +49,8 @@ hprscript -ident 'parse config' -glob '**/*.go' - `-p` / `-pi`: case-sensitive / case-insensitive regex, repeatable. - `-F` / `-Fi`: literal fixed strings. - `-name `: name the preceding pattern for output, relations, and `-file-where`. -- `-patterns-from `: load a JSONL rule pack. +- `-desc `: describe the preceding pattern; `-llm`/`-elide` output then opens with a query legend, keeping the result block self-describing for later readers. +- `-patterns-from `: load a JSONL rule pack (entries may carry a `description`). - `-ident ''`: find identifier variants such as `parseConfig`, `parse_config`, and `ConfigParser`. - `-w`: wrap all patterns in word boundaries; use inline `\b` for per-pattern control. diff --git a/src/cli.cpp b/src/cli.cpp index 0ce10cd..3e9a300 100644 --- a/src/cli.cpp +++ b/src/cli.cpp @@ -192,9 +192,11 @@ void print_help(FILE *out) { " -Fi Case-insensitive fixed-string pattern\n" " -name Name the preceding pattern (shown as pat/$PAT_ID, usable\n" " as the A/B side of relations and in -file-where)\n" +" -desc Describe the preceding pattern; descriptions print once\n" +" as a query-legend header in -llm/-elide output\n" " -patterns-from Load patterns from a JSONL rule file: one object per\n" -" line {id, regexp|literal, case_insensitive, word_boundary,\n" -" utf8}; '#' comment lines allowed (repeatable)\n" +" line {id, regexp|literal, description, case_insensitive,\n" +" word_boundary, utf8}; '#' comment lines allowed (repeatable)\n" " -extract n1,n2,… Re-extract capture groups from the preceding -p/-pi\n" " -ident 't1 t2 …' Match identifiers whose subtokens include ALL given\n" " terms, regardless of casing/separator (parseConfig ~\n" @@ -234,7 +236,9 @@ void print_help(FILE *out) { " each pattern (record-level absence, e.g. JSONL fields)\n" " -llm Token-efficient text for LLM consumption (auto-detects\n" " block/scope; dedupes file paths; prints a 'limit reached'\n" -" footer when -limit or -max-output-bytes truncates output)\n" +" footer when -limit or -max-output-bytes truncates output;\n" +" ends with a 'no matches' footer naming patterns that\n" +" matched nothing)\n" " -elide Scope-aware chunks: signature + matched lines with -A/-B\n" " context; untouched interior lines fold as \"… (+N lines)\"\n" " (implies -scope auto when no -scope config is given)\n" @@ -624,6 +628,26 @@ Cli parse_cli(int argc, char **argv) { cli.patterns.back().name = v; continue; } + if (eq(a, "-desc")) { + const char *v = take(i, argc, argv, a, cli); if (!v) return cli; + if (cli.patterns.empty()) { + cli.error = true; + cli.error_message = "-desc must follow a -p/-pi/-F/-Fi"; + return cli; + } + if (v[0] == '\0') { + cli.error = true; + cli.error_message = "-desc: empty description"; + return cli; + } + if (!cli.patterns.back().desc.empty()) { + cli.error = true; + cli.error_message = "-desc repeated for the same pattern"; + return cli; + } + cli.patterns.back().desc = v; + continue; + } if (eq(a, "-glob")) { const char *v = take(i, argc, argv, a, cli); if (!v) return cli; cli.globs.emplace_back(v); @@ -1229,7 +1253,7 @@ bool load_patterns_from(Cli &cli) { if (kv.first != "id" && kv.first != "regexp" && kv.first != "literal" && kv.first != "case_insensitive" && kv.first != "word_boundary" && kv.first != "utf8" && - kv.first != "ref") + kv.first != "ref" && kv.first != "description") return fail("unknown field '" + kv.first + "'"); } const json::Value *re = pr.value.find("regexp"); @@ -1270,6 +1294,11 @@ bool load_patterns_from(Cli &cli) { if (!v->is_bool()) return fail("'ref' must be a boolean"); p.ref = v->as_bool(); } + if (const json::Value *v = pr.value.find("description")) { + if (!v->is_string()) + return fail("'description' must be a string"); + p.desc = v->as_string(); + } cli.patterns.push_back(std::move(p)); } } diff --git a/src/cli.hpp b/src/cli.hpp index cb7822c..86ec9fc 100644 --- a/src/cli.hpp +++ b/src/cli.hpp @@ -23,6 +23,10 @@ struct CliPattern { // `-p`/`-pi`, or by `id` in a -patterns-from file). Empty → auto `p`. // Shown as `pat`/$PAT_ID and usable as the A/B side of relations. std::string name; + // Free-text meaning (set by `-desc` on the most recently declared + // pattern, or by `description` in a -patterns-from file). When any + // pattern has one, -llm/-elide output opens with a query-legend header. + std::string desc; // Comma-separated capture group names (set by `-extract` on the most // recently declared `-p`/`-pi`). Maps capture group i+1 → names[i]. std::vector extract_names; diff --git a/src/common.hpp b/src/common.hpp index 1b6c27c..ba4e7fb 100644 --- a/src/common.hpp +++ b/src/common.hpp @@ -24,6 +24,10 @@ struct Pattern { bool ucp = false; double weight = 1.0; // reserved for ranking mode (post-MVP) + // Free-text meaning of the pattern (-desc / rule-file `description`). + // Surfaced once as a query-legend line in -llm/-elide output. + std::string desc; + // Reference-only pattern (edit mode's -ref): participates in matching — // relations and -file-where see its matches — but never produces edit // sites. The qualifier in `-far hit:allow:0` needs this, or its own diff --git a/src/output.cpp b/src/output.cpp index e6cf756..b9003a4 100644 --- a/src/output.cpp +++ b/src/output.cpp @@ -454,9 +454,25 @@ void Formatter::emit_custom(const std::string &file, const Pattern &pattern, write_out(out); } +void Formatter::emit_header() { + if (header_done_) return; + header_done_ = true; + if (opts_.mode != OutputMode::Llm && opts_.mode != OutputMode::Elide) + return; + if (opts_.header_lines.empty()) return; + auto &s = scratch_; + s.clear(); + for (const auto &line : opts_.header_lines) { + s += line; + s += '\n'; + } + write_out(s); +} + void Formatter::emit_llm(const std::string &file, const Pattern &pattern, const Match &m, std::string_view buf, const LineIndex &idx, const ScopeIndex *scope) { + emit_header(); auto &s = scratch_; // Per-file header: print the path once per consecutive run on that file. @@ -542,6 +558,7 @@ void Formatter::on_file_elide(const std::string &file, const ScopeIndex *scope, const SeenStore *seen, std::vector *marks_out) { if (kept.empty()) return; + emit_header(); emitted_ += static_cast(kept.size()); bool first_block = true; @@ -761,6 +778,26 @@ void Formatter::on_file_end(const std::string &file, bool had_match) { void Formatter::on_complete() { if (opts_.mode == OutputMode::Llm || opts_.mode == OutputMode::Elide) { char buf[160]; + if (!zero_match_ids_.empty()) { + // A batched pattern that matched nothing must be stated, not + // inferred from its absence in the output. When the scan stopped + // early (limit / output budget) the claim is only valid for the + // scanned prefix, so it's qualified. + emit_header(); + std::string s = (limit_hit_ || over_budget_) + ? "--- no matches (scan stopped early): " + : "--- no matches: "; + for (size_t i = 0; i < zero_match_ids_.size(); ++i) { + if (i) s += ", "; + s += zero_match_ids_[i]; + } + s += " ("; + s += std::to_string(zero_match_ids_.size()); + s += " of "; + s += std::to_string(patterns_total_); + s += patterns_total_ == 1 ? " pattern) ---\n" : " patterns) ---\n"; + std::fwrite(s.data(), 1, s.size(), out_); + } if (limit_hit_) { int n = std::snprintf(buf, sizeof(buf), "--- limit reached: stopped at %llu matches; more may exist (re-run with -limit 0 for all) ---\n", diff --git a/src/output.hpp b/src/output.hpp index b2d90d3..8b79587 100644 --- a/src/output.hpp +++ b/src/output.hpp @@ -82,6 +82,12 @@ struct OutputOptions { // (so on_complete can emit a "limit reached" footer). size_t pattern_count = 1; int64_t global_limit = -1; + + // LLM-facing query header (-llm/-elide only): pre-rendered lines, no + // trailing newlines, printed once before the first output. The runner + // fills this when any pattern carries a -desc/description; empty = no + // header. + std::vector header_lines; }; class Formatter { @@ -138,6 +144,22 @@ class Formatter { // hit. LLM mode prints a "limit reached" footer in on_complete(). void mark_limit_hit() { limit_hit_ = true; } + // Print the query header (opts_.header_lines) now if it hasn't printed + // yet. Idempotent; no-op outside Llm/Elide or with no header configured. + // Emission paths call it internally before the first output; the runner + // also calls it before rows that bypass the formatter (-hotspots -llm). + void emit_header(); + + // Runner reports which patterns ended the scan with zero kept matches + // (post-filter, whole scan — not just the shown subset), so + // on_complete() can state absence explicitly instead of letting a + // batched pattern vanish from Llm/Elide output silently. `total` is the + // full pattern count. + void set_zero_match_patterns(std::vector ids, size_t total) { + zero_match_ids_ = std::move(ids); + patterns_total_ = total; + } + private: void emit_json(const std::string &file, const Pattern &pattern, const Match &m, std::string_view buf, const LineIndex &idx, @@ -171,6 +193,9 @@ class Formatter { uint64_t bytes_emitted_ = 0; // total bytes written to out_ bool over_budget_ = false; // sticky once max_output_bytes is exceeded bool limit_hit_ = false; // set by mark_limit_hit() (LLM footer) + bool header_done_ = false; // query header printed (emit_header dedupe) + std::vector zero_match_ids_; // patterns with 0 kept matches + size_t patterns_total_ = 0; // denominator for the "no matches" footer std::string llm_last_file_; // last file printed as a header (-llm/-elide dedupe) std::unordered_map per_file_counts_; diff --git a/src/pipeline.cpp b/src/pipeline.cpp index 3e1ff26..f9b4ecb 100644 --- a/src/pipeline.cpp +++ b/src/pipeline.cpp @@ -428,6 +428,7 @@ std::vector build_patterns(const Cli &cli) { p.ucp = p.utf8 && cli.ucp; p.extract_names = cp.extract_names; p.ref = cp.ref; + p.desc = cp.desc; patterns.push_back(std::move(p)); ++regex_i; } @@ -438,6 +439,7 @@ std::vector build_patterns(const Cli &cli) { p.id = cp.name.empty() ? "ident" + std::to_string(ident_i) : cp.name; // regexp deliberately left empty — ident-backed patterns never // reach Matcher::compile. + p.desc = cp.desc; patterns.push_back(std::move(p)); ++ident_i; } diff --git a/src/runner.cpp b/src/runner.cpp index a0f490c..7bd3206 100644 --- a/src/runner.cpp +++ b/src/runner.cpp @@ -93,6 +93,24 @@ BufferedFile buffer_file(const std::string &display_name, return bf; } +// One token per input source, in the walker's precedence order: globs, +// positional paths, file lists, git selections. Shared by -explain-plan's +// stage.inputs and the -llm/-elide query header so the two never drift. +std::vector input_tokens(const hpr::Cli &cli) { + std::vector in; + in.insert(in.end(), cli.globs.begin(), cli.globs.end()); + in.insert(in.end(), cli.positional.begin(), cli.positional.end()); + for (const auto &fl : cli.file_lists) + in.push_back(std::string(fl.nul ? "files0:" : "files:") + fl.path); + if (cli.git_changed) in.push_back("git:changed"); + if (cli.git_staged) in.push_back("git:staged"); + if (cli.git_untracked) in.push_back("git:untracked"); + for (const auto &range : cli.git_ranges) + in.push_back("git:range:" + range); + if (in.empty()) in.push_back(""); + return in; +} + // Render a buffered file's matches via -elide's logic into an owned string // instead of stdout, so -budget can measure the size before committing to // it. Reuses Formatter/on_file_elide verbatim through a memory-backed @@ -150,17 +168,7 @@ int run_search(const Cli &cli) { stage.id = "scan0"; stage.sets = {"quick"}; stage.patterns = patterns.size(); - stage.inputs.insert(stage.inputs.end(), cli.globs.begin(), cli.globs.end()); - stage.inputs.insert(stage.inputs.end(), cli.positional.begin(), - cli.positional.end()); - for (const auto &fl : cli.file_lists) - stage.inputs.push_back(std::string(fl.nul ? "files0:" : "files:") + fl.path); - if (cli.git_changed) stage.inputs.push_back("git:changed"); - if (cli.git_staged) stage.inputs.push_back("git:staged"); - if (cli.git_untracked) stage.inputs.push_back("git:untracked"); - for (const auto &range : cli.git_ranges) - stage.inputs.push_back("git:range:" + range); - if (stage.inputs.empty()) stage.inputs.push_back(""); + stage.inputs = input_tokens(cli); stage.scope = cli.scope_lang; if (stage.scope.empty() && (!cli.scope_pattern.empty() || !cli.in_scopes.empty())) @@ -256,6 +264,49 @@ int run_search(const Cli &cli) { oo.extract_table = extract_table.any() ? &extract_table : nullptr; oo.pattern_count = patterns.size(); oo.global_limit = cli.limit; + + // Query header: when any pattern carries a description, LLM-facing + // output opens with an echo of the interpreted query plus a one-line + // legend per pattern. Pre-rendered here so the Formatter stays ignorant + // of Cli; patterns without a description fall back to their regexp so + // the legend is complete on its own. + if (oo.mode == OutputMode::Llm || oo.mode == OutputMode::Elide) { + bool any_desc = false; + for (const auto &p : patterns) + if (!p.desc.empty()) { any_desc = true; break; } + if (any_desc) { + auto flatten = [](std::string_view sv) { + std::string out(sv); + for (char &c : out) + if (c == '\n' || c == '\r' || c == '\t') c = ' '; + return out; + }; + std::string head = "query: "; + head += std::to_string(patterns.size()); + head += patterns.size() == 1 ? " pattern over " : " patterns over "; + std::vector toks = input_tokens(cli); + for (size_t i = 0; i < toks.size(); ++i) { + if (i) head += ", "; + head += toks[i]; + } + oo.header_lines.push_back(std::move(head)); + for (const auto &p : patterns) { + std::string line = " "; + line += p.id; + line += " \xE2\x80\x94 "; // em dash + if (!p.desc.empty()) { + line += flatten(p.desc); + } else if (!p.regexp.empty()) { + line += '/'; + line += flatten(p.regexp); + line += '/'; + } else { + line += "(ident group)"; + } + oo.header_lines.push_back(std::move(line)); + } + } + } Formatter fmt(oo, stdout); // Block extraction populates `block_line_start`/`_end` from the line // index, so we need it whenever block delimiters are configured (even @@ -496,6 +547,13 @@ int run_search(const Cli &cli) { ident_groups); const bool have_rels = !cli.relations.empty(); + // Kept (post-filter) matches per pattern across the whole scan — the + // source of truth for the -llm/-elide "no matches" footer. Counted here + // rather than in the Formatter because buffering modes (-sample, + // -hotspots) only show a subset: "matched nowhere in the scan" must not + // depend on what happened to be displayed. + std::vector pattern_hits(patterns.size(), 0); + auto scan_buf = [&](const std::string &display_name, std::string_view content) -> bool { LineIndex idx; @@ -528,6 +586,8 @@ int run_search(const Cli &cli) { return true; } + for (const auto &m : kept) ++pattern_hits[m.pattern_index]; + // -records line (with -absent): emit one record per non-empty line // lacking each pattern, then skip the per-file emission path (and the // Absent-mode file listing) entirely. @@ -811,6 +871,9 @@ int run_search(const Cli &cli) { if (seen_active) for (const auto &m : marks) seen_store.mark(m); ++hs_emitted_rows; } else if (oo.mode == OutputMode::Llm) { + // Hotspot rows bypass the Formatter, so the query header + // has to be requested explicitly (idempotent). + fmt.emit_header(); std::string line = r.file; line += ':'; line += std::to_string(r.window_lo); @@ -990,6 +1053,12 @@ int run_search(const Cli &cli) { } } + if (oo.mode == OutputMode::Llm || oo.mode == OutputMode::Elide) { + std::vector zero; + for (size_t i = 0; i < patterns.size(); ++i) + if (pattern_hits[i] == 0) zero.push_back(patterns[i].id); + fmt.set_zero_match_patterns(std::move(zero), patterns.size()); + } fmt.on_complete(); if (cli.summary) { diff --git a/tests/run.sh b/tests/run.sh index b41e14a..c12f95e 100755 --- a/tests/run.sh +++ b/tests/run.sh @@ -1182,6 +1182,66 @@ OUT=$("$BIN" -p WARN -name warn -p ERROR -name err -llm "$NP_FIX/log.txt") expect_contains "-llm shows the name tag" "[warn]" "$OUT" rm -rf "$NP_FIX" +# --------------------------------------------------------------------------- +section "informative -llm output (-desc legend / no-matches footer)" +LD_FIX="$HERE/_tmp_llmdesc" +mkdir -p "$LD_FIX" +printf 'WARN a\nERROR b\nx\nWARN c\n' > "$LD_FIX/log.txt" + +# --- zero-match footer +OUT=$("$BIN" -p WARN -name warn -p NOPE_ZZZ -name ghost -llm "$LD_FIX/log.txt") +expect_contains "-llm zero-match footer names silent pattern" "--- no matches: ghost (1 of 2 patterns) ---" "$OUT" +OUT=$("$BIN" -p WARN -name warn -p ERROR -name err -llm "$LD_FIX/log.txt") +expect_not_contains "-llm no footer when all patterns match" "no matches" "$OUT" +OUT=$("$BIN" -p NOPE_A -p NOPE_B -llm "$LD_FIX/log.txt" ; echo "rc=$?") +expect_contains "-llm all-zero footer lists every pattern" "--- no matches: p0, p1 (2 of 2 patterns) ---" "$OUT" +expect_contains "-llm all-zero still exits 1" "rc=1" "$OUT" +OUT=$("$BIN" -p WARN -p NOPE_ZZZ -name ghost -llm -limit 1 "$LD_FIX/log.txt") +expect_contains "-llm footer qualified when limit stops scan" "no matches (scan stopped early): ghost" "$OUT" +OUT=$("$BIN" -p WARN -p NOPE_ZZZ -name ghost -elide "$LD_FIX/log.txt") +expect_contains "-elide zero-match footer" "--- no matches: ghost (1 of 2 patterns) ---" "$OUT" +OUT=$("$BIN" -p WARN -p NOPE_ZZZ "$LD_FIX/log.txt") +expect_not_contains "JSONL mode has no zero-match footer" "no matches" "$OUT" + +# --- -desc query legend +OUT=$("$BIN" -p WARN -name warn -desc 'warning lines' -p ERROR -llm "$LD_FIX/log.txt") +expect_contains "-desc legend described pattern" " warn — warning lines" "$OUT" +expect_contains "-desc legend regexp fallback" " p1 — /ERROR/" "$OUT" +FIRST=$(printf '%s\n' "$OUT" | head -1) +expect_eq "-desc header is the first line" "query: 2 patterns over $LD_FIX/log.txt" "$FIRST" +OUT=$("$BIN" -p WARN -p ERROR -llm "$LD_FIX/log.txt") +expect_not_contains "no header without -desc" "query:" "$OUT" +OUT=$("$BIN" -p WARN -desc 'warning lines' "$LD_FIX/log.txt") +expect_not_contains "JSONL mode has no header" "query:" "$OUT" +OUT=$("$BIN" -p NOPE_ZZZ -desc 'missing thing' -llm "$LD_FIX/log.txt") +expect_contains "header still prints before all-zero footer" "query: 1 pattern over" "$OUT" +expect_contains "singular footer wording" "(1 of 1 pattern) ---" "$OUT" +OUT=$("$BIN" -p WARN -desc 'warning lines' -hotspots 1 -llm "$LD_FIX/log.txt") +expect_contains "-hotspots -llm prints header before rows" "query: 1 pattern over" "$OUT" + +# --- -desc validation +OUT=$("$BIN" -desc x -p WARN "$LD_FIX/log.txt" 2>&1) ; RC=$? +expect_contains "-desc before pattern rejected" "-desc must follow" "$OUT" +[[ "$RC" == "2" ]] && report ok "-desc before pattern exit 2" || report fail "-desc before pattern exit (got $RC)" +OUT=$("$BIN" -p WARN -desc a -desc b "$LD_FIX/log.txt" 2>&1) ; RC=$? +expect_contains "-desc repeated rejected" "repeated for the same pattern" "$OUT" +OUT=$("$BIN" -p WARN -desc '' "$LD_FIX/log.txt" 2>&1) ; RC=$? +expect_contains "-desc empty rejected" "empty description" "$OUT" + +# --- patterns-from description field +cat > "$LD_FIX/rules.jsonl" <<'EOF' +{"id":"warnish","regexp":"WARN","description":"warning lines"} +{"id":"errish","regexp":"ERROR"} +EOF +OUT=$("$BIN" -patterns-from "$LD_FIX/rules.jsonl" -llm "$LD_FIX/log.txt") +expect_contains "rule-file description in legend" " warnish — warning lines" "$OUT" +expect_contains "rule-file undescribed pattern falls back to regexp" " errish — /ERROR/" "$OUT" +printf '{"id":"x","regexp":"y","description":5}\n' > "$LD_FIX/bad.jsonl" +OUT=$("$BIN" -patterns-from "$LD_FIX/bad.jsonl" -llm "$LD_FIX/log.txt" 2>&1) ; RC=$? +expect_contains "rule-file description type-checked" "'description' must be a string" "$OUT" +[[ "$RC" == "2" ]] && report ok "rule-file bad description exit 2" || report fail "rule-file bad description exit (got $RC)" +rm -rf "$LD_FIX" + # --------------------------------------------------------------------------- section "fixed strings (-F / -Fi / -patterns-from)" FS_FIX="$HERE/_tmp_fixed" From 0edb2416b79152a48eae19657de689b54c0ac8fa Mon Sep 17 00:00:00 2001 From: Pinkhas Nisanov Date: Thu, 13 Aug 2026 20:52:58 +0300 Subject: [PATCH 2/4] Add per-match role tags: def/comment/string/import Classify every match's lexical role so readers stop re-deriving it: - New RoleIndex (src/roles.cpp): a single-pass, escape-aware lexer builds comment/string spans per file (line + block comments with optional nesting, single-line/multiline/raw strings, Python triple quotes) and marks import-prefixed lines. Covers the scope-pack languages plus py/sh/rb/yaml/toml; unknown extensions get no tags. - ScopeIndex::anchor_on_line identifies matches sitting on a signature line; -llm renders those as '[def func X]' instead of the misleading '[in func X]'. Position-accurate roles win: a trailing comment on a signature line is a comment, not a def. - Surfaces: JSONL gains a "role" field (omitted for plain code), -llm appends [comment]/[string]/[import] tags, -format gains $ROLE. On by default for per-match output modes, built lazily only for files with kept matches; -no-roles disables. Co-Authored-By: Claude Fable 5 --- HPRSCRIPT.md | 33 ++++- skills/hprscript-search/SKILL.md | 1 + src/cli.cpp | 7 + src/cli.hpp | 5 + src/output.cpp | 66 +++++++-- src/output.hpp | 14 +- src/roles.cpp | 225 +++++++++++++++++++++++++++++++ src/roles.hpp | 68 ++++++++++ src/runner.cpp | 39 +++++- src/scope.cpp | 10 ++ src/scope.hpp | 6 + tests/run.sh | 86 ++++++++++++ 12 files changed, 543 insertions(+), 17 deletions(-) create mode 100644 src/roles.cpp create mode 100644 src/roles.hpp diff --git a/HPRSCRIPT.md b/HPRSCRIPT.md index 89627f8..d8234a3 100644 --- a/HPRSCRIPT.md +++ b/HPRSCRIPT.md @@ -116,6 +116,7 @@ simultaneous. | `-git-range ` | Scan the files changed in a diff range (`origin/main...HEAD`). Repeatable. | | `-git-added-lines` | Restrict matches to lines **added** by the selected diffs (quick mode; needs `-git-changed`/`-git-staged`/`-git-range`). | | `-w` | Whole-word matching (wraps the pattern as `\b(?:expr)\b`) | +| `-no-roles` | Disable per-match role classification (`role` JSONL field, `[def]`/`[comment]`/`[string]`/`[import]` tags in `-llm`, `$ROLE`). See [Per-match role tags](#per-match-role-tags). | | `-no-utf8` | Disable UTF-8 mode (byte-level matching — see [UTF-8 / Unicode](#utf-8--unicode-support)) | | `-ucp` | Enable Unicode property classes for `\w`/`\d`/`\s` (opt-in; may reject some patterns) | | `-limit ` | Maximum global results — scanning stops once reached | @@ -171,6 +172,7 @@ simultaneous. | `$MATCH` | Matched text | | `$CONTEXT` | Match line plus `-A`/`-B`/`-C` surrounding lines | | `$PAT_ID` | Pattern id (`p0`, `p1`, … unless overridden in script mode) | +| `$ROLE` | Match's lexical role: `def`/`comment`/`string`/`import`, empty for plain code. See [Per-match role tags](#per-match-role-tags). | | `$BLOCK` | Block content (when `-block-open`/`-close` is active) | | `$BLOCK_FULL` | Match start through block end (signature + body) | | `$BLOCK_START` / `$BLOCK_END` | Byte offsets of block start / end (exclusive) | @@ -2270,6 +2272,33 @@ hprscript -p 'BEGIN.*PRIVATE KEY' -summary -diagnostics -require-complete -glob --- +## Per-match role tags + +The first question a reader of search results resolves for almost every hit is "what kind of occurrence is this?" — a definition, a comment mention, a string literal, an import. hprscript answers it per match, by default, in every per-match output mode: + +| Role | Meaning | Detection | +|---|---|---| +| `def` | Match sits on a scope's signature line | The match's line is an anchor line in the scope index (needs an active `-scope`) | +| `comment` | Match is inside a line or block comment | Single-pass, escape-aware lexer over the file | +| `string` | Match is inside a string literal | Same lexer (quotes, raw/backtick strings, Python triple quotes) | +| `import` | Match's line is an import/include/use statement | Line-prefix check, only outside comments/strings | + +Surfaces: the default JSONL output gains a `"role":"comment"` field (omitted for plain code), `-llm` appends bracket tags, and `-format` recognises `$ROLE`. In `-llm`, a match on a signature line renders as `[def func Target]` **instead of** the misleading `[in func Target]`: + +``` +a.go + 3: import "fmt" [import] + 5: // TODO comment hit [comment] + 6: func Target() { [def func Target] + 7: s := "TODO in string" [string] [in func Target] +``` + +Precedence is position-accurate first: a trailing comment on a signature line is `comment`, not `def`. Comment/string lexing covers the scope-pack languages (Go, Rust, C, C++, Java, JS, TS) plus Python, shell, Ruby, YAML, and TOML; files with unrecognized extensions simply get no role tags. `def` additionally requires an active `-scope` config, since it reads the scope index's anchor lines. + +The classification is lexical, not syntactic — it will not distinguish a call from a reference, and block-delimiter oddities inside unrecognized constructs can skew it. It is computed lazily, only for files that produce output, so clean files cost nothing. `-no-roles` turns it off entirely. + +--- + ## LLM output mode `-llm` emits a compact, plain-text format intended for direct consumption by language models — no JSON parsing, no per-match metadata noise, just file → line → matched text. Far cheaper in tokens than `-j` for the same information. @@ -2280,9 +2309,11 @@ Layout: one **file header** per file (deduped — never repeated), then each mat |---|---| | (none) | ` : ` — with ≥2 patterns: ` : [] ` | | `-block-open`/`-block-close` | ` -` then the full block content on following lines | -| `-scope ` / `-scope-pattern …` | ` : [in ]` | +| `-scope ` / `-scope-pattern …` | ` : [in ]` — or ` [def ]` when the match sits on the signature line itself | | Both block + scope | block form, with a `[in ]` suffix on the header | +Line-shape matches also carry [role tags](#per-match-role-tags) — `[comment]`, `[string]`, `[import]` — unless `-no-roles` is given. + When `-limit` or `-max-output-bytes` truncates the output, a final `--- limit reached: ... ---` (or `--- output-byte budget reached ... ---`) footer line is emitted so the reader knows the result was cut, not finished. ### Query header and no-matches footer diff --git a/skills/hprscript-search/SKILL.md b/skills/hprscript-search/SKILL.md index 6073dcc..0f1d7d3 100644 --- a/skills/hprscript-search/SKILL.md +++ b/skills/hprscript-search/SKILL.md @@ -13,6 +13,7 @@ Invoke the binary through Bash as `hprscript`. Use one call per reasoning stage - Put distinguishable terms in separate `-p` or `-pi` flags so every hit retains its pattern ID. - Prefer `-llm` when reading results, `-f` for paths, `-c` for counts, and `-limit N` for existence checks. In `-llm`/`-elide` output, patterns with zero matches are named in a trailing `--- no matches: … ---` footer — treat that as explicit evidence of absence, qualified with "scan stopped early" when a limit cut the scan. +- Trust per-match role tags instead of re-deriving them: `[comment]`/`[string]`/`[import]` in `-llm` (a `role` field in JSONL, `$ROLE` in `-format`) classify each hit lexically, and with `-scope` active a hit on a signature line reads `[def func X]` while body hits read `[in func X]`. Untagged hits in a recognized language are plain code. - Use an absolute path or glob when the effective cwd is uncertain. Inspect the first emitted path and stop if it escapes the intended tree. - Add `-summary -require-complete` when a broad sweep must be exhaustive. Do not present a partial scan as complete. - Restructure unsupported lookarounds or backreferences, or express the relationship with `query` or script phases. Do not fall back to grep or rg. diff --git a/src/cli.cpp b/src/cli.cpp index 3e9a300..4a013ef 100644 --- a/src/cli.cpp +++ b/src/cli.cpp @@ -215,6 +215,9 @@ void print_help(FILE *out) { " -git-added-lines Restrict matches to lines ADDED by the selected diffs\n" " (needs -git-changed/-staged/-range; -p mode only)\n" " -w Whole-word match (\\b…\\b)\n" +" -no-roles Disable per-match role classification (the `role` JSONL\n" +" field, [def]/[comment]/[string]/[import] tags in -llm,\n" +" and $ROLE in -format)\n" " -no-utf8 Disable UTF-8 mode (byte-level matching)\n" " -ucp Enable Unicode \\w/\\d/\\s (may reject some patterns)\n" " -limit Max global results\n" @@ -648,6 +651,10 @@ Cli parse_cli(int argc, char **argv) { cli.patterns.back().desc = v; continue; } + if (eq(a, "-no-roles") || eq(a, "--no-roles")) { + cli.no_roles = true; + continue; + } if (eq(a, "-glob")) { const char *v = take(i, argc, argv, a, cli); if (!v) return cli; cli.globs.emplace_back(v); diff --git a/src/cli.hpp b/src/cli.hpp index 86ec9fc..fd259f2 100644 --- a/src/cli.hpp +++ b/src/cli.hpp @@ -179,6 +179,11 @@ struct Cli { std::string seen_path; bool word_boundary = false; bool no_utf8 = false; // -no-utf8: byte-mode matching + // -no-roles: disable per-match role classification (def/comment/string/ + // import — the `role` JSONL field, -llm bracket tags, and $ROLE). On by + // default for per-match output modes; roles are computed lazily per + // matched file, so the cost only exists where output is produced. + bool no_roles = false; bool ucp = false; // -ucp: enable Unicode \w/\d/\s (opt-in) int64_t limit = -1; // global match cap (-limit) int64_t per_file_limit = -1; // per-file cap (-m) diff --git a/src/output.cpp b/src/output.cpp index b9003a4..f4a4258 100644 --- a/src/output.cpp +++ b/src/output.cpp @@ -1,6 +1,7 @@ #include "output.hpp" #include "block.hpp" +#include "roles.hpp" #include "seen.hpp" #include @@ -166,6 +167,29 @@ std::string_view truncate_safe(std::string_view sv, uint64_t limit, return sv.substr(0, n); } +// Per-match role: "comment"/"string"/"def"/"import", or nullptr for plain +// code. Position-accurate lexer roles win over the line-based ones — a +// trailing comment on a signature line is a comment, not a def. `def_out` +// receives the defined scope when the result is "def". +const char *resolve_role(const RoleIndex *roles, const ScopeIndex *scope, + uint64_t offset, uint32_t line, + const ScopeRange **def_out) { + if (def_out) *def_out = nullptr; + if (roles) { + LexRole lr = roles->at(offset); + if (lr == LexRole::Comment) return "comment"; + if (lr == LexRole::Str) return "string"; + } + if (scope) { + if (const ScopeRange *sr = scope->anchor_on_line(line)) { + if (def_out) *def_out = sr; + return "def"; + } + } + if (roles && roles->import_line(line)) return "import"; + return nullptr; +} + } // namespace Formatter::Formatter(OutputOptions opts, FILE *out) : opts_(opts), out_(out) {} @@ -235,7 +259,8 @@ std::string_view Formatter::context_block(std::string_view buf, void Formatter::emit_json(const std::string &file, const Pattern &pattern, const Match &m, std::string_view buf, - const LineIndex &idx, const ScopeIndex *scope) { + const LineIndex &idx, const ScopeIndex *scope, + const RoleIndex *roles) { refresh_file_cache(file); refresh_pattern_cache(pattern); @@ -326,6 +351,11 @@ void Formatter::emit_json(const std::string &file, const Pattern &pattern, s += "}"; } } + if (const char *role = resolve_role(roles, scope, m.from, line, nullptr)) { + s += ",\"role\":\""; + s += role; + s += '"'; + } s += "}\n"; write_out(s); } @@ -347,7 +377,8 @@ void Formatter::emit_match_only(std::string_view buf, const Match &m, void Formatter::emit_custom(const std::string &file, const Pattern &pattern, const Match &m, std::string_view buf, - const LineIndex &idx, const ScopeIndex *scope) { + const LineIndex &idx, const ScopeIndex *scope, + const RoleIndex *roles) { uint32_t line = idx.line_of(m.from); uint32_t col = idx.col_of(m.from); std::string_view match_text(buf.data() + m.from, m.to - m.from); @@ -392,6 +423,7 @@ void Formatter::emit_custom(const std::string &file, const Pattern &pattern, return true; }; const ScopeRange *sr = (scope ? scope->find_innermost(m.from) : nullptr); + const char *role = resolve_role(roles, scope, m.from, line, nullptr); static const char EXTRACT_PREFIX[] = "$EXTRACT_"; static constexpr size_t EXTRACT_PREFIX_LEN = sizeof(EXTRACT_PREFIX) - 1; for (size_t i = 0; i < fmt.size();) { @@ -415,6 +447,7 @@ void Formatter::emit_custom(const std::string &file, const Pattern &pattern, if (try_num(i, "$FROM", m.from)) continue; if (try_num(i, "$TO", m.to)) continue; if (try_str(i, "$PAT_ID", pattern.id)) continue; + if (try_str(i, "$ROLE", role ? std::string_view(role) : std::string_view{})) continue; if (try_str(i, "$ENCLOSING_NAME", sr ? std::string_view(sr->name) : std::string_view{})) continue; if (try_str(i, "$ENCLOSING_KIND", sr ? std::string_view(sr->kind) : std::string_view{})) continue; if (try_num(i, "$ENCLOSING_LINE_START", sr ? sr->line_start : 0)) continue; @@ -471,7 +504,8 @@ void Formatter::emit_header() { void Formatter::emit_llm(const std::string &file, const Pattern &pattern, const Match &m, std::string_view buf, - const LineIndex &idx, const ScopeIndex *scope) { + const LineIndex &idx, const ScopeIndex *scope, + const RoleIndex *roles) { emit_header(); auto &s = scratch_; @@ -527,7 +561,22 @@ void Formatter::emit_llm(const std::string &file, const Pattern &pattern, if (!ctx_view.empty() && ctx_view.back() == '\n') ctx_view.remove_suffix(1); s.append(ctx_view.data(), ctx_view.size()); - if (scope) { + const ScopeRange *def_sr = nullptr; + const char *role = resolve_role(roles, scope, m.from, line, &def_sr); + if (def_sr) { + // The match sits on the signature line itself — `[def func X]` + // rather than the misleading `[in func X]`. + s += " [def "; + s.append(def_sr->kind); + s += ' '; + s.append(def_sr->name); + s += ']'; + } else if (role) { + s += " ["; + s += role; + s += ']'; + } + if (scope && !def_sr) { if (const ScopeRange *sr = scope->find_innermost(m.from)) { s += " [in "; s.append(sr->kind); @@ -723,10 +772,11 @@ void Formatter::on_file_elide(const std::string &file, void Formatter::on_match(const std::string &file, const Pattern &pattern, const Match &m, std::string_view buf, - const LineIndex &idx, const ScopeIndex *scope) { + const LineIndex &idx, const ScopeIndex *scope, + const RoleIndex *roles) { ++emitted_; switch (opts_.mode) { - case OutputMode::JsonLines: emit_json(file, pattern, m, buf, idx, scope); break; + case OutputMode::JsonLines: emit_json(file, pattern, m, buf, idx, scope, roles); break; case OutputMode::FilesOnly: // Print the file once, on first match. Counter tracked below. if (per_file_counts_[file]++ == 0) { @@ -738,8 +788,8 @@ void Formatter::on_match(const std::string &file, const Pattern &pattern, per_file_counts_[file]++; break; case OutputMode::MatchOnly: emit_match_only(buf, m, idx); break; - case OutputMode::Custom: emit_custom(file, pattern, m, buf, idx, scope); break; - case OutputMode::Llm: emit_llm(file, pattern, m, buf, idx, scope); break; + case OutputMode::Custom: emit_custom(file, pattern, m, buf, idx, scope, roles); break; + case OutputMode::Llm: emit_llm(file, pattern, m, buf, idx, scope, roles); break; case OutputMode::Elide: // Elide renders a whole file's matches at once via // on_file_elide(); it never streams through on_match(). Undo diff --git a/src/output.hpp b/src/output.hpp index 8b79587..06fc29f 100644 --- a/src/output.hpp +++ b/src/output.hpp @@ -22,6 +22,7 @@ namespace hpr { class SeenStore; // src/seen.hpp — -seen cross-invocation dedup struct SeenMark; +class RoleIndex; // src/roles.hpp — comment/string/import classification enum class OutputMode { JsonLines, // one JSON object per line (default) @@ -94,9 +95,14 @@ class Formatter { public: Formatter(OutputOptions opts, FILE *out); + // `roles` (optional) classifies the match position lexically; combined + // with `scope`'s anchor lines it yields a per-match role — + // def/comment/string/import — surfaced as `role` in JSONL, bracket tags + // in -llm, and $ROLE in -format. nullptr = no role classification. void on_match(const std::string &file, const Pattern &pattern, const Match &m, std::string_view buf, const LineIndex &idx, - const ScopeIndex *scope = nullptr); + const ScopeIndex *scope = nullptr, + const RoleIndex *roles = nullptr); // -elide mode: render one file's entire kept-match set in a single call, // grouped by innermost enclosing scope. Each scope prints its signature @@ -163,15 +169,15 @@ class Formatter { private: void emit_json(const std::string &file, const Pattern &pattern, const Match &m, std::string_view buf, const LineIndex &idx, - const ScopeIndex *scope); + const ScopeIndex *scope, const RoleIndex *roles); void emit_match_only(std::string_view buf, const Match &m, const LineIndex &idx); void emit_custom(const std::string &file, const Pattern &pattern, const Match &m, std::string_view buf, const LineIndex &idx, - const ScopeIndex *scope); + const ScopeIndex *scope, const RoleIndex *roles); void emit_llm(const std::string &file, const Pattern &pattern, const Match &m, std::string_view buf, const LineIndex &idx, - const ScopeIndex *scope); + const ScopeIndex *scope, const RoleIndex *roles); // Compute the joined context block (prev N lines + match line + next N // lines) as a single string view into buf. Trailing newline is excluded. diff --git a/src/roles.cpp b/src/roles.cpp new file mode 100644 index 0000000..cfe756e --- /dev/null +++ b/src/roles.cpp @@ -0,0 +1,225 @@ +#include "roles.hpp" + +#include "scope.hpp" + +#include +#include + +namespace hpr { + +namespace { + +const RoleConfig CFG_GO{ + "//", false, "/*", "*/", false, "\"'", "`", false, + {"import ", "import ("}}; +// Rust: `'` is excluded from string delims — `'a` lifetimes have no closing +// quote and would poison everything after them. Double-quote strings span +// lines; block comments nest. +const RoleConfig CFG_RUST{ + "//", false, "/*", "*/", true, "", "\"", false, + {"use ", "extern crate "}}; +const RoleConfig CFG_C{ + "//", false, "/*", "*/", false, "\"'", "", false, {"#include"}}; +const RoleConfig CFG_JAVA{ + "//", false, "/*", "*/", false, "\"'", "", false, {"import "}}; +const RoleConfig CFG_JS{ + "//", false, "/*", "*/", false, "\"'", "`", false, {"import "}}; +const RoleConfig CFG_PY{ + "#", false, nullptr, nullptr, false, "\"'", "", true, + {"import ", "from "}}; +const RoleConfig CFG_SH{ + "#", true, nullptr, nullptr, false, "\"'", "", false, {}}; +const RoleConfig CFG_RB{ + "#", false, nullptr, nullptr, false, "\"'", "", false, + {"require ", "require_relative "}}; +const RoleConfig CFG_HASH{ // yaml / toml: comments + quoted scalars only + "#", false, nullptr, nullptr, false, "\"'", "", false, {}}; + +bool ends_with(const std::string &s, const char *suffix) { + size_t n = std::strlen(suffix); + return s.size() >= n && s.compare(s.size() - n, n, suffix) == 0; +} + +} // namespace + +const RoleConfig *role_config_for_path(const std::string &path) { + const std::string lang = auto_lang_for_path(path); + if (lang == "go") return &CFG_GO; + if (lang == "rust") return &CFG_RUST; + if (lang == "c" || lang == "cpp") return &CFG_C; + if (lang == "java") return &CFG_JAVA; + if (lang == "js" || lang == "ts") return &CFG_JS; + if (ends_with(path, ".py")) return &CFG_PY; + if (ends_with(path, ".sh") || ends_with(path, ".bash")) return &CFG_SH; + if (ends_with(path, ".rb")) return &CFG_RB; + if (ends_with(path, ".yml") || ends_with(path, ".yaml") || + ends_with(path, ".toml")) + return &CFG_HASH; + return nullptr; +} + +void RoleIndex::build(std::string_view buf, const RoleConfig &cfg, + const LineIndex &idx) { + spans_.clear(); + import_lines_.clear(); + + enum class St { Code, LineComment, BlockComment, Str }; + St st = St::Code; + uint64_t span_start = 0; + int block_depth = 0; + char quote = 0; + bool raw = false; // no backslash escapes (Go/JS backtick) + bool multiline = false; // literal may cross newlines + bool triple = false; // delimiter is quote×3 (Python) + + const size_t n = buf.size(); + auto starts = [&](size_t i, const char *tok) { + if (!tok) return false; + size_t tl = std::strlen(tok); + return i + tl <= n && buf.compare(i, tl, tok) == 0; + }; + + size_t i = 0; + while (i < n) { + char c = buf[i]; + switch (st) { + case St::Code: + if (starts(i, cfg.block_open)) { + st = St::BlockComment; + span_start = i; + block_depth = 1; + i += std::strlen(cfg.block_open); + continue; + } + if (starts(i, cfg.line_comment) && + (!cfg.comment_needs_boundary || i == 0 || buf[i - 1] == ' ' || + buf[i - 1] == '\t' || buf[i - 1] == '\n')) { + st = St::LineComment; + span_start = i; + i += std::strlen(cfg.line_comment); + continue; + } + if (cfg.triple_quotes && (c == '"' || c == '\'') && + (starts(i, "\"\"\"") || starts(i, "'''"))) { + st = St::Str; + span_start = i; + quote = c; + raw = false; + multiline = true; + triple = true; + i += 3; + continue; + } + if (c != '\0' && std::strchr(cfg.ml_quotes, c) != nullptr) { + st = St::Str; + span_start = i; + quote = c; + raw = (c == '`'); + multiline = true; + triple = false; + ++i; + continue; + } + if (c != '\0' && std::strchr(cfg.sl_quotes, c) != nullptr) { + st = St::Str; + span_start = i; + quote = c; + raw = false; + multiline = false; + triple = false; + ++i; + continue; + } + ++i; + break; + case St::LineComment: + if (c == '\n') { + spans_.push_back({span_start, i, LexRole::Comment}); + st = St::Code; + } + ++i; + break; + case St::BlockComment: + if (cfg.nest_blocks && starts(i, cfg.block_open)) { + ++block_depth; + i += std::strlen(cfg.block_open); + continue; + } + if (starts(i, cfg.block_close)) { + i += std::strlen(cfg.block_close); + if (--block_depth == 0) { + spans_.push_back({span_start, i, LexRole::Comment}); + st = St::Code; + } + continue; + } + ++i; + break; + case St::Str: + if (!raw && c == '\\') { + i += 2; + continue; + } + if (!multiline && c == '\n') { + // Unterminated single-line literal: close it at the newline + // so one stray quote can't poison the rest of the file. + spans_.push_back({span_start, i, LexRole::Str}); + st = St::Code; + ++i; + continue; + } + if (c == quote && + (!triple || + starts(i, quote == '"' ? "\"\"\"" : "'''"))) { + i += triple ? 3 : 1; + spans_.push_back({span_start, i, LexRole::Str}); + st = St::Code; + continue; + } + ++i; + break; + } + } + if (st != St::Code) { + spans_.push_back({span_start, n, + st == St::Str ? LexRole::Str : LexRole::Comment}); + } + + if (!cfg.import_prefixes.empty()) { + const uint32_t nlines = idx.line_count(); + for (uint32_t L = 1; L <= nlines; ++L) { + std::string_view t = idx.line_text(L); + if (t.data() == nullptr) continue; + size_t ws = 0; + while (ws < t.size() && (t[ws] == ' ' || t[ws] == '\t')) ++ws; + if (ws >= t.size()) continue; + uint64_t off = static_cast(t.data() - buf.data()) + ws; + if (at(off) != LexRole::Code) continue; + std::string_view rest = t.substr(ws); + for (const auto &pref : cfg.import_prefixes) { + if (rest.size() >= pref.size() && + rest.compare(0, pref.size(), pref) == 0) { + import_lines_.push_back(L); + break; + } + } + } + } +} + +LexRole RoleIndex::at(uint64_t offset) const { + auto it = std::upper_bound(spans_.begin(), spans_.end(), offset, + [](uint64_t off, const Span &s) { + return off < s.from; + }); + if (it == spans_.begin()) return LexRole::Code; + --it; + return offset < it->to ? it->role : LexRole::Code; +} + +bool RoleIndex::import_line(uint32_t line) const { + return std::binary_search(import_lines_.begin(), import_lines_.end(), + line); +} + +} // namespace hpr diff --git a/src/roles.hpp b/src/roles.hpp new file mode 100644 index 0000000..56fdec6 --- /dev/null +++ b/src/roles.hpp @@ -0,0 +1,68 @@ +// Per-match lexical role classification: is a byte offset inside a comment +// or a string literal, and is its line an import-ish line? +// +// A single linear pass over the file builds comment/string spans with an +// escape-aware state machine (line comments, block comments with optional +// nesting, single-line and multiline string literals, Python triple quotes). +// Match offsets are then binary-searched against the span list. This is +// lexical classification, not parsing — but comment/string boundaries are +// exactly the distinctions a lexer gets right, and they're the ones a reader +// of search results otherwise has to re-derive per match. +// +// The fourth role, `def`, is not computed here: a match whose line is a +// scope-anchor line is classified by ScopeIndex::anchor_on_line (scope.hpp). +#pragma once + +#include "line_index.hpp" + +#include +#include +#include +#include + +namespace hpr { + +enum class LexRole : uint8_t { Code, Comment, Str }; + +struct RoleConfig { + const char *line_comment = nullptr; // "//" or "#" (nullptr = none) + // Shell only: "#" starts a comment only at line start or after + // whitespace ($#, ${#x} must not count). Python/YAML have no such rule. + bool comment_needs_boundary = false; + const char *block_open = nullptr; // "/*" (nullptr = none) + const char *block_close = nullptr; // "*/" + bool nest_blocks = false; // Rust nests /* /* */ */ + const char *sl_quotes = ""; // single-line delims; close at newline + const char *ml_quotes = ""; // multiline delims (Go/JS "`", Rust "\"") + bool triple_quotes = false; // Python """ / ''' (multiline) + std::vector import_prefixes; // "import ", "#include", … +}; + +// Language config for a path by extension, or nullptr when unknown. Covers +// the scope-pack languages (go, rust, c, cpp, java, js, ts) plus +// hash-comment ones (py, sh, rb, yaml, toml) that have no scope pack. +const RoleConfig *role_config_for_path(const std::string &path); + +class RoleIndex { +public: + // Single pass over `buf`; `idx` is only used for import-line marking. + void build(std::string_view buf, const RoleConfig &cfg, + const LineIndex &idx); + + // Role at a byte offset (Code when outside every span). + LexRole at(uint64_t offset) const; + + // True when `line` (1-based) starts — after leading whitespace, in code — + // with one of the language's import prefixes. + bool import_line(uint32_t line) const; + +private: + struct Span { + uint64_t from, to; // [from, to) + LexRole role; // Comment or Str only + }; + std::vector spans_; // sorted by `from`, non-overlapping + std::vector import_lines_; // sorted, 1-based +}; + +} // namespace hpr diff --git a/src/runner.cpp b/src/runner.cpp index 7bd3206..f25c60f 100644 --- a/src/runner.cpp +++ b/src/runner.cpp @@ -11,6 +11,7 @@ #include "pipeline.hpp" #include "planner.hpp" #include "rank.hpp" +#include "roles.hpp" #include "scope.hpp" #include "seen.hpp" #include "walker.hpp" @@ -69,6 +70,8 @@ struct BufferedFile { hpr::LineIndex idx; hpr::ScopeIndex scope; bool scope_built = false; + hpr::RoleIndex roles; + bool roles_built = false; std::vector kept; }; @@ -76,7 +79,7 @@ BufferedFile buffer_file(const std::string &display_name, std::string_view content, const std::string &eff_scope_lang, const hpr::ScopeConfig &user_scope_custom, - bool rebuild_scope) { + bool rebuild_scope, bool rebuild_roles = false) { BufferedFile bf; bf.path = display_name; bf.content.assign(content.data(), content.size()); @@ -90,6 +93,13 @@ BufferedFile buffer_file(const std::string &display_name, bf.scope_built = true; } } + if (rebuild_roles) { + if (const hpr::RoleConfig *rc = + hpr::role_config_for_path(display_name)) { + bf.roles.build(bf.content, *rc, bf.idx); + bf.roles_built = true; + } + } return bf; } @@ -308,6 +318,15 @@ int run_search(const Cli &cli) { } } Formatter fmt(oo, stdout); + + // Role classification is on by default for the per-match output modes + // that surface it (JSONL `role`, -llm tags, -format $ROLE). The index is + // built lazily, only for files that actually have kept matches. + const bool roles_enabled = + !cli.no_roles && + (oo.mode == OutputMode::JsonLines || oo.mode == OutputMode::Llm || + oo.mode == OutputMode::Custom); + // Block extraction populates `block_line_start`/`_end` from the line // index, so we need it whenever block delimiters are configured (even // for output modes that wouldn't otherwise build it). @@ -632,7 +651,8 @@ int run_search(const Cli &cli) { sample_files.push_back(buffer_file(display_name, content, eff_scope_lang, user_scope_custom, - scope_ptr != nullptr)); + scope_ptr != nullptr, + roles_enabled)); stats.buffered_bytes_peak += content.size(); for (size_t mi = 0; mi < kept.size(); ++mi) { if (sample_recs.size() >= SAMPLE_REC_CAP) { @@ -713,12 +733,22 @@ int run_search(const Cli &cli) { if (seen_active) for (const auto &m : marks) seen_store.mark(m); } } else { + RoleIndex role_idx; + const RoleIndex *roles_ptr = nullptr; + if (roles_enabled && !kept.empty()) { + if (const RoleConfig *rc = + role_config_for_path(display_name)) { + role_idx.build(content, *rc, idx); + roles_ptr = &role_idx; + } + } uint64_t per_file = 0; for (const auto &m : kept) { had_match = true; ++per_file; const Pattern &pat = patterns[m.pattern_index]; - fmt.on_match(display_name, pat, m, content, idx, scope_ptr); + fmt.on_match(display_name, pat, m, content, idx, scope_ptr, + roles_ptr); if (fmt.over_budget()) break; if (cli.per_file_limit > 0 && per_file >= static_cast(cli.per_file_limit)) break; @@ -842,7 +872,8 @@ int run_search(const Cli &cli) { const BufferedFile &sf = sample_files[r.file_idx]; const Pattern &pat = patterns[r.m.pattern_index]; fmt.on_match(sf.path, pat, r.m, sf.content, sf.idx, - sf.scope_built ? &sf.scope : nullptr); + sf.scope_built ? &sf.scope : nullptr, + sf.roles_built ? &sf.roles : nullptr); if (fmt.over_budget()) { if (stats.stop_reason.empty()) stats.stop_reason = "output_budget"; diff --git a/src/scope.cpp b/src/scope.cpp index b5f6a1c..ee7f2f9 100644 --- a/src/scope.cpp +++ b/src/scope.cpp @@ -170,6 +170,16 @@ bool ScopeIndex::build(std::string_view buf, const ScopeConfig &cfg, return true; } +const ScopeRange *ScopeIndex::anchor_on_line(uint32_t line) const { + // ranges_ is sorted by start_off, so line_start is non-decreasing. + auto it = std::lower_bound(ranges_.begin(), ranges_.end(), line, + [](const ScopeRange &r, uint32_t l) { + return r.line_start < l; + }); + if (it != ranges_.end() && it->line_start == line) return &*it; + return nullptr; +} + const ScopeRange *ScopeIndex::find_innermost(uint64_t offset) const { // Linear scan picks the smallest-by-end range that contains offset. Could // be tightened with an interval tree, but typical files have <1k scopes diff --git a/src/scope.hpp b/src/scope.hpp index d16b96f..53219ce 100644 --- a/src/scope.hpp +++ b/src/scope.hpp @@ -56,6 +56,12 @@ class ScopeIndex { // (smallest) containing range wins. const ScopeRange *find_innermost(uint64_t offset) const; + // Scope whose signature (anchor) line is `line` (1-based), or nullptr. + // A match on this line sits on the definition itself, not inside a body — + // role classification renders it as `def` rather than `in`. Signatures + // spanning multiple lines only register their first line. + const ScopeRange *anchor_on_line(uint32_t line) const; + // Full range list, sorted by start_off ascending (nested ranges are // interleaved). Used for ancestor-chain checks (-in-scope), anchorless // scope edits, and -list-scopes. diff --git a/tests/run.sh b/tests/run.sh index c12f95e..c536034 100755 --- a/tests/run.sh +++ b/tests/run.sh @@ -1242,6 +1242,92 @@ expect_contains "rule-file description type-checked" "'description' must be a st [[ "$RC" == "2" ]] && report ok "rule-file bad description exit 2" || report fail "rule-file bad description exit (got $RC)" rm -rf "$LD_FIX" +# --------------------------------------------------------------------------- +section "per-match role tags (def/comment/string/import)" +RT_FIX="$HERE/_tmp_roles" +mkdir -p "$RT_FIX" +cat > "$RT_FIX/a.go" <<'EOF' +package main + +import "fmt" + +// TODO comment hit +func Target() { + s := "TODO in string" + u := "http://x/TODO" + fmt.Println(s, u) // trailing TODO note +} +EOF + +OUT=$("$BIN" -p TODO -llm "$RT_FIX/a.go") +expect_contains "comment tag" "// TODO comment hit [comment]" "$OUT" +expect_contains "string tag" '"TODO in string" [string]' "$OUT" +expect_contains "trailing comment tag" "trailing TODO note [comment]" "$OUT" +expect_contains "// inside a string is not a comment" 'http://x/TODO" [string]' "$OUT" + +OUT=$("$BIN" -p 'func Target' -llm -scope go "$RT_FIX/a.go") +expect_contains "def tag on signature line" "[def func Target]" "$OUT" +expect_not_contains "def replaces [in] on the signature" "[in func Target]" "$OUT" + +OUT=$("$BIN" -p 'import' -llm "$RT_FIX/a.go") +expect_contains "import tag" 'import "fmt" [import]' "$OUT" + +# Position-accurate roles beat line-based ones: a trailing comment on a +# signature line is a comment, not a def. +cat > "$RT_FIX/b.go" <<'EOF' +package main + +func Helper() { // TODO here +} +EOF +OUT=$("$BIN" -p TODO -llm -scope go "$RT_FIX/b.go") +expect_contains "comment beats def on signature line" "[comment]" "$OUT" +expect_not_contains "no def tag for trailing comment" "[def" "$OUT" + +OUT=$("$BIN" -p TODO "$RT_FIX/a.go") +expect_contains "JSONL role field" '"role":"comment"' "$OUT" +OUT=$("$BIN" -p TODO -format '$LINE:$ROLE' "$RT_FIX/a.go") +expect_contains "\$ROLE format token" "5:comment" "$OUT" +OUT=$("$BIN" -p TODO -sample 2 -llm "$RT_FIX/a.go") +expect_contains "-sample keeps role tags" "[comment]" "$OUT" + +OUT=$("$BIN" -p TODO -llm -no-roles "$RT_FIX/a.go") +expect_not_contains "-no-roles drops -llm tags" "[comment]" "$OUT" +OUT=$("$BIN" -p TODO -no-roles "$RT_FIX/a.go") +expect_not_contains "-no-roles drops the JSONL field" '"role"' "$OUT" + +cat > "$RT_FIX/c.c" <<'EOF' +#include +/* start + TODO inside block + end */ +int x; +EOF +OUT=$("$BIN" -p TODO -llm "$RT_FIX/c.c") +expect_contains "block comment spans lines" "TODO inside block [comment]" "$OUT" +OUT=$("$BIN" -p 'stdio' -llm "$RT_FIX/c.c") +expect_contains "#include line tagged import" "[import]" "$OUT" + +cat > "$RT_FIX/p.py" <<'EOF' +import os +# TODO hash comment +s = "TODO quoted" +d = """ +TODO in docstring +""" +EOF +OUT=$("$BIN" -p TODO -llm "$RT_FIX/p.py") +expect_contains "python hash comment" "# TODO hash comment [comment]" "$OUT" +expect_contains "python string" '"TODO quoted" [string]' "$OUT" +expect_contains "python triple-quote string" "TODO in docstring [string]" "$OUT" +OUT=$("$BIN" -p 'import os' -llm "$RT_FIX/p.py") +expect_contains "python import tag" "[import]" "$OUT" + +printf '// TODO not code\n' > "$RT_FIX/x.txt" +OUT=$("$BIN" -p TODO -llm "$RT_FIX/x.txt") +expect_not_contains "unknown extension: no tags" "[comment]" "$OUT" +rm -rf "$RT_FIX" + # --------------------------------------------------------------------------- section "fixed strings (-F / -Fi / -patterns-from)" FS_FIX="$HERE/_tmp_fixed" From 0c6dddda6785a3d8855c0d2f8f293ddaf3666411 Mon Sep 17 00:00:00 2001 From: Pinkhas Nisanov Date: Thu, 13 Aug 2026 21:12:29 +0300 Subject: [PATCH 3/4] Add -rollup scope summaries and a co-occurrence footer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two more LLM-facing output features: - New -rollup output mode: one line per innermost enclosing scope with line range, kind/name, match count, a count-descending per-pattern breakdown, and the scope's first matched line as a representative. Scopeless matches group under a single "(top level)" row, so files without a scope pack collapse to a per-file summary. Implies -scope auto like -elide; shares the query header and result footers. - Co-occurrence footer in -llm/-elide/-rollup: when >=2 patterns matched, a trailing '--- files: ... ---' line gives per-pattern distinct-file counts and the overlap — an exact 'both:' crosstab for two patterns, 'multi-pattern:' files with their pattern sets for more. 'both: 0' is stated explicitly; counts are whole-scan and post-filter, qualified on early stops. Co-Authored-By: Claude Fable 5 --- HPRSCRIPT.md | 46 ++++++++++-- skills/hprscript-search/SKILL.md | 3 +- src/cli.cpp | 7 +- src/output.cpp | 119 +++++++++++++++++++++++++++++-- src/output.hpp | 24 +++++++ src/runner.cpp | 116 ++++++++++++++++++++++++++++-- tests/run.sh | 70 +++++++++++++++++- 7 files changed, 367 insertions(+), 18 deletions(-) diff --git a/HPRSCRIPT.md b/HPRSCRIPT.md index d8234a3..f84c98f 100644 --- a/HPRSCRIPT.md +++ b/HPRSCRIPT.md @@ -102,7 +102,7 @@ simultaneous. | `-pi ` | Case-insensitive search pattern (HS `CASELESS`; folds Unicode by default; repeatable, mixable with `-p`) | | `-F ` / `-Fi ` | Fixed-string pattern (case-sensitive / -insensitive) — matched literally, no regex interpretation. Repeatable, mixable with `-p`/`-pi`. | | `-name ` | Name the preceding `-p`/`-pi`/`-F`/`-Fi`/`-ident`: the id (`[A-Za-z_]\w*`) replaces the auto `p`/`ident` in `pat`, `$PAT_ID`, `-llm` tags, relations, and `-file-where`. | -| `-desc ` | Describe the preceding pattern in free text. When any pattern has one, `-llm`/`-elide` output opens with a [query legend](#query-header-and-no-matches-footer) so each result block is self-describing. | +| `-desc ` | Describe the preceding pattern in free text. When any pattern has one, `-llm`/`-elide`/`-rollup` output opens with a [query legend](#query-header-and-result-footers) so each result block is self-describing. | | `-patterns-from ` | Load additional patterns from a JSONL rule file — one `{"id","regexp"\|"literal","description","case_insensitive","word_boundary","utf8","ref"}` object per line, `#` comments allowed. Repeatable. See [Fixed strings & pattern files](#fixed-strings--pattern-files--f--fi--patterns-from). | | `-ident ''` | Match identifiers whose subtokens include ALL given space-separated terms, regardless of casing/separator (`parseConfig` ~ `parse_config`). Repeatable = OR. See [Identifier matching](#identifier-matching--ident). | | `-file-where ` | Per-file predicate: pattern ids, plus `count(pat) > n` / `churn(days) > n` / `lang == name` conditions (`'err AND NOT recovery'`, `'churn(30) > 2'`). See [Per-file conditions](#per-file-conditions--file-where). | @@ -159,6 +159,7 @@ simultaneous. | `-absent` | Files where the pattern is **not** found (like `grep -L`) | | `-llm` | Token-efficient plain text grouped by file (LLM-friendly). See [LLM output mode](#llm-output-mode). | | `-elide` | Scope-aware chunks with unmatched interior lines folded away. See [Elided scope output](#elided-scope-output--elide). | +| `-rollup` | One line per enclosing scope: line range, name, per-pattern match counts, one representative line. See [Scope rollup](#scope-rollup--rollup). | `-format` template tokens (substituted per match): @@ -372,7 +373,7 @@ rejected at startup. `-desc ` is the same kind of postfix modifier, but for meaning instead of identity: free text explaining what the pattern is *for*. It surfaces as a one-time query legend at the top of `-llm`/`-elide` output (see -[Query header and no-matches footer](#query-header-and-no-matches-footer)), +[Query header and result footers](#query-header-and-result-footers)), so a reader of the results — human or LLM — doesn't have to reverse-engineer intent from the regex: @@ -2316,9 +2317,9 @@ Line-shape matches also carry [role tags](#per-match-role-tags) — `[comment]`, When `-limit` or `-max-output-bytes` truncates the output, a final `--- limit reached: ... ---` (or `--- output-byte budget reached ... ---`) footer line is emitted so the reader knows the result was cut, not finished. -### Query header and no-matches footer +### Query header and result footers -Two additions make each `-llm`/`-elide` result block self-describing — useful when the output is read later (or by a different agent) than the command that produced it: +Three additions make each `-llm`/`-elide`/`-rollup` result block self-describing — useful when the output is read later (or by a different agent) than the command that produced it: **Query header.** When any pattern carries a description (`-desc`, or `description` in a `-patterns-from` rule file), the output opens with an echo of the interpreted query and a one-line legend per pattern — described patterns show their description, the rest fall back to their regexp so the legend is complete on its own: @@ -2338,6 +2339,20 @@ No `-desc` anywhere → no header, so existing output is unchanged unless you op The counts cover the whole scan, not just the displayed subset, so the footer stays truthful under `-sample`/`-hotspots`. When `-limit` or `-max-output-bytes` stopped the scan early the claim only holds for the scanned prefix, and the footer says so: `--- no matches (scan stopped early): … ---`. Exit codes are unaffected (still 1 when nothing matched at all). +**Co-occurrence footer.** Batched patterns are usually batched to be *correlated* — so the per-file join is handed over instead of left to the reader. When at least two patterns matched somewhere, a trailing footer gives each pattern's distinct-file count and the overlap. Two patterns get an exact crosstab: + +``` +--- files: warn 12, err 41; both: 3 (a.go, b.go, c.go) --- +``` + +Three or more report the multi-pattern files with their pattern sets: + +``` +--- files: warn 12, err 41, todo 7; multi-pattern: 2 (a.go warn+err, b.go warn+err+todo) --- +``` + +`both: 0` is stated explicitly — absence of overlap is a finding. At most 5 files are named (`+N more` covers the rest); counts are whole-scan and post-filter, qualified with `(scan stopped early)` like the no-matches footer when a limit cut the scan. + ### Examples ```bash @@ -2423,6 +2438,29 @@ hprscript -p 'cli\.limit' -elide -scope cpp src/runner.cpp --- +## Scope rollup (`-rollup`) + +Between per-match lines (`-llm`) and ranked files (`-hotspots`) there's a missing altitude: *which functions* are involved, and how heavily. Ten hits inside one function is one fact, not ten. `-rollup` renders one line per innermost enclosing scope — line range, kind and name, match count with a per-pattern breakdown (count-descending), plus the scope's first matched line as a representative: + +```bash +hprscript -p 'work\(\)' -name w -p 'other\(\)' -name o -rollup pkg/ +# → +# pkg/m.go +# 3-7 func Alpha — 3 hits (w×2, o×1) +# 4: work() +# 9-11 func Beta — 1 hit (w×1) +# 10: work() +# --- files: w 1, o 1; both: 1 (pkg/m.go) --- +``` + +Implies `-scope auto` when no `-scope` config is given (same convention as `-elide`). Matches outside any detected scope — including whole files in languages without a scope pack — group under a single `(top level)` row, so a config file or log collapses to a one-row-per-file summary. The per-pattern breakdown is omitted when only one pattern is active. + +`-rollup` is an output mode, mutually exclusive with `-j`/`-f`/`-c`/`-o`/`-format`/`-absent`/`-llm`/`-elide` and with `-sample` (it renders whole-file batches). It participates in the [query header and result footers](#query-header-and-result-footers) like `-llm` and `-elide`. `-m` caps how many of a file's matches participate. + +Use it as the first call of an investigation — it answers "where is this concentrated?" in a handful of tokens — then drill into specific scopes with `-in-scope` or `-elide`. + +--- + ## Cross-invocation dedup (`-seen`) Agents iterate: search, refine, search again — and re-pay tokens for the same unchanged function every round. `-seen ` fixes that for [`-elide`](#elided-scope-output--elide) and [`-budget`](#budget-packed-context--budget-n): before rendering a matched scope in full, its raw source bytes are hashed and checked against `path`; an unchanged chunk collapses to one line instead of its full body, and the file is rewritten at the end of every run with what was actually shown. diff --git a/skills/hprscript-search/SKILL.md b/skills/hprscript-search/SKILL.md index 0f1d7d3..69d817d 100644 --- a/skills/hprscript-search/SKILL.md +++ b/skills/hprscript-search/SKILL.md @@ -12,7 +12,7 @@ Invoke the binary through Bash as `hprscript`. Use one call per reasoning stage ## Non-negotiable defaults - Put distinguishable terms in separate `-p` or `-pi` flags so every hit retains its pattern ID. -- Prefer `-llm` when reading results, `-f` for paths, `-c` for counts, and `-limit N` for existence checks. In `-llm`/`-elide` output, patterns with zero matches are named in a trailing `--- no matches: … ---` footer — treat that as explicit evidence of absence, qualified with "scan stopped early" when a limit cut the scan. +- Prefer `-llm` when reading results, `-f` for paths, `-c` for counts, and `-limit N` for existence checks. In `-llm`/`-elide`/`-rollup` output, patterns with zero matches are named in a trailing `--- no matches: … ---` footer — treat that as explicit evidence of absence, qualified with "scan stopped early" when a limit cut the scan. With ≥2 matching patterns a `--- files: … ---` footer gives per-pattern file counts and the overlap (`both:`/`multi-pattern:`) — read the correlation from there instead of joining by hand. - Trust per-match role tags instead of re-deriving them: `[comment]`/`[string]`/`[import]` in `-llm` (a `role` field in JSONL, `$ROLE` in `-format`) classify each hit lexically, and with `-scope` active a hit on a signature line reads `[def func X]` while body hits read `[in func X]`. Untagged hits in a recognized language are plain code. - Use an absolute path or glob when the effective cwd is uncertain. Inspect the first emitted path and stop if it escapes the intended tree. - Add `-summary -require-complete` when a broad sweep must be exhaustive. Do not present a partial scan as complete. @@ -32,6 +32,7 @@ Invoke the binary through Bash as `hprscript`. Use one call per reasoning stage | Files containing / missing a pattern | `-f` / `-absent` | | File outline / enclosing function | `-list-scopes` / `-scope auto` | | Representative usages | `-sample N` | +| Which functions are involved, and how heavily | `-rollup` (one line per scope with per-pattern counts) | | Ranked files / packed evidence | `-hotspots N` / `-budget N` | | Compact scope excerpts | `-elide` | | Cross-run chunk deduplication | `-elide` or `-budget` with `-seen ` | diff --git a/src/cli.cpp b/src/cli.cpp index 4a013ef..d0c0453 100644 --- a/src/cli.cpp +++ b/src/cli.cpp @@ -63,7 +63,7 @@ bool parse_nonneg(const char *s, int64_t &out) { bool set_output_mode(Cli &cli, OutputMode mode) { if (cli.out_mode_set) { cli.error = true; - cli.error_message = "output modes -j/-f/-c/-o/-format/-absent/-llm/-elide are mutually exclusive"; + cli.error_message = "output modes -j/-f/-c/-o/-format/-absent/-llm/-elide/-rollup are mutually exclusive"; return false; } cli.out_mode = mode; @@ -245,6 +245,10 @@ void print_help(FILE *out) { " -elide Scope-aware chunks: signature + matched lines with -A/-B\n" " context; untouched interior lines fold as \"… (+N lines)\"\n" " (implies -scope auto when no -scope config is given)\n" +" -rollup One line per enclosing scope: line range, name, match\n" +" count with per-pattern breakdown, and one representative\n" +" line; scopeless matches group as \"(top level)\" (implies\n" +" -scope auto when no -scope config is given)\n" "\n" "Block extraction (with -p):\n" " -block-open Opening delimiter (e.g. \"{\")\n" @@ -761,6 +765,7 @@ Cli parse_cli(int argc, char **argv) { if (eq(a, "-absent")) { if (!set_output_mode(cli, OutputMode::Absent)) return cli; continue; } if (eq(a, "-llm")) { if (!set_output_mode(cli, OutputMode::Llm)) return cli; continue; } if (eq(a, "-elide")) { if (!set_output_mode(cli, OutputMode::Elide)) return cli; continue; } + if (eq(a, "-rollup")) { if (!set_output_mode(cli, OutputMode::Rollup)) return cli; continue; } if (eq(a, "-format")) { const char *v = take(i, argc, argv, a, cli); if (!v) return cli; if (!set_output_mode(cli, OutputMode::Custom)) return cli; diff --git a/src/output.cpp b/src/output.cpp index f4a4258..8b80ce9 100644 --- a/src/output.cpp +++ b/src/output.cpp @@ -490,7 +490,8 @@ void Formatter::emit_custom(const std::string &file, const Pattern &pattern, void Formatter::emit_header() { if (header_done_) return; header_done_ = true; - if (opts_.mode != OutputMode::Llm && opts_.mode != OutputMode::Elide) + if (opts_.mode != OutputMode::Llm && opts_.mode != OutputMode::Elide && + opts_.mode != OutputMode::Rollup) return; if (opts_.header_lines.empty()) return; auto &s = scratch_; @@ -770,6 +771,105 @@ void Formatter::on_file_elide(const std::string &file, flush(); } +void Formatter::on_file_rollup(const std::string &file, + const std::vector &kept, + std::string_view buf, const LineIndex &idx, + const ScopeIndex *scope) { + if (kept.empty()) return; + emit_header(); + emitted_ += static_cast(kept.size()); + + // Group by innermost enclosing scope in first-seen order. Unlike + // -elide's contiguous-run flushing, a map keeps a scope's count whole + // even when nesting interleaves its matches; every no-scope match lands + // in one shared "(top level)" group. + struct Group { + const ScopeRange *sr; + std::vector ms; + }; + std::vector groups; + std::unordered_map group_of; + for (const auto &m : kept) { + const ScopeRange *sr = scope ? scope->find_innermost(m.from) : nullptr; + auto it = group_of.find(sr); + if (it == group_of.end()) { + group_of.emplace(sr, groups.size()); + groups.push_back({sr, {&m}}); + } else { + groups[it->second].ms.push_back(&m); + } + } + + auto &s = scratch_; + if (llm_last_file_ != file) { + llm_last_file_ = file; + s.clear(); + s.append(file); + s += '\n'; + write_out(s); + } + + for (const auto &g : groups) { + s.clear(); + s += " "; + if (g.sr) { + append_uint32(s, g.sr->line_start); + s += '-'; + append_uint32(s, g.sr->line_end); + s += ' '; + s += g.sr->kind; + s += ' '; + s += g.sr->name; + } else { + s += "(top level)"; + } + s += " \xE2\x80\x94 "; // em dash + append_uint(s, g.ms.size()); + s += g.ms.size() == 1 ? " hit" : " hits"; + + // Per-pattern breakdown, count-descending — only worth the tokens + // when more than one pattern is in play. + if (opts_.pattern_count > 1 && opts_.patterns) { + std::vector> counts; // (pat, n) + for (const Match *m : g.ms) { + bool found = false; + for (auto &c : counts) + if (c.first == m->pattern_index) { + ++c.second; + found = true; + break; + } + if (!found) counts.push_back({m->pattern_index, 1}); + } + std::stable_sort(counts.begin(), counts.end(), + [](const auto &a, const auto &b) { + return a.second > b.second; + }); + s += " ("; + for (size_t i = 0; i < counts.size(); ++i) { + if (i) s += ", "; + s += (*opts_.patterns)[counts[i].first].id; + s += "\xC3\x97"; // × + append_uint(s, counts[i].second); + } + s += ')'; + } + s += '\n'; + + // One representative: the group's first match line. + uint32_t line = idx.line_of(g.ms.front()->from); + std::string_view text = idx.line_text(line); + text = truncate_safe(text, opts_.max_context_bytes, nullptr); + if (!text.empty() && text.back() == '\n') text.remove_suffix(1); + s += " "; + append_uint32(s, line); + s += ": "; + s.append(text.data(), text.size()); + s += '\n'; + write_out(s); + } +} + void Formatter::on_match(const std::string &file, const Pattern &pattern, const Match &m, std::string_view buf, const LineIndex &idx, const ScopeIndex *scope, @@ -791,9 +891,11 @@ void Formatter::on_match(const std::string &file, const Pattern &pattern, case OutputMode::Custom: emit_custom(file, pattern, m, buf, idx, scope, roles); break; case OutputMode::Llm: emit_llm(file, pattern, m, buf, idx, scope, roles); break; case OutputMode::Elide: - // Elide renders a whole file's matches at once via - // on_file_elide(); it never streams through on_match(). Undo - // the ++emitted_ above so callers can't double-count by mistake. + case OutputMode::Rollup: + // Elide/Rollup render a whole file's matches at once via + // on_file_elide()/on_file_rollup(); they never stream through + // on_match(). Undo the ++emitted_ above so callers can't + // double-count by mistake. --emitted_; break; case OutputMode::Absent: @@ -826,8 +928,15 @@ void Formatter::on_file_end(const std::string &file, bool had_match) { } void Formatter::on_complete() { - if (opts_.mode == OutputMode::Llm || opts_.mode == OutputMode::Elide) { + if (opts_.mode == OutputMode::Llm || opts_.mode == OutputMode::Elide || + opts_.mode == OutputMode::Rollup) { char buf[160]; + if (!cooccurrence_footer_.empty()) { + emit_header(); + std::string s = cooccurrence_footer_; + s += '\n'; + std::fwrite(s.data(), 1, s.size(), out_); + } if (!zero_match_ids_.empty()) { // A batched pattern that matched nothing must be stated, not // inferred from its absence in the output. When the scan stopped diff --git a/src/output.hpp b/src/output.hpp index 06fc29f..219dbed 100644 --- a/src/output.hpp +++ b/src/output.hpp @@ -33,6 +33,7 @@ enum class OutputMode { Absent, // grep -L: files where pattern is NOT present Llm, // -llm: token-efficient text for LLM consumption Elide, // -elide: scope-aware chunk rendering, whole file at once + Rollup, // -rollup: one line per enclosing scope with match counts }; // True iff the given output mode reads line/col/context from a LineIndex. @@ -43,6 +44,7 @@ inline bool needs_line_index(OutputMode m) { case OutputMode::Custom: case OutputMode::Llm: case OutputMode::Elide: + case OutputMode::Rollup: return true; case OutputMode::FilesOnly: case OutputMode::Counts: @@ -78,6 +80,10 @@ struct OutputOptions { // and surfaced in JSON / format-template output. Pointer is borrowed. const ExtractTable *extract_table = nullptr; + // Full pattern list (borrowed) — Rollup mode needs it to render + // per-pattern counts by id. nullptr disables the breakdown. + const std::vector *patterns = nullptr; + // LLM mode only: total pattern count (so the formatter can decide whether // to prefix records with the pattern id) and the effective global limit // (so on_complete can emit a "limit reached" footer). @@ -124,6 +130,16 @@ class Formatter { const ScopeIndex *scope, const SeenStore *seen = nullptr, std::vector *marks_out = nullptr); + // -rollup mode: render one file's kept-match set as one line per + // innermost enclosing scope — ` — N hits + // (×, …)` plus the first matched line as a representative. + // Matches outside any scope group under a single "(top level)" row, so + // files without scope detection collapse to a per-file summary. `kept` + // must be sorted by `from` (the collector's normal emission order). + void on_file_rollup(const std::string &file, const std::vector &kept, + std::string_view buf, const LineIndex &idx, + const ScopeIndex *scope); + // Record-level absence (-records line with -absent): emit one JSON line // for a record (line) of `file` that lacks `pattern`. Only meaningful in // Absent mode; `text` is the record's content (truncated to @@ -166,6 +182,13 @@ class Formatter { patterns_total_ = total; } + // Pre-rendered cross-pattern co-occurrence footer (no trailing newline), + // built by the runner from whole-scan per-file pattern sets. Printed by + // on_complete() in the LLM-facing modes; empty = none. + void set_cooccurrence_footer(std::string line) { + cooccurrence_footer_ = std::move(line); + } + private: void emit_json(const std::string &file, const Pattern &pattern, const Match &m, std::string_view buf, const LineIndex &idx, @@ -202,6 +225,7 @@ class Formatter { bool header_done_ = false; // query header printed (emit_header dedupe) std::vector zero_match_ids_; // patterns with 0 kept matches size_t patterns_total_ = 0; // denominator for the "no matches" footer + std::string cooccurrence_footer_; // set_cooccurrence_footer() std::string llm_last_file_; // last file printed as a header (-llm/-elide dedupe) std::unordered_map per_file_counts_; diff --git a/src/runner.cpp b/src/runner.cpp index f25c60f..17dd3a5 100644 --- a/src/runner.cpp +++ b/src/runner.cpp @@ -274,13 +274,20 @@ int run_search(const Cli &cli) { oo.extract_table = extract_table.any() ? &extract_table : nullptr; oo.pattern_count = patterns.size(); oo.global_limit = cli.limit; + oo.patterns = &patterns; // outlives the Formatter (borrowed) + + // The three LLM-facing modes share the query header and the trailing + // co-occurrence / no-matches footers. + const bool llm_facing = oo.mode == OutputMode::Llm || + oo.mode == OutputMode::Elide || + oo.mode == OutputMode::Rollup; // Query header: when any pattern carries a description, LLM-facing // output opens with an echo of the interpreted query plus a one-line // legend per pattern. Pre-rendered here so the Formatter stays ignorant // of Cli; patterns without a description fall back to their regexp so // the legend is complete on its own. - if (oo.mode == OutputMode::Llm || oo.mode == OutputMode::Elide) { + if (llm_facing) { bool any_desc = false; for (const auto &p : patterns) if (!p.desc.empty()) { any_desc = true; break; } @@ -368,7 +375,7 @@ int run_search(const Cli &cli) { // context anyway. std::string eff_scope_lang = cli.scope_lang; if ((tf.scope_needed() || oo.mode == OutputMode::Elide || - cli.budget_bytes > 0) && + oo.mode == OutputMode::Rollup || cli.budget_bytes > 0) && eff_scope_lang.empty() && cli.scope_pattern.empty()) eff_scope_lang = "auto"; bool scope_enabled = !eff_scope_lang.empty() || @@ -426,11 +433,13 @@ int run_search(const Cli &cli) { // Sample is incompatible with output modes that don't emit per-match // data — let the user know rather than silently doing nothing. if (oo.mode == OutputMode::FilesOnly || oo.mode == OutputMode::Counts || - oo.mode == OutputMode::Absent || oo.mode == OutputMode::Elide) { + oo.mode == OutputMode::Absent || oo.mode == OutputMode::Elide || + oo.mode == OutputMode::Rollup) { std::fprintf(stderr, "hprscript: -sample requires a per-match output mode " - "(default JSONL, -o, -format, -llm) — -elide renders " - "whole-file batches and doesn't compose with -sample\n"); + "(default JSONL, -o, -format, -llm) — -elide/-rollup " + "render whole-file batches and don't compose with " + "-sample\n"); return 2; } } @@ -573,6 +582,19 @@ int run_search(const Cli &cli) { // depend on what happened to be displayed. std::vector pattern_hits(patterns.size(), 0); + // Cross-pattern co-occurrence, per file: distinct-file counts per + // pattern, plus which files matched more than one pattern. Feeds the + // trailing co-occurrence footer in the LLM-facing modes. + std::vector pattern_file_counts(patterns.size(), 0); + uint64_t multi_pattern_files = 0; + struct MultiFile { + std::string file; + std::vector pats; + }; + std::vector multi_sample; // first few, for the footer's list + static constexpr size_t MULTI_SAMPLE_CAP = 5; + std::vector file_pats(patterns.size()); // per-file scratch + auto scan_buf = [&](const std::string &display_name, std::string_view content) -> bool { LineIndex idx; @@ -607,6 +629,28 @@ int run_search(const Cli &cli) { for (const auto &m : kept) ++pattern_hits[m.pattern_index]; + if (!kept.empty()) { + std::fill(file_pats.begin(), file_pats.end(), 0); + for (const auto &m : kept) file_pats[m.pattern_index] = 1; + uint32_t distinct = 0; + for (size_t pi = 0; pi < file_pats.size(); ++pi) { + if (!file_pats[pi]) continue; + ++pattern_file_counts[pi]; + ++distinct; + } + if (distinct >= 2) { + ++multi_pattern_files; + if (multi_sample.size() < MULTI_SAMPLE_CAP) { + MultiFile mf; + mf.file = display_name; + for (size_t pi = 0; pi < file_pats.size(); ++pi) + if (file_pats[pi]) + mf.pats.push_back(static_cast(pi)); + multi_sample.push_back(std::move(mf)); + } + } + } + // -records line (with -absent): emit one record per non-empty line // lacking each pattern, then skip the per-file emission path (and the // Absent-mode file listing) entirely. @@ -732,6 +776,16 @@ int run_search(const Cli &cli) { // and discarded like -budget — so every mark commits. if (seen_active) for (const auto &m : marks) seen_store.mark(m); } + } else if (oo.mode == OutputMode::Rollup) { + std::vector capped = kept; + if (cli.per_file_limit > 0 && + capped.size() > static_cast(cli.per_file_limit)) + capped.resize(static_cast(cli.per_file_limit)); + if (!capped.empty()) { + had_match = true; + fmt.on_file_rollup(display_name, capped, content, idx, + scope_ptr); + } } else { RoleIndex role_idx; const RoleIndex *roles_ptr = nullptr; @@ -1084,11 +1138,61 @@ int run_search(const Cli &cli) { } } - if (oo.mode == OutputMode::Llm || oo.mode == OutputMode::Elide) { + if (llm_facing) { std::vector zero; for (size_t i = 0; i < patterns.size(); ++i) if (pattern_hits[i] == 0) zero.push_back(patterns[i].id); fmt.set_zero_match_patterns(std::move(zero), patterns.size()); + + // Co-occurrence footer: batched patterns are usually batched to be + // correlated — hand over the per-file join instead of leaving it to + // the reader. Only meaningful when at least two patterns matched + // somewhere; `both: 0` is stated explicitly (absence of overlap is + // a finding). Counts are whole-scan, qualified on early stops. + size_t active = 0; + for (uint64_t c : pattern_file_counts) + if (c > 0) ++active; + if (patterns.size() >= 2 && active >= 2) { + const bool stopped = stats.stop_reason == "limit" || + stats.stop_reason == "output_budget"; + std::string f = stopped ? "--- files (scan stopped early): " + : "--- files: "; + bool first = true; + for (size_t i = 0; i < patterns.size(); ++i) { + if (pattern_file_counts[i] == 0) continue; + if (!first) f += ", "; + first = false; + f += patterns[i].id; + f += ' '; + f += std::to_string(pattern_file_counts[i]); + } + f += "; "; + f += (patterns.size() == 2) ? "both: " : "multi-pattern: "; + f += std::to_string(multi_pattern_files); + if (multi_pattern_files > 0) { + f += " ("; + for (size_t i = 0; i < multi_sample.size(); ++i) { + if (i) f += ", "; + f += multi_sample[i].file; + if (patterns.size() > 2) { + f += ' '; + for (size_t k = 0; k < multi_sample[i].pats.size(); ++k) { + if (k) f += '+'; + f += patterns[multi_sample[i].pats[k]].id; + } + } + } + if (multi_pattern_files > multi_sample.size()) { + f += ", +"; + f += std::to_string(multi_pattern_files - + multi_sample.size()); + f += " more"; + } + f += ')'; + } + f += " ---"; + fmt.set_cooccurrence_footer(std::move(f)); + } } fmt.on_complete(); diff --git a/tests/run.sh b/tests/run.sh index c536034..b89af2e 100755 --- a/tests/run.sh +++ b/tests/run.sh @@ -1328,6 +1328,74 @@ OUT=$("$BIN" -p TODO -llm "$RT_FIX/x.txt") expect_not_contains "unknown extension: no tags" "[comment]" "$OUT" rm -rf "$RT_FIX" +# --------------------------------------------------------------------------- +section "scope rollup (-rollup) and co-occurrence footer" +RU_FIX="$HERE/_tmp_rollup" +mkdir -p "$RU_FIX" +cat > "$RU_FIX/m.go" <<'EOF' +package main + +func Alpha() { + work() + work() + other() +} + +func Beta() { + work() +} +EOF + +OUT=$("$BIN" -p 'work\(\)' -name w -p 'other\(\)' -name o -rollup "$RU_FIX/m.go") +expect_contains "rollup line with counts" "3-7 func Alpha — 3 hits (w×2, o×1)" "$OUT" +expect_contains "rollup second scope" "9-11 func Beta — 1 hit (w×1)" "$OUT" +expect_contains "rollup representative line" " 4: work()" "$OUT" +expect_contains "rollup file header" "$RU_FIX/m.go" "$OUT" + +OUT=$("$BIN" -p 'work\(\)' -rollup "$RU_FIX/m.go") +expect_contains "single pattern: no breakdown" "3-7 func Alpha — 2 hits" "$OUT" +expect_not_contains "single pattern: no ×" "×" "$OUT" + +printf 'TODO one\nx\nTODO two\n' > "$RU_FIX/notes.txt" +OUT=$("$BIN" -p TODO -rollup "$RU_FIX/notes.txt") +expect_contains "scopeless matches roll up as top level" "(top level) — 2 hits" "$OUT" + +OUT=$("$BIN" -p 'zz_nope' -rollup "$RU_FIX/m.go" ; echo "rc=$?") +expect_contains "rollup no matches exit 1" "rc=1" "$OUT" +expect_contains "rollup zero-match footer" "--- no matches: p0 (1 of 1 pattern) ---" "$OUT" + +OUT=$("$BIN" -p x -rollup -llm 2>&1) ; RC=$? +expect_contains "-rollup is an exclusive output mode" "mutually exclusive" "$OUT" +[[ "$RC" == "2" ]] && report ok "-rollup + -llm exit 2" || report fail "-rollup + -llm exit (got $RC)" +OUT=$("$BIN" -p x -rollup -sample 3 "$RU_FIX/m.go" 2>&1) ; RC=$? +expect_contains "-rollup + -sample rejected" "don't compose with -sample" "$OUT" + +# --- co-occurrence footer +cat > "$RU_FIX/a.log" <<'EOF' +WARN x +ERROR y +EOF +printf 'WARN only\n' > "$RU_FIX/b.log" +printf 'ERROR only\n' > "$RU_FIX/c.log" +OUT=$("$BIN" -p WARN -name warn -p ERROR -name err -llm "$RU_FIX/a.log" "$RU_FIX/b.log" "$RU_FIX/c.log") +expect_contains "co-occurrence crosstab" "--- files: warn 2, err 2; both: 1 ($RU_FIX/a.log) ---" "$OUT" + +OUT=$("$BIN" -p WARN -name warn -p ERROR -name err -llm "$RU_FIX/b.log" "$RU_FIX/c.log") +expect_contains "zero overlap stated explicitly" "both: 0" "$OUT" + +OUT=$("$BIN" -p WARN -p 'zz_nope' -llm "$RU_FIX/b.log") +expect_not_contains "one active pattern: no co-occurrence footer" "--- files:" "$OUT" + +OUT=$("$BIN" -p WARN -name warn -p ERROR -name err -p 'x$' -name xx -llm "$RU_FIX/a.log" "$RU_FIX/b.log" "$RU_FIX/c.log") +expect_contains ">2 patterns: multi-pattern form with sets" "multi-pattern: 1 ($RU_FIX/a.log warn+err+xx)" "$OUT" + +OUT=$("$BIN" -p WARN -name warn -p ERROR -name err "$RU_FIX/a.log") +expect_not_contains "JSONL mode: no co-occurrence footer" "--- files:" "$OUT" + +OUT=$("$BIN" -p 'work\(\)' -name w -p 'other\(\)' -name o -rollup "$RU_FIX/m.go") +expect_contains "rollup gets the co-occurrence footer" "--- files: w 1, o 1; both: 1" "$OUT" +rm -rf "$RU_FIX" + # --------------------------------------------------------------------------- section "fixed strings (-F / -Fi / -patterns-from)" FS_FIX="$HERE/_tmp_fixed" @@ -2083,7 +2151,7 @@ OUT=$("$BIN" -p x -elide -llm 2>&1) ; RC=$? expect_contains "-elide output-mode conflict" "mutually exclusive" "$OUT" [[ "$RC" == "2" ]] && report ok "-elide output-mode conflict exit 2" || report fail "-elide output-mode conflict exit (got $RC)" OUT=$("$BIN" -p x -elide -sample 5 2>&1) ; RC=$? -expect_contains "-elide + -sample rejected" "doesn't compose with -sample" "$OUT" +expect_contains "-elide + -sample rejected" "don't compose with -sample" "$OUT" [[ "$RC" == "2" ]] && report ok "-elide + -sample exit 2" || report fail "-elide + -sample exit (got $RC)" # --- -m caps how many matches participate in the render (pattern excludes From a4d2376915fd0c0b70109119bc8a7e642371a170 Mon Sep 17 00:00:00 2001 From: Pinkhas Nisanov Date: Thu, 13 Aug 2026 21:32:35 +0300 Subject: [PATCH 4/4] Add stable refs (-refs) and the expand verb MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The drill-down half of the search → expand loop: - New 'hprscript expand ' subcommand: prints the full enclosing scope of a search hit (scope pack resolved per file, -scope override honored) without the caller reconstructing a pattern and re-scanning. Multiple refs batch in one call; scopeless lines fall back to a numbered context window with the ref line marked; -max-block-bytes caps renders at a line boundary. - New -refs search flag (-llm/-rollup): appends a 6-hex content hash of the line's whitespace-trimmed text to each line number. expand verifies it before rendering — a moved line is recovered by content and noted as '(ref line moved: A -> B)'; a vanished one reports in-band as stale with exit 3, never silently expanding wrong code. ref_hash6 in src/expand.hpp is the single definition of ref identity, shared by search output and expand verification. Co-Authored-By: Claude Fable 5 --- HPRSCRIPT.md | 38 +++++ skills/hprscript-search/SKILL.md | 3 + src/cli.cpp | 30 ++++ src/cli.hpp | 12 ++ src/expand.cpp | 243 +++++++++++++++++++++++++++++++ src/expand.hpp | 29 ++++ src/main.cpp | 5 + src/output.cpp | 9 ++ src/output.hpp | 4 + src/runner.cpp | 1 + tests/run.sh | 80 ++++++++++ 11 files changed, 454 insertions(+) create mode 100644 src/expand.cpp create mode 100644 src/expand.hpp diff --git a/HPRSCRIPT.md b/HPRSCRIPT.md index f84c98f..f4bdc73 100644 --- a/HPRSCRIPT.md +++ b/HPRSCRIPT.md @@ -65,6 +65,7 @@ hprscript -s '' [files...] hprscript -script [files...] hprscript script.json [files...] # positional arg as script file cat script.json | hprscript # script piped on stdin +hprscript expand [...refs] # print a hit's enclosing scope hprscript edit -p [files...] # discover/preview/plan edits hprscript apply [apply flags] # verify and apply exact stored edits ``` @@ -117,6 +118,7 @@ simultaneous. | `-git-added-lines` | Restrict matches to lines **added** by the selected diffs (quick mode; needs `-git-changed`/`-git-staged`/`-git-range`). | | `-w` | Whole-word matching (wraps the pattern as `\b(?:expr)\b`) | | `-no-roles` | Disable per-match role classification (`role` JSONL field, `[def]`/`[comment]`/`[string]`/`[import]` tags in `-llm`, `$ROLE`). See [Per-match role tags](#per-match-role-tags). | +| `-refs` | Append a `@hash` content check to line numbers in `-llm`/`-rollup` output, making each hit a ref that `hprscript expand` verifies. See [Stable refs & expand](#stable-refs--expand--refs--hprscript-expand). | | `-no-utf8` | Disable UTF-8 mode (byte-level matching — see [UTF-8 / Unicode](#utf-8--unicode-support)) | | `-ucp` | Enable Unicode property classes for `\w`/`\d`/`\s` (opt-in; may reject some patterns) | | `-limit ` | Maximum global results — scanning stops once reached | @@ -2461,6 +2463,42 @@ Use it as the first call of an investigation — it answers "where is this conce --- +## Stable refs & expand (`-refs` / `hprscript expand`) + +Search output names every hit as `file:line`. `hprscript expand` turns that pointer into the full enclosing scope — the drill-down half of the search → expand loop — without the caller reconstructing a pattern and re-scanning: + +```bash +hprscript -p 'work\(\)' -llm pkg/m.go +# pkg/m.go +# 4: work() + +hprscript expand pkg/m.go:4 +# pkg/m.go:3-7 func Alpha +# func Alpha() { +# work() +# work() +# other() +# } +``` + +Multiple refs expand in one call (blank-line separated). The scope pack resolves per file (`-scope auto` semantics; `-scope`/`-scope-pattern` override). A line outside any detected scope falls back to a numbered context window with the ref line marked `>` (window size from `-C`, default 5). `-max-block-bytes` caps each render at a line boundary with an explicit `… (+N more lines)` marker. + +**Verified refs.** Files change between a search and the follow-up — especially mid-edit. `-refs` (in `-llm`/`-rollup` search output) appends a content check to each line number: + +``` + 4@1a87e3: work() +``` + +The hash covers the line's whitespace-trimmed text. `expand file:line@hash` verifies it before rendering: an unchanged line expands normally; a line whose content moved (code inserted above it) is found again by content and reported as `(ref line moved: 4 → 7)`; a line that no longer exists anywhere reports in-band as stale and exits 3 — expand never silently renders the wrong code: + +``` +pkg/m.go:4@ffffff: stale — line 4 changed and no line with this hash exists +``` + +Exit codes: 0 = every ref expanded, 3 = at least one stale ref (reported on stdout), 2 = bad ref syntax or unreadable file (stderr). `expand` is read-only, takes refs rather than patterns, and rejects output-mode flags. + +--- + ## Cross-invocation dedup (`-seen`) Agents iterate: search, refine, search again — and re-pay tokens for the same unchanged function every round. `-seen ` fixes that for [`-elide`](#elided-scope-output--elide) and [`-budget`](#budget-packed-context--budget-n): before rendering a matched scope in full, its raw source bytes are hashed and checked against `path`; an unchanged chunk collapses to one line instead of its full body, and the file is rewritten at the end of every run with what was actually shown. diff --git a/skills/hprscript-search/SKILL.md b/skills/hprscript-search/SKILL.md index 69d817d..81b3f23 100644 --- a/skills/hprscript-search/SKILL.md +++ b/skills/hprscript-search/SKILL.md @@ -33,6 +33,7 @@ Invoke the binary through Bash as `hprscript`. Use one call per reasoning stage | File outline / enclosing function | `-list-scopes` / `-scope auto` | | Representative usages | `-sample N` | | Which functions are involved, and how heavily | `-rollup` (one line per scope with per-pattern counts) | +| Full function around a known hit | `hprscript expand file:line[@hash]` (batch refs in one call) | | Ranked files / packed evidence | `-hotspots N` / `-budget N` | | Compact scope excerpts | `-elide` | | Cross-run chunk deduplication | `-elide` or `-budget` with `-seen ` | @@ -81,6 +82,8 @@ hprscript -p 'func\s+(\w+)\(([^)]*)\)' -extract name,args \ Prefer `-in-scope` over line numbers when code may move. Balanced block tracking is lexical, not language-aware; delimiters inside strings or comments can skew it. +To read the whole function around a hit, run `hprscript expand :` on the hit's location instead of re-searching with block flags; batch several refs in one call. Add `-refs` to the search so hits carry `file:line@hash` — expand then verifies the line, recovers it by content if it moved, and reports `stale` (exit 3) instead of rendering the wrong code after an edit. + Relate named patterns by distance or enclosing scope: ```bash diff --git a/src/cli.cpp b/src/cli.cpp index d0c0453..c80f04f 100644 --- a/src/cli.cpp +++ b/src/cli.cpp @@ -182,6 +182,7 @@ void print_help(FILE *out) { " hprscript investigate -p [options] [inputs]\n" " hprscript query -q '' [input overrides]\n" " hprscript query -query [input overrides]\n" +" hprscript expand [...refs]\n" " hprscript edit -p [files...]\n" " hprscript apply [apply flags]\n" "\n" @@ -218,6 +219,9 @@ void print_help(FILE *out) { " -no-roles Disable per-match role classification (the `role` JSONL\n" " field, [def]/[comment]/[string]/[import] tags in -llm,\n" " and $ROLE in -format)\n" +" -refs Append a @hash content check to line numbers in\n" +" -llm/-rollup output; the resulting file:line@hash refs\n" +" are verified by 'hprscript expand'\n" " -no-utf8 Disable UTF-8 mode (byte-level matching)\n" " -ucp Enable Unicode \\w/\\d/\\s (may reject some patterns)\n" " -limit Max global results\n" @@ -314,6 +318,12 @@ void print_help(FILE *out) { " record per function/class (JSONL, or -llm flat).\n" " Honors -in-scope/-in-scope-kind; takes no patterns\n" "\n" +"Expand mode (hprscript expand ...):\n" +" Print the enclosing scope of each ref (a search hit's file:line).\n" +" With @hash (from -refs) the line is verified first: a moved line is\n" +" found again by content; a vanished one reports 'stale' (exit 3).\n" +" Honors -scope*/-C (context for scopeless lines)/-max-block-bytes.\n" +"\n" "Script mode:\n" " -s Inline script\n" " -script Script file\n" @@ -407,6 +417,9 @@ Cli parse_cli(int argc, char **argv) { } else if (argc > 1 && eq(argv[1], "query")) { cli.query.active = true; first = 2; + } else if (argc > 1 && eq(argv[1], "expand")) { + cli.expand.active = true; + first = 2; } for (int i = first; i < argc; ++i) { const char *a = argv[i]; @@ -659,6 +672,10 @@ Cli parse_cli(int argc, char **argv) { cli.no_roles = true; continue; } + if (eq(a, "-refs") || eq(a, "--refs")) { + cli.refs = true; + continue; + } if (eq(a, "-glob")) { const char *v = take(i, argc, argv, a, cli); if (!v) return cli; cli.globs.emplace_back(v); @@ -1232,6 +1249,19 @@ Cli parse_cli(int argc, char **argv) { cli.error_message = "query cannot combine with quick-search output selectors"; } } + if (cli.expand.active && !cli.error && !cli.show_help && + !cli.show_version) { + if (!cli.patterns.empty() || !cli.patterns_from.empty() || + !cli.script_inline.empty() || !cli.script_path.empty()) { + cli.error = true; + cli.error_message = + "expand takes refs, not patterns/scripts"; + } else if (cli.out_mode_set) { + cli.error = true; + cli.error_message = + "expand output is plain text; output-mode flags do not apply"; + } + } return cli; } diff --git a/src/cli.hpp b/src/cli.hpp index fd259f2..17f69a5 100644 --- a/src/cli.hpp +++ b/src/cli.hpp @@ -122,6 +122,13 @@ struct QueryOptions { std::string path; }; +// `hprscript expand …` — print the enclosing scope of a +// search hit. Refs travel in Cli::positional; scope/-C/-max-block-bytes +// flags are shared with search mode. See src/expand.hpp. +struct ExpandOptions { + bool active = false; +}; + struct Cli { // Original argv, retained for immutable edit-plan provenance. std::vector command; @@ -184,6 +191,10 @@ struct Cli { // default for per-match output modes; roles are computed lazily per // matched file, so the cost only exists where output is produced. bool no_roles = false; + // -refs: append a @hash content check to line numbers in -llm/-rollup + // output, making each hit a verified `file:line@hash` ref that + // `hprscript expand` can check for drift before expanding. + bool refs = false; bool ucp = false; // -ucp: enable Unicode \w/\d/\s (opt-in) int64_t limit = -1; // global match cap (-limit) int64_t per_file_limit = -1; // per-file cap (-m) @@ -320,6 +331,7 @@ struct Cli { ApplyOptions apply; InvestigateOptions investigate; QueryOptions query; + ExpandOptions expand; // Misc. bool show_version = false; diff --git a/src/expand.cpp b/src/expand.cpp new file mode 100644 index 0000000..6b93cc9 --- /dev/null +++ b/src/expand.cpp @@ -0,0 +1,243 @@ +#include "expand.hpp" + +#include "cli.hpp" +#include "file_io.hpp" +#include "line_index.hpp" +#include "pipeline.hpp" // looks_binary +#include "scope.hpp" +#include "seen.hpp" // fnv1a + +#include +#include +#include +#include +#include + +namespace hpr { + +std::string ref_hash6(std::string_view line_text) { + size_t b = 0, e = line_text.size(); + auto ws = [](char c) { + return c == ' ' || c == '\t' || c == '\r' || c == '\n'; + }; + while (b < e && ws(line_text[b])) ++b; + while (e > b && ws(line_text[e - 1])) --e; + uint64_t h = fnv1a(line_text.substr(b, e - b)); + char buf[8]; + std::snprintf(buf, sizeof(buf), "%06llx", + static_cast(h & 0xFFFFFFull)); + return buf; +} + +namespace { + +struct Ref { + std::string path; + uint32_t line = 0; + std::string hash; // empty = unverified +}; + +// `file:line[@hash]`, parsed from the right so paths containing ':' work. +bool parse_ref(const std::string &s, Ref &out, std::string &err) { + std::string body = s; + size_t at = s.rfind('@'); + if (at != std::string::npos) { + std::string h = s.substr(at + 1); + bool hex = !h.empty() && h.size() <= 16; + for (char &c : h) { + c = static_cast(std::tolower(static_cast(c))); + if (!std::isxdigit(static_cast(c))) hex = false; + } + if (!hex) { + err = "bad ref '" + s + "': '@' must be followed by a hex hash"; + return false; + } + out.hash = h; + body = s.substr(0, at); + } + size_t colon = body.rfind(':'); + if (colon == std::string::npos || colon == 0 || + colon + 1 >= body.size()) { + err = "bad ref '" + s + "': expected :[@hash]"; + return false; + } + uint64_t line = 0; + for (size_t i = colon + 1; i < body.size(); ++i) { + char c = body[i]; + if (c < '0' || c > '9') { + err = "bad ref '" + s + "': line must be a number"; + return false; + } + line = line * 10 + static_cast(c - '0'); + if (line > 0xFFFFFFFFull) { + err = "bad ref '" + s + "': line out of range"; + return false; + } + } + if (line == 0) { + err = "bad ref '" + s + "': lines are 1-based"; + return false; + } + out.path = body.substr(0, colon); + out.line = static_cast(line); + return true; +} + +} // namespace + +int run_expand(const Cli &cli) { + if (cli.positional.empty()) { + std::fprintf(stderr, + "hprscript: expand: at least one ref " + "required\n"); + return 2; + } + if (!cli.scope_lang.empty() && cli.scope_lang != "auto" && + !builtin_scope_pack(cli.scope_lang)) { + std::fprintf(stderr, + "hprscript: unknown -scope pack '%s' (supported: auto, " + "go, rust, c, cpp, java, js, ts)\n", + cli.scope_lang.c_str()); + return 2; + } + + ScopeConfig custom; + custom.anchor_regex = cli.scope_pattern; + custom.open = cli.scope_open; + custom.close = cli.scope_close; + custom.kind = cli.scope_kind; + const std::string eff_lang = + cli.scope_lang.empty() && cli.scope_pattern.empty() ? "auto" + : cli.scope_lang; + + int exit_code = 0; + bool first_out = true; + + // Print lines [lo, hi], stopping at -max-block-bytes with an explicit + // truncation marker rather than a mid-line cut. + auto print_lines = [&](const LineIndex &idx, uint32_t lo, uint32_t hi, + bool numbered, uint32_t mark) { + uint64_t used = 0; + for (uint32_t L = lo; L <= hi; ++L) { + std::string_view t = idx.line_text(L); + if (cli.max_block_bytes > 0 && + used + t.size() > cli.max_block_bytes) { + std::printf("\xE2\x80\xA6 (+%u more lines, -max-block-bytes " + "reached)\n", + hi - L + 1); + return; + } + used += t.size(); + if (numbered) + std::printf("%s%u: ", L == mark ? "> " : " ", L); + std::fwrite(t.data(), 1, t.size(), stdout); + if (t.empty() || t.back() != '\n') std::printf("\n"); + } + }; + + for (const auto &raw : cli.positional) { + Ref ref; + std::string perr; + if (!parse_ref(raw, ref, perr)) { + std::fprintf(stderr, "hprscript: expand: %s\n", perr.c_str()); + return 2; + } + MappedFile mf; + if (!mf.open(ref.path)) { + std::fprintf(stderr, "hprscript: cannot read %s: %s\n", + ref.path.c_str(), std::strerror(errno)); + exit_code = std::max(exit_code, 2); + continue; + } + std::string_view buf = mf.view(); + if (looks_binary(buf)) { + std::fprintf(stderr, "hprscript: expand: %s is binary\n", + ref.path.c_str()); + exit_code = std::max(exit_code, 2); + continue; + } + LineIndex idx; + idx.build(buf); + + if (!first_out) std::printf("\n"); + first_out = false; + + // Verify / recover the ref's line. Stale refs are reported in-band — + // for the reader, "the thing you saw is gone" is a result, not a + // diagnostic. + uint32_t line = ref.line; + std::string moved_note; + const bool line_exists = line <= idx.line_count(); + if (!ref.hash.empty()) { + if (!line_exists || ref_hash6(idx.line_text(line)) != ref.hash) { + uint32_t found = 0; + int count = 0; + for (uint32_t L = 1; L <= idx.line_count() && count < 2; ++L) + if (ref_hash6(idx.line_text(L)) == ref.hash) { + found = L; + ++count; + } + if (count == 1) { + moved_note = " (ref line moved: " + + std::to_string(ref.line) + " \xE2\x86\x92 " + + std::to_string(found) + ")"; + line = found; + } else { + std::printf("%s: stale \xE2\x80\x94 line %u changed and " + "%s\n", + raw.c_str(), ref.line, + count == 0 + ? "no line with this hash exists" + : "multiple lines share this hash"); + exit_code = std::max(exit_code, 3); + continue; + } + } + } else if (!line_exists) { + std::printf("%s: stale \xE2\x80\x94 file has only %u lines\n", + raw.c_str(), idx.line_count()); + exit_code = std::max(exit_code, 3); + continue; + } + + ScopeIndex scope; + bool have_scope = false; + ScopeConfig sc = resolve_scope_for_file(eff_lang, custom, ref.path); + if (!sc.anchor_regex.empty()) { + std::string serr; + if (scope.build(buf, sc, idx, &serr)) have_scope = true; + } + const ScopeRange *sr = nullptr; + if (have_scope) { + sr = scope.anchor_on_line(line); + if (!sr) { + std::string_view t = idx.line_text(line); + if (t.data() != nullptr) + sr = scope.find_innermost( + static_cast(t.data() - buf.data())); + } + } + + if (sr) { + std::printf("%s:%u-%u %s %s%s\n", ref.path.c_str(), + sr->line_start, sr->line_end, sr->kind.c_str(), + sr->name.c_str(), moved_note.c_str()); + print_lines(idx, sr->line_start, sr->line_end, + /*numbered=*/false, 0); + } else { + // No enclosing scope: a numbered context window with the ref + // line marked, since there's no signature to anchor the eye. + uint32_t before = static_cast(cli.context_before); + uint32_t after = static_cast(cli.context_after); + if (before == 0 && after == 0) before = after = 5; + uint32_t lo = line > before ? line - before : 1; + uint32_t hi = std::min(line + after, idx.line_count()); + std::printf("%s:%u (no enclosing scope)%s\n", ref.path.c_str(), + line, moved_note.c_str()); + print_lines(idx, lo, hi, /*numbered=*/true, line); + } + } + return exit_code; +} + +} // namespace hpr diff --git a/src/expand.hpp b/src/expand.hpp new file mode 100644 index 0000000..a33b37e --- /dev/null +++ b/src/expand.hpp @@ -0,0 +1,29 @@ +// `hprscript expand ` — the drill-down half of the +// search → expand loop. +// +// Search output already names every hit as file:line; expand turns that +// pointer into the full enclosing scope without the caller reconstructing a +// pattern and re-scanning. The optional `@hash` (printed by search's -refs +// flag) is a content check on the ref's line: if the file changed +// underneath, expand either recovers the line at its new position (unique +// hash match elsewhere → "ref line moved") or reports the ref as stale +// instead of silently expanding the wrong code. +#pragma once + +#include +#include + +namespace hpr { + +struct Cli; + +// 6-hex-char content hash of a line's text, whitespace-trimmed so pure +// re-indentation doesn't invalidate refs. The single definition of ref +// identity — search's -refs output and expand's verification both use it. +std::string ref_hash6(std::string_view line_text); + +// Exit codes: 0 = every ref expanded; 3 = at least one ref was stale +// (reported in-band on stdout); 2 = usage/read errors (stderr). +int run_expand(const Cli &cli); + +} // namespace hpr diff --git a/src/main.cpp b/src/main.cpp index ccb3241..af16270 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -11,6 +11,7 @@ #include "cli.hpp" #include "edit.hpp" #include "edit_plan.hpp" +#include "expand.hpp" #include "investigate.hpp" #include "query.hpp" #include "runner.hpp" @@ -63,6 +64,10 @@ int main(int argc, char **argv) { return hpr::run_investigate(cli); } + if (cli.expand.active) { + return hpr::run_expand(cli); + } + // Edit subcommand (`hprscript edit …`) — the only mode that can write. if (cli.edit.active) { return hpr::run_edit(cli); diff --git a/src/output.cpp b/src/output.cpp index 8b80ce9..571ea8c 100644 --- a/src/output.cpp +++ b/src/output.cpp @@ -1,6 +1,7 @@ #include "output.hpp" #include "block.hpp" +#include "expand.hpp" // ref_hash6 (-refs) #include "roles.hpp" #include "seen.hpp" @@ -551,6 +552,10 @@ void Formatter::emit_llm(const std::string &file, const Pattern &pattern, s.clear(); s += " "; append_uint32(s, line); + if (opts_.refs) { + s += '@'; + s += ref_hash6(idx.line_text(line)); + } s += ": "; if (opts_.pattern_count > 1 && !pattern.id.empty()) { s += '['; @@ -863,6 +868,10 @@ void Formatter::on_file_rollup(const std::string &file, if (!text.empty() && text.back() == '\n') text.remove_suffix(1); s += " "; append_uint32(s, line); + if (opts_.refs) { + s += '@'; + s += ref_hash6(idx.line_text(line)); + } s += ": "; s.append(text.data(), text.size()); s += '\n'; diff --git a/src/output.hpp b/src/output.hpp index 219dbed..d4cb482 100644 --- a/src/output.hpp +++ b/src/output.hpp @@ -90,6 +90,10 @@ struct OutputOptions { size_t pattern_count = 1; int64_t global_limit = -1; + // -refs: append @hash (see ref_hash6 in src/expand.hpp) to line numbers + // in Llm/Rollup output, making each hit an expand-verifiable ref. + bool refs = false; + // LLM-facing query header (-llm/-elide only): pre-rendered lines, no // trailing newlines, printed once before the first output. The runner // fills this when any pattern carries a -desc/description; empty = no diff --git a/src/runner.cpp b/src/runner.cpp index 17dd3a5..d9053cd 100644 --- a/src/runner.cpp +++ b/src/runner.cpp @@ -275,6 +275,7 @@ int run_search(const Cli &cli) { oo.pattern_count = patterns.size(); oo.global_limit = cli.limit; oo.patterns = &patterns; // outlives the Formatter (borrowed) + oo.refs = cli.refs; // The three LLM-facing modes share the query header and the trailing // co-occurrence / no-matches footers. diff --git a/tests/run.sh b/tests/run.sh index b89af2e..146c92a 100755 --- a/tests/run.sh +++ b/tests/run.sh @@ -1396,6 +1396,86 @@ OUT=$("$BIN" -p 'work\(\)' -name w -p 'other\(\)' -name o -rollup "$RU_FIX/m.go" expect_contains "rollup gets the co-occurrence footer" "--- files: w 1, o 1; both: 1" "$OUT" rm -rf "$RU_FIX" +# --------------------------------------------------------------------------- +section "stable refs (-refs / hprscript expand)" +EX_FIX="$HERE/_tmp_expand" +mkdir -p "$EX_FIX" +cat > "$EX_FIX/m.go" <<'EOF' +package main + +func Alpha() { + work() + work() + other() +} + +func Beta() { + work() +} +EOF + +# fnv1a is a pure function, so ref hashes are deterministic and can be +# asserted literally: trimmed "work()" → 1a87e3, "other()" → 737a8a. +OUT=$("$BIN" -p 'work\(\)' -llm -refs "$EX_FIX/m.go") +expect_contains "-refs appends @hash in -llm" "4@1a87e3: work()" "$OUT" +OUT=$("$BIN" -p 'other\(\)' -rollup -refs "$EX_FIX/m.go") +expect_contains "-refs on rollup representative" "6@737a8a: other()" "$OUT" +OUT=$("$BIN" -p 'work\(\)' -llm "$EX_FIX/m.go") +expect_not_contains "no @hash without -refs" "@" "$OUT" + +OUT=$("$BIN" expand "$EX_FIX/m.go:4") +expect_contains "expand: bare file:line resolves scope" "$EX_FIX/m.go:3-7 func Alpha" "$OUT" +expect_contains "expand: prints full body" " other()" "$OUT" +OUT=$("$BIN" expand "$EX_FIX/m.go:4@1a87e3" ; echo "rc=$?") +expect_contains "expand: verified ref ok" "func Alpha" "$OUT" +expect_contains "expand: verified ref exit 0" "rc=0" "$OUT" + +OUT=$("$BIN" expand "$EX_FIX/m.go:4" "$EX_FIX/m.go:10") +expect_contains "expand: batch second ref" "func Beta" "$OUT" +OUT=$("$BIN" expand "$EX_FIX/m.go:1") +expect_contains "expand: scopeless line falls back to context" "(no enclosing scope)" "$OUT" +expect_contains "expand: ref line is marked" "> 1: package main" "$OUT" + +OUT=$("$BIN" expand "$EX_FIX/m.go:4@ffffff" ; echo "rc=$?") +expect_contains "expand: wrong hash reports stale" "stale" "$OUT" +expect_contains "expand: stale exit 3" "rc=3" "$OUT" +OUT=$("$BIN" expand "$EX_FIX/m.go:999" ; echo "rc=$?") +expect_contains "expand: line beyond EOF is stale" "stale" "$OUT" +expect_contains "expand: beyond-EOF exit 3" "rc=3" "$OUT" + +# Content moved: insert a comment above Alpha, then expand with the OLD +# line number + hash — recovery must find the line again by content. +cat > "$EX_FIX/moved.go" <<'EOF' +package main + +// new comment +func Alpha() { + work() + work() + other() +} +EOF +OUT=$("$BIN" expand "$EX_FIX/moved.go:6@737a8a" ; echo "rc=$?") +expect_contains "expand: moved line recovered by hash" "(ref line moved: 6 → 7)" "$OUT" +expect_contains "expand: moved ref still expands scope" "func Alpha" "$OUT" +expect_contains "expand: moved ref exit 0" "rc=0" "$OUT" + +OUT=$("$BIN" expand "not-a-ref" 2>&1) ; RC=$? +expect_contains "expand: bad ref syntax rejected" "bad ref" "$OUT" +[[ "$RC" == "2" ]] && report ok "expand bad ref exit 2" || report fail "expand bad ref exit (got $RC)" +OUT=$("$BIN" expand 2>&1) ; RC=$? +expect_contains "expand: refs required" "at least one" "$OUT" +[[ "$RC" == "2" ]] && report ok "expand no refs exit 2" || report fail "expand no refs exit (got $RC)" +OUT=$("$BIN" expand -p x "$EX_FIX/m.go:4" 2>&1) ; RC=$? +expect_contains "expand rejects patterns" "not patterns" "$OUT" +[[ "$RC" == "2" ]] && report ok "expand -p exit 2" || report fail "expand -p exit (got $RC)" +OUT=$("$BIN" expand -llm "$EX_FIX/m.go:4" 2>&1) ; RC=$? +expect_contains "expand rejects output modes" "do not apply" "$OUT" + +OUT=$("$BIN" expand -max-block-bytes 20 "$EX_FIX/m.go:4") +expect_contains "expand honors -max-block-bytes" "-max-block-bytes reached" "$OUT" +rm -rf "$EX_FIX" + # --------------------------------------------------------------------------- section "fixed strings (-F / -Fi / -patterns-from)" FS_FIX="$HERE/_tmp_fixed"