Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .claude/agents/class-start-scout.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
name: class-start-scout
description: Mechanical diff-reporting and verification-sequence execution for epic #1295 (extending gitgalaxy/core/detector.py's _CLASS_START_NAMED_EXTRACTION_LANGS allowlist). Runs tests/tools/class_start_diff.py and the tree_sitter_accuracy_audit.py/crucible_check.py/audit_check.py verification sequence, reports structured results -- it does not judge whether an extra/missing name is a real bug, decide how to fix a regex, or edit the allowlist itself. Use before a per-language triage pass (to get one language's exact extra/missing name diff without reading language_standards.py cold) and after a regex fix + allowlist flip (to run the full verification sequence and report pass/fail).
description: Mechanical diff-reporting and verification-sequence execution for epic #1295 (extending gitgalaxy/core/detector.py's _CLASS_START_NAMED_EXTRACTION_LANGS allowlist). Runs tests/tools/class_start_diff.py and the tree_sitter_accuracy_audit.py/crucible_check.py/audit_check.py verification sequence, reports structured results -- it does not judge whether an extra/missing name is a real bug, decide how to fix a regex, or edit the allowlist itself. Use before a per-language triage pass (to get one language's exact extra/missing name diff without reading language_standards/languages/<lang>.py cold) and after a regex fix + allowlist flip (to run the full verification sequence and report pass/fail).
tools: Bash, Read
model: haiku
---

You run scripts and report their output cleanly. You do not judge whether a `class_start` regex
is buggy, decide whether an extra/missing name is a real bug versus a ground-truth scope mismatch,
or edit `gitgalaxy/core/detector.py`/`gitgalaxy/standards/language_standards.py` yourself -- that
or edit `gitgalaxy/core/detector.py`/`gitgalaxy/standards/language_standards/` yourself -- that
judgment belongs to whoever asked for the scout pass (the main conversation or a case-authoring
subagent). If something looks ambiguous, report it plainly rather than guessing what it means.

Expand Down
2 changes: 1 addition & 1 deletion .claude/agents/strict-signature-scout.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: strict-signature-scout
description: Mechanical gap-reporting and candidate-verification for epic #1069 (hardening tests/extraction/languages/test_<lang>_strict.py). Runs tests/extraction/tools/audit_strict_coverage.py and tests/extraction/tools/verify_candidates.py and reports structured results -- it does not draft test cases or judge realism, only executes scripts and formats their output. Use before a case-writing pass (to get one language's exact gap list without loading language_standards.py cold) and after drafting candidate cases (to batch-verify them against the real compiled regex before they're written into a test file).
description: Mechanical gap-reporting and candidate-verification for epic #1069 (hardening tests/extraction/languages/test_<lang>_strict.py). Runs tests/extraction/tools/audit_strict_coverage.py and tests/extraction/tools/verify_candidates.py and reports structured results -- it does not draft test cases or judge realism, only executes scripts and formats their output. Use before a case-writing pass (to get one language's exact gap list without loading language_standards/languages/<lang>.py cold) and after drafting candidate cases (to batch-verify them against the real compiled regex before they're written into a test file).
tools: Bash, Read
model: haiku
---
Expand Down
7 changes: 4 additions & 3 deletions .claude/skills/harden-language-extraction/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: harden-language-extraction
description: Deepen or fix a language's structural-extraction accuracy in gitgalaxy/standards/language_standards.py (func_start/args/class_start/_dependency_capture regexes) using the epic #813 methodology. Use when the user asks to "harden extraction for X", "add strict tests for language X", "find bugs in X's parsing", or similar per-language regex-correctness work -- not for adding a brand-new language from scratch (that's how_to_add_a_language.md's LLM generation prompt) and not for the broader ReDoS/boundary-correctness rules covered there either.
description: Deepen or fix a language's structural-extraction accuracy in gitgalaxy/standards/language_standards/languages/<lang>.py (func_start/args/class_start/_dependency_capture regexes) using the epic #813 methodology. Use when the user asks to "harden extraction for X", "add strict tests for language X", "find bugs in X's parsing", or similar per-language regex-correctness work -- not for adding a brand-new language from scratch (that's how_to_add_a_language.md's LLM generation prompt) and not for the broader ReDoS/boundary-correctness rules covered there either.
---

Source of truth is `tests/extraction/how_to_harden_extraction.md` -- read it directly, don't work
Expand All @@ -12,7 +12,8 @@ a language-specific quirk.

## Process (translating that doc's 5-stage pipeline to this tool's Agent tool)

1. **Load the target.** Read the language's full `rules` dict from `language_standards.py` and its
1. **Load the target.** Read the language's full `rules` dict from
`gitgalaxy/standards/language_standards/languages/<lang>.py` and its
existing cases in `tests/extraction/languages/test_<lang>.py` (or the old monolithic
`test_*_extraction_strict.py` files if it hasn't migrated yet). Work one language, all four
gauntlets, in one sitting -- not gauntlet-by-gauntlet across languages.
Expand All @@ -30,7 +31,7 @@ a language-specific quirk.
unrealistic payload), not an automatic fix.
5. **Fix real bugs with the full discipline**: ReDoS scaling check on any quantifier change,
`python tests/tools/audit_check.py` (add `--regenerate` for pure line-shifts), then
`pytest tests/extraction/languages/test_<lang>.py`. If the fix touches `language_standards.py`,
`pytest tests/extraction/languages/test_<lang>.py`. If the fix touches `language_standards/`,
`detector.py`, or `prism.py`: `python tests/tools/crucible_check.py` -- confirm any real diff is
confined to the language(s) actually changed (check `_dependency_capture` fixes for legitimate
cross-language DAG ripple, per the doc's note) before `--update --yes`.
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/tree-sitter-accuracy-audit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ on:
paths:
- "gitgalaxy/core/detector.py"
- "gitgalaxy/core/prism.py"
- "gitgalaxy/standards/language_standards.py"
- "gitgalaxy/standards/language_standards/**"
- "tests/tools/tree_sitter_accuracy_audit.py"
- "tests/tree_sitter_accuracy_baseline_*.json"
- ".github/workflows/tree-sitter-accuracy-audit.yml"
Expand Down Expand Up @@ -63,8 +63,8 @@ jobs:
run: |
cd gitgalaxy
python tests/tools/tree_sitter_accuracy_audit.py --summary-table
if ! git diff --exit-code -- gitgalaxy/standards/language_standards.py; then
echo "::error::gitgalaxy/standards/language_standards.py's summary table is stale."
if ! git diff --exit-code -- gitgalaxy/standards/language_standards/__init__.py; then
echo "::error::gitgalaxy/standards/language_standards/__init__.py's summary table is stale."
echo "Run 'python tests/tools/tree_sitter_accuracy_audit.py --summary-table' and commit the result."
exit 1
fi
12 changes: 6 additions & 6 deletions .github/workflows/tree-sitter-accuracy-history.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ name: Tree-sitter Accuracy History
# baselined language again and append one row each to
# docs/self_scan/tree_sitter_accuracy_history.csv (an accumulating time series, unlike the
# gitignored self-scan DB -- this is meant to be graphed later), regenerate the Markdown
# summary table embedded in language_standards.py's docstring so it can't drift from the
# summary table embedded in language_standards/__init__.py's docstring so it can't drift from the
# committed baselines between now and the next manual --summary-table run, and re-render
# docs/self_scan/tree_sitter_accuracy_chart.svg (a small-multiples bar chart of the most
# recent history batch -- always the one this same job just appended). Opens an auto-merged
Expand All @@ -19,7 +19,7 @@ name: Tree-sitter Accuracy History
# protection), and why the loop guard below is load-bearing.
#
# Adaptive by design, not just by trigger path: the trigger below fires on any push touching
# detector.py/prism.py/language_standards.py, but not every such push actually moves a
# detector.py/prism.py/language_standards/, but not every such push actually moves a
# language's measured accuracy. --history itself detects that (see
# tree_sitter_accuracy_audit.py's `_batch_matches_measured`) and skips appending a row when the
# fresh measurement is identical to the last recorded batch -- which means --chart re-renders
Expand All @@ -34,7 +34,7 @@ on:
paths:
- "gitgalaxy/core/detector.py"
- "gitgalaxy/core/prism.py"
- "gitgalaxy/standards/language_standards.py"
- "gitgalaxy/standards/language_standards/**"
- "tests/tools/tree_sitter_accuracy_audit.py"
- "tests/tree_sitter_accuracy_baseline_*.json"

Expand All @@ -44,7 +44,7 @@ permissions:
jobs:
tree-sitter-accuracy-history:
# Guards against the same infinite-loop shape documented in gitgalaxy.yml's
# full-report job: this workflow's own bot commit touches language_standards.py and
# full-report job: this workflow's own bot commit touches language_standards/__init__.py and
# the baseline-adjacent history CSV, which are both in the trigger's `paths` filter,
# so without this it would re-trigger itself on every merge of its own PR.
if: |
Expand Down Expand Up @@ -114,7 +114,7 @@ jobs:
${{ steps.blurbs.outputs.blurbs }}

- Appends one row per language to `docs/self_scan/tree_sitter_accuracy_history.csv`.
- Regenerates the summary table in `gitgalaxy/standards/language_standards.py`.
- Regenerates the summary table in `gitgalaxy/standards/language_standards/__init__.py`.
- Re-renders `docs/self_scan/tree_sitter_accuracy_chart.svg` from that same batch.

Never touches `tests/tree_sitter_accuracy_baseline_*.json` -- those stay
Expand All @@ -124,7 +124,7 @@ jobs:
add-paths: |
docs/self_scan/tree_sitter_accuracy_history.csv
docs/self_scan/tree_sitter_accuracy_chart.svg
gitgalaxy/standards/language_standards.py
gitgalaxy/standards/language_standards/__init__.py

# Same rationale as gitgalaxy.yml's "Merge the LLM Brief PR" step -- nothing else
# gates this PR, so without an explicit merge it just sits mergeable-but-unmerged.
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/tri-comparison-audit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ on:
paths:
- "gitgalaxy/core/detector.py"
- "gitgalaxy/core/prism.py"
- "gitgalaxy/standards/language_standards.py"
- "gitgalaxy/standards/language_standards/**"
- "tests/tools/tri_comparison_chart.py"
- "tests/tools/tri_comparison_gatherer.py"
- "tests/tools/tri_comparison_reconcile.py"
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/tri-comparison-history.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ name: Tri-Comparison History
# whatever actually changed -- same mechanism gitgalaxy.yml uses for the LLM architecture brief.
#
# Adaptive by design, not just by trigger path: the trigger below fires on any push touching
# detector.py/prism.py/language_standards.py, but not every such push moves a language's measured
# detector.py/prism.py/language_standards/, but not every such push moves a language's measured
# numbers or an existing ledger entry's counts. Unlike the tree-sitter history CSV (an
# append-only file that needs its own _batch_matches_measured dedup check to stay a no-op), the
# three files here are overwritten in place -- "no diff" already IS the adaptivity, so
Expand All @@ -29,7 +29,7 @@ on:
paths:
- "gitgalaxy/core/detector.py"
- "gitgalaxy/core/prism.py"
- "gitgalaxy/standards/language_standards.py"
- "gitgalaxy/standards/language_standards/**"
- "tests/tools/tri_comparison_chart.py"
- "tests/tools/tri_comparison_gatherer.py"
- "tests/tools/tri_comparison_reconcile.py"
Expand Down
2 changes: 1 addition & 1 deletion gitgalaxy/standards/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Each file in this directory serves a distinct calibration purpose for the downst

* **`gitgalaxy_config.py` (Global Ingestion Firewall):** Defines Zero-Trust ingestion boundaries. It houses the Supply Chain Firewall configurations (approved vs. blacklisted imports), global file denylists, X-Ray binary scanner bypasses, and physical file-size clamps.
* **`language_lens.py` (Identity Classifier):** The Bayesian engine that assigns definitive "Identity Locks" to files. It defines the multi-tiered confidence hierarchy, resolving extension collisions by weighing exact filename matches against Contextual Baselines.
* **`language_standards.py` (Structural Signature Registry):** The massive, highly optimized structural mapping registry for 50+ languages. It defines exactly how to slice a language into Branch Logic, State Flux, High-Risk Execution, and Object Declarations using ReDoS-proof regular expressions.
* **`language_standards/` (Structural Signature Registry):** A package with one file per language under `languages/` (59 today), assembled by `__init__.py` into the `LANGUAGE_DEFINITIONS` registry every downstream engine imports. Each per-language file defines exactly how to slice that language into Branch Logic, State Flux, High-Risk Execution, and Object Declarations using ReDoS-proof regular expressions; cross-language shared pieces (`LENS_CONFIG`, `PRISM_CONFIG`, the `GLOBAL_*` debt/AI-SDK detectors, `PROJECT_OVERRIDES`) live in their own sibling modules. See `how_to_add_a_language.md` for the registration mechanics.
* **`analysis_lens.py` (Mathematical Constants & Threat Policies):** The repository of Threat Policies, Sigmoid Curve tuning, and K-Means clustering medians. It dictates how raw structural signals are mathematically converted into normalized 0-100% risk exposure vectors.
* **`how_to_add_a_language.md` (Extension Protocol):** Contains the strict prompt engineering protocols required to generate ReDoS-proof language dictionaries using advanced LLMs, bypassing the need for manual parser development.

Expand Down
2 changes: 1 addition & 1 deletion gitgalaxy/standards/gitgalaxy_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@
"yarn.lock",
"composer.lock",
"gitgalaxy/core/aperture.py",
"gitgalaxy/standards/language_standards.py",
"gitgalaxy/standards/language_standards/",
"gitgalaxy/security/security_lens.py",
"gitgalaxy/tools/network_auditing/full_api_network_map.py",
"gitgalaxy/tools/cobol_to_cobol/cobol_schema_forge.py",
Expand Down
26 changes: 21 additions & 5 deletions gitgalaxy/standards/how_to_add_a_language.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,32 @@ To add a new language to the Language Classifier, you will use an advanced LLM (
---

### Step 1: Initialize the LLM Context
Before asking the LLM to generate the new language signatures, upload the `gitgalaxy/standards/language_standards.py` file to the chat window. Issue this exact command:
> *"Read this file to understand how the GitGalaxy Structural Signature Analysis Engine uses bounded regex to guarantee ReDoS immunity. Pay close attention to how C++ and Python are mapped to prevent Catastrophic Backtracking."*
`gitgalaxy/standards/language_standards/` is a package with one file per language (each a
`DEFINITION: dict[str, Any] = {...}` literal, assembled into the `LANGUAGE_DEFINITIONS` registry
by `__init__.py`), not a single monolithic file. Before asking the LLM to generate new language
signatures, upload `gitgalaxy/standards/language_standards/__init__.py` plus one or two
representative per-language files (e.g. `languages/python.py` and `languages/cpp.py`) to the chat
window. Issue this exact command:
> *"Read these files to understand how the GitGalaxy Structural Signature Analysis Engine uses bounded regex to guarantee ReDoS immunity, and how each language's rules dict is registered in `__init__.py`. Pay close attention to how C++ and Python are mapped to prevent Catastrophic Backtracking."*

### Step 2: Inject the Structural Signature Prompt
Copy the **Generation Prompt** below and paste it into the LLM. Replace `[TARGET LANGUAGE]` with the exact language you want to map.

### Step 3: Register the Signatures
1. Open `gitgalaxy/standards/language_standards.py`.
2. Locate the `LANGUAGE_DEFINITIONS` registry.
3. Paste the generated Python dictionary directly into the registry to instantly grant the engine native support for the new language architecture.
1. Create `gitgalaxy/standards/language_standards/languages/<lang>.py` containing
`DEFINITION: dict[str, Any] = {...}` with the generated dictionary as its value (see any
existing file in that directory for the exact shape — license header, `import re`, and a
`from .._shared_patterns import ...` line only if the rules reference one of the shared
`GLOBAL_*` cross-language constants).
2. In `gitgalaxy/standards/language_standards/__init__.py`, add an
`from .languages import <lang> as _<lang>` import (isort-sorted alphabetically among the
others) and a `"<lang>": _<lang>.DEFINITION,` line inside the `LANGUAGE_DEFINITIONS` dict, in
the same position other entries follow (insertion order is preserved end-to-end, so add new
languages at the end unless you have a specific reason not to).
3. If the language's key isn't a valid Python identifier (spaces, hyphens, etc. — the only
existing case is `"objective-c"`, whose module is `languages/objectivec.py`), the file's own
module name still needs to be identifier-safe even though the dict key itself doesn't; pick a
concatenated or underscored slug and keep the dict key exactly as the LLM/consumers expect it.

<br><br>

Expand Down
Loading
Loading