From a1d942b71d0c6a58dfd8c40a9ffeacfb2917990a Mon Sep 17 00:00:00 2001 From: squid-protocol Date: Mon, 31 Aug 2026 15:46:46 -0400 Subject: [PATCH 1/2] docs: the repo-constellation map (docs/ecosystem.md) + sync all three agent files Work here routinely spans 4-5 intermingled repos (engine, language-crucible, keyword-rosetta, raw-output, telemetry/analyses) plus local-only directories (the data/ source pool; stale v1-v5/temp/threat_hunter copies of engine files), and nothing mapped it: which repo does what, where each skill lives, how the cross-repo workflows sequence, or that a PR should name its companions. docs/ecosystem.md is now the canonical agent-neutral map (repo table, skills inventory, workflow merge-order table incl. the rosetta draft-until-engine-merges choreography, cross-repo PR body convention, stale-copy warning), and CLAUDE.md / ANTIGRAVITY.md / AGENTS.md each gain a thin synced section pointing at it -- ANTIGRAVITY.md also gains the skills discovery pointer (.agents/skills symlink) it never had, so Antigravity sees the same skill inventory Claude does. Companion PRs add the same thin agent files to keyword-rosetta and language-crucible. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PUfXMUECX4Vq1vqh9mMQqt --- AGENTS.md | 2 ++ ANTIGRAVITY.md | 18 +++++++++++ CLAUDE.md | 13 ++++++++ docs/ecosystem.md | 82 +++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 115 insertions(+) create mode 100644 docs/ecosystem.md diff --git a/AGENTS.md b/AGENTS.md index c689f52bd..8f6c2153c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,3 +20,5 @@ Whenever you file issues or create pull requests, you **MUST** adhere to the fol 1. **Detailed PR Bodies**: When using `gh pr create`, always provide a detailed, well-formatted PR body describing the problem, the root cause, and the fix. Do not leave the body brief or empty. 2. **Auto-closing Issues**: Always include issue-closing keywords (e.g., `Resolves #123`, `Fixes #456`) in the **initial** PR body during creation. Do not rely on `gh pr edit` to add them later, as the PR may auto-merge before you do so, leaving the issues open. 3. **Issue Labels**: When using `gh issue create`, always apply appropriate labels using the `--label` flag (e.g., `--label "bug"`, `--label "upstream"`). + +2. **Cross-Repo PRs**: This repo is the hub of a multi-repo constellation (`docs/ecosystem.md` is the canonical map — repos, skills, workflows, merge ordering). Any PR participating in a cross-repo workflow MUST include a "Cross-repo" section in its body naming the companion PR/issue in the other repo(s), which side merges first and why, and what must be re-run after the other side lands. Example: squid-protocol/gitgalaxy#2611 ↔ squid-protocol/keyword-rosetta#4. diff --git a/ANTIGRAVITY.md b/ANTIGRAVITY.md index 504399750..8adf9f4fb 100644 --- a/ANTIGRAVITY.md +++ b/ANTIGRAVITY.md @@ -125,6 +125,23 @@ To ensure rigorous, adversarial testing, structure your work into a strict **5-s *(Tip: You can use the `/teamwork-preview` slash command to help automate and visualize complex multi-agent teams for large projects).* +## 8b. The Repo Constellation & Skills + +- GitGalaxy is the hub of several sibling repos (language-crucible, keyword-rosetta, + gitgalaxy-raw-output, squid-telemetry, gitgalaxy-population-analyses) and local-only + directories (the `gitgalaxy/data/` source pool; stale `v1`–`v5`/`temp/`/`threat_hunter/` + copies — **only `gitgalaxy/v6` is the live engine checkout**, verify paths before trusting a + grep hit). **`docs/ecosystem.md` is the canonical, agent-neutral map** — read it before any + cross-repo work (crucible pin bumps, keyword-rosetta sweeps). Note especially: + keyword-rosetta's CI checks out gitgalaxy **main**, so a rosetta corpus PR depending on new + engine rules stays **draft** until the engine PR merges. +- **Skills** (step-by-step workflow docs any agent can follow) live at `.agents/skills/` + (a symlink to `.claude/skills/`) in this repo, and the same layout in keyword-rosetta + (`rosetta-language-sweep`) and language-crucible (`expand-language-coverage`). Before + re-deriving a workflow (extraction hardening, tri-comparison sweeps, language status docs, + CI push checklist, release notes), check the relevant repo's skills directory first — + `docs/ecosystem.md` has the full inventory. + ## 9. Submitting Pull Requests When working in this repository, **you MUST ALWAYS work on a side branch and submit a PR to `main`. NEVER merge or push your changes directly to `main` without a PR.** This strict workflow ensures that tests and multi-agent pipelines are run in isolation. @@ -141,6 +158,7 @@ When generating or submitting a Pull Request for this repository, it is critical - **Metrics & Limitations:** You must explicitly list: (1) How many adversarial tests were created, (2) How many errors/failures were initially found by these tests, and (3) Any known regex limitations or edge-cases that remain for this language. - **Do not leave the PR body blank, sparse, or lame.** A poor description will cause the PR to be rejected. - **Add relevant labels:** Ensure the PR has descriptive labels attached so it integrates correctly into the project's tracking and CI processes. +- **Cross-repo note:** If the PR participates in a cross-repo workflow (see `docs/ecosystem.md`), the body MUST name the companion PR/issue in the other repo, which side merges first and why, and what must re-run after the other side lands. ## 10. Scratch Files & Working Directory diff --git a/CLAUDE.md b/CLAUDE.md index 4d644e2bb..68d6f25ab 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,6 +9,19 @@ repository, extracts **Structural Signatures** via bounded, ReDoS-proof regexes toolchain, no ASTs), and builds a mathematical knowledge graph used for risk scoring, SBOM generation, dependency mapping, and 3D visualization. CLI entry points: `galaxyscope` / `blast`. +## The repo constellation (read before cross-repo work) + +This engine is the hub of several sibling repos — the language-crucible benchmark corpus +(pinned by CI), the keyword-rosetta cross-language control corpus (whose CI checks out THIS +repo's main), gitgalaxy-raw-output (scan evidence), squid-telemetry, and population-analyses — +plus local-only directories (the `gitgalaxy/data/` source pool; the stale `v1`–`v5`/`temp/`/ +`threat_hunter/` copies — only `v6` is live). **`docs/ecosystem.md` is the canonical map**: per- +repo purpose, where every skill lives, the cross-repo workflows and their merge order, and the +PR convention — any PR in a cross-repo workflow must carry a "Cross-repo" note naming its +companion PRs, which side merges first, and what re-runs after. Read it before a crucible pin +bump, a rosetta sweep, or any change whose companion lives in another repo, and keep it (not +this file) updated when the constellation changes. + ## Commands ```bash diff --git a/docs/ecosystem.md b/docs/ecosystem.md new file mode 100644 index 000000000..bae7f370f --- /dev/null +++ b/docs/ecosystem.md @@ -0,0 +1,82 @@ +# The GitGalaxy repo constellation + +GitGalaxy work routinely spans four or five intermingled repositories, and most of the recurring +workflows (a crucible pin bump, a rosetta sweep, a language addition) have steps in more than one +of them. This doc is the canonical, **agent-neutral** map: what each repo is for, where it lives, +what skills and agent guidance it carries, and how the cross-repo workflows sequence. It is +written for any coding agent (Claude, Antigravity/Gemini, or the next one) and for humans; the +per-agent files (`CLAUDE.md`, `ANTIGRAVITY.md`, `AGENTS.md`) point here rather than each carrying +their own drifting copy. + +## The repos + +Canonical local layout on the dev machine — all siblings under `/srv/storage_16tb/projects/`: + +| Repo (GitHub, org `squid-protocol`) | Local checkout | What it is | +|---|---|---| +| **gitgalaxy** | `gitgalaxy/v6` | **The engine** (this repo). AST-free, LLM-free static analysis: bounded-regex structural signatures → knowledge graph → risk scoring / SBOM / 3D map. Everything else in this table exists to feed, verify, or showcase it. | +| **language-crucible** | `all_language_repo` | Zero-execution structural-parser **benchmark corpus** (`data///`, per-category `SOURCES.md`, machine-readable `PROVENANCE.json`). gitgalaxy CI pins it to a release tag (`LANGUAGE_CRUCIBLE_REF` GH Actions var + `tests/_crucible_pin.py`) and diffs golden masters against it. Releases per its `RELEASING.md`. | +| **keyword-rosetta** | `keyword-rosetta` | **Control corpus**: one identical 12-probe program shell in all 46 signature-bearing languages, exact planted keyword counts — measures whether the engine treats identical intent identically across languages (cross-language bias). Gates via `tools/verify_language.py`; deviations live in `deviation_ledger.json` per its `docs/GATING.md`. Its CI checks out gitgalaxy **main** (see choreography below). | +| **gitgalaxy-raw-output** | `gitgalaxy-raw-output` | Real, unedited **scan outputs** on independently-chosen production repos (`v//_galaxy_llm.md` + gzipped audit/SBOM) plus speed charts. Evidence source for README claims and `docs/language_status/` §8 sections. | +| **squid-telemetry** | `squid-telemetry` | Automated **data warehouse / visualization pipeline** (the engine itself is air-gapped and phones nothing home; this repo aggregates published artifacts). Hosts the live WebGL architecture map. | +| **gitgalaxy-population-analyses** | `gitgalaxy-population-analyses` | Offline **statistical analyses** over scan populations (risk-distribution ridgeplots, archetype clustering, threat-prediction distribution studies). Consumes scan DBs; never on any CI path. | + +**Local-only directories that are NOT repos** (but matter): + +- `gitgalaxy/data/` — the full-repo **source pool** (~113 clones, `corpus_/` collections, + npm/pypi mirrors). Feeds the crucible; not pinned, not a git repo itself. Its `README.md` + documents the layout. `gitgalaxy/data/gitgalaxy/` is a clone of the engine kept in the pool so + the engine can scan itself — **never edit engine code there**. +- `gitgalaxy/v1` … `v5`, `museum*`, `temp/`, `threat_hunter/` — historical/scratch copies. + **Only `gitgalaxy/v6` is the live engine checkout.** A repo-wide grep from `/srv/.../projects` + or `/srv/.../gitgalaxy` will hit stale copies of files like `language_standards.py` in + `temp/`, `threat_hunter/`, and the pool's self-scan clone — check the path before trusting or + editing a hit. + +## Where the skills live + +Skills are markdown workflows (`SKILL.md`) usable by any agent that reads them; each repo carries +the skills that operate **on that repo**, under `.claude/skills/` with an `.agents/skills` symlink +to the same directory so non-Claude agents find them at a vendor-neutral path. + +| Repo | Skills | +|---|---| +| gitgalaxy | `harden-language-extraction`, `harden-strict-signatures`, `harden-class-start-extraction`, `tri-comparison-ledger-sweep`, `tree-sitter-accuracy-sweep`, `language-status`, `ci-push-checklist`, `self-scan-query`, `issue-generation`, `pipeline-check`, `readme-maintenance`, `release-notes` | +| keyword-rosetta | `rosetta-language-sweep` (work one language's cross-language-consistency tracking issue end to end) | +| language-crucible | `expand-language-coverage` (fill a `data//` category from the source pool) | + +gitgalaxy additionally has `.claude/rules/` (always-on constraints: planning approval, +golden-master hygiene, CI self-healing, sandbox/permission discipline, tri-comparison regen) — +`ANTIGRAVITY.md` and `AGENTS.md` restate the same constraints for other agents; if you change a +rule, sync all three. + +## Cross-repo workflows (and their merge order) + +| Workflow | Repos touched (in merge order) | Documented in | +|---|---|---| +| **Crucible corpus growth → release → pin bump** | language-crucible (data PRs, tag per `RELEASING.md`) → gitgalaxy (`docs/self_scan/BUMPING_THE_CRUCIBLE_PIN.md`: regen golden masters + tri-comparison + tree-sitter artifacts, bump `LANGUAGE_CRUCIBLE_REF` + `PINNED_TAG`) | crucible `RELEASING.md`; gitgalaxy `BUMPING_THE_CRUCIBLE_PIN.md` | +| **Rosetta sweep** (work one language's bias issue) | gitgalaxy engine PR first (its CI is self-contained) → keyword-rosetta corpus PR **stays draft until the engine PR merges**, because rosetta CI checks out gitgalaxy *main*; then `gh run rerun --failed` + `gh pr ready`. Capstone lands back in gitgalaxy `docs/language_status/.md` §10. | keyword-rosetta `rosetta-language-sweep` skill | +| **Adding a language to the engine** | gitgalaxy (`standards/how_to_add_a_language.md`, includes authoring the rosetta control folder) → keyword-rosetta (`SPEC.md` shell + manifest) → optionally language-crucible (`expand-language-coverage`) | those three docs | +| **Tri-comparison / accuracy verification** | gitgalaxy only (ledger, chart, `manual_verification.json`), but reads the pinned crucible corpus | gitgalaxy `docs/self_scan/tri_comparison_README.md` | +| **README / evidence claims** | gitgalaxy README cites gitgalaxy-raw-output artifacts and the keyword-rosetta chart (embedded from that repo's raw main URL — it self-updates when rosetta main moves) | gitgalaxy `readme-maintenance` skill | + +## PR convention for cross-repo work + +Any PR that participates in a cross-repo workflow **must carry a "Cross-repo" note in its body** +stating: (1) the companion PR/issue links in the other repo(s), (2) which side merges first and +why (e.g. "draft here until squid-protocol/gitgalaxy#NNNN merges — this repo's CI checks out +gitgalaxy main"), and (3) what must be re-run after the other side lands (a CI rerun, a +re-baseline, a pin bump). A reviewer — human or agent — landing on either PR alone must be able +to reconstruct the whole change without hunting. Worked example: gitgalaxy#2611 ↔ +keyword-rosetta#4 (the first rosetta jcl sweep). + +## Agent-guidance file conventions + +- **`AGENTS.md`** — vendor-neutral hard policies for the repo (any agent must follow). +- **`CLAUDE.md`** — Claude-specific guidance (loads automatically in Claude Code sessions). +- **`ANTIGRAVITY.md`** — Antigravity/Gemini-specific guidance, mirroring CLAUDE.md's constraints. +- **`.agents/skills` → `.claude/skills`** symlink — one skills directory, two discovery paths. +- Satellite repos keep their agent files **thin**: repo-specific gates plus a pointer to this doc + — the constellation map is maintained *here only* (`docs/ecosystem.md` in gitgalaxy), so it + cannot fork across repos. When the constellation changes (a new repo, a new cross-repo + workflow, a moved skill), update this file and the satellites' pointers in the same pass. From 0afea94f21a676daa041ad9f61be2077b1c1493e Mon Sep 17 00:00:00 2001 From: squid-protocol Date: Mon, 31 Aug 2026 16:01:40 -0400 Subject: [PATCH 2/2] docs: extend the constellation map with the full repo web + public assets; README evidence table gains the missing strands Full org scan surfaced strands the first map draft missed: cobol_to_java_examples (the legacy-modernization showcase -- 10 COBOL repos auto-translated to compiling Spring Boot), the squid-protocol profile README (the public front door), and the whole public-surface inventory -- the ~100-page gh-pages docs site (Museum of Code, cookbook, Legacy Bridge chapter, risk-equation methodology), gitgalaxy.io, PyPI, the demo video, and the three families of auto-regenerating chart assets (rosetta bias SVG, raw-output speed charts, telemetry adoption PNGs) with their raw embed URLs. ecosystem.md also codifies the satellite-README 'constellation section' convention (every repo links its neighbors with a you-are-here marker; the engine README stays governed by how_to_maintain_the_readme.md). README 'Explore the evidence' table: +4 rows (COBOL->Java examples, population analyses, Museum of Code, distribution telemetry+PyPI) -- evidence-shaped one-liners per the five rules, no new superlatives. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PUfXMUECX4Vq1vqh9mMQqt --- README.md | 4 ++++ docs/ecosystem.md | 29 +++++++++++++++++++++++++++-- 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index cd3bcaeea..444bcc533 100644 --- a/README.md +++ b/README.md @@ -370,6 +370,10 @@ guide](github-action-readme.md). | [Language Crucible](https://github.com/squid-protocol/language-crucible) | Cross-language benchmark and golden corpus | | [Keyword Rosetta](https://github.com/squid-protocol/keyword-rosetta) | 46-language planted control corpus and bias reports | | [Raw Output](https://github.com/squid-protocol/gitgalaxy-raw-output) | Unedited scans of real repositories | +| [COBOL → Java examples](https://github.com/squid-protocol/cobol_to_java_examples) | 10 COBOL repos auto-translated to compiling Spring Boot architectures (`mvn clean compile` works out of the box) | +| [Population analyses](https://github.com/squid-protocol/gitgalaxy-population-analyses) | Statistical analyses over the raw-output scan population: archetype clustering, risk distributions, threat-classifier studies | +| [Museum of Code](https://squid-protocol.github.io/gitgalaxy/museum-of-code/) | Full architectural teardowns of real codebases (Apollo 11, IBM CICS benchmarks) | +| [Distribution telemetry](https://github.com/squid-protocol/squid-telemetry) | Public fetch metrics across GitHub/GitLab/[PyPI](https://pypi.org/project/gitgalaxy/), regenerated daily | | [`tests/README.md`](tests/README.md) | Regression and golden-master methodology | | [`tri_comparison_ledger.json`](docs/self_scan/tri_comparison_ledger.json) | Disagreement-by-disagreement validation record | | [`manual_verification.json`](docs/self_scan/manual_verification.json) | Reviewed cases where comparator coverage is unavailable | diff --git a/docs/ecosystem.md b/docs/ecosystem.md index bae7f370f..eaab04623 100644 --- a/docs/ecosystem.md +++ b/docs/ecosystem.md @@ -18,8 +18,10 @@ Canonical local layout on the dev machine — all siblings under `/srv/storage_1 | **language-crucible** | `all_language_repo` | Zero-execution structural-parser **benchmark corpus** (`data///`, per-category `SOURCES.md`, machine-readable `PROVENANCE.json`). gitgalaxy CI pins it to a release tag (`LANGUAGE_CRUCIBLE_REF` GH Actions var + `tests/_crucible_pin.py`) and diffs golden masters against it. Releases per its `RELEASING.md`. | | **keyword-rosetta** | `keyword-rosetta` | **Control corpus**: one identical 12-probe program shell in all 46 signature-bearing languages, exact planted keyword counts — measures whether the engine treats identical intent identically across languages (cross-language bias). Gates via `tools/verify_language.py`; deviations live in `deviation_ledger.json` per its `docs/GATING.md`. Its CI checks out gitgalaxy **main** (see choreography below). | | **gitgalaxy-raw-output** | `gitgalaxy-raw-output` | Real, unedited **scan outputs** on independently-chosen production repos (`v//_galaxy_llm.md` + gzipped audit/SBOM) plus speed charts. Evidence source for README claims and `docs/language_status/` §8 sections. | -| **squid-telemetry** | `squid-telemetry` | Automated **data warehouse / visualization pipeline** (the engine itself is air-gapped and phones nothing home; this repo aggregates published artifacts). Hosts the live WebGL architecture map. | -| **gitgalaxy-population-analyses** | `gitgalaxy-population-analyses` | Offline **statistical analyses** over scan populations (risk-distribution ridgeplots, archetype clustering, threat-prediction distribution studies). Consumes scan DBs; never on any CI path. | +| **squid-telemetry** | `squid-telemetry` | **Distribution/adoption analytics** (the engine itself is air-gapped and phones nothing home; this pipeline scrapes public GitHub/GitLab/PyPI fetch metrics daily via Actions and commits regenerated chart PNGs). | +| **gitgalaxy-population-analyses** | `gitgalaxy-population-analyses` | Offline **statistical analyses** over scan populations (risk-distribution ridgeplots, archetype clustering, threat-prediction distribution studies). Reads raw inputs from gitgalaxy-raw-output; never on any CI path. | +| **cobol_to_java_examples** | *(not usually checked out locally)* | **Legacy-modernization showcase**: 10 COBOL repos auto-translated by the engine's `cobol_to_java` pipeline into compiling Spring Boot architectures (entities/controllers/services + `ai_agent_jobs/` tickets). Evidence for the docs site's Legacy Bridge chapter. | +| **squid-protocol** (profile repo) | *(not usually checked out locally)* | The org/user **profile README** — the public front door linking the flagship projects. Update it when a new constellation repo becomes showcase-worthy. | **Local-only directories that are NOT repos** (but matter): @@ -33,6 +35,29 @@ Canonical local layout on the dev machine — all siblings under `/srv/storage_1 `temp/`, `threat_hunter/`, and the pool's self-scan clone — check the path before trusting or editing a hit. +## Public-facing surfaces & reusable assets + +Things to link (never duplicate) when writing READMEs, docs, issues, or showcase material: + +- **The docs site** — https://squid-protocol.github.io/gitgalaxy/ (source: this repo's `gh-pages` + branch). ~100 pages: architecture chapters (pipeline 02-*, risk-equation methodology 08-*, + visual-encoding 07-*), the ten claims (03-*), security landscape (04-*), the **Legacy Bridge + chapter** (05-* — refraction controller, Spring Boot scaffolding, JCL forge/auditor, agent + tickets), plus the [Museum of Code](https://squid-protocol.github.io/gitgalaxy/museum-of-code/) + (full architectural teardowns), a cookbook, LLM-report examples, and the CLI reference. +- **The visualizer / product site** — https://gitgalaxy.io/ · **PyPI** — + https://pypi.org/project/gitgalaxy/ · **Demo video** — https://www.youtube.com/watch?v=XWWSd8LmoCM +- **Auto-regenerating chart assets** (embed by raw URL; they update themselves, so never + hand-copy the image): + - keyword-rosetta bias chart: `https://raw.githubusercontent.com/squid-protocol/keyword-rosetta/main/docs/bias_variance_chart.svg` (regen by `tools/bias_report.py`) + - raw-output speed charts: `https://raw.githubusercontent.com/squid-protocol/gitgalaxy-raw-output/main/speed_charts/latest/{loc_vs_time,rate_model}.png` (regen per scan batch) + - telemetry adoption charts: `https://raw.githubusercontent.com/squid-protocol/squid-telemetry/main/{cumulative_downloads,conversion_funnel,discovery_channels,feature_intent,release_correlation}.png` (regen daily by Actions) +- **README convention**: every constellation repo's README carries a short "GitGalaxy + constellation" section linking its neighbors (with a *you-are-here* marker) and the docs site — + so a reader landing anywhere can navigate the whole web. The engine README's own linking is + governed by `docs/how_to_maintain_the_readme.md` (evidence-shaped, five rules) — follow that + doc, not this section, when editing it. + ## Where the skills live Skills are markdown workflows (`SKILL.md`) usable by any agent that reads them; each repo carries