feat: RepoDiagnoser — clone & analyse any public Git repo via CLI or Python API

.github/copilot-instructions.md

@@ -0,0 +1,85 @@
+# Copilot Agent Instructions
+
+## Repository overview
+
+This is the **Orthogonal Engineering** repository — a deterministic audit and
+governance engine.  Key primitives live in:
+
+| Path | Purpose |
+|------|---------|
+| `toolkit/oe/hasher.py` | SHA-256 file/byte hashing with optional HMAC |
+| `toolkit/oe/merkle.py` | Binary Merkle tree, inclusion proofs, JSONL export |
+| `minimal_ai_ide/repository_scanner.py` | Structural repository scanner |
+| `cli.py` | Main CLI (subcommands: `index`, `merkle`, `handling-clamp`, `verify`, `diagnose`) |
+| `tools/repo_diagnoser/` | RepoDiagnoser tool (see below) |
+
+---
+
+## External repository analysis — RepoDiagnoser
+
+To clone and analyse **any public GitHub repository** from a terminal:
+
+```bash
+# Dry-run (shows what would happen, no writes)
+python -m tools.repo_diagnoser.cli_diagnose --url https://github.com/owner/repo
+
+# Clone and analyse (requires --apply)
+python -m tools.repo_diagnoser.cli_diagnose \
+    --url https://github.com/owner/repo \
+    --apply
+
+# Full clone, specific branch, export Merkle proofs
+python -m tools.repo_diagnoser.cli_diagnose \
+    --url https://github.com/owner/repo \
+    --depth 0 --ref main \
+    --out-proofs /tmp/proofs.jsonl \
+    --apply
+
+# Analyse an already-cloned directory
+python -m tools.repo_diagnoser.cli_diagnose --local /tmp/repo_analysis/myrepo
+
+# Via the main CLI
+python cli.py diagnose --url https://github.com/owner/repo --apply
+python cli.py diagnose --local /tmp/myrepo --apply
+```
+
+### Python API
+
+```python
+from tools.repo_diagnoser import RepoDiagnoser
+
+diagnoser = RepoDiagnoser()
+result = diagnoser.diagnose("https://github.com/owner/repo")
+
+print(result["merkle_root"])                # integrity fingerprint
+print(result["file_hashes"]["README.md"])   # per-file SHA-256
+proof = result["tree"].get_inclusion_proof("README.md")
+result["tree"].export_proofs_jsonl(Path("/tmp/proofs.jsonl"))
+```
+
+---
+
+## CLI safety contract
+
+- **Default behaviour is always dry-run** — no files are written without `--apply`.
+- Use `--apply` to perform writes (clones, manifests, proof files).
+- No credentials are stored or transmitted; only public repositories are supported.
+
+---
+
+## Adding new subcommands to cli.py
+
+Follow the existing pattern:
+
+1. Add `cmd_<name>(args) -> int` function.
+2. Register the subparser in `main()` with `subparsers.add_parser(...)`.
+3. Add `elif args.command == '<name>': return cmd_<name>(args)` to the dispatch block.
+
+---
+
+## Coding conventions
+
+- No bare `except: pass` — always catch specific exceptions and log them.
+- Dry-run by default; `--apply` required for mutations.
+- Reuse `toolkit.oe.hasher` and `toolkit.oe.merkle` rather than re-implementing SHA-256 or Merkle logic.
+- Imports from this repo use absolute package paths (e.g. `from toolkit.oe.hasher import hash_file`).

cli.py

@@ -13,6 +13,7 @@
     python cli.py process file.txt --dry-run
     python cli.py manifest create files/*.txt
 """
+"""
 CLI module - Main entrypoint for the deterministic pipeline scaffold.
 
 This module provides command-line interface with subcommands:
@@ -178,6 +179,7 @@ def cmd_manifest(args):
 from merkle import MerkleTreeBuilder, verify_inclusion_proof
 from handling_pipeline import process_handling_file
 from logger import create_logger
+from tools.repo_diagnoser.diagnoser import RepoDiagnoser
 
 
 VERSION = "1.0.0"
@@ -368,6 +370,54 @@ def cmd_verify(args):
         return 0
 
 
+def cmd_diagnose(args) -> int:
+    """Diagnose command — clone and analyse a public Git repository."""
+    diagnoser = RepoDiagnoser(clone_dir=args.clone_dir)
+
+    if args.url:
+        prefix = "DRY RUN — " if not args.apply else ""
+        print(f"{prefix}Cloning {args.url} …")
+        if not args.apply:
+            print("  (pass --apply to perform the clone and analysis)")
+            return 0
+        try:
+            result = diagnoser.diagnose(args.url, depth=args.depth, ref=args.ref)
+        except RuntimeError as exc:
+            print(f"Error: {exc}", file=sys.stderr)
+            return 1
+    else:
+        local_path = Path(args.local)
+        if not local_path.is_dir():
+            print(f"Error: local path not found: {local_path}", file=sys.stderr)
+            return 1
+        print(f"Analysing local repository: {local_path}")
+        result = diagnoser.analyze(local_path)
+        result["repo_path"] = str(local_path)
+
+    if args.out_proofs:
+        if not args.apply:
+            print(f"DRY RUN — would write proofs to: {args.out_proofs}")
+        else:
+            result["tree"].export_proofs_jsonl(Path(args.out_proofs))
+            print(f"Inclusion proofs written to: {args.out_proofs}")
+
+    print("\nDiagnosis complete.")
+    summary = {
+        "repo_path": result.get("repo_path", ""),
+        "merkle_root": result["merkle_root"],
+        "file_count": len(result["file_hashes"]),
+        "scan_timestamp": result["scan"].get("scan_timestamp", ""),
+    }
+    if args.json:
+        print(json.dumps(summary, indent=2))
+    else:
+        print(f"  Repo path  : {summary['repo_path']}")
+        print(f"  Files      : {summary['file_count']}")
+        print(f"  Merkle root: {summary['merkle_root']}")
+        print(f"  Scanned at : {summary['scan_timestamp']}")
+    return 0
+
+
 def main():
     """Main CLI entry point."""
     parser = argparse.ArgumentParser(
@@ -428,37 +478,7 @@ def main():
                                 help="Output path (for create)")
     manifest_parser.add_argument("--verbose", action="store_true",
                                 help="Verbose output")
-
-        description='Deterministic Pipeline Scaffold - Default behavior is DRY-RUN',
-        formatter_class=argparse.RawDescriptionHelpFormatter,
-        epilog="""
-Examples:
-  # Index repository (dry-run)
-  python cli.py index --repo /path/to/repo
-
-  # Index and create manifest (apply)
-  python cli.py index --repo /path/to/repo --apply --out manifest.jsonl
-
-  # Build Merkle tree from manifest
-  python cli.py merkle --manifest manifest.jsonl --apply
-
-  # Process handling file (dry-run)
-  python cli.py handling-clamp --handling-path handling.meta
-
-  # Process handling file (apply)
-  python cli.py handling-clamp --handling-path handling.meta --apply --out ./output
-
-  # Verify proofs
-  python cli.py verify --manifest merkle_proofs.jsonl
-
-Safety Notes:
-  - Default behavior is DRY-RUN (no writes)
-  - Use --apply flag to perform actual writes
-  - Backups are created automatically before overwrites
-  - No network calls or credentials used
-"""
-    )
-
+
     parser.add_argument('--version', action='version', version=f'%(prog)s {VERSION}')
 
     subparsers = parser.add_subparsers(dest='command', help='Available commands')
@@ -486,6 +506,38 @@ def main():
     # Verify command
     verify_parser = subparsers.add_parser('verify', help='Verify manifest or proofs')
     verify_parser.add_argument('--manifest', type=str, required=True, help='Manifest or proof file to verify')
+
+    # Diagnose command
+    diagnose_parser = subparsers.add_parser(
+        'diagnose', help='Clone and analyse a public Git repository'
+    )
+    diagnose_source = diagnose_parser.add_mutually_exclusive_group(required=True)
+    diagnose_source.add_argument('--url', metavar='URL', help='Public Git repository URL to clone')
+    diagnose_source.add_argument(
+        '--local', metavar='PATH', help='Path to an already-cloned local repository'
+    )
+    diagnose_parser.add_argument(
+        '--depth', type=int, default=1, metavar='N',
+        help='Shallow-clone depth (default: 1).  Use 0 for a full clone.'
+    )
+    diagnose_parser.add_argument(
+        '--ref', metavar='REF', default=None,
+        help='Branch or tag to check out (default: repository default branch)'
+    )
+    diagnose_parser.add_argument(
+        '--clone-dir', metavar='DIR', default='/tmp/repo_analysis',
+        help='Base directory for clones (default: /tmp/repo_analysis)'
+    )
+    diagnose_parser.add_argument(
+        '--out-proofs', metavar='FILE', default=None,
+        help='Write Merkle inclusion proofs to this JSONL file (requires --apply)'
+    )
+    diagnose_parser.add_argument(
+        '--apply', action='store_true', help='Apply changes (default is dry-run)'
+    )
+    diagnose_parser.add_argument(
+        '--json', action='store_true', help='Print summary as JSON'
+    )
 
     # Parse arguments
     args = parser.parse_args()
@@ -496,19 +548,27 @@ def main():
 
     try:
         if args.command == "hash":
-            cmd_hash(args)
+            return cmd_hash(args) or 0
         elif args.command == "process":
-            cmd_process(args)
+            return cmd_process(args) or 0
         elif args.command == "backup":
-            cmd_backup(args)
+            return cmd_backup(args) or 0
         elif args.command == "manifest":
-            cmd_manifest(args)
+            return cmd_manifest(args) or 0
+        elif args.command == 'index':
+            return cmd_index(args)
+        elif args.command == 'merkle':
+            return cmd_merkle(args)
+        elif args.command == 'handling-clamp':
+            return cmd_handling_clamp(args)
+        elif args.command == 'verify':
+            return cmd_verify(args)
+        elif args.command == 'diagnose':
+            return cmd_diagnose(args)
         else:
             parser.print_help()
             return 1
-
-        return 0
-
+
     except Exception as e:
         logger = get_logger("cli")
         logger.error(f"Command failed: {e}")
@@ -517,22 +577,5 @@ def main():
         return 1
 
 
-if __name__ == "__main__":
-        return 1
-
-    # Dispatch to command handler
-    if args.command == 'index':
-        return cmd_index(args)
-    elif args.command == 'merkle':
-        return cmd_merkle(args)
-    elif args.command == 'handling-clamp':
-        return cmd_handling_clamp(args)
-    elif args.command == 'verify':
-        return cmd_verify(args)
-    else:
-        parser.print_help()
-        return 1
-
-
 if __name__ == '__main__':
     sys.exit(main())

tools/repo_diagnoser/README.md

@@ -0,0 +1,93 @@
+# RepoDiagnoser
+
+Clone and analyse any public Git repository from the command line or Python API.
+
+## What it does
+
+1. **Clones** the target repository (shallow by default; full clone optional).
+2. **Scans** its structure using the existing `RepositoryScanner` (detects system types, categorises files, analyses imports and dependencies).
+3. **Hashes** every file with `toolkit.oe.hasher.hash_file` (SHA-256).
+4. **Builds a Merkle tree** with `toolkit.oe.merkle.MerkleTree` and returns the root hash as a compact fingerprint.
+5. Optionally **exports per-file inclusion proofs** as a JSONL file.
+
+## File layout
+
+```
+tools/repo_diagnoser/
+├── __init__.py        # Exports RepoDiagnoser
+├── diagnoser.py       # RepoDiagnoser class
+├── cli_diagnose.py    # Standalone CLI entry point
+└── README.md          # This file
+```
+
+## CLI usage
+
+```bash
+# Analyse a remote repo — dry-run (no clone, just shows what would happen)
+python -m tools.repo_diagnoser.cli_diagnose --url https://github.com/owner/repo
+
+# Clone and analyse (--apply to actually perform the clone)
+python -m tools.repo_diagnoser.cli_diagnose \
+    --url https://github.com/owner/repo \
+    --apply
+
+# Full clone, specific branch, write proofs
+python -m tools.repo_diagnoser.cli_diagnose \
+    --url https://github.com/owner/repo \
+    --depth 0 --ref main \
+    --out-proofs proofs.jsonl \
+    --apply
+
+# Analyse an already-cloned local directory
+python -m tools.repo_diagnoser.cli_diagnose --local /tmp/repo_analysis/repo
+
+# Output summary as JSON
+python -m tools.repo_diagnoser.cli_diagnose --local /tmp/myrepo --json
+```
+
+## Via the main cli.py
+
+```bash
+# Dry-run (shows what would be done)
+python cli.py diagnose --url https://github.com/owner/repo
+
+# Apply
+python cli.py diagnose --url https://github.com/owner/repo --apply
+
+# Local path
+python cli.py diagnose --local /tmp/myrepo --apply
+```
+
+## Python API
+
+```python
+from tools.repo_diagnoser import RepoDiagnoser
+
+diagnoser = RepoDiagnoser()                  # clones to /tmp/repo_analysis/
+result = diagnoser.diagnose("https://github.com/owner/repo")
+
+print(result["merkle_root"])                 # hex root hash
+print(len(result["file_hashes"]))            # number of files hashed
+
+# Inclusion proof for a specific file
+proof = result["tree"].get_inclusion_proof("README.md")
+
+# Export all proofs
+result["tree"].export_proofs_jsonl(Path("proofs.jsonl"))
+```
+
+## Integration points
+
+| Component | File | Usage |
+|-----------|------|-------|
+| File hashing | `toolkit/oe/hasher.py` | `hash_file()` — SHA-256 raw bytes |
+| Merkle tree | `toolkit/oe/merkle.py` | `MerkleTree.add_leaf()`, `.build()`, `.get_inclusion_proof()`, `.export_proofs_jsonl()` |
+| Structural scan | `minimal_ai_ide/repository_scanner.py` | `RepositoryScanner.scan_entire_repository()` |
+| Main CLI | `cli.py` | `diagnose` subcommand |
+
+## Safety notes
+
+- `--apply` is required for any write operation (clone, proofs file).  
+  Default behaviour is always **dry-run**.
+- Clones are placed in `/tmp/repo_analysis` and can be overridden with `--clone-dir`.
+- Only public repositories are supported; no credentials are stored or used.

tools/repo_diagnoser/__init__.py

@@ -0,0 +1,11 @@
+"""
+RepoDiagnoser — clone and analyse any public Git repository.
+
+Public API::
+
+    from tools.repo_diagnoser import RepoDiagnoser
+"""
+
+from .diagnoser import RepoDiagnoser
+
+__all__ = ["RepoDiagnoser"]