Add discovery graph v1 for Helm-Path

README.md

@@ -9,6 +9,24 @@ Helm-Path is a local-first CTF flight recorder. It scaffolds a per-challenge wor
 3. Generate report artifacts from a selected run or from all runs.
 4. Verify the audit chain, manifests, and generated outputs.
 
+## Feature Matrix
+
+| Capability | Status | Notes |
+| --- | --- | --- |
+| Challenge workspace scaffolding | Implemented | `helm-path init` creates the per-challenge directory structure, metadata, notes, and artifact folders. |
+| Dockerized run capture | Implemented | `helm-path start` records a run into `sessions/<run-id>/raw.log` using the lite or full image. |
+| Immutable raw logs + derived clean logs | Implemented | Raw logs are preserved; `clean.log` is generated by stripping ANSI noise and redacting obvious secrets. |
+| Per-run manifests | Implemented | Each run writes `manifest.json` with timestamps, image metadata, file locations, and hashes. |
+| Command ledger capture | Implemented | Each run also writes `commands.jsonl` with command IDs, timestamps, cwd, raw command text, and exit codes. |
+| Append-only audit chain | Implemented | `.ffr/audit.db` chains run manifests and is checked by `helm-path verify`. |
+| Host-side AI writeup generation | Implemented | `helm-path report` uses Ollama on the host to generate structured Markdown and JSON artifacts. |
+| Multi-run aggregation | Implemented | Reports can target the latest run, a specific run, or all runs. |
+| Discovery graph build | Implemented | `helm-path graph build` parses command/output evidence and writes a per-challenge graph with linked infra entities. |
+| Embedded graph UI | Implemented | `helm-path graph serve` launches a local static UI with filters, search, evidence drill-down, and a lightweight `vis-network` renderer. |
+| Environment diagnostics | Implemented | `helm-path doctor` checks Docker, Ollama, model presence, and optional Pandoc support. |
+| Optional PDF export | Experimental | Supported through `report --format pdf`, but remains best-effort and non-blocking. |
+| Release publishing | Implemented | GitHub Actions build Python artifacts and publish container images on version tags. |
+
 ## Challenge Layout
 
 ```text
@@ -28,9 +46,15 @@ challenges/<competition>/<category>/<challenge>/
 │   ├── payloads.json
 │   ├── report_manifest.json
 │   └── timeline.json
+├── graph/
+│   ├── commands.json
+│   ├── graph.json
+│   ├── index.html
+│   └── manifest.json
 └── sessions/
     └── <run-id>/
         ├── clean.log
+        ├── commands.jsonl
         ├── manifest.json
         └── raw.log
 ```
@@ -41,6 +65,8 @@ challenges/<competition>/<category>/<challenge>/
 helm-path init "HTB Cyber Apocalypse" Web "Flag Command Injection"
 helm-path start challenges/htb-cyber-apocalypse/web/flag-command-injection --lite
 helm-path report challenges/htb-cyber-apocalypse/web/flag-command-injection --all-runs
+helm-path graph build challenges/htb-cyber-apocalypse/web/flag-command-injection --all-runs
+helm-path graph serve challenges/htb-cyber-apocalypse/web/flag-command-injection
 helm-path verify challenges/htb-cyber-apocalypse/web/flag-command-injection
 helm-path doctor
 ```
@@ -57,7 +83,9 @@ The Docker images are for terminal capture only. AI generation now runs on the h
 ## Notes
 
 - `report` defaults to the latest run unless `--all-runs` or `--run-id` is provided.
+- `graph build` defaults to the latest run unless `--all-runs` or `--run-id` is provided.
 - `verify` checks the append-only audit chain, log hashes, and report output hashes.
+- `graph serve` opens a local interactive graph focused on `IPAddress -> Port -> Service -> Product -> ProductRelease`.
 - PDF export is best-effort and does not block the Markdown workflow.
 
 ## CI/CD
@@ -67,6 +95,17 @@ The Docker images are for terminal capture only. AI generation now runs on the h
 - `Release` runs on tags matching `v*` and on manual dispatch.
 - It builds Python distribution artifacts, uploads them to the GitHub release, and publishes `helm-path-lite` and `helm-path-kali` images to GitHub Container Registry.
 
+## Roadmap
+
+| Stage | Priority | Outcome |
+| --- | --- | --- |
+| First remote CI validation | Next | Confirm the GitHub Actions workflows pass on the repository and fix any hosted-run drift. |
+| Real end-to-end smoke run | Next | Validate `init -> start -> report -> verify` against a real challenge with Docker and Ollama. |
+| Reporting integration test | Next | Add a mocked Ollama test that proves canned logs produce the expected report bundle. |
+| Tagged release | Next | Cut `v0.2.0` after the live smoke run and CI validation succeed. |
+| Reproducible exploit verification | Later | Add optional CI replay for challenge artifacts without making it part of the local-first core. |
+| Cloud execution for heavy workloads | Later | Add remote execution only for cases where local verification is insufficient. |
+
 ## Support
 
 Support the project: [buymeacoffee.com/santiagogow](https://buymeacoffee.com/santiagogow)

Test/test_graph.py

@@ -0,0 +1,228 @@
+import json
+from urllib.request import urlopen
+
+from helm_path.constants import COMMAND_MARKER_END, COMMAND_MARKER_START
+from helm_path.graph.build import (
+    build_graph_bundle,
+    load_command_records,
+    parse_curl_headers,
+    parse_nc_banner,
+    parse_nmap_xml,
+    parse_openssl_s_client,
+    parse_ssh_banner,
+    parse_whatweb,
+    segment_raw_log,
+    write_graph_artifacts,
+)
+from helm_path.graph.models import CommandRecord
+from helm_path.graph.server import serve_graph_dir
+from helm_path.processing import build_clean_log, calculate_file_hash, write_json_file
+from helm_path.workspace import create_run_layout, graph_output_paths, init_challenge_workspace, load_challenge_metadata, run_file_paths
+
+
+def create_recorded_run(challenge_path, metadata, command_raw, output_excerpt, command_id, extra_files=None):
+    run_dir, manifest = create_run_layout(challenge_path, metadata, image_tag="helm-path:lite", image_id="sha256:test")
+    paths = run_file_paths(challenge_path, manifest["run_id"])
+    raw_content = (
+        f"{COMMAND_MARKER_START}::{command_id}\n"
+        f"{output_excerpt}\n"
+        f"{COMMAND_MARKER_END}::{command_id}::0\n"
+    )
+    paths["raw_log"].write_text(raw_content, encoding="utf-8")
+    build_clean_log(paths["raw_log"], paths["clean_log"])
+    paths["commands_log"].write_text(
+        json.dumps(
+            {
+                "schema_version": 1,
+                "command_id": command_id,
+                "run_id": manifest["run_id"],
+                "started_at": "2026-03-13T10:00:00Z",
+                "finished_at": "2026-03-13T10:00:05Z",
+                "cwd": "/workspace",
+                "command_raw": command_raw,
+                "exit_code": 0,
+            }
+        )
+        + "\n",
+        encoding="utf-8",
+    )
+    if extra_files:
+        for relative_path, content in extra_files.items():
+            target = challenge_path / relative_path
+            target.parent.mkdir(parents=True, exist_ok=True)
+            target.write_text(content, encoding="utf-8")
+    manifest["captured_at"]["end"] = "2026-03-13T10:00:05Z"
+    manifest["hashes"]["raw_log"] = calculate_file_hash(paths["raw_log"])
+    manifest["hashes"]["clean_log"] = calculate_file_hash(paths["clean_log"])
+    manifest["hashes"]["commands_log"] = calculate_file_hash(paths["commands_log"])
+    write_json_file(paths["manifest"], manifest)
+    return run_dir
+
+
+def test_load_command_records_and_segment_raw_log(tmp_path):
+    commands_log = tmp_path / "commands.jsonl"
+    raw_log = tmp_path / "raw.log"
+    commands_log.write_text(
+        json.dumps(
+            {
+                "command_id": "run-1-1",
+                "run_id": "run-1",
+                "started_at": "2026-03-13T10:00:00Z",
+                "finished_at": "2026-03-13T10:00:03Z",
+                "cwd": "/workspace",
+                "command_raw": "nmap -sV 10.10.11.42",
+                "exit_code": 0,
+            }
+        )
+        + "\n",
+        encoding="utf-8",
+    )
+    raw_log.write_text(
+        f"{COMMAND_MARKER_START}::run-1-1\n22/tcp open ssh OpenSSH 8.2p1 Ubuntu 4ubuntu0.5\n{COMMAND_MARKER_END}::run-1-1::0\n",
+        encoding="utf-8",
+    )
+
+    commands = load_command_records(commands_log, "run-1")
+    segments = segment_raw_log(raw_log)
+
+    assert commands[0].executable == "nmap"
+    assert "OpenSSH 8.2p1" in segments["run-1-1"]
+
+
+def test_core_parsers_extract_expected_observations(tmp_path):
+    xml_file = tmp_path / "scan.xml"
+    xml_file.write_text(
+        """<nmaprun><host><address addr="10.10.11.42" addrtype="ipv4"/><hostnames><hostname name="target.htb"/></hostnames><ports><port protocol="tcp" portid="22"><state state="open"/><service name="ssh" product="OpenSSH" version="8.2p1"/></port></ports></host></nmaprun>""",
+        encoding="utf-8",
+    )
+    xml_result = parse_nmap_xml(xml_file)
+    assert xml_result.observations[0]["ip"] == "10.10.11.42"
+    assert xml_result.observations[0]["product"] == "OpenSSH"
+
+    whatweb_result = parse_whatweb(
+        CommandRecord(
+            command_id="c1",
+            run_id="r1",
+            started_at="",
+            finished_at="",
+            cwd="/workspace",
+            command_raw="whatweb http://10.10.11.42",
+            exit_code=0,
+            executable="whatweb",
+            output_excerpt="http://10.10.11.42 [200 OK] HTTPServer[nginx/1.18.0], IP[10.10.11.42]",
+        )
+    )
+    assert whatweb_result.observations[0]["service"] == "http"
+    assert whatweb_result.observations[0]["product"] == "nginx"
+
+    curl_result = parse_curl_headers(
+        CommandRecord(
+            command_id="c2",
+            run_id="r1",
+            started_at="",
+            finished_at="",
+            cwd="/workspace",
+            command_raw="curl -I http://10.10.11.42",
+            exit_code=0,
+            executable="curl",
+            output_excerpt="HTTP/1.1 200 OK\nServer: nginx/1.18.0\nX-Powered-By: PHP/8.1.2",
+        )
+    )
+    assert any(item["product"] == "nginx" for item in curl_result.observations)
+    assert any(item["product"] == "PHP" for item in curl_result.observations)
+
+    openssl_result = parse_openssl_s_client(
+        CommandRecord(
+            command_id="c3",
+            run_id="r1",
+            started_at="",
+            finished_at="",
+            cwd="/workspace",
+            command_raw="openssl s_client -connect 10.10.11.42:443",
+            exit_code=0,
+            executable="openssl",
+            output_excerpt="Protocol  : TLSv1.3\nCipher    : TLS_AES_256_GCM_SHA384",
+        )
+    )
+    assert openssl_result.observations[0]["service"] == "tls"
+
+    ssh_result = parse_ssh_banner(
+        CommandRecord(
+            command_id="c4",
+            run_id="r1",
+            started_at="",
+            finished_at="",
+            cwd="/workspace",
+            command_raw="ssh 10.10.11.42",
+            exit_code=0,
+            executable="ssh",
+            output_excerpt="debug1: remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.5",
+        )
+    )
+    assert ssh_result.observations[0]["product"] == "OpenSSH"
+
+    nc_result = parse_nc_banner(
+        CommandRecord(
+            command_id="c5",
+            run_id="r1",
+            started_at="",
+            finished_at="",
+            cwd="/workspace",
+            command_raw="nc 10.10.11.42 22",
+            exit_code=0,
+            executable="nc",
+            output_excerpt="SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5",
+        )
+    )
+    assert nc_result.observations[0]["service"] == "ssh"
+
+
+def test_graph_bundle_deduplicates_entities_across_runs(tmp_path):
+    challenge_path = init_challenge_workspace(tmp_path, "HTB", "Pwn", "Graphbox")
+    metadata = load_challenge_metadata(challenge_path)
+    run1 = create_recorded_run(
+        challenge_path,
+        metadata,
+        "nmap -sV 10.10.11.42",
+        "Nmap scan report for 10.10.11.42\n22/tcp open ssh OpenSSH 8.2p1 Ubuntu 4ubuntu0.5\n80/tcp open http nginx 1.18.0",
+        "cmd-1",
+    )
+    run2 = create_recorded_run(
+        challenge_path,
+        metadata,
+        "nc 10.10.11.42 22",
+        "SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5",
+        "cmd-2",
+    )
+
+    graph, commands_payload, manifest = build_graph_bundle(challenge_path, [run1, run2])
+
+    ip_nodes = [node for node in graph["nodes"] if node["type"] == "IPAddress" and node["label"] == "10.10.11.42"]
+    ssh_services = [node for node in graph["nodes"] if node["type"] == "Service" and node["label"] == "ssh"]
+    assert len(ip_nodes) == 1
+    assert len(ssh_services) == 1
+    assert commands_payload["summary"]["commands"] == 2
+    assert manifest["run_ids"] == [run1.name, run2.name]
+
+
+def test_write_graph_artifacts_and_serve(tmp_path):
+    challenge_path = init_challenge_workspace(tmp_path, "HTB", "Web", "Render")
+    metadata = load_challenge_metadata(challenge_path)
+    run_dir = create_recorded_run(
+        challenge_path,
+        metadata,
+        "curl -I http://10.10.11.42",
+        "HTTP/1.1 200 OK\nServer: nginx/1.18.0",
+        "cmd-1",
+    )
+
+    manifest = write_graph_artifacts(challenge_path, [run_dir])
+    output_paths = graph_output_paths(challenge_path)
+
+    assert output_paths["graph.json"].exists()
+    assert output_paths["assets_dir"].is_dir()
+    assert "graph.json" in manifest["outputs"]
+
+    with serve_graph_dir(challenge_path / "graph", host="127.0.0.1", port=0, open_browser=False) as (_, url):
+        html = urlopen(url).read().decode("utf-8")
+        assert "Discovery Graph" in html

Test/test_logic.py

@@ -26,6 +26,7 @@ def test_init_challenge_workspace_creates_expected_layout(tmp_path):
     assert (challenge_path / ".metadata.json").exists()
     assert (challenge_path / "sessions").is_dir()
     assert (challenge_path / "reports").is_dir()
+    assert (challenge_path / "graph").is_dir()
     assert (challenge_path / "notes" / "FAILURES.md").exists()
     assert (challenge_path / "notes" / "WORKING_NOTES.md").exists()
     metadata = load_challenge_metadata(challenge_path)
@@ -103,9 +104,11 @@ def test_audit_chain_detects_manifest_tampering(tmp_path):
 
     paths["raw_log"].write_text("nc target 31337\n", encoding="utf-8")
     paths["clean_log"].write_text("nc target 31337\n", encoding="utf-8")
+    paths["commands_log"].write_text("", encoding="utf-8")
     manifest["captured_at"]["end"] = "2026-03-13T10:00:00+00:00"
     manifest["hashes"]["raw_log"] = calculate_file_hash(paths["raw_log"])
     manifest["hashes"]["clean_log"] = calculate_file_hash(paths["clean_log"])
+    manifest["hashes"]["commands_log"] = calculate_file_hash(paths["commands_log"])
     write_json_file(paths["manifest"], manifest)
 
     db_path = challenge_path / ".ffr" / "audit.db"

docker/Dockerfile.kali

@@ -2,6 +2,8 @@ FROM kalilinux/kali-rolling
 
 ENV DEBIAN_FRONTEND=noninteractive
 
+COPY docker/helm-path-hooks.zsh /opt/helm-path/helm-path-hooks.zsh
+
 RUN echo "wireshark-common wireshark-common/install-setuid boolean true" | debconf-set-selections
 
 # Update and install the Watcher's Arsenal (Apt Packages)
@@ -93,6 +95,7 @@ RUN mkdir -p /workspace && chown durk:durk /workspace
 USER durk
 
 RUN echo 'if [ "${SCRIPT_LOGGED:-0}" != "1" ]; then export SCRIPT_LOGGED=1; script -f -q /workspace/${LOG_FILE:-sessions/default/raw.log}; exit; fi' >> /home/durk/.zshrc
+RUN echo 'source /opt/helm-path/helm-path-hooks.zsh' >> /home/durk/.zshrc
 
 RUN echo 'echo -e "\n\033[1;33mHelm-Path full capture environment ready.\033[0m"' >> /home/durk/.zshrc
 RUN echo 'echo -e "\033[1;34mLocal capture is active. AI runs on the host.\033[0m\n"' >> /home/durk/.zshrc

docker/Dockerfile.lite

@@ -2,6 +2,8 @@ FROM kalilinux/kali-rolling
 
 ENV DEBIAN_FRONTEND=noninteractive
 
+COPY docker/helm-path-hooks.zsh /opt/helm-path/helm-path-hooks.zsh
+
 RUN apt-get update && apt-get install -y \
     curl \
     wget \
@@ -42,6 +44,7 @@ RUN mkdir -p /workspace && chown durk:durk /workspace
 USER durk
 
 RUN echo 'if [ "${SCRIPT_LOGGED:-0}" != "1" ]; then export SCRIPT_LOGGED=1; script -f -q /workspace/${LOG_FILE:-sessions/default/raw.log}; exit; fi' >> /home/durk/.zshrc
+RUN echo 'source /opt/helm-path/helm-path-hooks.zsh' >> /home/durk/.zshrc
 
 RUN echo 'echo -e "\n\033[1;33mHelm-Path Lite capture environment ready.\033[0m\n"' >> /home/durk/.zshrc