feat: repository-supplied threat models, application-type context, and multi-language scanning

.gitignore

@@ -10,3 +10,6 @@ libs/openant-core/parsers/javascript/.openant-npm-install.lock
 _docs/
 docs/
 .worktrees/
+
+# Local-only scratch and generated artifacts (not part of the codebase)
+*knowledge-graph.*

ARCHITECTURE.md

@@ -0,0 +1,201 @@
+# OpenAnt architecture
+
+How the scanner is put together, what depends on what, and where to stand when you
+need to change something.
+
+> **Where documentation lives.** In tracked files at the repository root and under
+> `libs/openant-core/`. **Not** in `docs/` — that directory is gitignored
+> (`.gitignore:11`), which is why twelve `docs/*.md` links elsewhere in this repo
+> point at files that do not exist. Anything written there is silently discarded.
+
+---
+
+## 1. The shape of the thing
+
+OpenAnt is a Go CLI wrapping a Python engine. The Go side owns the user's
+workspace — projects, config, checkpoints, process lifecycle. The Python side owns
+everything about analysis. They speak over a deliberately narrow contract.
+
+```mermaid
+flowchart TB
+    subgraph go["Go CLI — apps/openant-cli"]
+        cmd["cmd/*.go<br/>init · scan · parse · report"]
+        invoke["internal/python/invoke.go<br/>process lifecycle, timeouts, signals"]
+        gocfg["internal/config<br/>~/.config/openant/config.json"]
+        golang_["internal/languages<br/>reads config/languages.json"]
+    end
+
+    subgraph py["Python engine — libs/openant-core"]
+        cli["openant/cli.py<br/>argparse + cmd_* entry points"]
+        scanner["core/scanner.py<br/>orchestration"]
+        parsers["parsers/&lt;lang&gt;/<br/>7 language front-ends"]
+        ctx["context/<br/>app context + threat model"]
+        prompts["prompts/<br/>Stage 1 + Stage 2 prompt construction"]
+        llm["utilities/llm/<br/>provider adapters, rate limit, cost"]
+    end
+
+    reg[("config/languages.json<br/>single source of truth")]
+
+    cmd --> invoke --> cli --> scanner
+    scanner --> parsers & ctx & prompts
+    prompts --> llm
+    golang_ --> reg
+    parsers --> reg
+    gocfg -.->|"both sides read it<br/>independently"| cli
+
+    classDef seam stroke-width:3px
+    class reg,invoke seam
+```
+
+The two thick-bordered nodes are the seams that break quietly. `config/languages.json`
+is read by both runtimes; `invoke.go` is where the JSON envelope contract lives and
+where drift between the two sides shows up as silence rather than an error.
+
+---
+
+## 2. A scan, end to end
+
+```mermaid
+sequenceDiagram
+    participant U as user
+    participant Go as Go CLI
+    participant Py as openant/cli.py
+    participant S as core/scanner.py
+    participant P as parsers/&lt;lang&gt;
+    participant C as context/
+    participant L as LLM
+
+    U->>Go: openant scan ./repo
+    Go->>Py: python -m openant scan … (argv only)
+    Py->>S: scan_repository(...)
+    S->>P: detect languages, parse each
+    P-->>S: per-language datasets
+    S->>S: merge_datasets → one dataset.json
+    S->>C: load OPENANT.THREATMODEL.md, else generate app context
+    C-->>S: ApplicationContext (custom or built-in)
+    S->>L: Stage 1 — detection, per unit
+    L-->>S: findings
+    S->>L: Stage 2 — attacker simulation (--verify)
+    L-->>S: verdicts
+    S-->>Py: ScanResult
+    Py-->>Go: one JSON envelope on stdout; human text on stderr
+    Go-->>U: rendered summary; exit 0 clean / 1 vulns found / 2 error
+```
+
+**The contract in one paragraph.** stdout carries exactly one JSON envelope
+(`{status, data, errors}`); stderr carries human output and is streamed unparsed;
+exit codes are 0 clean, 1 vulnerabilities found, 2 error. Only `ANTHROPIC_API_KEY`
+crosses as an environment variable. Config is *not* passed — both sides read
+`~/.config/openant/config.json` independently, which is a known drift vector
+(see §6).
+
+---
+
+## 3. Multi-language: fan out, merge once
+
+Parsing fans out per language into `<run>/<lang>/`, because every parser writes the
+same seven flat filenames and would otherwise overwrite its siblings. The datasets
+then merge into one, and the expensive LLM stages run **once** over the merged set.
+
+```mermaid
+flowchart LR
+    repo[/"scanned repo"/] --> det["detect_languages()"]
+    det --> sel["select_languages()<br/>thresholds, dominant always kept"]
+    sel --> py2["python/"] & js2["javascript/"] & go2["go/"]
+    py2 & js2 & go2 --> merge["merge_datasets()<br/>stamps unit.language"]
+    merge --> ds[("dataset.json<br/>merged")]
+    ds --> stages["enhance → analyze → verify"]
+    sel -.->|excluded| warn["report_exclusions()<br/>loud coverage gap"]
+```
+
+**Why merge rather than fan the whole pipeline out:** one `--limit`, one cost
+budget, one dedup pass, one report. The cost is that the LLM stages see a mixed-
+language corpus — see §7 for the caveat that carries.
+
+`auto` means *every detected language above threshold*, and that is the default.
+`-l <lang>` is the opt-out.
+
+---
+
+## 4. Threat models: replacing "app type"
+
+The original design classified a repository into one of four values
+(`web_app | cli_tool | library | agent_framework`), each mapping to a hardcoded
+attack model, and collapsed the whole attacker question into one boolean. A
+repository can now ship `OPENANT.THREATMODEL.md` and describe itself.
+
+```mermaid
+flowchart TB
+    scan["core/scanner.py"] --> exists{"OPENANT.THREATMODEL.md<br/>present?"}
+    exists -->|"absent"| gen["generate_application_context()<br/>4-value enum"]
+    exists -->|"present"| load["load_threat_model()"]
+    load --> val{"valid?"}
+    val -->|"no"| abort["ABORT the scan<br/>never silently downgrade"]
+    val -->|"yes"| warn["warn_permissive_threat_model()<br/>all-trusted? no remote attacker?"]
+    warn --> ctxo["ApplicationContext<br/>threat_model_version set"]
+    gen --> ctxo
+    ctxo --> s1["Stage 1 prompt"] & s2["Stage 2 personas"]
+```
+
+Two properties worth knowing before you touch this:
+
+- **Absence falls back; malformed aborts.** A missing file is a choice. A present
+  but broken file is an error, because a scan that silently reverts to `web_app`
+  looks entirely successful while analysing under the wrong security model.
+- **The file is attacker-controlled.** It comes from the scanned repository. Its
+  prose is *not* prompt-injection fenced — a documented, accepted gap. The
+  permissive-model warnings exist because a schema-valid model can legitimately
+  suppress findings, and that suppression must at least be visible.
+
+---
+
+## 5. Where to stand to change something
+
+| I want to… | Start here | Then |
+|---|---|---|
+| add a language | `config/languages.json` | `parsers/<lang>/`, then run `tests/test_scanner_contract.py` — it discovers parsers from the registry, so a new language enters the suite automatically |
+| change what a prompt says | `prompts/vulnerability_analysis.py` (Stage 1), `prompts/verification_prompts.py` (Stage 2) | `prompts/threat_model_render.py` if it concerns attacker personas |
+| change the threat-model schema | `context/threat_model.py` (`REQUIRED_TOP_LEVEL`, the `_validate_*` helpers) | `context/OPENANT_THREATMODEL_TEMPLATE.md`, and `tests/test_threat_model_rejection.py` |
+| touch repository traversal | `core/repo_walk.py` — **one** walker for all Python parsers | never per-parser; that is why traversal bugs used to land in 1 of 5 |
+| open a file from a scanned repo | `utilities/file_io.py` — `read_repo_file` / `write_repo_file` / `repo_path_state` | never bare `open()`; these guard symlinks, FIFOs and size |
+| add a provider | `utilities/llm/providers/` | the adapter Protocol in `utilities/llm/adapter.py` |
+| change the Go↔Python contract | `core/schemas.py` **and** `apps/openant-cli/internal/` together | §6 — they are not bound by anything mechanical |
+
+---
+
+## 6. Known structural hazards
+
+These are documented rather than fixed. Each has bitten at least once.
+
+| Hazard | Where | Consequence |
+|---|---|---|
+| Go/Python contract is convention, not schema | `core/schemas.py` ↔ `internal/types/results.go` | Already failed: `formatter.go:120` reads `data["reports"]`, Python emits `step_reports`, so the CLI's Reports section has never rendered. `types.ReportData` has zero references — a struct that looks like type safety while production reads untyped maps |
+| Config path resolved independently on both sides | `internal/config/config.go` vs `utilities/llm/registry.py` | On Windows Go writes `%APPDATA%` and Python reads `~/.config` — the engine never sees the wizard's config |
+| Model defaults duplicated | `utilities/llm/builtins.py` ↔ `cmd/setup.go` | The Go wizard pre-fills retired model IDs; a fresh user's config 404s on every phase |
+| Import cycle | 11-module SCC via `utilities/__init__.py` | Held together by deferred in-function imports; initialization order is invisible to static reading |
+| `scan_repository` is 841 lines / 26 params | `core/scanner.py` | The change hotspot for anything pipeline-shaped |
+| Rate limiter is process-local | `utilities/rate_limiter.py` | Coordinated backoff works within one process; N concurrent scans do not coordinate |
+| No temperature set anywhere | repo-wide | Two scans of one commit can disagree; findings are not run-over-run comparable |
+| No spend ceiling | — | `--limit` caps units, not dollars |
+
+---
+
+## 7. The premise this design rests on, and its caveat
+
+Merging languages into one dataset is justified by the claim that the LLM stages
+are language-agnostic. That is true of Stage 1 and Stage 2 prompts. It is **not**
+true of the enhancer: `utilities/agentic_enhancer/prompts.py` lists sinks
+(`eval`, `exec`, SQL, `innerHTML`) and entry-point examples that are Python/JS
+only — one is Streamlit-specific. That prompt produces the classification gating
+the `exploitable` cost filter, so non-Python/JS languages are systematically
+under-classified.
+
+The architecture survives this; the docstring that asserted the premise did not.
+Extending those exemplars per language is a known, unclosed gap.
+
+---
+
+## See also
+
+- `libs/openant-core/context/THREAT_MODEL_AUTHORITY_DESIGN.md` — the unimplemented
+  authority model for repository-supplied threat models

CHANGELOG.md

@@ -3,6 +3,102 @@
 
 All notable changes to OpenAnt are documented in this file.
 
+## [2026-07-22] — Efficacy harness rescoped to a smoke test
+
+### Changed
+
+- **The "efficacy" harness no longer produces a recall/precision number.** It was
+  reporting `recall=1.0 precision=1.0` against a fixture whose docstrings captioned
+  each unit `VULN` / `NOT A VULN` — text that reaches the analysis prompt (via the
+  unit's `code.primary_code` and `metadata.docstring`), so the measurement only
+  proved a model can read an English label. `tests/efficacy/score.py` is now a
+  strict pipeline **smoke test** with a binary per-unit contract (flag the three
+  planted vulns, clear the three traps) and no averaged headline number.
+
+### Fixed
+
+- **Blinded the fixture.** Removed the answer-bearing verdict docstrings; the two
+  files (formerly an all-vulnerable `handlers.py` and an all-clean
+  `maintenance.py`, a file-split that itself signalled the answer) are replaced by
+  two neutrally-named modules with vulnerable and clean units interleaved. Expected
+  outcomes live only in the sidecar oracle `tests/efficacy/oracles/<fixture>.json`,
+  kept OUTSIDE the scanned fixture directory so the app-context survey agent (which
+  can `list_dir`/`read_repo_file` anything under the scanned tree) cannot reach the
+  answer key — a stronger guarantee than relying on the parser skipping non-`.py`
+  files. Verified: no verdict marker remains in the parsed dataset, and no
+  answer-key file remains under the scanned tree.
+- **Deleted the fabricated baseline** `tests/efficacy/baselines/webapp.json`, whose
+  `code_revision` named a tree where `score.py` did not yet exist.
+- **Fixed the scorer's silent-failure modes.** A missing `results.json` (scan
+  failure), an expected unit absent from the results, an unknown flagged id, and a
+  duplicate id each now raise and exit non-zero instead of scoring a confident
+  `recall=0.0`. The pure checking functions are unit-tested offline in
+  `tests/test_efficacy_score.py`.
+
+## [2026-07-22] — Go parser symlink containment (security)
+
+### Fixed
+
+- **The Go parser followed file symlinks out of the scanned repo (containment
+  hole).** `filepath.Walk` lstat's each entry, so a symlinked file
+  `leak.go -> /outside/secret` had `IsDir()==false` and a `.go` extension and was
+  added to the file list and read *through* the link — the same host-file→
+  `dataset.json`→model-provider exfiltration the Python engine's refuse-all
+  policy already closed, never applied to the Go binary. The scanner now refuses
+  every symlink (file and directory) inside the repo (the repo root itself is
+  exempt), and an unreadable directory is a counted `directories_unreadable`
+  coverage gap rather than a silently swallowed error. Proven by an executable
+  containment test (`scanner_symlink_test.go`) with a leak canary + negative
+  control; a mutation disabling the guard re-scans the symlinked file.
+- **The `test_pipeline.py` Go build now rebuilds on staleness, not just
+  absence.** Rebuild-only-if-absent left this security fix inert on any machine
+  with a cached (gitignored) binary. The build now triggers when any `.go`
+  source is newer than the binary.
+
+### Changed
+
+- **JavaScript parser reports symlink refusals as `symlinks_skipped`** (snake_case,
+  matching the other parsers) instead of folding them into `directoriesExcluded`,
+  and counts unreadable directories. Containment was already correct; this makes
+  the coverage aggregator see the gap. Added the missing JS symlink-containment
+  test (`TestSymlinkContainment`). With Go and JS now instrumented, they drop off
+  `languages_without_coverage_data` automatically. Corrected the false claim in
+  `test_scanner_contract.py` that Go/Node containment was covered by their own
+  suites — those suites did not exist, which is how the Go hole shipped.
+
+## [2026-07-22] — Artifact provenance + coverage; threat-model known risk
+
+### Added
+
+- **Context provenance in artifacts.** `scan.report.json` and
+  `pipeline_output.json` now carry `context_source` (`"threat_model"` |
+  `"generated"` | `"none"`) so a scan run under a repo-supplied security model
+  is distinguishable from one under the built-in generator. Both artifacts also
+  gained the multi-language coverage fields (`per_language`, `parse_errors`,
+  `excluded_languages`, `degraded`) that `scan.report.json` previously omitted,
+  and an aggregate `coverage` block reporting symlinks refused and directories
+  that could not be read. All keys are additive; Go consumers use comma-ok
+  access, so nothing breaks.
+- **Threat-model provenance (R5 controls).** When a repository supplies
+  `OPENANT.THREATMODEL.md`, the scan records `threat_model_sha256` (over the raw
+  file bytes) in both artifacts, persists the previously-discarded
+  over-permissive-model warnings (`threat_model_warnings`), and prepends a
+  deterministic, non-LLM banner to the summary report stating that the security
+  model came from a repo-controlled file. The sha key is **absent** — never the
+  empty-string hash — when no threat model is present.
+
+### Known risk (accepted, documented)
+
+- **Prompt injection via a repo-supplied threat model is NOT prevented.**
+  `OPENANT.THREATMODEL.md` ships inside the scanned (untrusted) repository and
+  shapes the attacker model applied to every finding; a hostile repo can declare
+  that nothing is a vulnerability. Per the project decision this is an accepted
+  gap. The controls above are *visibility*, not prevention: the sha, the
+  persisted permissive-model warnings, and the report banner let an operator see
+  that a repo-controlled file supplied the security model, and detect the
+  most-permissive case. See Risk R5 in the plan and the `load_threat_model`
+  call site.
+
 ## [2026-05-24] — Pluggable LLM providers (per-phase llm-configs)
 
 ### Added

README.md

@@ -31,6 +31,7 @@ To submit your repo for scanning:
 - C/C++ (beta)
 - PHP (beta)
 - Ruby (beta)
+- Zig (beta)
 
 ## Credits