feat: repository-supplied threat models, application-type context, and multi-language scanning - #199
Merged
gadievron merged 44 commits intoJul 30, 2026
Conversation
…call test_F3_verdict_taxonomy_shared_constant.py deleted every core.* entry from sys.modules without restoring it. Any test collected afterwards that bound `import core.X as m` and monkeypatched m.attr was silently defeated: the code under test re-imported and got a DIFFERENT module object, so the patch applied to an orphan. That is why test_enhance_limit bypassed its stub and issued a live Anthropic API call during an offline run — a billing hazard, not merely untidy. The eviction is now save/restore-wrapped, with a regression lock. The same file defaulted its root to an absolute path inside another machine session's scratchpad. Where that directory still exists the test does not error — it silently asserts against a stale, different tree, which is why it reported core/reporter.py as missing a constant it has always imported. Seven test files carried machine-specific absolute paths; all now derive from __file__. tests/test_no_hardcoded_paths.py guards every test file against recurrence, with a self-test so the matcher cannot silently rot. test_F4_entry_root_additive_v3 required an out-of-tree pre-knostic#165 checkout and a .patch file that was never committed, so it has failed since the day it landed — as a hard assert, not a skip. Now skipped with a precise reason and replaced by forward-looking tests asserting the current tree seeds the F4 entry points (APIRouter, aiohttp RouteTableDef, Starlette websocket_route, Django CBV), which had zero coverage: it was the only test naming those markers and it aborted before reaching any assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two features that share the same integration points (core/scanner.py,
openant/cli.py, prompts/) and so land together; splitting them further would
require hunk-level surgery on those files for no reviewer benefit.
=== Multi-language ===
detect_language() counted source files and returned max(counts), discarding
the rest — a 60% Go / 40% TypeScript repo was scanned as Go and the absence of
the TypeScript was never reported anywhere.
config/languages.json is now the single source of truth. Four places
independently described the supported set and had drifted: the config, the
parser dispatch chain, argparse choices (twice), and Go flag help in three
commands — scan.go, parse.go and README.md had all silently fallen behind when
Zig was added. Everything now derives from the config, with consistency tests
pinning each description to it. Resolution searches an env override then
upward from module and CWD and DEGRADES rather than raising:
supported_languages() runs during argparse construction, so a missing config
previously took down `openant --help` entirely.
parse_repository_multi() parses each selected language into <run>/<lang>/,
because every parser writes the SAME seven flat filenames into whatever
directory it is handed — two languages sharing one leaves exactly one
survivor. dataset_merge.py merges them so every later stage still runs ONCE.
Call graphs are NOT merged: no parser emits cross-language edges, so a union
would assert connectivity that does not exist; call_graphs.json indexes them,
built by probing the filesystem rather than a hardcoded list.
Partial success is data, not exceptions — one broken toolchain must not cost
every other language. The catch list is narrow so KeyboardInterrupt and
MemoryError still abort. Sequential by design: cost tracking deltas against a
process-global tracker, the Python parser mutates sys.path in-process, and six
concurrent parsers on a monorepo is a realistic OOM.
Excluded languages are reported loudly — stderr, step report, and JSON
envelope. Measured, the default threshold silently dropped a PHP file holding
a real path traversal to save 0.13s; for a security scanner a silent skip is a
silently missed vulnerability class.
=== Threat models ===
A repository can commit OPENANT.THREATMODEL.md declaring its own free-form
classification, components, attacker profiles, three-level input trust, what
is and is not a vulnerability, and impact. When present it replaces the
built-in application context entirely.
The built-in model had four application types and ONE hardcoded attacker ("an
attacker on the internet with a browser and nothing else"), with a single
boolean deciding whether local-only findings counted. A deployment
orchestrator whose real threat is "a developer with commit access to a watched
manifest repo, no shell on the host" could not be expressed at all. Declared
profiles now replace that persona in BOTH analysis stages, each with explicit
capabilities and — load-bearing — explicit CANNOT limits.
load_threat_model returns None only when the file is ABSENT; a malformed file
RAISES, and the scanner calls it outside the context step's warn-and-continue
handler on purpose. Degrading a malformed model into a default web_app context
would silently apply the wrong security model to every finding.
The file is attacker-authored when scanning third-party code, so it is guarded
before opening: lstat rejects symlinks and non-regular files (a FIFO hung the
scanner indefinitely — confirmed) and caps size at 1 MiB.
`openant threat-model <repo>` generates one; --validate-only makes no LLM call.
context/THREAT_MODEL_AUTHORITY_DESIGN.md records the unsolved risk: the danger
is not prompt injection but that the audited repository is granted authority
over its own threat model.
=== Bugs found by adversarial review and fixed here ===
- Python and Ruby multi-file units were analysed WHOLESALE. Parsers emit the
boundary marker in each language's comment syntax (`#` for Python/Ruby) but
all four consumers matched `//` literally, so the split never fired and the
model was handed the entire concatenation as the target function with the
"do NOT analyze" context section silently dropped. Verified by isolating
control: 3 function definitions inside the ANALYZE-ONLY block before, 1
after.
- Entry-point seeds computed across ALL languages were passed into EACH
per-language reachability filter, defeating its empty-seed blackout guard —
every unit of any language lacking its own entry points was silently
dropped while the scan reported success.
- `--languages python` parsed the dominant language instead.
- `--languages go` on a repo with no Go silently scanned Python.
- Code fences were keyed by the scan-wide language, so a .ts file in a
"javascript" scan was fenced as ```javascript.
- The dynamic tester defaulted to "Python" for any unmapped language; it now
resolves per finding and SKIPS untemplated ones, with SKIPPED registered as
a first-class status so skipped findings are not invisible.
=== Documentation ===
OPENANT_THREATMODEL_TEMPLATE.md listed four mitigations for the accepted
prompt-injection gap in the present tense. None existed — worse than an
undocumented gap, because a reviewer would approve the risky configuration on
the strength of controls that are not there. They are now an unchecked TODO
list. internal/languages/registry.go claimed a shared golden fixture pinning
the two detectors together; no such fixture exists. LanguageSelection.excluded
documented itself as surfaced while being discarded; it is now genuinely
surfaced. internal notes.md records the request, plan, result and lessons.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… audit
OpenAnt scans untrusted third-party repositories, so every byte of a scanned
repo -- file names, directory shapes, comment text, OPENANT.md -- is attacker
input. A nine-reviewer audit of the multi-language work found four defects that
a repository can trigger against the scanner itself. Two were introduced by
earlier fixes in the same work.
1. Boundary-marker forgery (regression, introduced by the multi-language fix).
split_on_boundary matched a fixed literal that scanned source can contain, so
one comment line cut a unit in half and relabelled everything after it as
"Context (do NOT analyze these)" -- hiding a vulnerability from both stages.
Before this work the matcher required "//", a syntax error in Python, so the
marker was unforgeable by accident; accepting "#" fixed a real splitting bug
(Python/Ruby units never split, so dependency code was analysed as the target)
and made it forgeable in exactly the languages just fixed. Fixed at the
producers, not the pattern: all 8 now neutralize boundary-shaped lines before
concatenating, so every marker in a blob is parser-emitted by construction.
Tightening the regex again is what caused this, so the fix does not.
2. Traversal followed directory symlinks and silently lost deep files.
"escape -> /tmp/outside" put a file from outside the repository into
dataset.json, which ships to the model provider; "loop -> .." ran for minutes
at full CPU on a one-file repo. Separately, Path.is_dir() converts OSError to
a silent False, so a path past PATH_MAX was classified as neither file nor
directory and skipped -- a planted file at depth 600 vanished from a scan that
reported success. That is a false-negative injection primitive, and worse than
a crash because it manufactures assurance. The Python scanner is now iterative
(preserving traversal order via an iterator stack), classifies with an explicit
stat, and records unreadable subtrees as a coverage gap in statistics rather
than dropping them. The guard is shared and now present in all 7 parsers; it
previously existed only in ruby and zig, whose own comment noted the deviation.
3. ReDoS in the threat-model JSON extractor. ```json\s*(.*?)\s*``` is ambiguous
and backtracks cubically on an unclosed fence: 4 KB of trailing whitespace
took 32.9s, and the 1 MiB cap does not bound it. Eight bytes wedge a run.
The same pattern was also present in check_manual_override -- found while
fixing the first one, not reported by any reviewer.
4. Arbitrary write from LLM-controlled short_name. .replace(" ","_") is not
sanitization; "/" and ".." survived it into os.path.join, escaping output_dir.
Also: a dangling OPENANT.THREATMODEL.md symlink read as absent (exists() follows
links), silently downgrading the scan to built-in heuristics and inverting the
module's absence-falls-back/malformed-aborts rule; a threat-model JSON block
hidden in an HTML comment beat the visible one, so a reviewer and the scanner
saw different documents; and the FIFO/S_ISREG guard existed at exactly one of
four repo-file open sites, so a FIFO named OPENANT.md blocked indefinitely.
Centralized as read_repo_file rather than patched per site.
Testing: new tests/test_hostile_repo.py builds one repository that mounts every
attack above and asserts the scanner survives. Written red first -- 12 failing
assertions reproduced each defect before any fix. It includes two enumerated
parity tests (every producer neutralizes; every parser guards symlinks) because
the recurring defect in this codebase is a fix landing at one of N sites, and the
recurring test defect is asserting on the helper instead of the call site.
Suite: 2013 passed, 2 failed (both pre-existing: a CPython 3.14 fixture that no
longer raises RecursionError), 27 skipped. ruff clean; go build/vet/test clean.
… registry
Three remaining findings from the adversarial audit, plus the two long-standing
test failures.
Threat-model validation was shape-only where it mattered. Mutation testing found
18-21 surviving mutants: every type check, every empty check and every nested
missing-field check could be deleted with the suite still green, because the
existing tests only ever delete a top-level key from a valid model and never
supply a wrong type or a malformed nested object. This file is authored by the
scanned repository, so a validator whose rejection paths never execute is not a
trust boundary. Adds 57 negative controls (tests/test_threat_model_rejection.py),
each asserting a rejection rather than an acceptance, and verified to kill
mutants rather than merely pass: disabling _require_nonempty_str fails 11 of
them, disabling the duplicate-id check fails 1.
Genuine schema holes closed alongside: "schema_version": true was accepted as
version 1 (True == 1, and the bool guard existed on confidence but not here);
duplicate attacker-profile ids and duplicate component names were accepted, the
latter weakening the handled_by cross-check by collapsing into a name set; a
capability asserted in both capabilities and cannot was rendered verbatim to the
verifier, telling it the attacker both can and cannot do the same thing, which
silently decides a NOT-EXPLOITABLE verdict either way; and a non-string
input_sources key could make the dangling-reference error path itself raise,
since it sorts mixed key types.
Un-defers 2.4 of THREAT_MODEL_AUTHORITY_DESIGN.md. A schema-valid threat model
can suppress findings with no prompt injection at all: declare every input
trusted, model no reachable attacker, blanket-exclude the classes a scanner would
report. Nothing is malformed, so validation cannot catch it. warn_permissive_
threat_model does not refuse such a model -- the operator may have written it --
but makes the suppression visible instead of silent. Per-finding suppression
accounting (2.3) is NOT implemented: suppression happens inside the prompt, so
there is no code-level event to count, and doing it properly needs the A/B arm.
missing_headings had zero callers in production and in tests while its docstring
claimed callers surfaced warnings, so the documented "the md must follow a
specific structure" requirement was enforced nowhere. Now called on load.
Packaging: config/languages.json is the single source of truth for both runtimes
but lives at the monorepo root, two levels above the Python project root, and the
wheel shipped without it -- the registry resolved to {}, "-l python" became
"invalid choice", and detection found zero files in every repository. Adds a
force-include stanza, proven by building a wheel and installing it into a clean
venv outside the checkout: all 7 languages resolve. Also splits the contract that
earlier degradation blurred: supported_languages() still returns [] so --help
survives, but extension_map() now raises and names the installation as the cause.
The previous behaviour reported "repository has no supported source files",
sending the operator to inspect their repository over a broken install -- a worse
outcome than the crash it replaced, and the opposite of Go's loud contract for
the same file. Two tests encoding the old contract are updated deliberately, with
the reasoning recorded in them rather than silently relaxed.
Test fixture: _STACK_BLOWER no longer raises on CPython 3.14, whose PEG parser
handles left-recursive attribute chains iteratively. The production guard was
never at fault (test_post_parse_extraction_error_is_isolated injects the error
and has always passed), so the fix is the fixture. This also repairs a third test
in that file that had begun passing vacuously, isolating an error that no longer
occurred -- a false green.
Suite: 2078 passed, 0 failed, 27 skipped. ruff clean; go build/test clean.
…r run Acting on independent consultations from sol (gpt-5.6-sol) and fable on the whole arc. Both ruled traversal before feature work, for the same reason: inverting the `auto` default is a reachability multiplier for the vulnerable scanners, so shipping "scan all languages" first would amplify a live defect. ONE SHARED WALKER (core/repo_walk.py). python/c/php/ruby/zig each had their own recursion, symlink handling, stat-error behaviour and gap accounting. That is the mechanical cause of this codebase's most persistent defect shape: traversal was fixed three times in this arc and each fix reached a different subset. The walker is iterative (no recursion limit), refuses escaping symlinks, and counts anything it cannot classify or read into the structured result rather than dropping it. Symlink policy is now three-branch, and the middle branch is a user decision: - resolves inside the repo -> SKIP (canonical path is walked anyway; descending duplicates every file under the alias) - resolves inside the parent -> FOLLOW, inode-bounded (monorepo sibling vendoring) - anything further out -> REFUSE (the exfiltration path) This surfaced a real conflict rather than a stale test: parsers/zig carries a regression test requiring external symlinked dirs be followed, because .zig files reachable only that way were once silently dropped. Flattening it would have traded an exfiltration bug for a false-negative bug. Accepted residual risk, recorded in safe_to_descend: anything under the parent directory is reachable; clone into a dedicated parent. WRITER HARDENED BEFORE THE GENERATOR WAS MADE REACHABLE. threat_model_agent used exists() + write_text(), both of which follow symlinks: a dangling OPENANT.THREATMODEL.md read as absent, and the write then followed it OUT of the repository. Unreachable only because the generator was broken -- the one-line fix below would have activated it. Same defect I fixed in load_threat_model and left here, so it is the third instance of the 1-of-N pattern inside my own fix for it. Added write_repo_file (O_NOFOLLOW) and repo_path_state (lstat) to file_io. THE GENERATOR NOW RUNS. threat_model_agent called `adapter.complete(prompt=..., max_tokens=...)`; the protocol is keyword-only `complete(*, model, system, messages, max_tokens, tools=None) -> CompletionResult`. Every real adapter raised TypeError, wrapped into a polite generation error, so the feature had NEVER once executed. Now goes through simple_text, which also records the call against the token tracker -- generation was previously invisible to cost accounting. This is "the generator runs", NOT "request 2 is done": it remains a single shot over a truncated README and a shallow listing, so it can emit a schema-valid model of a repository it has largely not read. TEST MECHANISMS, replacing the ones that failed: - Deleted the token-grep parity tests. They assert a string exists in a source file, which proves neither that the guard executes nor that incompleteness reaches the result. Both failure directions occurred within a day: all 7 parsers passed while 3 had an unrelated hole, then compliant parsers FAILED once the guard moved to a shared module. - tests/test_scanner_contract.py executes each scanner against hostile fixtures and discovers them from the language registry, so language knostic#8 is covered on arrival. A skip is a failure. This immediately found that zig followed external symlinks on first encounter -- a hole the audit's grep census and my parity test both scored compliant. - tests/test_no_banned_patterns.py sweeps the whole tree for defect patterns rather than asserting guards in an author-remembered list. The author's list is reliably short by one: this found a THIRD copy of the ambiguous json-fence ReDoS at application_context.py:664 (fable), after I had reported it fixed at "1 of 2". - The permissive fake `def complete(self, *a, **k)` is banned and replaced with one matching the real keyword-only signature. That fake is the entire reason 672 tests were green over a generator that never ran; tightening it turned 6 tests red before the fix and green after. ALSO: the dev venv held a stale NON-editable copy of the package in site-packages, shadowing source -- so standalone parser runs imported pre-fix code, and commit b8dc4ed had actually broken `python parsers/c/repository_scanner.py` at HEAD without the suite noticing. Reinstalled editable; all 5 scanners run standalone again. Dropped a stale stash containing a one-line debug probe. Suite: 2107 passed, 0 failed, 26 skipped. ruff clean; go build/test clean; node ok.
This is the headline requirement and it was inverted. The request was "all languages we currently support should be detected in the repo and it should scan all of them, not just the main one" -- a statement about DEFAULT behaviour. What shipped made multi-language opt-in, with a comment in cli.py stating the inversion outright. A deprecation path was considered and rejected on both consultations' advice: it preserves exactly the behaviour the user rejected. Python: `_select_languages_for` returned None unless a multi flag was passed, which forced the legacy dominant-language path and skipped detection entirely. It now returns None only for an explicit `-l <lang>`, which remains the escape hatch. The rest of the machinery already existed and was already reviewed -- select_languages applies thresholds, always keeps the dominant language so a selection cannot be empty, and report_exclusions prints anything dropped. Only that early return stood between them and the default. Go: `init` resolved auto to ONE detected language and persisted it; `scan` then passed the stored value as an explicit -l, and explicit beats auto. An init-created project was therefore pinned to a single language permanently, so the product's PRIMARY flow was the one place the engine's default could never take effect -- fixing Python alone would have let me declare the requirement met while the main path stayed single-language. `init` now stores "auto" and detection runs only to tell the user what is present. Consequence: new projects resolve their language set per-scan, so a language added to the repo later is picked up without re-running init. Existing projects keep their stored concrete language, so back-compat holds; their scan directory layout is unchanged. Also fixes the exclusivity check, which omitted `--multi-language` while a sibling function treated it as a trigger: `-l go --multi-language` silently dropped the flag and then reported the exclusion as "not requested via --languages", a flag the user never passed. Behaviour change, stated plainly: multi-language repositories now cost more to scan and produce more findings. That is the intended effect. The loud selection and exclusion banner is the mitigation the user already ratified, so the coverage change is never silent. `-l <lang>` opts out. Two llm_reachability tests used an EMPTY tmp_path as the repo and passed only because the default never walked it; they now create a source file, since detection running by default is the point of this change. Suite: 2114 passed, 0 failed, 26 skipped. ruff clean; go build/vet/test clean.
Request 2 asked for "an AI agent [that] will go over the repo, understand its components, structure, architecture, classify each component". What existed after the adapter fix was a single completion over a README truncated to 4KB and a two-level directory listing -- enough to emit a schema-valid document about a repository it had largely not read, naming components that may not exist. Schema validity proves structure, never understanding, and both consultations (sol, fable) independently called the one-line fix necessary but not sufficient. context/repo_explorer.py gives the model bounded read-only tools -- list_dir, read_file, search -- and loops until it calls `finish`, whose input_schema IS the v1 threat-model contract, so structure is enforced at generation time rather than only by the validator afterwards. Modeled on the in-house loop already used by utilities/finding_verifier.py rather than a new pattern. The repository is the same untrusted third-party code the scanner exists to analyse, so: every path resolved and confined beneath the repo root (a tool argument is an injection sink, and the repo's prose is an unfenced channel for steering it), symlinks never listed or followed, reads through the hardened read_repo_file, no shell, no write, no network. The system prompt states that file contents are DATA and never instructions to the model. Bounds are reported, not hidden: turns, bytes and per-file truncation land in generated_by.exploration inside the document. A threat model written from a partial survey must say so, or it silently claims coverage it does not have. Adapters without tool support still work via the single shot, recorded as survey: "single_shot_summary" rather than passed off as equivalent. read_repo_file gains an explicit oversize="raise"|"truncate" mode. Refusing is right for an override file, where half a document is worse than none; truncating is right for source during exploration, where refusing would hide every large file from the survey. Making it a parameter rather than guessing keeps both correct. LIVE SMOKE TEST (user-authorized spend), first successful end-to-end run of this feature ever. Against a 4-file repo, model claude-sonnet-4-5-20250929: survey : repository_exploration (tool loop, not the fallback) turns : 7, read all 4 files, budget not exhausted output : 3 components, every cited path verified to EXIST on disk 4 attacker profiles incl. supply_chain and adjacent -- positions the replaced 4-value enum could not express at all correctly classified git_repository_contents as untrusted and caught a planted shell injection in the manifest applier loads through load_threat_model, deriving application_type "custom:gitops-cluster-reconciliation-daemon-with-http-webhook-interface" Note for whoever runs this next: the checked-in llm config names claude-sonnet-4-20250514, which now 404s. The smoke test used a temporary config via XDG_CONFIG_HOME; the user's global config was not modified. Test fake now models BOTH protocol shapes (tool-use and text). A fake that knew only one path would have left the other untested -- the same shape of hole that let a never-executing generator sit behind a green suite. Wiring bug caught by that: the exploration path dropped `force`, so --force silently skipped the backup. Suite: 2129 passed, 0 failed, 26 skipped. ruff clean.
.
did. `-l/--language` was MarkFlagRequired with an empty default, so the
`if initLanguage == "" { initLanguage = "auto" }` branch added last commit was
dead code — empty was unreachable. `openant init <repo>` errored, every user had
to name a language, and the help examples all show `-l go`, so the natural choice
still pinned the project to one language forever.
Now defaults to auto and is not required: `openant init <repo>` detects and
stores "auto". Also corrects the flag's own help, which still described auto as
an "experimental dominance heuristic" — the behaviour it no longer has.
Adds cmd/init_language_default_test.go asserting the default and the absence of
the required-annotation. Deliberately at the flag level: the bug was invisible to
logic tests because the unreachable branch read as correct.
An `openant` invocation could install and execute code from the repository it was pointed at. `findOpenantCore` walked up from the CURRENT WORKING DIRECTORY looking for `libs/openant-core/pyproject.toml`, and its result is handed to `installOpenant` -> `pip install -e`, which executes the target's build backend. So: run openant at or below a repository that ships that marker path, with the `import openant` probe failing, and the CLI installs from — and runs code out of — the untrusted repository. OpenAnt's entire purpose is being pointed at untrusted third-party repositories, which makes the scan target an installation source. The trigger is conditional (the probe must fail first), which makes it latent rather than acceptable: a broken venv, a partial upgrade or a Python version bump is enough. Found by an independent product-architecture review and ranked P0 by a judge over two reviews; I verified the CWD walk and the pip call before acting. Resolution is now narrow and fails closed: 1. $OPENANT_CORE_PATH — an explicit operator override, validated, no silent fallback if it is wrong (a silent fallback is how the CWD path stayed reachable). 2. Walking up from the executable — trusted because the operator chose which binary to run; the scan target has no say in where it lives. 3. Otherwise an error naming the remediation, and saying explicitly that the working directory is NOT searched and why. Tests: internal/python/runtime_untrusted_cwd_test.go builds a hostile repository carrying the marker path plus a setup.py that would run on install, chdirs into it, and asserts resolution refuses it. Mutation-verified rather than assumed — restoring the CWD fallback makes it fail with the resolved hostile path in the message. Also covers the override working and a bad override failing closed. Unrelated observation while verifying: TestInvoke_LateInterruptDoesNotDiscard Envelope failed once under package-wide load and passed 8 consecutive runs after. It is timing-sensitive (a SIGINT race with a 15s deadline) and does not touch findOpenantCore. Recording it as a pre-existing flaky test, not a regression. go build/vet/test clean; Python 2129 passed.
…ilities
2129 tests establish that the plumbing works. None of them establish the product's
central claim. A judge over two independent product reviews called this the largest
unanswered question: a SAST scanner can be well packaged, deterministic, observable
and correctly exit-coded, and still be a poor scanner. The test count says nothing
about recall.
The scoring machinery for this already half-existed: experiment.py loads ground
truth and compares verdicts — but its datasets live under hardcoded
/Users/nahumkorda/ paths, so the data never shipped and the capability was
unreachable. This is a self-contained replacement.
Fixture (tests/efficacy/fixtures/webapp): three remotely-reachable vulnerabilities
behind Flask routes — command injection, SQL injection, path traversal — and three
traps SHAPED to look alarming without being reachable: a subprocess call with a
non-request argument, a sqlite call that parameterises, an f-string with no sink.
Clean code that trivially looks clean measures nothing; the traps are where a
scanner earns its precision claim.
Stage 1 and Stage 2 are scored SEPARATELY, which is the point. The product thesis
is that attacker simulation raises precision by discarding what an attacker cannot
reach. Scoring only the final output cannot distinguish Stage 2 removing false
positives (the feature) from Stage 2 removing TRUE positives (strictly worse than
never finding them, because the cost was already paid). Both look like "fewer
findings".
FIRST MEASURED RESULT (claude-sonnet-4-5-20250929, recorded in
tests/efficacy/baselines/webapp.json):
stage1 detection recall=1.0 precision=1.0
stage2 verification recall=1.0 precision=1.0
All three vulnerabilities found; all three traps correctly cleared.
Honest limit of that result: Stage 1 produced no false positives, so this fixture
CANNOT evaluate the precision-improvement claim — there was nothing for Stage 2 to
remove. "No measurable effect" here means the fixture is too easy to test the
thesis, not that the thesis is false. A fixture where Stage 1 over-flags is needed
next, and the baseline file says so rather than implying a clean bill.
Two bugs in the harness itself, both caught before reporting, both recorded in the
code because each produced a confidently wrong number:
- rglob("*.json") merged results.json and results_verified.json, which BOTH
carry `finding` and `verdict`. The two stages were mathematically guaranteed
to score identically, so the verification-effect measurement could never show
anything. Now reads the specific artifact per stage.
- `if entry.get("finding")` is true for the string "safe" as well as
"vulnerable". Every correctly-cleared trap scored as a false alarm, reporting
precision 0.5 for a run that scored 1.0 — a harness bug masquerading as a
product defect, in the direction that looks like diligence. Now compares the
value, never the truthiness.
--level all is required rather than optional: the default reachability filter
removes the trap units before analysis, and they would then score as free true
negatives — a precision the scanner never earned.
Deliberately not a pytest test: it costs money and needs a provider. Wire it into
a scheduled canary, not a merge gate. Emits a run manifest (fixture version, model,
code revision) because a number without its provenance cannot be compared to next
month's, and comparability is the entire reason to measure.
Suite unchanged: 2129 passed. ruff clean.
The user asked for "document well, kg, navigability" many turns ago. It was never started, while four audit rounds, a nine-reviewer audit and three external consultations ran. Two independent drift reviews reached the same verdict — the reviewing had become the work — and both ranked this first. ARCHITECTURE.md — how a scan actually flows, the Go/Python contract, and a "where to stand to change something" table. Includes the caveat that the merge-at-dataset design rests on a premise (language-agnostic prompts) that holds for Stage 1 and 2 but NOT for the enhancer, whose sink list and entry-point examples are Python/JS only. The architecture survives that; the docstring asserting it did not. AFFECTED_PLACES.md — the "trace all the affected places in the code (remote attacker, and such)" deliverable from the original request. It had existed only in an out-of-repo plan file, which is not a project artifact. Now a maintenance map marking each site changed / deliberately-unchanged / still-assumes-the-old-model / planned-but-unbuilt. It records a finding neither I nor the first drift review caught: **plan risk R5 accepted the prompt-injection gap CONDITIONAL on four visibility measures, and only one shipped.** No SHA-256 of the threat-model file, no "context supplied by repo-controlled file" report header, no named xfail. Verified by grep. A conditional acceptance whose conditions went unmet is an unmitigated risk, not an accepted one. knowledge-graph.json + .md — the module graph derived from real AST imports rather than hand-drawn, so it cannot quietly stop describing the code. 114 modules, 271 internal edges, 117 of them deferred (written inside functions) — that last number is a diagnosis: it is how an import cycle gets managed rather than removed. internal notes.md rewritten. It was not merely stale but actively false — "0 commits made / nothing is committed" and "2 failed, 1988 passed" against ten commits and 2129 passing. A next session would have inherited that as fact. Preserved the still-true static-scanner results (CodeQL 0 findings, Semgrep 13, Bandit 25) rather than discarding evidence along with the falsehoods. Also published as a browsable page for navigability; the repository remains the source of truth. Practical note recorded in all three documents: docs do NOT go in docs/ — it is gitignored at .gitignore:11, which is why twelve docs/*.md links in this repo point at files that do not exist. Writing there loses the work silently. No code changed. Suite unchanged: 2129 passed, 26 skipped.
…y directory Found by running the suite from the repository root while verifying the docs commit: 2 failed there and 2129 passed from libs/openant-core. The suite was cwd-dependent and nobody knew. tests/test_threat_model_hardening.py did `sys.path.insert(0, "tests")` twice, then imported a sibling fixture module. Two defects in one line: - The path is RELATIVE. Invoked from anywhere but libs/openant-core it does not resolve, and the two tests fail with ModuleNotFoundError while 2127 others pass — which reads as a product regression rather than a harness bug. That is exactly how it presented. - The entry is never removed, so tests/parsers/ shadows the real top-level parsers package for the rest of the process. The suite stayed green only because alphabetical collection happened to run the victim first. This was flagged in an earlier audit round and left unfixed; it is the same class of bug commit 2f68162 exists to fix. Now resolved with importlib against a path derived from __file__: no global state, no ordering dependency, no cwd dependency. Verified in all three configurations, each with a real exit code rather than a `tail`-swallowed one: repo root exit=0 2129 passed libs/openant-core exit=0 2129 passed hardening BEFORE parse_multi exit=0 31 passed (previously the ordering that broke) Process note, recorded because it caused the preceding commit to ship unverified: I ran `pytest ... | tail -2`, which returns tail's exit status, so `&&` proceeded over a RED run and the commit body asserted a passing suite. The claim happened to be true from the canonical directory, but the verification did not establish it. Pipe to a file and check `$?`, or never chain a commit behind a piped test run.
…directories
An untrusted repository could exfiltrate arbitrary host files. Reproduced end to
end before the fix: a repo containing `leak.py -> <outside>/secret.py` and
`abs.py -> /etc/hosts` yielded scanned files ['abs.py','leak.py','main.py'], and
FunctionExtractor.read_file returned the out-of-repo content verbatim (a planted
API key). Whatever the scanner returns reaches dataset.json, which is sent to the
model provider. Reachable with one committed symlink.
Every guard in the tree was DIRECTORY-only. repo_walk calls safe_to_descend inside
the S_ISDIR branch; the file branch takes its mode from entry.stat(), which
FOLLOWS symlinks, so a file symlink reports S_ISREG and went straight to on_file.
Same shape in the bespoke parsers/python scanner. Affects python, zig, c, ruby,
php — i.e. the shared walker and its one hold-out.
Adds utilities.file_io.safe_to_read, the file-side counterpart of
safe_to_descend, applying the SAME policy: resolve, allow inside the repository or
its immediate parent, refuse beyond. Deliberately NOT "refuse every file symlink"
— parsers/zig carries a regression test whose fixture is source reachable only
through a symlink, and for a SAST tool a false negative is the worse direction.
The security property is "never leave the repository", not "never follow a link".
An external reviewer recommended refusing both files and directories outright;
that would reverse the operator's explicit parent-scoped decision for directories
and break that fixture, so it was not adopted. Residual risk is unchanged and
stated in the docstring: anything under the parent is reachable.
WHY FIFTEEN REVIEW ROUNDS MISSED IT. The test named
test_traversal_does_not_ingest_files_outside_the_repository asserts exactly this
invariant and has always passed, because it was blind three separate ways:
1. Its fixture built only a DIRECTORY symlink, certifying one shape of an
attack with two shapes.
2. Its assertion matched on the path STRING ("host_secret" in f), which can
never fire for a file symlink — the scanner reports the in-repo name it
walked (src/leak.py) and the target name appears nowhere in it.
3. _iter_scanned_paths returned the scanner's row DICTS, and the caller did
str(f), so every comparison ran against "{'path': 'src/leak.py', ...}" — a
string that matches no filename and, joined to the repo root, resolves
INSIDE it. The check could not fail whatever the scanner did.
Each layer independently guaranteed a pass. All three are fixed, and the fixture
now targets a path beyond the parent (mkdtemp), because a target under tmp_path is
a sibling of the repo and is permitted by policy — a fixture that lands inside the
allowed zone asserts nothing.
Negative control added: a parent-scoped symlink (src/sibling.py) must STILL be
scanned. Without it, "refuse every symlink" passes the escape assertion while
silently dropping legitimate vendored source. The escape assertion itself was also
corrected — it tested "outside the repo", stricter than the implemented policy,
and flagged its own negative control.
Mutation-verified: replacing both guards with `if False:` turns the test red;
restored, and grep confirms zero residue.
Suite: 2129 passed, 26 skipped, exit 0 (unpiped, $? checked). ruff clean. Go clean.
`pip install openant` produced a distribution that could not start. Proven before the fix by building a wheel, installing it into an empty venv, and running from outside the checkout: ModuleNotFoundError: No module named 'experiment' (import core.analyzer) Two defects, one cause — shipped code depending on files that are not shipped. 1. `core/analyzer.py` imported analyze_unit, parse_response and _normalize_result from `experiment.py`, a research harness that is a loose top-level module, not one of the seven packages in pyproject. Production depended on research code, and the dependency ran the wrong way round. Moved those three functions (~190 lines) into `core/analysis_core.py`. They were clean to extract: AST analysis showed they depend only on each other, stdlib, and shipped packages (utilities/, prompts/). `experiment.py` now imports them FROM core, inverting the direction so research depends on product. 2. `core/reporter.py` subprocessed `generate_report.py` and `export_csv.py` by source-tree-relative path with cwd=_CORE_ROOT. Neither shipped, and `openant report --format csv|html` is reachable from the CLI (cli.py:670), so both failed on an installed wheel. Moved into the already-shipped `report/` package (git mv, history preserved) and invoked as `python -m report.csv_export` / `report.html_report`. Dropped cwd=_CORE_ROOT: depending on the working directory is what made them unrunnable when installed. Deliberately did NOT do the C1 namespace rename in this commit. An external adjudication suggested folding it in; blast-radius analysis put it at 469 import sites across 206 of 267 test files, and — the deciding factor — the producer-parity guard in tests/test_hostile_repo.py resolves paths via __file__ and pytest.skips when a file is missing, so a rename without moving tests in lockstep would SILENTLY skip all 8 parametrizations and evaporate the boundary-marker security invariant into a green run. Namespacing a distribution that could not import was premature; correctness first, namespace as its own PR. WHY 2129 TESTS COULD NOT SEE THIS. The editable install is a bare .pth containing the checkout root, so every import resolves from source regardless of what the wheel contains. The existing wheel test asserted only that config/languages.json appeared in the archive — a substring check that passes wherever the file lands. A packaging defect is structurally invisible to a suite that never leaves the checkout. New test builds a wheel, installs into an empty venv, and imports core.analyzer, both report generators and the language registry from a directory OUTSIDE the repository. It fails against the pre-fix tree. Also caught in passing: my first pass at updating test imports used a regex broad enough to rewrite `from utilities.dynamic_tester.reporter import generate_report` — an unrelated function that merely shares a name — producing syntax errors. Reverted and redone as exact-line replacements; that call site is untouched. Verified: installed wheel imports and runs from outside the checkout, 7 languages resolve. Suite 2129 passed, 26 skipped, exit 0 (unpiped, $? checked). ruff clean. Go build/test clean.
Operator decision, replacing the parent-scoped policy. Symlinked files and directories are now refused outright rather than followed when they resolve inside the repository or its parent. WHY THE STRICTER RULE. The parent-scoped policy was chosen to admit monorepo sibling-vendoring, and it worked, but it requires containment arithmetic on every entry and can be defeated by a target that resolves somewhere unexpected. "Never follow a link" needs no arithmetic and has no such edge. Given the payload here — a followed link puts host-file contents into dataset.json, which is sent to the model provider — the simpler property is worth the coverage it costs. WHAT IT COSTS, MADE VISIBLE. Source reachable only through a symlink is now unscanned code. That is precisely the false negative a SAST tool can least afford, so the refusal is COUNTED, not silent: `symlinks_skipped` plus up to five `symlink_examples` land in the structured scan statistics, not only on stderr which CI discards. All five scanners report the same key — verified empirically across python/c/php/ruby/zig rather than assumed from the shared walker. That verification caught a 1-of-5 gap in the making: the four shared-walker scanners inherited the counter automatically, but parsers/python has its own walk and was folding refusals into `directories_excluded`, hiding them among ordinary prunes like node_modules. Same defect shape this codebase keeps producing; caught by running all five rather than reasoning that the shared walker covered them. TESTS INVERTED DELIBERATELY, NOT DELETED. Two asserted the old behaviour and both were right when written: - parsers/zig/test_repository_scanner_symlink_dir.py asserted a symlinked directory IS followed, because a .zig file reachable only that way had been silently dropped. It now asserts refusal AND that the gap is counted — the original concern is honoured by converting a silent loss into a visible one, not by dismissing it. - test_hostile_repo.py's negative control asserted a parent-scoped link stays scanned, guarding against over-refusing. Under refuse-all it instead proves the refusal is recorded. safe_to_descend/safe_to_read keep their `repo_real` and `seen` parameters though both are now unused, documented as such: the containment arithmetic is the first thing anyone needs if this is relaxed again, and the cycle guard has nothing to bound once nothing is followed. Mutation-verified: replacing both guards with `return True` turns 4 tests red across two files; restored with zero residue. Suite 2129 passed, 26 skipped, exit 0 (unpiped, $? checked). ruff clean. Go clean.
The wheel shipped things it should not, and shipped someone's home directory. Removed from the repo (regenerable artifacts, zero production references — each token grepped separately across the shipped packages to confirm): - report/test_output/disclosures/DISCLOSURE_01_CREDENTIAL_THEFT.md — a third-party exploit writeup (paperless-ngx, CWE-639) WITH reproduction steps. Shipping an exploit disclosure to PyPI is a disclosure problem, not clutter. - report/test_output/SUMMARY_REPORT.md — a stale sample; production writes SUMMARY_REPORT.md to a runtime path (core/scanner.py:910), not this committed copy. - report/test_data/paperless_ngx_pipeline.json — a captured pipeline. Belt-and-suspenders: a wheel `exclude` for **/test_output and **/test_data so a future test run cannot re-leak generated output into the distribution. Personal paths that SHIPPED, replaced with opt-in env vars (all default-inert): - utilities/context_corrector.py:644 and utilities/context_reviewer.py:363 — `/Users/nahumkorda/code/dvna` in __main__ demo blocks, in shipped library code. - parsers/javascript/test_pipeline.py — two `/Users/nahumkorda/...` IDE-run constants, dead unless IDE_MODE=True, but test_pipeline.py ships (verified: it is in the wheel, excluded from pytest collection, not imported by production). The register named experiment.py for this; that was wrong in both directions — experiment.py does NOT ship (it is the ModuleNotFoundError from the prior commit), and the paths that DO ship are the utilities/ ones it did not name. TWO GUARD TESTS, closing the gap that let this through: - Wheel-content test builds a wheel and asserts, two-sided, that the junk is absent AND the runtime files remain (core/analyzer.py, report/generator.py, report/prompts/disclosure.txt, config/languages.json). An over-broad exclude that dropped a runtime module would pass a junk-absent check and break install. - The hardcoded-path sweep only scanned tests/ — which is exactly why a personal path in utilities/ shipped unnoticed. Extended to the seven shipped packages (390 files) with a vacuity floor asserting >100 sources scanned, so a glob that stops matching fails loudly instead of passing on an empty set. Verified: fresh wheel contains no test_output/test_data/DISCLOSURE/paperless entries and keeps disclosure.txt (a live prompt) and languages.json. Suite 2248 passed (+119: the parametrized source sweep), 26 skipped, exit 0 (unpiped). ruff clean. Go build clean.
…on -m A cwd-shadow RCE I introduced in the wheel-repair commit (24cc639), found by an independent review of that commit. core/reporter.py runs `python -m report.csv_export` / `report.html_report`. `-m` prepends the process working directory to sys.path[0], and the engine inherits the user's shell CWD — which in the ordinary `git clone X && cd X && openant report` flow is INSIDE the scanned, untrusted repository. A hostile `report/__init__.py` there shadows the installed package and executes on import: repo-authored code execution, the same class as the working-directory `pip install -e` hole closed in 5de6cf0. The exact invariant-destroying shape this codebase keeps producing. The prior form passed `cwd=_CORE_ROOT` for script-path resolution; that INCIDENTALLY pinned the child's sys.path away from the untrusted CWD. Removing it to fix the wheel dropped the accidental guard. Reproduced: with a hostile report/ package in CWD, the unguarded `python -m report.csv_export` runs the attacker's __init__; under -P it resolves the installed package. Fixed at all three -m sites: - core/reporter.py:492,527 — the two report subprocesses I introduced. - apps/openant-cli/internal/python/invoke.go:42 — `python -m openant`, the PARENT engine invocation. Pre-existing (not from commits 1-3), same class: a hostile `openant/` dir in the scanned repo would shadow the whole engine. Setting -P here also propagates via env to the report subprocesses. -P verified not to break anything: the editable install and `openant --help` both resolve under -P (installed code lives on the .pth / site-packages, which -P does not touch; only the untrusted CWD entry is removed). tests/test_report_subprocess_no_cwd_shadow.py, three tests: -P refuses the shadow; WITHOUT -P the shadow executes (the control proving -P is load-bearing, not decorative); and reporter.py emits -P at both call sites. Mutation-verified: dropping -P from reporter reddens the structural test. Suite 2252 passed, 26 skipped, exit 0 (unpiped). Go build + internal/python test clean.
TestInvoke_LateInterruptDoesNotDiscardEnvelope slept a hardcoded 400ms before delivering SIGINT, then asserted the child had already printed its envelope. Under CPU load the child hadn't — reproduced at 27/30 failures with 8 background `yes` processes, every failure the exact symptom (`got Status="interrupted", want "success"`). It is a TEST bug: the production precedence (a parsed envelope wins over a late interrupt) is correct; the sleep was a wall-clock guess, not synchronisation. Fix per adjudication: keep signalling the runner PID (the test exists to exercise Invoke's signal.Notify -> forward path; signalling the child's own process group would bypass exactly that and assert nothing), and replace the sleep with a sentinel-readiness barrier. The child touches a sentinel file ONLY after it has installed its trap and printed the envelope; the test polls for it (bounded 5s, failing loudly on timeout) before signalling. That gates on the precise preconditions the scenario needs, and transitively guarantees signal.Notify is registered (the child can only reach the touch after cmd.Start returned). The sibling TestInvoke_InterruptedFlagHasNoRace was measured robust under the same load, but gets the same sentinel pattern for hygiene so neither relies on a fixed sleep. Verified: 30/30 under 8x CPU load (old form: 27/30 FAIL under identical load). Mutation: removing the `touch` makes waitForSentinel time out loudly — proving the barrier is load-bearing, not a longer sleep. Full go test ./... clean.
scan.report.json hand-built its summary and dropped context_source and every multi-language field that ScanResult.to_dict() carries, so a scan run under a repo-supplied threat model, or a degraded/merged multi-language scan, produced a report byte-indistinguishable from a clean single-language one under the default security model. build_pipeline_output had no context_source at all (Plan DoD knostic#9). Both artifacts now carry context_source, the multi-language fields (per_language, parse_errors, excluded_languages, degraded), and an aggregate coverage block. R5 provenance for a repo-supplied threat model: threat_model_sha256 over the raw file bytes (absent, never the empty hash, when no threat model), the previously discarded warn_permissive_threat_model output as threat_model_warnings, and a deterministic non-LLM report banner a hostile threat model cannot suppress. CHANGELOG records the accepted prompt-injection known risk (R5). Coverage uses a presence PROBE, not a hardcoded language allowlist: a language is instrumented iff its scan-result statistics carry a coverage count key, else it is disclosed in languages_without_coverage_data. The probe fails safe (an uninstrumented parser is disclosed, never summed as a false symlinks_skipped:0); a stale allowlist would fail dangerous. Fixes a latent bug where the Python parser's scan() reset dropped the coverage-counter seed, so a clean Python scan was indistinguishable from an uninstrumented one. JavaScript and Go do not yet instrument coverage and appear in the disclosure list until a follow-up commit. Verified on disk, not via to_dict(): tests read the actual scan.report.json / pipeline_output.json files; a mutation reverting _write_scan_report to the hand-built summary reddens the disk tests while to_dict() stays green, and a mutation disabling the coverage disclosure reddens the false-0 tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Go parser followed file symlinks out of the scanned repository. filepath.Walk lstat's each entry, so a symlinked file `leak.go -> /outside/secret` had IsDir()==false and a .go extension, was added to the file list, and was read THROUGH the link by the extractor — exfiltrating host files into dataset.json and thence to the model provider. This is the same class the Python engine's refuse-all symlink policy already closed; it was never applied to the Go binary, and no executable Go/Node containment test existed to catch it (the shared contract test's docstring falsely claimed those runtimes were covered elsewhere — which is how the hole shipped). scanner.go now refuses every symlink (file OR directory) inside the repo, with the root exempt (the threat is attacker-committed links inside the repo, not an operator scanning a symlinked path — matching the Python walker). An unreadable directory is now a counted directories_unreadable gap instead of a silently swallowed error. New snake_case coverage fields on ScanStatistics match the Python-family parsers so the coverage aggregator's presence probe reads them. Proven by execution: scanner_symlink_test.go asserts a leak canary is refused (symlinks_skipped=2) with a clean-repo negative control (present at 0) and an unreadable-dir case; a mutation disabling the guard re-scans the symlinked file. test_pipeline.py now rebuilds the Go binary on staleness (any .go source newer than the binary), not just absence — rebuild-if-absent left this fix inert on any machine with a cached (gitignored) binary. JavaScript was already contained (isSymbolicLink checked before isDirectory) but folded refusals into camelCase directoriesExcluded, invisible to the aggregator. It now emits snake_case symlinks_skipped/directories_unreadable (no double-count) with a new TestSymlinkContainment test. With Go and JS instrumented, both drop off languages_without_coverage_data automatically. Corrected the false-comfort docstring in test_scanner_contract.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The harness reported recall=1.0 precision=1.0 against a fixture whose docstrings captioned each unit `VULN` / `NOT A VULN`. Re-derived by parsing the fixture: that text reaches the analysis prompt via the unit's code.primary_code AND metadata.docstring, so the "measurement" only proved a model can read an English label — it could not support a recall, precision, or external-validity claim. Blinded the fixture: removed the verdict docstrings, and replaced the all-vulnerable handlers.py / all-clean maintenance.py split (a file grouping that itself signalled the answer) with two neutrally-named modules, app_a.py and app_b.py, each mixing reachable-route vulnerabilities and unreachable-plain-function traps. Expected outcomes live only in the sidecar ground_truth.json, which the scanner never sees (it analyses only src/*.py). Verified by re-parse: no verdict marker remains in the dataset, and all six oracle units are present under --level all. Deleted the fabricated baseline baselines/webapp.json — its code_revision named 5de6cf0, a tree that (verified via git ls-tree) contains no score.py, so the recall=1.0 it recorded could not have been produced by the scorer it claims. Rewrote score.py as a strict smoke checker, not a metric generator: a binary per-unit contract (flag the three planted vulns, clear the three traps) with no averaged number. The silent-failure modes are now errors that exit non-zero — a missing results.json (scan failure, previously scored recall=0.0), an expected unit absent from the results, a flagged id the oracle does not cover, and a duplicate id. The pure checking functions (read_results, index_results, evaluate) take no provider and are unit-tested offline in tests/test_efficacy_score.py (13 tests); a mutation restoring either silent-zero mode reddens exactly its targeting test. Sequenced before the provider-registry work so the fabricated baseline's model id cannot contaminate that validation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the rescope. source removed the docstring leak, but the oracle (ground_truth.json, the answer key with `vulnerable: true` per unit) still sat at the root of the scanned directory. score.py's scan runs app-context generation, and its repo_explorer agent has list_dir / read_repo_file tools that can reach ANY file under the scanned tree — not just .py source. So "the scanner never sees it because it only parses .py" was a false guarantee: the survey agent could read the answer key and fold it into the context that shapes every unit's analysis. Moved the oracle to tests/efficacy/oracles/<fixture>.json, outside the scanned directory, making "never shown to the scanner" structural rather than a property of file-extension filtering. score.py gains oracle_path(); a guard test asserts the oracle is outside the fixture tree and that no answer-key-shaped .json remains under it (a stray json in the scanned tree reddens it). Verified: gather_context_sources over the fixture now yields no answer-key content, and no .json remains under the scanned tree. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive scaffolding for the provider-model registry cutover — no consumer reads
it yet, so nothing changes at runtime and the existing model_config pricing is
untouched. Landed alone because it is purely additive and cannot create the
half-migrated intermediate state (empty wizard default / corrupt pricing) that
the consumer cutover in the following commit must stay atomic to avoid.
config/models.json is the single source of truth for provider model IDs, status,
and pricing, read by both runtimes exactly like config/languages.json. Each
record carries status (current | retired | unknown) plus source + retrieved, so
no provider fact is asserted without a receipt. The 22 existing models are
migrated with their shipped prices; the models OpenAnt's own source labels
retired (claude-opus-4-20250514, claude-sonnet-4-20250514) and the contested
claude-opus-4-6 carry price:null and assert nothing about liveness.
core/model_registry.py mirrors core/language_registry.py (upward config search,
lru_cache, fail-loud require_models). Two invariants it exists to hold:
- pricing_map OMITS null-priced records — a retired/unknown model is absent from
the map, never emitted as a truthy {"input":0} dict, so it can never resolve to
a silent $0 (verified: claude-opus-4-6 is absent from pricing_map).
- require_models raises an installation-problem error on a missing config rather
than degrading to {} (which would price every model at $0). models.json feeds
no argparse choices, so there is no --help path to keep alive.
Ships in the wheel via a force-include stanza mirroring languages.json; the slow
installed-layout test builds a wheel and asserts models.json is inside it.
Structural tests (shape, unique ids, price-nullness-matches-status, and configured
default phases resolve to a non-retired entry) replace the eternal DEAD_MODEL_IDS
assertion in a later step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the config/models.json cutover begun in 050c981 (step 1). Python
pricing and the Go setup wizard's model prefills now resolve from the shared
registry instead of hardcoded literals. Lands atomically: there is no safe
intermediate state (a half-cut wizard would prefill a retired id, or pricing
would read $0).
Two invariants held throughout (both proven on live code before commit):
- A null price is NEVER $0. pricing_map() omits retired/unknown records, so a
lookup misses -> record_call's warn+$0 path, never a truthy {"input":0} dict.
- A missing config fails LOUD, never silent $0. require_models() raises; the new
lazy pricing descriptor lets that RuntimeError propagate (it never raises
AttributeError, so helpers.lookup_pricing's getattr(...,{}) cannot swallow it).
Python (step 2):
- New utilities/llm/_pricing.py: LazyProviderPricing descriptor resolving
pricing_map(provider) on first access (class + instance), not at import.
- 3 adapters, llm_client.MODEL_PRICING and the utilities package re-export now
serve pricing lazily (PEP 562 __getattr__); record_call calls pricing_map
directly (a bare module name won't trigger __getattr__).
- model_config.py: the 3 pricing dicts removed; model-ID constants kept as the
compile-time defaults, now validated against the registry.
Go (steps 3-4):
- New internal/models/registry.go: FindConfig/Load (fail loud) + DefaultModel
(validated current) + KnownModels (registry-sourced). setup.go's per-phase
defaultModels map and knownModels hint list — which prefilled the RETIRED
claude-opus-4-6 / claude-sonnet-4-20250514 so a fresh config 404'd — are gone.
- internal/models/models_test.go: network-free proof every phase x provider
default resolves to a current id (never runs the billed wizard probe).
Tests (step 5):
- Deleted test_builtin_model_ids_current.py (eternal DEAD_MODEL_IDS list); its
protection is preserved structurally by test_model_registry
(test_configured_default_phase_models_resolve_to_non_retired) and strengthened
by a new ..._are_current companion, without naming which ids are dead.
- Migrated test_model_config_centralize (pricing now from the registry; locks
that the dicts left model_config), test_token_tracker (retired-id cost
fixtures repointed to current ids of identical price), and the Go
pr69_round3 hint test (now reads models.KnownModels).
- Added a positive-content assertion to the drift guard so it can't pass
vacuously on two empty maps.
Amendments folded in from the pre-cutover review (test_token_tracker retired-id
gate breakage; the helpers.py RuntimeError-vs-AttributeError trap; the
report/generator MODEL_PRICING consumer). config_test.go's retired-id strings
are config-roundtrip fixtures (not prefills) and left as-is, out of scope.
Gates (all green, re-run at commit time):
.venv/bin/python -m pytest tests/ -q -> 2285 passed, 28 skipped, 0 failed
.venv/bin/ruff check . -> All checks passed
go build -o bin/openant . && go vet ./... && go test ./... -> ok (cmd, internal/models, ...)
(parsers/go/go_parser) go test ./... -> ok
git diff --cached --stat -> 16 files changed, 526 insertions(+), 255 deletions(-)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ct stale exclusion docstring D3b: core/reporter.py set finding location.function = the full route_key (e.g. "src/app_a.py:render_snippet") while location.file already carried the file — a redundant duplication in every finding. Make function the exact complement of file so file + ":" + function == route_key still round-trips. Coordinated fix in openant/cli.py (the dynamic-test report-data bridge) which reads location.function AS a route_key: it now reconstructs route_key = file + ":" + function, so dynamic-test results keep attaching (verified against the running code, not the sol/fable claim — fable had missed this consumer; sol caught it). C9: core/language_selection.py `LanguageSelection.excluded` docstring claimed exclusions are "computed and discarded ... still silent" — stale/false: the scanner threads excluded_languages into the scan report (scanner.py:337,1153). Corrected the comment (doc-only; behavior already pinned by test_threshold_exclusions_are_loud + test_artifact_serialization_contract). D4 (library_seed_ids ignoring Ruby/PHP visibility) considered and REJECTED: excluding `private` from library-mode seeds would drop orphan private methods reachable only via unmodeled dynamic dispatch (send(user_input), framework callbacks) — a false negative in a security tool. The over-seed is the recall-safe intended behavior; the real cost fix is a visible budget ceiling (), not hidden seed-pruning. No code change. Tests (TDD RED->GREEN): - new test_reporter_exploit_path_shape.py::test_location_function_is_bare_qualified_name (equality + file:function round-trip, not the substring checks that lock in nothing) - updated 4 test_disclosure_source_fidelity.py assertions that encoded the buggy ":name" shape -> exact "name" Gates: .venv/bin/python -m pytest tests/ -q -> 2286 passed, 28 skipped, 0 failed ruff check <changed> -> All checks passed free rebuild of pipeline_output on the oracle's real results -> all 3 findings have a bare function; round-trip == original route_key. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
: investigated first-hand — repo-supplied threat models suppress findings via TWO surfaces: the deterministic suppress_local_only flag (already tested) AND free-text injection of not_a_vulnerability/purpose/attacker-profiles into both stage prompts (untested, the real R5 teeth). Decision: de-scope the paid --ab-context A/B instrument (its output is a non-deterministic count of an LLM-mediated effect; the actionable remedy is trust-hardening, not measurement). Documented in THREAT_MODEL_SUPPRESSION_SURFACE.md. add the missing multi-turn-loop test for explore_repository — pins the previously UNVERIFIED nudge fix (old code sent ToolResultBlock(tool_use_id='nudge') = a guaranteed 400) plus finish/tool-result/exhaustion paths. 4 tests, fake adapter, no network. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…de stale API-key ACTION header exercise _finalize()'s non-dict generated_by guard (no test hit the else-branch). 5 tests, fake binding, tmp file, no network: string/list coerced (not AttributeError-crashed), dict preserved+stamped, missing created, single-shot label. Doc-hygiene: internal notes's '⚠ ACTION REQUIRED: rotate key' header is superseded (operator accepted + de-gated, 2026-07-27); disclosure retained as historical record, status appended. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
. Both runtimes' language detection was tested separately with independent
expectations (could drift). Bind them to ONE shared golden
(config/testdata/detection_parity.json): a Python harness (8 passed) and a Go harness
(7 subtests passed) compare the ranked detected-language list / error outcome across 7
drift dimensions. Parity holds.
Surfaced (see A7_DETECTION_PARITY.md): D6 = a LIVE Go<->Python divergence on a dot-only
filename (Go filepath.Ext('.py')=='.py' counts it; Python splitext=='' ignores it) —
confirmed both halves first-hand, deliberately NOT pinned pending a semantics decision.
Also corrected sol's 'no threshold' claim (it lives in language_selection.py, a
selection-stage concern, not detection).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
43 internal .md (BUGHUNT_*, PILOT_*, P1-P4, TM_*, COLLISION_CHECK_*, KB_*, internal notes, internal notes, STATUS_LEDGER, pr-bodies/*, ...) + bughunt-repros/ + 3 internal .json untracked via git rm --cached and added to .gitignore, so they stay on disk but leave the branch/PR. Product docs kept: ARCHITECTURE.md, knowledge-graph.md/.json, OPENANT_THREATMODEL_TEMPLATE.md, THREAT_MODEL_AUTHORITY_DESIGN.md, webapp fixture README/oracle, config/*.json. Classification: independent + judge verified (keep 5, remove 43). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…l warning + round-trip, non-str finding
- GO-1 html_report: escape verdict badge (model-supplied finding text) — HTML/JS injection
- GO-2 html_report: coerce null reasoning ((x or '')[:n]) — report-build crash on reasoning:null
- GO-3 html_report: surface 'error' (unanalyzed) verdict in stat card + remediation msg — silent-FN
- TM-1 threat_model: case-insensitive trust compare in warn_permissive — 'Trusted' evaded warning
- TM-2 threat_model: adaptive-length json fence + backref parse — backtick in a field broke round-trip
- ML-1 analysis_core: str(finding) coercion — non-string finding no longer AttributeErrors
- TM-3/TM-4 repo_explorer: list_dir truncated flag; dict(input or {}) on finish
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Round-1 GO-3 fix added the Errored stat card + remediation message but the pie charts still iterated FINDING_VERDICT_ORDER (which excludes 'error'), so errored units stayed absent from the charts — a partial fix. Now append any producer verdict not in the canonical order to verdict_order so 'error' is charted. Regression test strengthened to assert chart inclusion, not just the stat card. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ot arbitrary verdicts The prior GO-3 completion (d78b2dc) appended ALL verdict_counts keys to verdict_order; a model-supplied verdict string then flowed unescaped into the json.dumps chart labels, re-opening the GO-1 injection (caught by test_go1_verdict_badge_is_escaped). Now append only the known ERROR_VERDICT literal; unknown verdicts remain in the escaped findings table. +test_go3_chart_does_not_reintroduce_injection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- R2D-5 (analysis_core): a non-string finding now maps to ERROR (countable by the error/manual-review accounting) instead of a garbage verdict from str(finding).upper() that silently escaped it. - R2D-3 (threat_model): parse regex now requires the closing json fence on its own line at exact length (^\1[ \t]*$, MULTILINE) so a mixed 3-tick-decoy + 4-tick-real doc no longer desyncs and swallows the real block. + regression tests (non-str->ERROR, string-still-maps, mixed-length-fence). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sh, child leak - R2A-1 (agent.py): degenerate-exit paths now record_call + carry input/output/cost, so the tracker and per-unit metadata don't undercount spend (was total_tokens>0 but cost=0). - R2B-1 (schemas.py): ParseResult.to_dict() now includes the derived 'degraded' flag (asdict omitted the @Property) — parity with ScanResult; parse envelope no longer hides a degraded/partial parse. - R2B-2 (generator.py): disclosure safe_name falls back to id when short_name is null — presence-only validation let a null short_name crash disclosure generation mid-run. - R2C-2 (invoke.go): reap the child process with cmd.Wait() on a stdout read error (no leak). + regression tests (degenerate records spend; parse envelope includes degraded). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- R3A-1 (threat_model): normalize CRLF->LF before the json-fence match. The R2D-3 ^\1[ \t]$ closing anchor (MULTILINE) matches before \n but not \r, so a Windows/ autocrlf checkout of a valid threat model aborted the scan. Regression from R2D-3. - R3B-1 (dynamic_tester): reconstruct the TestEvidence array on checkpoint resume — it was serialized but dropped on restore, so a resumed CONFIRMED finding silently lost its exploit-proof (command_output/file_read/http_response) evidence. + regression tests (CRLF parse, evidence survives resume). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- R4B-1 (application_context): the untrusted-boundary check matched with exact equality
(== 'untrusted'), tolerating case but not qualifiers — so an LLM-written level like
'untrusted (attacker-controlled)' slipped past and re-enabled suppression of the
untrusted-input bug class (silent FN in the suppression backstop). Now substring-matches
'untrusted' (case/qualifier tolerant); 'trusted'/'semi-trusted' correctly don't match.
- R4B-3 (application_context): coerce a non-dict trust_boundaries (LLM hallucination) to
{} in __post_init__, so it can't AttributeError at the unguarded analyze-phase prompt build.
+ regression tests. (R4B-2 repo-supplied-override trust deferred — documented design limitation.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Push-relevance sweep (independent + auditor + judge): knowledge-graph.json/.md are auto-generated/regenerable AST-derived artifacts with 0 test coupling and their durable content lives in ARCHITECTURE.md; pilot-knowledge-graph.md is an internal pilot artifact (missed by the earlier scrub). Untracked via git rm --cached + gitignored (kept on disk). Also removed ARCHITECTURE.md's now-dangling 'See also' bullets (AFFECTED_PLACES.md + knowledge-graph.json). PRINCIPLES.md kept (judge: high-value parser-authoring guidance). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Test-exhaustiveness audit (independent+tester-expert+auditor+judge+sol) found coverage effectively exhaustive (98.8% of fork public fns) and tests REAL not fake; filled the concrete gaps: - repo_explorer MAX_TOTAL_BYTES budget exhaustion + list_dir truncation flag (cost/visibility bounds the module's 'bounds not optional' contract left untested) - threat_model provenance carry-through (ctx.source_sha256 tamper-evidence + permissive_warnings self-whitelisting audit trail reach the context) - R2B-2 null short_name disclosure (was fixed but had no regression test) - GO-2 strengthened from assertion-free no-crash to assert the finding actually renders Deferred: R3B-1 real-resume-path test (needs a run_dynamic_tests harness; low-sev, to_dict partially guarded). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…the WHY)
Pre-upstream archeology scrub (sol+fable): the fix comments referenced private session
round indices (R2A-1/R2B-*/R2C-2/R2D-*/R3A-*/R3B-*/R4B-*/GO-*/SW-1/TM-2 and pre-existing R5)
that are dead references upstream. Stripped the index tokens in place and reworded round-only
phrasings ('the GO-3 surface' -> 'the silent-false-negative surface') while keeping every WHY/
invariant intact (e.g. 'must not re-enable suppression', 'don't revert to == match'). Comment-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…test threat_model_exists() had 0 production callers — the scanner and generator decide "does a model already exist" via repo_path_state() at the point of use; the only reference was this helper's own unit test. Confirmed dead by independent + auditor + judge review during upstreaming prep. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Removes obvious narration comments (# Statistics, # Run scan, # Build the unit, etc.) across the multi-language parsers and a few utilities. Comment-only; no executable line changes. Load-bearing WHY/safety rationale is deliberately kept (the .ts-fence bug rationale in reporter.py, the cross-parser "ported from zig" provenance in python/repository_scanner.py, and the issue-knostic#65 migration pointer in llm_client.py were restored, not scrubbed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…portability
- init: emit "no supported source files" for a source-less repo instead of
formatting a nil error as "<nil>" (apps/openant-cli/cmd/init.go)
- test_go_cli: TestInit asserts the multi-language init output ("Detected: X",
stored mode "auto")
- repo_walk, dataset_merge: emit "/"-separated relative paths in scan artifacts so
they are stable across OSes (a Windows "\" regression)
- test_hostile_repo: skip the POSIX-only cases (os.mkfifo, dir_fd deep-nest) on win32
- test_no_banned_patterns: POSIX allowlist keys (as_posix)
- test_model_config_centralize: default to the in-tree core (previously resolved
out-of-tree and passed without asserting)
- experiment.py: resolve test-repo paths from $OPENANT_TEST_REPOS, not hardcoded paths
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TestChangedFilesNonASCII creates a file whose name contains a newline, which Windows filesystems forbid, so the fixture cannot be built there. Skip on Windows to match the sibling POSIX-only tests in this package (invoke_test.go, config_perms_test.go, runtime_test.go). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gadievron
marked this pull request as ready for review
July 30, 2026 07:12
gadievron
requested review from
dgeyshis,
shahar-davidson and
sounil
as code owners
July 30, 2026 07:12
gadievron
added a commit
that referenced
this pull request
Jul 30, 2026
Adds a tree-sitter-based Swift parser (repository scan, function/type extraction, call-graph construction with overload/trailing-closure/actor resolution, reachability filtering, analysis-unit generation), integrated into the post-#199 registry architecture: a single config/languages.json entry provides dispatch, CLI --language choice, markdown fence, Go flag help and detection. Adds Swift @objc/@IBAction entry-point + XPC/CLI input detection, the .swift context-corrector extension, the tree-sitter-swift dependency, per-unit reachability prune telemetry, and the Swift test suite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gadievron
added a commit
that referenced
this pull request
Jul 30, 2026
Adds a tree-sitter-based Swift parser (repository scan, function/type extraction, call-graph construction with overload/trailing-closure/actor resolution, reachability filtering, analysis-unit generation), integrated into the post-#199 registry architecture: a single config/languages.json entry provides dispatch, CLI --language choice, markdown fence, Go flag help and detection. Adds Swift @objc/@IBAction entry-point + XPC/CLI input detection, the .swift context-corrector extension, the tree-sitter-swift dependency, per-unit reachability prune telemetry, and the Swift test suite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gadievron
added a commit
that referenced
this pull request
Jul 30, 2026
Adds a tree-sitter-based Swift parser (repository scan, function/type extraction, call-graph construction with overload/trailing-closure/actor resolution, reachability filtering, analysis-unit generation), integrated into the post-#199 registry architecture: a single config/languages.json entry provides dispatch, CLI --language choice, markdown fence, Go flag help and detection. Adds Swift @objc/@IBAction entry-point + XPC/CLI input detection, the .swift context-corrector extension, the tree-sitter-swift dependency, per-unit reachability prune telemetry, and the Swift test suite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds repository-supplied threat models, an application-type (
custom:) context system, and multi-language scanning (Python, JavaScript, Go, C, PHP, Ruby, Zig) to OpenAnt, with security hardening of the new repo-traversal/generator paths and regression tests throughout.Multi-language scanning
Problem: the scanner handled one language per run; polyglot repos were under-covered and excluded languages were silently dropped.
Solution:
--language autodetects and scans every language above the size threshold;core/language_selection.py+core/language_registry.pydrive per-language parser dispatch (core/parser_adapter.py);core/dataset_merge.pymerges per-language datasets. Excluded languages are recorded on the result, not just stderr.Verification: all 7 parsers produce units on a fresh clone (go 35→12 reachable, python 290, js 131→100, c 51, ruby 651, + a planted-vuln python repo).
Repository-supplied threat models
Problem: a scan under the wrong security model looks identical to a correct one; operators had no way to state the trust model.
Solution:
context/threat_model.pyloads/validatesOPENANT.THREATMODEL.md(schema-enforced, permissive-model warnings surfaced into the artifact) and feeds it into Stage-1/Stage-2 prompts (prompts/threat_model_render.py);context/threat_model_agent.py+context/repo_explorer.pygenerate a model via a real repository survey.context_source+threat_model_sha256are serialized into scan artifacts so a run is tied to the exact model that shaped it.Verification: a repo with
OPENANT.THREATMODEL.mdscans asapplication_type=custom:<slug>,context_source=threat_model; a malformed model aborts loudly rather than degrading to a default web_app context.Application-type context
Problem: false positives from analyzing library/CLI/agent code under a web-app attack model.
Solution:
context/application_context.pyclassifies app type and appliessuppress_local_onlyso intentional behaviors aren't flagged.Verification:
tests/test_application_context_backcompat.py+ scanner integration tests.Security hardening (new paths only)
Symlink/traversal refusal, working directory kept off
sys.pathfor the Python subprocess, importable-wheel packaging with test artifacts stripped — contained to the new repo-survey/threat-model surface.Adversarial review checklist
context_source,threat_model_sha256,threat_model_warnings)Tests
Full suite green on a fresh clone install (
python -m pytest): 0 failed; feature subset 1318 passed. Regression tests cover multi-language selection/registry/merge, threat-model schema/rejection/provenance/prompts/agent/integration, app-context, and repo-traversal hardening.Compatibility
Additive to the scan-artifact schema (
context_source,threat_model_sha256,threat_model_warnings); no verdict values added;verdict_taxonomy.pyuntouched.Relationship to #197 (Auto Patcher): additive-only overlap in
cli.py(distinct subcommandthreat-modelvspatch) andREADME.md— no textual conflict, no ordering dependency, no shared logic. Whoever merges second takes the trivialbuild_parser()merge.