fix(url-crawler): SSRF — the scanner fetched attacker-chosen addresses

[Ap6pack]

The vulnerability

The URL crawler follows links found in skill content. Skill content is hostile input by definition — it is the thing being analysed. URLs are extracted from the body, from source_url and author_url, and from arbitrary frontmatter keys, so the destination is entirely attacker-controlled.

This is the default path, not an opt-in one:

async def scan(..., use_urls: bool = True, ...)
scan_default_layers = ["rule_engine", "url_crawler", "llm_analyzer", "threat_intel"]

There was no address check anywhere in fetcher.py, extractor.py or detector.py. Scanning a skill containing

http://169.254.169.254/latest/meta-data/iam/security-credentials/

made the scanner fetch cloud instance credentials — from CI, or from whatever machine a user ran it on. The only constraint was the extractor limiting schemes to http/https.

The fix

safety.check_url resolves the hostname before connecting and refuses any answer that is private, loopback, link-local, multicast, reserved or unspecified, unwrapping IPv4-mapped and 6to4 forms so a private v4 address cannot ride inside a v6 literal.

Resolution is the point. An attacker controls DNS for their own domain, so totally-normal.example with an A record of 169.254.169.254 walks past any textual hostname check.

httpx no longer follows redirects. Hops are walked manually and each one is validated before it is requested:

for _ in range(self.max_redirects + 1):
    safe, reason = check_url(current, resolver=self.resolver)
    if not safe:
        raise UnsafeURLError(reason)
    resp = await client.request(method, current)
    ...

A URL that passes on the first request can redirect inward on the second, so validating only the entry point catches nothing. A refused fetch comes back as a FetchResult carrying the reason rather than being dropped, so it shows in the report instead of looking like a URL with nothing behind it.

Residual risk, stated

The name is resolved here and resolved again by httpx at connect time. A DNS record that changes between the two (rebinding) defeats the check. Closing it needs a transport pinned to the validated address. Recorded in the module docstring as a known limit rather than left implied.

Why the resolver is injectable

respx mocks at the transport layer, so mocked hosts never resolve and the guard would reject every existing fetcher test. The alternative — a flag to skip checking in tests — would leave those tests covering nothing. Injecting the DNS answer keeps scheme checks and address classification fully live.

Verification

Removed only the guard call, leaving the manual redirect walk in place, so the tests measure "no SSRF check" rather than a different design:

FAILED TestUrlSafety::test_fetcher_refuses_metadata_address
FAILED TestUrlSafety::test_redirect_to_internal_address_is_refused
2 failed, 14 passed

test_public_redirect_is_still_followed passes both ways, confirming the guard doesn't break ordinary redirects.

Full suite: 1,780 passed, 16 skipped. Ruff clean.

Also in here

Wires crawler_max_urls, crawler_timeout, crawler_max_redirects, crawler_max_response_bytes, crawler_concurrency — five documented settings that nothing read, so tuning them silently did nothing. Values match SafeFetcher's constructor defaults, so behaviour is unchanged; the knobs now do what they claim.

How this was found

Auditing config for dead fields after the ambiguous-slug discovery. 14 of 56 settings are never referenced outside config.py; five of them clustered in the crawler, which is what prompted reading it. Related and not fixed here: scan_max_file_size (512 KB) is also dead. That one needs thought rather than a quick wire-up — Unit 42 found omnicogg evading scanners with 22 MB of junk padding specifically to exceed file-size limits, so adding a cap that silently skips oversized files would import the exact weakness it exploited. Oversized input should probably be scanned and flagged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL

---

Generated by Claude Code