Netlas is the only optional passive intelligence provider. Its three workers — phishing-netlas, dns-netlas and exposure-netlas, one per module phishing, dns and exposure — use the core's pull/lease protocol and are defined in docker-compose.yml behind the netlas profile. A second, separate family of optional workers asks a provider which services a host exposes: exposure-netlas-endpoints and exposure-shodan-endpoints, behind the netlas-endpoints and shodan-endpoints profiles. Censys, FOFA and Criminal IP are not shipped: the baseline seeds no registry row for them and their provider code is not in the repository. Source observations already recorded for them are kept as evidence.
-
Upgrade the core with the normal migration procedure and take a verified backup first. Nothing is enabled automatically.
-
Set the provider key in
.env(or in your secret manager):NETLAS_API_TOKEN. Never place credentials in connector config, API responses or the browser. -
Issue a core credential for each Netlas worker you intend to run:
docker compose exec -T backend python -m scripts.manage_connector_tokens issue exposure-netlas --type exposure --env CONNECTOR_TOKEN_NETLAS_EXPOSURERepeat for
phishing-netlas(--type phishing,CONNECTOR_TOKEN_NETLAS_PHISHING) anddns-netlas(--type dns,CONNECTOR_TOKEN_NETLAS_DNS_SECURITY). The endpoint workers issue the same way with--type exposureand their own variables,CONNECTOR_TOKEN_EXPOSURE_NETLAS_ENDPOINTSandCONNECTOR_TOKEN_EXPOSURE_SHODAN_ENDPOINTS. The--typeis the module id; the--envname is listed in.env.example. -
Enable the profile and start the workers:
COMPOSE_PROFILES=netlas make up
The
netlasprofile starts all three workers together. A worker started without its provider key still registers: its capability list comes from the provider adapter rather than the key, so it reportscredentials_configured: falseand is never admitted a scan. -
In Settings, check credential readiness, capabilities and last success. Explicitly approve the request/minute and daily budgets for the shared quota account, and enable the registry rows you want. Registration and heartbeat do not validate the provider key;
validis reported only after a successful provider request. -
Add active domain/IP assets and use Exposure Rescan. Exposure schedules start disabled; enable one explicitly if you need it.
Provider credentials reach the worker from its own environment or from the core. A key stored once per source account is encrypted with ENCRYPTION_KEY and read only by the connector bound to that account, through GET /api/v1/connectors/me/credential, which is never cacheable and is audited by connector and account without the value. credential_ref only names an environment variable.
connectors/exposure_endpoints is a second optional image behind the
netlas-endpoints and shodan-endpoints profiles. EXPOSURE_PROVIDER selects the
provider and refuses any value other than netlas or shodan; the worker reads
NETLAS_API_TOKEN or SHODAN_API_KEY for that provider and charges the lookup
quota bucket before every request. Each scan is bounded by its own policy
(max_requests and max_targets, each 1–200, default 20): an IP target is an
exact match, and a domain target is associated with the address its name resolved
to, so a shared address alone never establishes ownership. live_check.py in that
directory is an operator-run check that spends real quota — at most five requests,
retries included — and prints counts and ports, never a key, a request URL or a
response body.
Settings → Connectors → Provider accounts holds the metadata-only quota policy for each upstream account. The migrated default Netlas account is used by unassigned Netlas workers once its budgets are approved. Unapproved or missing accounts are blocked fail-closed. Adding a second account does not switch credentials: the matching worker must also receive that key. Scan jobs freeze account policy and connector config when queued; later edits apply to new scans only.
Connector ingestion reports received, created, updated, unchanged, accepted, rejected and bounded per-item errors. Replays are unchanged, not rejected. Findings, evidence and notification outbox events commit together. Notification delivery is at-least-once across the database-to-channel boundary.
Provenance comes only from the authenticated connector identity and its bound account, never from a field in the finding. Connector rows use <module>-<source>[-<task>] labels — phishing-netlas, dns-netlas, exposure-netlas-endpoints — and each seeded row also carries its earlier spelling as an alias, which is what keeps authentication and tooling on the same identity.
GET/POST /api/v1/connectors/accountslists or creates non-secret account identities.PATCH /api/v1/connectors/accounts/{account_id}/policytakes an explicit approval boolean and integer minute/day budgets.PATCH /api/v1/connectors/{connector_id}/account?account_id=...binds a module connector to a matching account.GET /api/v1/connectorsincludes each connector's UUID,source_account_idand display label.
Each provider request needs a live job lease, an enabled module, account minute/day capacity, a per-job request slot and an unexpired deadline. Redis failure denies requests. Response bytes, pages, targets, findings and HTTP timeouts are bounded. Transient network errors, HTTP 429 and 5xx get at most two retries, with capped backoff and a Retry-After wait of up to five seconds. Every physical attempt consumes quota. No provider redirect is followed. Provider errors map to safe codes; raw response bodies are not logged or stored.
Disabling a source cancels pending jobs; re-enabling does not resurrect them. A phishing match is a candidate, not a malicious verdict. Exposure is provider-observed inventory, not verified vulnerability or exploitation. Exposure association requires exact IP or hostname matching; a shared IP, CDN or certificate SAN alone does not establish ownership. Failed or truncated scans never resolve historical evidence.
- Bearer authentication against the documented
/responses/search,/host/{host}/summary,/domains/collection (passive A/AAAA/NS) and/certs/collection (certificate names and fingerprints). - Responses and domains share the 60/min general account budget; certificate collection has its own 3/min bucket plus the shared account, day and job budgets.
- The DNS Security worker uses the Domains and Certificates collections. Host summary and certificate SAN names are not ownership assertions.
- Malformed records are isolated from valid siblings; a scan with malformed records reports
partial, not a clean result. - The phishing worker asks the breadth questions:
/certs/withcertificate.subject.organization:"…" OR certificate.names:*…*(a brand, not our own domain — a double does not name us),/responses/withhttp.bodyand withhttp.favicon.hash_sha256/http.favicon.perceptual_hash:…~1,/whois_domains/(created_date,registrar) and/domains/withdomain:"…" AND mx:*. The last two are asked about confirmed candidates — names an analyst judged, handed to the scan in its params — so they are evidence about a finding rather than another way to discover one. - The two broad expressions (a brand in a certificate, a brand in page text) are counted with
/responses_count/before any page of them is fetched, and a result set abovemax_broad_resultsis skipped by name instead of paged: one request instead of a scan's whole budget. - Certificate queries — the brand one included, and its count — are reserved from the certificate bucket the provider's 3/min limit applies to, not from the general one.
- A scan reports what it could not do beside what it found:
skippedcounts the reasons (expression_too_broad,target_refused,favicon_hash_unavailable,whois_unavailable,provider_quota_exhausted,source_disabled,provider_access_denied), andspent,budget_left,daily_budget,daily_budget_leftandretry_after_secondssay what the allowed work cost. A budget or kill-switch refusal ends the scan as a named reason rather than as a provider error, because the credentials and the provider are fine and a readiness row claiming otherwise outlives the condition.
connectors/intelligence/live_check.py is the operator-run spike for exactly these
questions. It spends real quota — at most six requests, retries included — and prints
counters, never the key, a request URL or a response body, so its output can be pasted
into an issue:
NETLAS_API_TOKEN=... python live_check.py --question brand --value "ACME Corp."
NETLAS_API_TOKEN=... python live_check.py --question body --value "ACME Corp."
NETLAS_API_TOKEN=... python live_check.py --question own_hashes --value example.com
NETLAS_API_TOKEN=... python live_check.py --question favicon --value <sha256 from above>One question per run: brand, count, body, own_hashes, favicon,
favicon_similar, whois and mx, each answering with counts (certificates=,
responses=, count=, sha256=/perceptual=, registrar=/created=,
mail_capable=) rather than rows. A field the provider names differently shows up here
as an empty answer and in a scan summary as a skip reason, because a value that arrives
under none of the documented names is refused rather than guessed at. A brand, a count
and a body expression are phrases the query builder quotes; only own_hashes, whois
and mx are normalised as hosts, so a documented example like "ACME Corp." is not
refused as an invalid endpoint.
Confirmed against a live account (2026-10-17, SPIKE-04). http.body:"OpenDRP"
answered count=8 rows=8 — the count route and the responses route agree, so the body
expression WI-24 rests on is confirmed, and WI-25's acknowledgement gate is wired in
front of it. whois on a reserved domain answered rows=1 registrar=no created=yes, and
mx answered rows=0 mail_capable=False — both fields parse. The certificate collection
is not confirmed on this account: /certs/ with a certificate.* expression (the
brand question, and even the provider's own name) timed out after 30 seconds, while
host: on the same path answered immediately. Until it is re-checked against the plan
in use, the brand-certificate capability must be treated as unverified: a scan reports it
as a provider error and does not fall back to another shape. This is the kind of result
the spike exists for — it is a property of the account, not of the parser.
Readiness uses a small safe error set (invalid_credentials, credits_exhausted, rate_limited, provider_rejected_request, network_or_invalid_response, or an HTTP status class).
POST /api/v1/modules/{module}/scan: per-source admission.GET /api/v1/exposure/observations: paginated inventory with optional source filter.GET /api/v1/phishing/threats/{id}/evidence: source evidence (up to 200 records). Each row carriesfields— the allowlisted payload facts (capability,verdict, registrar and registration date, mail facts) — beside the rawpayload, so a row that found a name and one that describes a name the platform already had are distinguishable.- The Exposure UI at
/exposurelets analysts triage observations and promote a concrete hostname or IP into Assets after a confirmation that names the asset type and criticality. - PDF reports include bounded Exposure and DNS observation sections with truncation metadata.
Provider tests use mocked HTTP transports. They cover parsing, bounds, retries, quota reservations and error handling, but do not prove live credentials, plan entitlement or provider availability. Before relying on live results, run a bounded, budgeted validation against your Netlas account.
- Netlas capabilities depend on the account plan and the operator's configuration.
- Provider contracts and plans can change; re-check the Netlas API reference before enabling a source in production.