Checks which AI-crawler hits in your server log are real. A user-agent string is a claim anyone can type. The source IP, checked against the range each vendor publishes, is the evidence.
Node 18 or newer. No dependencies.
node verify-crawler-hits.mjs sample-access.log
node verify-crawler-hits.mjs /var/log/nginx/access.log* # rotated files, .gz is fine
node verify-crawler-hits.mjs hits.csv # ip,timestamp,path,user_agent
Input: the Combined Log Format that nginx and Apache write by default, or a
CSV of ip,timestamp,path,user_agent (quoted fields are fine). The first field
must be an IP address; the ::ffff: form a dual-stack server writes is read as
the IPv4 address inside it.
JSON logs (Caddy's default, most CDN exports) and hostname logs are not read:
those lines are counted as unreadable and never judged, because a guessed
address would turn a real crawler into a spoof.
Behind a CDN, load balancer or reverse proxy, your server log may record the
proxy's address instead of the crawler's, and every claim will read as spoof.
Log the original client IP first (nginx: the realip module). The script
prints a warning when more claims fail than pass.
Output, per bot: verified (IP inside the published range), spoof (the UA
claimed a crawler whose published range does not contain the IP), unknown
(that crawler publishes no range, so neither honest nor spoofed). Then the
two authenticity checks for verified live-agent traffic: hits per UTC hour, and
distinct days per path.
Published range sources the script fetches:
- OpenAI: https://openai.com/gptbot.json, https://openai.com/searchbot.json, https://openai.com/chatgpt-user.json
- Anthropic: https://claude.com/crawling/bots.json
- Perplexity: https://www.perplexity.ai/perplexitybot.json, https://www.perplexity.ai/perplexity-user.json
- Google: https://developers.google.com/search/apis/ipranges/googlebot.json
- Bing: https://www.bing.com/toolbox/bingbot.json
- Apple: https://search.developer.apple.com/applebot.json
sample-access.log holds 20 Combined Log Format lines built from addresses
inside those published ranges, plus three deliberate spoofed hits from two
addresses and one unverifiable bot, so a first run reproduces the table shape
from the article.
npm test checks that the sample still prints the report above and runs the
regression checks in test.mjs. Both fetch the live vendor ranges.
A verified hit is a retrieval. It is not a citation and not a click.
Written for the freeCodeCamp article "How to Verify Which AI Crawler Hits in Your Server Logs Are Real". Author: Chudi Nnorukam, https://chudi.dev