Website Tracking Tools
AI Crawler Log Analyzer
See which AI crawlers are asking for your pages. Drop in an Apache or Nginx access log and the tool counts requests from the AI crawlers that announce themselves in the user-agent field, then breaks them down by crawler, purpose, response code, path and day. The log is read in your browser and never uploaded. Because a user-agent name can be faked, the results show what requests claimed to be, not proof of who sent them.

How to use it
- Download your server's access log in the combined format (the one that includes the user agent), or copy a few lines from it. A .gz file works too.
- Choose the file, or paste lines into the box, and press “Analyse”. Large files are read in the background so the page stays responsive.
- Read the tables, filter by crawler, purpose or status code, and download the CSV files if you want to keep the numbers.
Example
This combined-format line is counted as one GPTBot request that returned 200 for /blog/: 203.0.113.5 - - [10/Nov/2026:13:55:36 +0000] "GET /blog/ HTTP/1.1" 200 5120 "-" "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)". The IP address on the line is read but not stored or shown.
How to get your access log
On many Linux servers the logs are in /var/log/apache2/ (Apache) or /var/log/nginx/ (Nginx), usually in files called access.log with older days rotated and compressed as .gz. Paths vary by host and setup. Shared hosting panels often offer a “raw access log” download, and managed platforms or CDNs may need logging switched on first. The format must include the user agent, which is the default “combined” format on both servers.
How to read the results
- Requests are single fetches, not visits or people, and not evidence of rankings or citations.
- Purpose groups crawlers the way their operators describe them: training or data collection, search, or fetching a page because a user asked an assistant to. These descriptions can change, so check the operator's documentation.
- Status codes show how your server answered: 2xx success, 3xx redirects, 4xx errors such as 403 and 404, and 5xx server errors.
- Other bot-like user agents lists names containing words like bot or crawler that the tool does not recognise, which is a starting point for investigation and not a verdict.
What this tool cannot tell you
- Whether a crawler is genuine: user agents can be spoofed.
- Whether a crawler obeyed your robots.txt. You can compare the paths it requested with your rules yourself.
- Anything about traffic that never reached your server, such as CDN-cached responses.
- Which crawlers are new: the list covers well-known declared crawlers and will not include every bot.
To see how a crawler might read one of your pages, try the search engine spider simulator.
Limits
- Files up to 1 GB (plain or .gz); pasted text up to 5 million characters.
- Lines longer than 64 KB are skipped.
- Up to 5,000 different paths are listed per crawler; totals still count every request.
- Days are shown as written in the log, in your server's time zone.
- Query strings are removed from paths unless you choose to keep them.
Questions
Is my log file uploaded?
No. The file is read by your browser (in a background worker when available) and only totals are kept in memory. Nothing is sent to us or to anyone else. Close or reload the page and the results are gone.
Does a match prove a crawler is genuine?
No. Anyone can send a request with GPTBot or ClaudeBot in the user-agent field. To confirm a crawler, check the requesting IP address against the ranges or reverse-DNS method that the operator publishes. This tool does not do that.
Why is Google-Extended not in the results?
Google-Extended is a robots.txt token that controls whether content may be used for some Google AI uses. It does not send requests with its own user agent, so it never appears in a log. Applebot-Extended works the same way. The tool lists Googlebot and Applebot, which are ordinary search crawlers, only for context.
Does a request mean my content was used to train an AI model?
No. A request shows that something fetched a URL. What the operator does with it depends on its own policies and which of its crawlers made the request, and those policies can change. The purpose column reflects how the operators describe each crawler at the time of writing.
Which log formats work, and how big can the file be?
Apache and Nginx access logs in the common or combined format, including the vhost variant, one request per line. Only the combined format includes the user agent, so it is the one you need. Files up to 1 GB are accepted, plain or gzip-compressed, and lines that do not fit the format are counted and skipped.
Why might AI crawlers be missing from my log?
A CDN or cache may answer the request before it reaches your server, your host may log in a different format or keep logs only for a short time, or the crawler may simply not have visited in that period. An empty result is not proof that no crawler visited.
Get practical SEO tips by email
Occasional emails with new guides and tools. Confirm by email first; unsubscribe in one click.
