Tool · server logs

Check AI bots in your logs

Robots.txt tells you which bots you let in. Your server log tells you which ones actually came.

Drop a log file and see which AI bots send requests to your site, which of them come from confirmed IP addresses and whether your server turns any of them away.

The tool at a glance

117 bots

recognised by the tool, from search engines to AI assistants.

20 IP lists

from operators such as Google, OpenAI, Anthropic and Perplexity, used to confirm bot addresses.

0 bytes

of your log reach the server. Everything runs in your browser.

Under a second

to analyse 186,000 lines in a test on podrez.pl logs.

How it works

What the tool reads from a single log line

Your server log records every request on its own line. The tool reads them one by one and takes five things from each.

20.171.207.181 - - [17/Oct/2026:10:12:01 +0200]2 "GET /en/bots/3 HTTP/2" 2004 84512 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)5"

IP address is checked against the operator’s list, here OpenAI’s. It is on the list, so the address is confirmed.
Date and time go into the day-by-day chart.
URL shows what the bot asked for: a page, robots.txt, a sitemap or a Markdown version.
Server response 200 means success, 403 or 429 a refusal.
Bot name in the user-agent field says who it claims to be. The name alone proves nothing.

Result for this line: GPTBot · OpenAI IP confirmed 200 · page served

Bot names come from the dictionary behind my list of bots and user agents, and IP addresses are compared with 20 lists published by bot operators.

Privacy

Your file stays with you

Your log file never leaves your computer. The tool runs in your browser, just like a program installed on your disk.

When you open the page, your browser downloads the tool’s code and the official bot IP lists from podrez.pl. Everything else happens on your side: unpacking the file, reading it line by line, comparing addresses with the lists and building the results table. The tool asks no external service about IP addresses and stores nothing in cookies or browser storage. Close the tab and nothing is left.

Don’t take my word for it. Open the page, wait a few seconds, switch off your internet connection and drop the file. The result appears offline. If you prefer developer tools, open the Network tab: dropping the file triggers no request with its content.

What I measure. Google Analytics only learns that someone used the tool and the rough size of the log, e.g. 10,000⁠–⁠100,000 lines. No file name, no IP addresses, no content, and only as your cookie settings allow.

Personal data

Logs contain visitors’ IP addresses. With this tool, that data doesn’t reach me; it stays with you. That’s why the tool also works for logs you are not allowed to send anywhere.

Result

What the result shows

Confirmed and unconfirmed addresses – a request is confirmed when its IP address is on the operator’s list. An address outside the list is usually impersonation, because anyone can type a bot name.
Server refusals – codes 403, 429 or 503 don’t come from robots.txt. They usually point to a firewall, CDN or rate limits, and 503 can also mean overload or maintenance. On podrez.pl, the hosting firewall was turning some AI bots away with 429 until I spotted it in the logs.
What bots ask for – pages, robots.txt, sitemaps, files and Markdown versions, plus each bot’s ten most requested URLs and how many of them the server actually delivered.
Day by day – a chart of AI bot requests from confirmed addresses and from bots without an IP list, plus a CSV file for your own analysis.

Before you start

Where to find your server log

You need the access log in the combined format, which Apache, Nginx and LiteSpeed write by default. The error log won’t do.

cPanel: Metrics → Raw Access, a .gz file with the last few days.
DirectAdmin: archives in /domains/your-domain.com/logs/, one .tar.gz file per day.
Your own server: usually /var/log/nginx/access.log or /var/log/apache2/access.log.
Cloudflare and other CDNs: the log must contain real visitor addresses (e.g. from the CF-Connecting-IP header). If it holds Cloudflare addresses, the tool will tell you.

You can drop several files at once, e.g. a whole week. Unpack .zip archives first.

More tools

Other tools

Free, no sign-up. Each one answers one specific question.

Tool
List of bots and user agents
A searchable list of bots: who runs them, what they do and how to handle them in robots.txt.
Tool
AI knowledge cutoff dates
How far the knowledge of ChatGPT, Claude, Gemini and Grok reaches. Cutoff dates in one place.

Bots don’t come to your site? Maybe they can’t! Check it in High Five, AI! by Maciej Chmurkowski.

Articles on bots, logs and AI visibility

Article
Server log analysis
Before you work on AI visibility, check whether AI bots actually reach your content. What the logs show.
Article
Common Crawl
Does AI have anything to remember you by? About the public archive that language models learn from.
Article
Grounding queries in Bing Webmaster Tools
Free AI visibility data most SEOs ignore: how often, and for which queries, your content gets cited.

What next

Did the result surprise you?

Refusals for bots from confirmed addresses, visits to robots.txt only, or no AI search bots at all are signals worth a closer look. Log analysis is part of my AI search visibility work: from the firewall and robots.txt to what bots actually download.

Want me to take a look?

Send a short brief: your site and what you saw in the log. I’ll tell you what is worth checking first.

Send a short brief AI search visibility

Frequently asked questions

Is my file sent anywhere?

No. The analysis runs in your browser. You can check it by switching off your internet connection before dropping the file. Google Analytics only learns that someone used the tool.

How is this different from the list of bots?

The list of bots and user agents tells you what a bot is and how to handle it in robots.txt. This tool shows which bots actually came to your site, whether their IP addresses are on the operators’ lists and what your server gave them.

What does “no IP list” mean?

Some operators, such as Meta, ByteDance or Yandex, publish no IP address lists. I count those requests by name only, so there is no way to tell the real bot from an impersonator.

What should I do about requests from outside the IP list?

Usually nothing. These are mostly scanners and scrapers posing as well-known bots to slip past filters. With older logs, check the list date first, because operators change their address ranges. If you want to block them, do it by IP address or network, never by bot name alone. Otherwise you’ll block the real bot as well. I show how many such requests reach podrez.pl in the verification results.

Why is a bot I expect missing from the result?

Either it didn’t visit in that period, or robots.txt or a firewall is stopping it. Check a longer period. If the bot never shows up, look at your robots.txt and firewall settings.

How often do you update the lists?

Together with the list of bots. You’ll find the date of each list under the result, in the “How we know which bot is real” section.

Prepared by Ewelina Podrez-Siama, in SEO since 2009. Latest version of the tool: 1.0.14, 31 October 2026.