Server Log Analysis: Do AI Bots Actually Reach Your Content?

You are thinking about GEO – about paragraphs that hold up when lifted out of context, about BLUF at the top of a section, about entities, about whether your content ever made it into Common Crawl. Those are the right questions to be asking in 2026. But… pause for a moment and go one floor down, to the foundations of technical SEO: your server logs. All that work only pays off on one condition – that the bot got onto the page at all and was able to fetch what is on it.

In early June my hosting provider let me know it was starting to “protect my content from being used to train artificial intelligence.” Nobody asked me what I thought about it, so I sat down with the logs to find out what that protection meant in practice.

It turned out that from the start of my available archive, 2 February, through to mid-June, AI bots were regularly getting a 429 from my server. Across the whole period analysed, up to 5 August, GPTBot received that response 2,615 times, Amazonbot 1,839, OAI-SearchBot 834, and ClaudeBot 400. Googlebot – not once. When the blocking actually started I cannot say, because older logs no longer exist. What I can say is that I had no idea it was happening until I opened the files myself.

Which is the most uncomfortable part of this story, given what I do for a living. The cobbler’s children go barefoot. All the more so because what pushed me towards auditing my own setup was client work – I had seen the same pattern on a few of their sites before I ever looked at my own server. I am only showing what I can show without compromise, meaning my own logs and my own numbers, to make one case: reading your logs belongs among the first steps of any work on AI visibility.

Content optimisation assumes that… AI can fetch it

Every technique we file under GEO rests on a fairly obvious chain of events: something in the ecosystem has to fetch the page – the provider’s crawler, a search index, or a partner supplying results. Only then is a passage cut out of that content (a fragment of text, usually a few sentences or a paragraph) to compete for a place in the answer. Content structure, headings as a map for the algorithm, the answer placed at the top of a section – all of that works on links two and three of the chain.

We treat the first link as a given, and that is exactly why it is the weakest. You can have the best-organised content in your niche, tidy structured data, and a well-ordered entity, but if the bot gets “too many requests, come back later” from your server, there is nothing to build a passage from and none of the rest of the work has any way to take effect.

A 429 (“Too Many Requests”) is a refusal to serve a request because a rate limit was exceeded. By design it is a signal to slow down and come back shortly. Every single request that ends in a 429 receives no content, so when those responses keep landing for four months straight, a bot’s access becomes unpredictable and badly constrained.

Why the answer is only in the logs

No external tool will tell you whether GPTBot visited your site yesterday and what it got back. AI visibility monitoring tools query the models and check whether you show up in the generated answer, which is an entirely different question, asked at the opposite end of the process. An access log records every request separately: IP address, timestamp, requested URL, status code, and user agent. You look for the answer on whichever layer the decision is made – in the server log, or, if the block happens earlier, in your CDN logs or your WAF panel.

One step back: what a WAF is

The catch is that traffic can be filtered in several places at once, and you do not set the rules in all of them:

• At the hosting layer

  Rules switched on by default in the server configuration, usually as part of the service, and nobody mentions them to you separately.

• At the CDN

  Since 1 July 2026 Cloudflare gives every customer, including the Free plan, separate controls for Search, Agent, and Training bots. From 15 September 2026 all newly onboarded domains are set to block Agent and Training by default on pages that display ads, while Search stays allowed.

• In a plugin

  Wordfence, iThemes Security, and other WordPress security tools, the ones guides describe as “set it and forget it.”

• In your .htaccess

  Not a WAF as such, but a server configuration file where blocking and rate-limiting rules can live. Usually the only place you know exists.

A rule that turns bots away is not a mistake in itself. The only question is whether it serves your goals.

What I found in my own logs

I took everything I still had archived, covering 2 February to 5 August 2026: more than 1.8 million requests in the access log and 119,794 entries in the error log. Within that window the blocks grew month over month. In February AI bots received 752 responses with a 429, in May already 2,020, and the record day was 12 June, when more than four hundred requests bounced off the server within twenty-four hours.

Daily 429 responses to AI bots. The highlighted band is 13–15 June, when I was switching the rules off one by one.

The rules behind those blocks were issued separately for individual bots, with the bot’s name written into the rule itself, which tells you plainly enough that they did not appear by accident.

Why this is so easy to miss

Because we check Search Console, we monitor visibility, and Googlebot usually gets through. Across 119,794 lines of error log its name does not appear once: no blocks, no limits, no rules written for it. There is a logic to that. A hosting provider blocking Googlebot would be shooting itself in the foot, since customers would notice the drop within a week and Google still sends the overwhelming majority of search traffic. So the other bots take the hit – sometimes entirely deservedly, because some of them really do behave aggressively, and sometimes not at all.

The practical effect is that every conventional metric looks fine: organic traffic steady, indexing healthy, Search Console quiet. No warning sign appears anywhere. Except in the logs.

Server log analysis and the diagnosis: AI bots cannot reach the site. Now what?

Between 13 and 15 June I worked through batch after batch of logs and switched off rules in .htaccess, one bot at a time, checking after each change whether the blocks actually stopped. I started with bots tied to the generative ecosystem, although they serve different purposes: training crawlers such as GPTBot and ClaudeBot, and search-index crawlers such as OAI-SearchBot and PerplexityBot.

That distinction matters, because unblocking GPTBot is not the same as making yourself eligible to appear in live ChatGPT Search results – OpenAI points to OAI-SearchBot for that second job. With each bot I watched not only whether the blocks disappeared, but also whether it started eating the server alive, because the original rule did not come from nowhere.

What it changed

Blocks on AI bots fell from 1,920 a month in June to 33 in July. Successful AI bot requests that landed on actual content rose from 1,852 in May to 2,446 in July.

The fix happened on 13–15 June, so July is the first full month without the rules. Source: podrez.pl access logs.

I am deliberately counting AI bots only here. Fold in all bot traffic and SEO tooling joins the table – the Ahrefs site auditor alone makes over twenty thousand requests a month on my site – and the growth would look far more impressive while meaning nothing at all.

Worth saying: not every block is a mistake

A second layer of rules, the one rejecting traffic with a 406, is still running on my server, and for a moment it looked like the next problem to solve. In July and August it turned away requests signed as GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, CCBot, ChatGPT-User, and Google-Extended. All it took was checking which addresses they were asking for.

They were asking for /.env, /.env.bak, /config/.env.php, /wp-config.php, and similar files, the kind that hold database passwords. No genuine AI crawler goes looking for those, and nearly all of the requests came from a single IP address that simply rotated its user agent. One more detail settles it: Google-Extended appeared among the signatures, and according to Google’s own documentation it has no separate HTTP user agent string at all – it exists purely as a control token in robots.txt.

Real AI crawlers have not been getting a 406 on content since July. One crawler is still bouncing off the rules, mostly on PDF files, and that one stays on my list. Earlier, between February and May, the same layer stopped ClaudeBot from fetching robots.txt 267 times – the very file meant to tell it what it is allowed to read.

Crawl volume is not a measure of success

I expected a simple story: unblock the bots, they come flooding back, the chart goes up. That is not how it went, and on reflection it makes sense. After being unblocked, GPTBot visits less often – 1,066 successful requests in June, 538 in July – while OAI-SearchBot has been declining steadily since February regardless of whether it was blocked. ClaudeBot and Amazonbot went up. So raw crawl volume turned out to be useless as a standalone measure of AI visibility. For judging server load it remains perfectly useful.

Far more interesting is what these bots come for. I measured how much of their successful traffic lands on an actual content page, and how much on CSS files, images, sitemaps, and API endpoints.

The split is clear. Bots fired by something a user just did go more directly at the content: ChatGPT-User at almost seventy-four per cent, Claude-User at sixty. CCBot’s high figure, eighty-two per cent, has a different explanation – it is the Common Crawl crawler, building an open archive of the web rather than responding to anyone’s question. So I would not judge these groups by one yardstick, any more than I would measure crawl budget by counting favicon downloads.

Knock-on effects

Alongside this I track visibility in generative systems: tools query the models with a fixed set of prompts and check whether my name comes up, and how large a share I hold against the competition. The prompt set changed over time, so I show two versions of the same measure – across all prompts, and across the twelve that were present throughout. The second is the cautious version, and it keeps a change in the prompt set from masquerading as a change in visibility.

All three lines move the same way. Average share of voice per probe sat around one per cent through May and early June, then settled between seven and eight per cent from the turn of June and July. Median position, calculated the same way the tool does it and only across answers where I actually appear, fell from five in the second half of May to two in the most recent week. In a typical answer that mentions me, I am now in second place.

The dashed line is the same measure across the twelve prompts present throughout the period. It rises more slowly, from around one per cent to three, and I am leaving it on the chart on purpose: in June the set gained prompts closer to what I actually do, so part of the growth on the full set comes from better-matched questions. The truth sits between those two lines, and in both cases it is growth.

Ninth among a hundred and fifty names

What convinces me most, though, is the comparison against the competition. Share of voice within a single answer adds up to one hundred per cent, so every percentage point I gain is a point somebody else loses. Roughly a hundred and fifty names of Polish and international specialists show up each month in the answers to those twelve prompts.

In May I was sixteenth and accounted for 1.3 per cent of the entire space the models devoted to naming experts. In June I moved up to tenth with 2.1 per cent, and in July to ninth with four per cent, which means my share tripled in two months.

And one figure straight from Analytics that impresses me most of all: on 23 June 2026 the first user attributed to the “AI assistant” channel arrived on podrez.pl in the entire period analysed. The traffic is still small, but it is now something you can count.

How to check this on your own site

You need access to your logs and a bit of time. On the server, look for a logs directory: it holds the access log, usually as access.log or yourdomain.com.log, and separately an error.log, which records why a request was rejected. Older days are typically packed into .gz or .tar.gz archives, one per day or per week, so analysing a longer period means unpacking and merging them first. On some hosts logging has to be switched on in the control panel, and occasionally you have to ask support for it.

For the analysis itself you have a few routes, easiest first:

• Download the logs directory

  That is what I did: pulled the whole directory down, then unpacked and merged the archives. With console access you can read the logs in place instead – six months of logs can run to several hundred megabytes.

• Screaming Frog Log File Analyser

  Drop the file in and you get a breakdown by bot, status code, and URL. The free tier handles a thousand events, which is enough for a first look.

• A language model with file access

  I merged my logs and analysed the whole set that way. At hundreds of thousands of lines it saves hours of work.

What to look for in the output:

• 1

  Recurring 4xx and 5xx codes

  Not just 429. My second layer of blocks returned 406, and 403 and 503 also turn up. The odd 301 or 302 is usually fine; long redirect chains are the ones to worry about.

• 2

  A bot that is missing entirely

  Zero lines for GPTBot while Googlebot is crawling happily is a clear signal to dig further.

• 3

  Proportions between bots

  If Googlebot sails through and everything else collects refusals, you have your answer before you even open the rules.

• 4

  The reason for the refusal

  Depending on your setup it will show up in the error log, the WAF audit log, or the CDN panel. That is where you find out whether the rule is yours or your provider’s.

• 5

  The requested URL

  If an “AI bot” is asking for configuration files, the block is entirely appropriate.

Two closing notes:

• If you sit behind a CDN, part of the decision happens before traffic reaches your server, so those blocks will not appear in your log and you have to look for them in the provider’s panel.
• A user agent is trivial to forge, so verify the source IP against the ranges each operator publishes. For Google you can additionally run a reverse DNS lookup and then a forward lookup on the result.

What to do about it

1. 1Find out whether you have a problem at all

   Fifteen minutes with the logs usually answers that without guesswork. If traffic is being stopped earlier, you also need to check the CDN or WAF panel.

2. 2Establish who set the rule

   Your .htaccess, a plugin, the CDN, or the host’s default configuration – each is unblocked differently, and in each case you are talking to somebody else.

3. 3Decide deliberately who gets in

   Blocking AI crawlers can be a sound business decision, especially if you sell the content you publish. The point is that it should be your decision, made knowing what it costs.

4. 4Switch rules off one at a time and watch the logs

   Rate-limiting rules exist for a reason, so if the server starts struggling once a bot is let back in, what you need is a limit rather than a ban.

5. 5Come back to the logs a month later

   Rules have a way of returning when a provider updates its configuration, and nobody will tell you about that either.

And only then go back to passages, BLUF, and entities, because those genuinely matter.

Frequently asked questions