Do AI Crawlers Slow Your Site? We Counted Every Hit

The Short Answer

For a small, cached site, no. Over fourteen days ending 26 September 2026, AI crawlers made 933 requests to vitalsfixer.com that reached our server, more than Googlebot's 133 and Bingbot's 32 combined many times over. Yet all of them together downloaded just 4.3 MB. That is less than one unoptimised hero image a day.

The frightening numbers in the news are real, but they come from large sites with huge page counts or heavy uncached pages. Read the Docs, for example, reported that blocking AI crawlers cut its traffic from 800 GB to 200 GB a day and saved about 1,500 dollars a month. The honest answer depends on which kind of site you run, and this page shows you how to tell.

Our Log Data by the Numbers

These are counts from our nginx access logs for 13 to 26 September 2026, matched by user agent. Bytes are what our server sent, so requests Cloudflare answered from its cache never reached us and are not in these numbers.

Two things stand out. AI crawlers now visit us about seven times as often as Googlebot. And the busiest of them, ClaudeBot, pulled a quarter of the data GPTBot did, a reminder that request counts alone tell you little about cost.

Which Bot Is Which

Lumping every AI bot together is where most bad blocking decisions start. They do very different jobs.

• Training crawlers such as GPTBot, ClaudeBot, CCBot, Bytespider and meta-externalagent collect pages to train future models. Blocking them keeps your content out of future training data. It does not remove you from answers today.
• Search index crawlers such as OAI-SearchBot and PerplexityBot build the index that AI search answers draw from. Block these and you drop out of those answers.
• User triggered fetchers such as ChatGPT-User visit a page because a real person asked ChatGPT something and it decided to read your page to answer. OpenAI documents ChatGPT-User as acting on behalf of users, not crawling automatically.

That last group is the one to watch. Our 88 ChatGPT-User hits were 88 moments where someone's question led ChatGPT to our pages. It is the closest thing to a visible citation counter most site owners will get, and blocking it would cut you out of exactly the answers you want to appear in. Our AI bot optimization guide covers how to be the page it chooses.

Why the Cost Was So Small

Three things kept our bill near zero, and all three are in your control.

1. A CDN cache in front of the server. Cloudflare answers repeat requests for static pages itself, so many crawler hits never touch the origin at all.
2. Light pages. Our articles are static HTML with compressed images. A crawler that reads the text downloads tens of kilobytes, not megabytes. Heavy pages cost more for every visitor, bots included, which is one more reason to compress your images.
3. A small site. We publish a few dozen pages. A crawler cannot spend much on a site with little to crawl. A store with a hundred thousand product and filter URLs is a completely different case.

When You Should Worry

The pattern is the same as with any crawler: cost scales with how many URLs you expose and how expensive each one is to produce. The speed damage appears when bot load competes with real visitors for the same server, which shows up as slower TTFB and, through that, worse LCP.

Check Your Own Logs

If you have shell access, this counts AI crawler requests and data for any nginx or Apache access log in the combined format:

for bot in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot PerplexityBot \
           Amazonbot meta-externalagent Bytespider CCBot Googlebot bingbot; do
  awk -v b="$bot" 'tolower($0) ~ tolower(b) { n++; s += $10 }
    END { printf "%-20s %6d requests %8.1f MB\n", b, n, s/1048576 }' access.log
done

Run it over at least a week, since crawlers arrive in bursts. If AI crawlers account for a large share of your data sent and your TTFB rises at the same times, you have a real problem. If the totals look like ours, you do not.

What to Block and What to Keep

Blocking should be a decision per bot, not a reflex. A reasonable default for most sites that want AI visibility:

# Keep AI search and live answers
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Keep crawlers out of pages that cost you and help no one
User-agent: *
Disallow: /search
Disallow: /cart
Disallow: /*?filter=

Block training crawlers only if you have a clear reason, such as licensing concerns or genuine load problems you have measured. Remember that robots.txt is a request, not a wall, so a crawler that ignores it has to be stopped at your CDN or firewall. And whatever you decide, caching and light pages do more for your server than any robots.txt line, because they help with every visitor at once.

Frequently Asked Questions

Do AI crawlers slow down websites?

They can, but mostly on large or uncached sites. On vitalsfixer.com, 933 AI crawler requests over 14 days in September 2026 transferred only 4.3 MB from our server. Sites with thousands of URLs or pages built fresh on every request, such as uncached WordPress or stores with filters, carry far more load.

Does GPTBot affect website speed?

GPTBot alone made 164 requests and downloaded 1.6 MB from our server in 14 days, which had no measurable effect. It can affect speed on sites where each page is expensive to generate and there is no cache, because bot requests then compete with real visitors for server time.

What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?

GPTBot collects pages for training OpenAI models. OAI-SearchBot builds the index for ChatGPT search. ChatGPT-User fetches a page live because a user asked ChatGPT a question. Blocking GPTBot does not remove you from ChatGPT answers, blocking the other two can.

Should I block AI crawlers?

Block by purpose, not by habit. Keep search and live fetch bots if you want to appear in AI answers. Consider blocking training bots only for licensing reasons or measured load problems. Keep all bots out of search, cart and filter URLs.

How can I see how many AI bots visit my site?

Count them in your server access logs by user agent. The shell loop on this page prints requests and data sent per bot. Requests your CDN answers from cache will not appear in origin logs.

Do AI crawlers use more resources than Googlebot?

Per request it depends on the page, not the bot. What changed is volume: on our site AI crawlers visited about seven times as often as Googlebot in September 2026, so their share of total bot traffic is now larger.

Does blocking AI crawlers improve Core Web Vitals?

Only if bot load was slowing your server. Core Web Vitals are measured on real visitors, so blocking bots helps only when it lowers server response time for those visitors. Caching and lighter pages usually help more.

The Bottom Line

AI crawlers are now the most frequent bots on a small site like ours, but frequency is not cost. With a CDN cache and light pages, 933 AI crawler requests cost us 4.3 MB in two weeks. Measure before you block, keep the bots that put you in AI answers, fence off the URLs that cost you money, and spend your effort on caching and page weight, which speed things up for bots and people alike.