AI Crawler Traffic Grew 300% This Year -- Here's What Your WordPress Site Should Do About It

Open your server logs. You'll find names you didn't expect: GPTBot, ClaudeBot, PerplexityBot, Bytespider, AppleBot. A year ago, most of them barely registered. Now they account for billions of requests per day across the web -- and your WordPress site is almost certainly being crawled by at least a few of them.

This isn't speculation. Cloudflare processes roughly 50 billion AI crawler requests per day across its network. Vercel reported that GPTBot, ClaudeBot, AppleBot, and PerplexityBot combined account for 1.3 billion fetches per month -- about 28% of Googlebot's total volume. OpenAI's GPTBot alone surged from 5% to 30% of AI crawling share between May 2024 and May 2025.

The age of AI crawling is here. The question isn't whether these bots are visiting your site. It's whether you have a strategy for dealing with them.

The Data: How Much AI Crawler Traffic Has Actually Grown

Let's start with the numbers, because they're striking.

Overall growth: AI-oriented bots now make up 4.2% of all HTML page requests on the web, according to Cloudflare's 2025 data. Crawling by AI bots rose 32% year-over-year by April 2025. User-triggered AI crawling -- where an AI agent fetches a page because a human asked a question -- grew 15x in 2025 alone.

GPTBot: OpenAI's crawler jumped from roughly 5% to 30% of all AI crawling share in one year. That's a 305% increase. It made 569 million requests per month across Vercel's network alone.

ClaudeBot: Anthropic's crawler made 370 million fetches per month on Vercel. While its share of total AI traffic has fluctuated (rising from 6% to 10% of AI crawling by one measure, dropping by another), its absolute volume remains massive.

PerplexityBot: Smaller in volume at 24.4 million monthly fetches on Vercel, but growing fast -- Perplexity itself grew 370% year-over-year as a platform.

The bigger picture: Bots now account for 49-51% of all internet traffic. AI bots are a growing slice of that. And the trend is accelerating, not flattening.

Why These Bots Are Visiting Your Site

Not all AI crawlers want the same thing. Understanding the difference is critical to making smart decisions about access.

Training Crawlers

The majority -- roughly 80% of all AI crawling, according to Cloudflare -- is for model training. These bots are scraping your content to feed it into the next version of an AI model. They consume large volumes of data, visit repeatedly (some returning every 6 hours), and send essentially zero referral traffic back to your site.

OpenAI's crawl-to-referral ratio in June 2025 was 1,700:1. Anthropic's was 73,000:1. That means for every visitor these companies send back to a website, they crawl thousands of pages. For training crawlers, your content is the product.

Search Crawlers

Search-oriented AI bots crawl your site to retrieve information for real-time answers. When someone asks ChatGPT "best WordPress hosting 2026" and it browses the web, that request hits your server via a search crawler. These bots do drive referral traffic, and that traffic is valuable -- AI search visitors convert at 14.2% compared to Google's 2.8%.

User-Action Crawlers

The fastest-growing category. These are triggered directly by a user interacting with an AI tool -- for example, pasting a URL into ChatGPT and asking "summarize this article." User-action crawling grew 15x in 2025, and it's the most targeted type of AI crawling.

The Impact on Your WordPress Site

AI crawler traffic isn't free. Here's what it's actually costing you.

Bandwidth and Hosting Costs

AI crawlers are aggressive. They don't crawl a page once and move on -- they return repeatedly, sometimes every few hours. The Read the Docs project found that blocking AI crawlers reduced their bandwidth from 800GB to 200GB per day, saving approximately $1,500 per month.

Your WordPress site probably isn't serving 800GB daily, but the impact scales. If you're on a shared hosting plan with bandwidth limits, AI crawlers can push you into overage fees. On metered cloud hosting, every request costs money.

Server Performance

Unlike Googlebot, which follows crawl-delay directives and adjusts its rate based on server response times, many AI crawlers lack these courtesies. They can hammer your server with parallel requests, slowing down the experience for actual human visitors.

For WordPress sites without aggressive caching, each AI crawler request can trigger PHP execution and database queries -- the same server load as a human visit, but with zero business value if it's a training bot.

Crawl Budget Interference

This one's subtle but important. If AI crawlers are consuming significant server resources, your site may respond slower to Googlebot. That can affect your Google crawl budget and, indirectly, your search rankings. The bots you didn't invite might be hurting the bots you depend on.

Wasted Crawl Requests

Vercel's data revealed that ChatGPT and Claude spend over 34% of their crawl requests on 404 pages. If your site has broken links, redirect chains, or orphaned URLs, AI crawlers will find and hit them repeatedly -- wasting your server resources on pages that don't even exist.

5 Things WordPress Site Owners Should Do Right Now

1. Monitor Your AI Crawler Traffic

You can't manage what you can't measure. Before making any decisions, find out which AI bots are visiting your site and how much traffic they're consuming.

Option A: Server logs. If you have access to raw access logs, search for user agent strings like "GPTBot," "ClaudeBot," "PerplexityBot," "Bytespider," "CCBot," and "Google-Extended."

Option B: Use a monitoring tool. Prime SEO's AI Crawler Stats module automatically detects and logs visits from 16 known AI crawlers, showing you which bots are visiting, how frequently, and which pages they're hitting. No server log access required.

Once you have the data, you can make informed decisions about the next four steps.

2. Set Up robots.txt Rules for AI Bots

Your robots.txt file is the first line of control. Most WordPress sites have a basic robots.txt that says nothing about AI crawlers, which means they're all allowed by default.

You have three strategic options:

Allow all AI bots: If you want maximum AI search visibility and don't mind training crawlers using your content.

Allow search bots, block training bots: The balanced approach. Allow PerplexityBot and ChatGPT-User (which drive referral traffic) while blocking training-only crawlers like CCBot and Google-Extended.

Block all AI bots: The nuclear option. Protects your content from AI training but also prevents your site from appearing in AI search results.

For most WordPress sites, the balanced approach makes the most sense. This guide has ready-made robots.txt templates you can copy and paste.

3. Add llms.txt to Guide AI Crawlers

If you're allowing AI search bots to crawl your site (and you probably should), give them a map. An llms.txt file tells AI systems which pages on your site are most important and how your content is organized.

Think of it as the difference between letting someone wander through your house versus giving them a guided tour. Without llms.txt, AI crawlers treat every page equally -- your homepage, your privacy policy, that test page from three years ago. With llms.txt, they know where the good stuff is.

Setting one up takes less than 10 minutes on WordPress. You can do it manually or use a plugin that generates it automatically.

4. Decide: Block Training Bots, Allow Search Bots

This is the strategic decision every site owner needs to make, and there's no universal right answer.

Block training crawlers if:

• You create original, proprietary content (original research, unique data, premium guides)
• You don't want your content used to train AI models without compensation
• Your hosting resources are limited and training bots consume significant bandwidth

Allow training crawlers if:

• You want maximum brand presence in AI model knowledge (the model "knows" your brand)
• You publish public documentation or open-source resources
• You believe broader AI exposure drives long-term brand authority

Search bots are almost always worth allowing. AI search referral traffic is growing 357% year-over-year, converts at 5x the rate of Google traffic, and this channel is still in its early stages. Blocking search bots means voluntarily opting out of the fastest-growing referral channel on the web.

The AI Bots Manager in Prime SEO lets you control access for 16 individual AI crawlers, so you can allow PerplexityBot while blocking Bytespider -- without editing config files manually.

5. Optimize Your Content for AI Citation

Allowing AI crawlers is only half the equation. You also need to make your content worth citing.

AI search engines don't just grab random text from your page. They evaluate authority, structure, and specificity. Content that includes concrete data, cites credible sources, and provides clear, quotable answers gets cited more frequently. The 15-point checklist for getting cited by ChatGPT covers this in detail.

The short version: write content that an AI system would want to quote. Definitive statements. Specific numbers. Clear expertise. If your content reads like it was written to fill a word count, no AI is going to cite it.

The Bottom Line

AI crawler traffic isn't a curiosity anymore. It's a significant, fast-growing category of web traffic that impacts your server costs, your SEO performance, and your visibility in the fastest-growing search channel on the internet.

The sites that get this right early -- monitoring their AI traffic, making strategic decisions about crawler access, and optimizing content for AI citation -- will have a structural advantage as AI search captures an increasing share of how people find information online.

The sites that ignore it will find themselves paying the bandwidth costs of AI crawling while capturing none of the benefits.

Start with step one: find out who's crawling your site. Everything else follows from there.

Frequently Asked Questions

How do I check if AI crawlers are visiting my WordPress site?

You have two options. First, you can check your server's raw access logs (usually available through your hosting control panel) and search for user agent strings like "GPTBot," "ClaudeBot," "PerplexityBot," and "Bytespider." Second, you can use a WordPress plugin with AI crawler monitoring. Prime SEO's AI Crawler Stats automatically detects and logs visits from 16 known AI bots without requiring server log access.

Will blocking AI crawlers hurt my Google rankings?

No. Google's search rankings are determined by Googlebot, which is separate from AI training crawlers. Blocking GPTBot, ClaudeBot, or other AI crawlers has no direct effect on your Google search position. However, blocking AI search bots will prevent your content from appearing in AI-generated answers on platforms like ChatGPT and Perplexity, which means missing out on a high-converting traffic channel.

How much bandwidth do AI crawlers actually use?

It varies dramatically by site size and visibility. The Read the Docs project saw AI crawlers consuming 600GB per day before blocking them. For smaller WordPress sites, the impact might be a few gigabytes per month. The key issue is that AI crawlers often revisit pages repeatedly (some every 6 hours) rather than crawling once and caching. If you're on metered hosting, monitor your bandwidth usage after identifying AI crawler activity.

Should I block all AI bots or only some of them?

For most WordPress sites, a selective approach works best. Allow search-oriented bots like PerplexityBot and ChatGPT-User that drive referral traffic, while considering blocking training-only crawlers like CCBot, Google-Extended, and Bytespider that consume bandwidth without sending visitors. This guide includes ready-made robots.txt templates for different strategies.