AI Bots Manager

Overview

Prime SEO includes a built-in AI Bots Manager that lets you control which AI crawlers can access your site. Unlike traditional SEO that focuses on Google, the AI Bots Manager deals specifically with AI search engines (ChatGPT, Perplexity, Claude) and AI training crawlers (GPTBot, CCBot, Google-Extended).

As AI-powered search becomes more prevalent, controlling how these bots interact with your content is critical. The AI Bots Manager gives you granular control over 16 known AI crawlers, letting you decide which ones can access your site and which ones should be blocked.

This is one of the features that makes Prime SEO unique — most traditional SEO plugins don't offer any AI crawler management at all.

Accessing AI Bots Manager

Navigate to Prime SEO → AI Settings in your WordPress admin panel. The AI Bots Manager is the first section on the page, displayed above the LLMs.txt settings.

You'll see a preset selector at the top, followed by a list of all 16 supported AI bots with their current status.

Presets

Prime SEO provides four presets to quickly configure your AI bot rules. Select a preset from the dropdown at the top of the AI Bots Manager section:

• Allow All — All AI crawlers can access your site. This is the default setting and is recommended for most sites that want maximum visibility in AI search results.
• Block Training — Allows AI search bots (like ChatGPT-User and PerplexityBot) but blocks training-only crawlers (like GPTBot and CCBot). Your content will still appear in AI search results, but it won't be used to train AI models.
• Block All — Blocks all AI crawlers completely. Your content will not appear in any AI search results and will not be used for AI training.
• Custom — Configure each bot individually. Use this when you need fine-grained control over specific crawlers.

Individual Bot Control

When using the Custom preset, you can toggle each of the 16 supported AI bots individually. Each bot entry displays the bot name, the company behind it, its type (Search or Training), and the current status (Allowed or Blocked).

Here is the complete list of AI bots that Prime SEO manages:

To block a specific bot, switch to the Custom preset, then toggle the bot off. Click Save Changes to apply your configuration.

Bot Categories

AI bots are divided into two main categories based on their purpose:

Search Bots

Search bots are used by AI-powered search products to find answers for users in real time. When someone asks ChatGPT a question, asks Perplexity for research, or uses Google Gemini, these bots crawl the web to find relevant content.

Examples include ChatGPT-User, OAI-SearchBot, PerplexityBot, Gemini, Amazonbot, Diffbot, and Timesbot.

Blocking search bots means your content will not appear in AI-powered search results. For most sites, this is not desirable since AI search is becoming an increasingly important source of traffic and citations.

Training Bots

Training bots crawl websites to collect data for training AI language models. This data is used to improve the AI's knowledge and capabilities, but it does not directly result in your content being shown to users in search results.

Examples include GPTBot, ClaudeBot, Google-Extended, Bytespider, CCBot, FacebookBot, Applebot-Extended, and Cohere-ai.

Blocking training bots prevents your content from being used to train AI models while still allowing AI search engines to find and cite your content. This is a good middle ground for sites that want AI visibility without contributing to model training.

Robots.txt Integration

Prime SEO automatically adds the appropriate rules to your site's robots.txt file based on your AI bot settings. You do not need to manually edit your robots.txt file.

When you block a bot, Prime SEO adds a Disallow: / rule for that bot's user-agent. For example, if you block GPTBot and CCBot, the following rules are automatically added:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

When you allow a bot, no rule is needed since bots are allowed by default when no specific rule exists in robots.txt.

These rules are injected dynamically and do not modify any physical robots.txt file on your server. They work through WordPress's virtual robots.txt system, which means they apply automatically and are removed cleanly if you deactivate the plugin.

Best Practices

Here are some recommendations for configuring the AI Bots Manager based on different types of websites:

• If you want maximum AI visibility — use the "Allow All" preset. This ensures your content can appear in all AI search products and is the best option for growing your reach.
• If you want AI search but not training — use the "Block Training" preset. Your content will still appear in ChatGPT, Perplexity, and Gemini search results, but it won't be used to train future AI models.
• For news and media sites — allowing search bots is crucial for appearing in AI-powered news summaries and answers. Consider using "Allow All" or "Block Training" to maximize citations.
• For sites with proprietary content — if your content is behind a paywall or is highly proprietary, "Block All" may be appropriate. However, keep in mind this also removes you from AI search results.
• Review your settings periodically — new AI crawlers emerge regularly. Prime SEO updates its bot list with plugin updates, so make sure to check your settings after each update to ensure new bots are configured to your preference.