Block AI crawlers in WordPress: what the free code of five SEO plugins does

Of five free WordPress SEO plugins checked on September 21 and re-tested on September 24, 2026, two write robots.txt rules that block AI crawlers: All in One SEO Lite for three crawler names, Prime SEO for 15. Yoast SEO stores a GPTBot setting in its free build, and no code in that build reads it. Rank Math and SEOPress contain no AI crawler names at all.

Disclosure: the author develops Prime SEO, the fifth plugin in this comparison. Every claim below comes from the free packages downloaded from the WordPress.org plugin directory, searched for crawler names and for the code that acts on each setting; file paths are given so that anyone can repeat the check.

Correction, September 24, 2026. The September 21 version of this article said that All in One SEO Lite stores its GPTBot setting and never reads it. The claim was wrong. On a test install with 5.0.2 Lite, one tick and a save put User-agent: GPTBot into robots.txt.

The code that writes the rule sits in the JavaScript of the settings screen, not in the PHP folders the September 21 check covered, and the older 5.0.1.1 package carries the same code. The table, the All in One SEO section and the closing paragraph are updated below.

Yoast SEO free stores three switches it never reads

Yoast SEO, active on 10 million sites according to the directory, defines three options in inc/options/class-wpseo-option-wpseo.php: deny_gptbot_crawling, deny_ccbot_crawling and deny_google_extended_crawling. All default to false. The settings screen even carries the label "Prevent OpenAI GPTBot from crawling" under Crawl optimization.

What the free build lacks is a reader. Its robots.txt code, src/integrations/front-end/robots-txt-integration.php, checks three neighbouring options, among them internal search and Google AdsBot, and stops there. In the settings script the GPTBot toggle receives isDummy whenever the site's isPremium preference is false, and the same component loads a link named block-unwanted-bots-upsell. A free user sees the switch as a preview of Premium.

All in One SEO Lite writes the rule from its JavaScript

All in One SEO, installed on 2 million sites, declares an unwantedBots group in app/Common/Options/Options.php with openAiGptBot, commonCrawlCcBot and googleGeminiVertexAiBots next to a Google AdsBot entry. Across the PHP folders those names appear in two option files and nowhere else, which is where the September 21 version of this article stopped. The reader is in dist/Lite/assets/js/RobotsTxt.*.js: when Save Changes is pressed, the settings screen turns each ticked bot into a rule (userAgent, disallow, /, flagged bot: true) and stores it under tools.robots.rules.

app/Common/Tools/RobotsTxt.php then prints every rule that carries the bot flag, through extractSearchAppearanceRules() on the robots_txt filter, whether the Robots.txt Editor is on or off. On a WordPress 7.1 test install with version 5.0.2 Lite, one tick on OpenAI GPTBot under Search Appearance, Advanced, Crawl Cleanup, Unwanted Bots produced User-agent: GPTBot and Disallow: / on the next request. The 5.0.1.1 package on the WordPress.org SVN carries the same JavaScript, so the switch already worked in the version checked on September 21.

The list is short. GPTBot, CCBot and Google-Extended (labelled Google Gemini and Vertex AI bots) are on it, plus AdsBot, which is not an AI crawler; ClaudeBot, PerplexityBot, Bytespider and the other names in the Prime SEO list are not, and the Lite build keeps no log of which of them visit.

Rank Math and SEOPress free have no AI crawler code

A search of Rank Math (4 million active installs) for 14 crawler names, from GPTBot to Amazonbot, returns nothing outside a bundled user agent library. SEOPress returns nothing at all. Rank Math free generates an llms.txt file, and SEOPress keeps that feature for its PRO version.

SEOPress PRO, a paid product, goes further: the 9.5 copy checked for this article offers a robots.txt template with rules for CCBot, GPTBot, Google-Extended and ClaudeBot. None of the four free builds logs visits from AI crawlers. Each package reads the visitor's user agent in at most one file, and no crawler name appears in any of them.

Three lines of robots.txt need no plugin

Blocking a crawler does not require any of the five. OpenAI's crawler documentation says "GPTBot is used to make our generative AI foundation models more useful and safe" and accepts an ordinary robots.txt rule:

User-agent: GPTBot
Disallow: /

The same two lines work for ClaudeBot or for Google-Extended, the token that, in Google's words, "does not impact a site's inclusion in Google Search." On WordPress the catch is practical. The default robots.txt is virtual, many hosts offer no file access, and a list of 15 names kept by hand goes stale as vendors rename their agents.

Few site owners bother either way. Among the robots.txt files that Cloudflare could read on top domains, GPTBot was the most refused AI agent and was still, João Tomé of Cloudflare wrote in a July 2025 study, "disallowed by 312 domains" only. A rule also binds nobody: agents that fetch a page for a live user, ChatGPT-User and Perplexity-User among them, are documented by their vendors as outside robots.txt, a point covered in the guide to AI crawlers in WordPress logs.

Prime SEO 1.2.0 writes rules for 15 crawlers

Prime SEO puts the feature in its free version. The AI Bots Manager lists 15 crawlers with a switch each, plus three presets (allow all, block training crawlers only, block all). Matching robots.txt lines are appended to the virtual file, and a separate screen logs which AI agents visited and which page they opened.

Its weak side is plain. Prime SEO has 100+ active installs against Yoast's 10 million and a record that starts in January 2026. Two of the names on its list (anthropic-ai and Claude-Web) are ones Anthropic no longer documents. Those two rules block nothing today.

A site that needs one blocked crawler and nothing else is served as well by the three-line rule above (no plugin, no updates to follow). For the rest of the setup, see the llms.txt plugin for WordPress page.

The option keys are already in place

Yoast would need little work to change this table, because the setting exists and only the reader is missing from its free code. All in One SEO Lite already reads its three names; what it lacks is the rest of the list and a log. If Yoast moves the switch out of the paid tier, or All in One SEO extends its list past three names, free AI crawler control stops being a reason to choose a smaller plugin. Until then the comparison stands at two in five.