This article shows you how to stop AI companies from crawling your stories while keeping Google, Bing and other search engines working normally. It takes about 15 minutes.
Before you start
- You need administrator access to your WordPress dashboard.
- Part 1 covers the Yoast SEO and All in One SEO (AIOSEO) plugins. Use the steps for whichever plugin your site has. If your site uses neither, see Troubleshooting.
- Part 2 applies only if your site uses Cloudflare. You need access to your Cloudflare account. If you’re not sure whether you use Cloudflare, contact PaywallProject support.
Important: Never block Googlebot, Bingbot, Applebot or DuckDuckBot. Blocking them removes your site from search results.
Part 1: Update your robots.txt file
Your robots.txt file tells bots which parts of your site they may crawl. The major AI companies say they follow it.
Here’s the robots.txt text you’ll add. Copy it, then follow the steps for your SEO plugin below.
# ===== AI training crawlers: blocked =====
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Diffbot
User-agent: Omgilibot
User-agent: Timpibot
User-agent: ImagesiftBot
User-agent: PanguBot
User-agent: AI2Bot
User-agent: FacebookBot
Disallow: /
# ===== AI search / answer / on-demand fetchers: blocked =====
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-externalfetcher
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: YouBot
Disallow: /
# ===== Search engines and everyone else: allowed =====
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xmlOption A: Yoast SEO
- Log in to your WordPress dashboard.
- Go to Yoast SEO → Tools → File editor.
- If you see a Create robots.txt file button, click it. Otherwise, the editor opens with your current file.
- Find the line that starts with
Sitemap:and copy it somewhere. You’ll need it in step 6. - Delete everything in the robots.txt box and paste in the robots.txt text from above.
- Replace the last line (
Sitemap: https://example.com/sitemap_index.xml) with the sitemap line you copied in step 4. - Click Save changes to robots.txt.
Option B: All in One SEO (AIOSEO)
- Log in to your WordPress dashboard.
- Go to All in One SEO → Tools and open the Robots.txt Editor tab.
- Turn on Enable Custom Robots.txt.
- In the robots.txt text from above, replace
example.comin the last line with your own domain. - Click Import. Paste the robots.txt text into the Paste Robots.txt text box, then click Import.
- Check the rules list. Each AI bot should appear with the Block directive. AIOSEO uses “Block” where robots.txt says “Disallow”; they mean the same thing.
- Click Save Changes.
AIOSEO keeps WordPress’s default rules and adds yours to them, so you’ll see a few extra lines in the preview. That’s expected. To view the live file, click Open Robots.txt.
Note: AIOSEO’s rules only take effect when your site has no physical robots.txt file. If AIOSEO shows a notice about an existing robots.txt file (for example, one created earlier with Yoast), contact PaywallProject support and we’ll remove it for you.
To confirm it worked: visit yoursite.com/robots.txt in your browser. You should see the new rules. If you still see the old version, clear your site cache (and your Cloudflare cache, if you use it) and reload.
Part 2: Set up Cloudflare’s AI bot controls
Some AI bots ignore robots.txt. Cloudflare can block them before they reach your site. Skip this part if your site doesn’t use Cloudflare.
Warning: Do not set Training to Block or Block on pages with ads. Cloudflare counts Googlebot, Bingbot and Applebot as training crawlers too, so either option can remove your site from search. Block on pages with ads is the default for many sites, so check this setting even if you’ve never changed it.
- Log in to the Cloudflare dashboard and select your domain.
- Go to Security → Settings.
- Find the AI bot settings and set them as follows:
| Setting | Choose | What it does |
|---|---|---|
| Search | Allow | Keeps Google, Bing and other search engines crawling your site. |
| Training | Disallow AI Training | Blocks AI training crawlers and tells Google, Bing and Apple not to train on your content, while still letting them index you for search. |
| Agent | Block | Stops AI assistants from fetching your stories on demand. |
- Save your changes.
- Optional: turn on AI Labyrinth on the same page. It sends AI crawlers that ignore your rules into a maze of decoy pages.
Cloudflare also adds its own AI rules to the top of your robots.txt file. This is expected and works alongside the rules from Part 1.
Optional: Block specific AI bots by name
Some AI search bots can still get through under Cloudflare’s Search setting. To block them by name:
- In Cloudflare, go to Security → Security rules (on some accounts, WAF → Custom rules).
- Click Create rule and name it “Block AI bots by name.”
- Click Edit expression and paste in the text below.
- Under Choose action, select Block.
- Click Deploy.
(http.user_agent contains "GPTBot") or
(http.user_agent contains "OAI-SearchBot") or
(http.user_agent contains "ChatGPT-User") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "Claude-SearchBot") or
(http.user_agent contains "Claude-User") or
(http.user_agent contains "PerplexityBot") or
(http.user_agent contains "Perplexity-User") or
(http.user_agent contains "meta-externalagent") or
(http.user_agent contains "meta-externalfetcher") or
(http.user_agent contains "Bytespider") or
(http.user_agent contains "CCBot") or
(http.user_agent contains "Diffbot") or
(http.user_agent contains "MistralAI-User")Part 3: Check that search engines can still reach your site
- Open Google Search Console and select your site.
- Paste the address of a recent story into the URL Inspection bar at the top.
- Click Test live URL. The result should say the page is available to Google.
- Check Search Console again over the next week for new crawl errors.
If Google reports that it’s blocked, go back to Part 2 and make sure Training is set to Disallow AI Training, not Block.
Add a free registration wall (recommended)
robots.txt and Cloudflare stop bots that identify themselves. A free registration wall adds another layer: readers see the opening of each story, then create a free account with their email address to keep reading. Bots don’t create accounts, and every new account grows your newsletter list.
To turn on a free registration wall for your site, contact PaywallProject support or book a call and we’ll set it up with you.
Troubleshooting
I don’t see “File editor” in Yoast or the Robots.txt Editor in AIOSEO
Your host may have turned off file editing in WordPress. Contact PaywallProject support and we’ll update your robots.txt for you. You can also upload a plain text file named robots.txt to your site’s root folder using FTP or your host’s file manager.
My site doesn’t use Yoast or AIOSEO
Most other SEO plugins, such as Rank Math, include their own robots.txt editor. Paste the same text there. If you’re not sure, contact support.
robots.txt still shows the old version
Clear your WordPress cache, then purge your Cloudflare cache (Caching → Configuration → Purge Everything), and reload the page.
Does this stop Google AI Overviews?
No. AI Overviews use Google’s regular search crawler, so you can’t opt out of them without leaving Google Search. The Google-Extended line in your robots.txt does keep your content out of Google’s Gemini AI models.
How often should I update this?
New AI bots appear regularly. Review your robots.txt and Cloudflare rules every few months. The ai.robots.txt project on GitHub keeps a current list of AI bot names.