1. Build your rules
Start from a preset, then adjust. A preset replaces the current rules.
Sitemap lines apply to the whole file and must be absolute URLs.
2. Your robots.txt
Upload the file to the root of your site so it is reachable at /robots.txt.
Validation
3. Test a URL against these rules
Uses Google-style matching: the longest matching rule wins, and Allow wins a tie.
Should you block AI crawlers?
It is a business decision, and it does not change how you rank in Google or Bing. What it changes is whether AI companies can use your pages for model training or for answering questions. The presets above split the two groups.
| Group | User agents in the preset | What blocking does |
|---|---|---|
| Training and data collection | GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, Bytespider | Low SEO risk Asks these crawlers not to use your content for training. Google-Extended and Applebot-Extended are control tokens: they do not change Google Search or regular Applebot crawling. |
| AI search and assistants | OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User | Visibility trade-off These fetch pages to power AI search answers or to read a page a user asked about. Blocking them can mean your site is not cited or summarised there. Some user-triggered fetchers may not follow robots.txt at all. |
| Google and Bing search | Googlebot, Bingbot | Do not block Blocking these removes your pages from normal search results. The AI presets never touch them. |
It is a request, not a lock
robots.txt is voluntary. Well-behaved crawlers follow it; others may ignore it. To enforce a block you need server or firewall rules, which this file does not do.
It does not remove what is already collected
Blocking stops future crawling by compliant bots. It does not delete content already used or indexed elsewhere.
Bot names and policies change
Vendors add and rename crawlers. Check each vendor's current documentation before relying on this list, and review the file periodically.
Practical guidance
The mistakes that cause real damage are usually simple ones.
robots.txt controls crawling, not indexing
A blocked URL can still appear in results if other sites link to it. To keep a page out of search, leave it crawlable and add a noindex tag. If you block it, Google cannot see that tag.
Do not block CSS, JavaScript or images
Search engines need them to render your pages. Blocking /wp-content/ or /wp-includes/ wholesale can hurt how pages are understood.
WordPress basics
- Disallow
/wp-admin/but allow/wp-admin/admin-ajax.php. - Never disallow
/wp-content/uploads/if you want images indexed. - List your sitemap index, for example
/sitemap_index.xml.
Check before you publish
- Test your important URLs in the tester above.
- After uploading, open
yoursite.com/robots.txtin a browser. - Use the robots.txt report in Google Search Console to confirm Google can fetch it.
Where does the file go?
In the root of the host it applies to, so https://example.com/robots.txt. A file in a sub-folder is ignored, and each host (including www versus non-www, and subdomains) needs its own.
Is Crawl-delay worth setting?
Google ignores it. Bing and some other crawlers honour it. Only set it if a specific bot is overloading your server.
What does an empty Disallow mean?
Disallow: with nothing after it means nothing is blocked for that user agent. Disallow: / blocks the whole site.