Skip to content
SuperSEOPlus Technical SEO audit
Free technical SEO tool

Robots.txt Generator

Decide what crawlers may fetch. Set rules per user agent, add your sitemaps, test any URL against the rules, then copy or download the file. Everything runs in your browser and nothing is uploaded.

1. Build your rules

Start from a preset, then adjust. A preset replaces the current rules.

Sitemap lines apply to the whole file and must be absolute URLs.

2. Your robots.txt

    
            

    Upload the file to the root of your site so it is reachable at /robots.txt.

    Validation

      3. Test a URL against these rules

      Uses Google-style matching: the longest matching rule wins, and Allow wins a tie.

      Should you block AI crawlers?

      It is a business decision, and it does not change how you rank in Google or Bing. What it changes is whether AI companies can use your pages for model training or for answering questions. The presets above split the two groups.

      AI crawler groups and what blocking them does
      GroupUser agents in the presetWhat blocking does
      Training and data collectionGPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, BytespiderLow SEO risk Asks these crawlers not to use your content for training. Google-Extended and Applebot-Extended are control tokens: they do not change Google Search or regular Applebot crawling.
      AI search and assistantsOAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-UserVisibility trade-off These fetch pages to power AI search answers or to read a page a user asked about. Blocking them can mean your site is not cited or summarised there. Some user-triggered fetchers may not follow robots.txt at all.
      Google and Bing searchGooglebot, BingbotDo not block Blocking these removes your pages from normal search results. The AI presets never touch them.

      It is a request, not a lock

      robots.txt is voluntary. Well-behaved crawlers follow it; others may ignore it. To enforce a block you need server or firewall rules, which this file does not do.

      It does not remove what is already collected

      Blocking stops future crawling by compliant bots. It does not delete content already used or indexed elsewhere.

      Bot names and policies change

      Vendors add and rename crawlers. Check each vendor's current documentation before relying on this list, and review the file periodically.

      Practical guidance

      The mistakes that cause real damage are usually simple ones.

      robots.txt controls crawling, not indexing

      A blocked URL can still appear in results if other sites link to it. To keep a page out of search, leave it crawlable and add a noindex tag. If you block it, Google cannot see that tag.

      Do not block CSS, JavaScript or images

      Search engines need them to render your pages. Blocking /wp-content/ or /wp-includes/ wholesale can hurt how pages are understood.

      WordPress basics

      • Disallow /wp-admin/ but allow /wp-admin/admin-ajax.php.
      • Never disallow /wp-content/uploads/ if you want images indexed.
      • List your sitemap index, for example /sitemap_index.xml.

      Check before you publish

      • Test your important URLs in the tester above.
      • After uploading, open yoursite.com/robots.txt in a browser.
      • Use the robots.txt report in Google Search Console to confirm Google can fetch it.
      Where does the file go?

      In the root of the host it applies to, so https://example.com/robots.txt. A file in a sub-folder is ignored, and each host (including www versus non-www, and subdomains) needs its own.

      Is Crawl-delay worth setting?

      Google ignores it. Bing and some other crawlers honour it. Only set it if a specific bot is overloading your server.

      What does an empty Disallow mean?

      Disallow: with nothing after it means nothing is blocked for that user agent. Disallow: / blocks the whole site.

      Related tools, guides and services