Skip to content
Start free trial

Free robots.txt generator

Choose what crawlers may fetch, block GPTBot, ClaudeBot and other AI crawlers (or only the ones that train models), and add your sitemap. You get a robots.txt file to copy or download, checked against Google's rules. Free, no sign-up, and it all runs in your browser.

Short answer

A robots.txt file tells crawlers which URLs they may fetch. It lives at the root of your site (yoursite.com/robots.txt) and holds groups: a User-agent line naming a crawler, then Disallow and Allow lines with paths. Pick your rules below and the generator writes the file, then reads it back the way Google does so you can see who gets in.

Every crawler

One per line. A path blocks everything that starts with it, so /admin/ covers every page in that folder. * matches anything and $ marks the end of the URL.

Blocks /wp-admin/ but allows admin-ajax.php, like WordPress does.

AI crawlers

Model training and datasets

AI search answers

Pages a user asks for

Optional, one per line. Use the full URL. Any host works, but list each one you want crawlers to find.

Your robots.txt

User-agent: *
Allow: /

Upload it to the root of your site, so it opens at yoursite.com/robots.txt. A file in a folder is ignored.

Who may crawl your home page

Read back from the file above with the robots.txt tester's rules (Google's).

  • GooglebotAllowed
  • BingbotAllowed
  • GPTBotAllowed
  • OAI-SearchBotAllowed
  • ChatGPT-UserAllowed
  • ClaudeBotAllowed
  • Claude-SearchBotAllowed
  • Claude-UserAllowed
  • PerplexityBotAllowed
  • Perplexity-UserAllowed
  • Google-ExtendedAllowed
  • Applebot-ExtendedAllowed
  • CCBotAllowed
  • meta-externalagentAllowed
  • Meta-WebIndexerAllowed
  • meta-externalfetcherAllowed

Want to check other URLs, or a file you already have? Paste it into the robots.txt tester.

Serpwright finds the searches your site shows up for but has no page for, then writes and publishes the articles you approve, then checks in Search Console whether Google has indexed each one.

Start free trial

3 days free, no card.

AI crawlers and what each one does

From each company's own crawler page, checked November 13, 2026. Tokens are not case-sensitive.

  • GPTBot

    OpenAI · Training

    OpenAI's crawler for content that may be used to train its AI models.

  • OAI-SearchBot

    OpenAI · AI search

    Shows sites in ChatGPT search. OpenAI says sites that block it won't be shown in ChatGPT search answers, though they can still appear as navigational links.

  • ChatGPT-User

    OpenAI · User fetch

    Opens pages for actions a ChatGPT user asks for. OpenAI says robots.txt rules may not apply.

  • ClaudeBot

    Anthropic · Training

    Anthropic's crawler for content that may be used to train its AI models.

  • Claude-SearchBot

    Anthropic · AI search

    Indexes pages to improve Claude's search results. Blocking it stops that indexing.

  • Claude-User

    Anthropic · User fetch

    Opens a page when a Claude user's question needs it. Anthropic says it follows robots.txt.

  • PerplexityBot

    Perplexity · AI search

    Shows and links sites in Perplexity's search results. Perplexity says it isn't used to train AI models.

  • Perplexity-User

    Perplexity · User fetch

    Opens a page when a Perplexity user asks. Perplexity says it generally ignores robots.txt.

  • Google-Extended

    Google · Training

    Not a separate crawler: it decides whether pages Google crawls may train Gemini and ground Gemini's answers, so blocking it does both. Google says it doesn't affect Google Search.

  • Applebot-Extended

    Apple · Training

    Doesn't crawl: it decides whether pages Applebot crawls may train Apple's AI models. Apple says blocked pages can still show in its search.

  • CCBot

    Common Crawl · Open dataset

    Builds Common Crawl's free, open archive of web pages, which anyone can download and use, AI researchers included.

  • meta-externalagent

    Meta · Training

    Meta's crawler for training AI models and for indexing content for its products.

  • Meta-WebIndexer

    Meta · AI search

    Improves Meta AI's search results and helps it cite and link your pages.

  • meta-externalfetcher

    Meta · User fetch

    Fetches a link when a user asks, including for AI that completes tasks on sites. Meta says it may bypass robots.txt.

How the generator writes the file

  1. One group for everyone

    Your site-wide rules go in a single User-agent: * group. Googlebot, Bingbot and every AI crawler you leave alone follow it.

  2. One group per blocked AI crawler

    Each AI crawler you block gets its own group with Disallow: /. A crawler that finds a group with its name ignores the * group, so that group has to block everything on its own.

  3. Paths written the way crawlers read them

    Every path starts with /. A full URL is cut to its path, a # comment is dropped, a space becomes %20 (a crawled URL never has a bare space), and a pasted Allow or Disallow line keeps its meaning whichever box it's in.

  4. Read back before you copy

    The file is checked with the same code as our robots.txt tester, which follows Google's rules: the longest matching rule wins, and Allow wins a tie.

What the companies say

Checked November 13, 2026.

  • OpenAI: “Each setting is independent of the others,” so a site can allow OAI-SearchBot to appear in search while blocking GPTBot. For ChatGPT-User, “robots.txt rules may not apply” (OpenAI's crawlers).
  • Anthropic: blocking ClaudeBot “signals that the site's future materials should be excluded from our AI model training datasets”; blocking Claude-SearchBot or Claude-User stops search indexing or user fetches (Anthropic's crawler help).
  • Google: Google-Extended “does not impact a site's inclusion in Google Search nor is it used as a ranking signal” (Google's common crawlers). For AI Overviews and AI Mode, robots.txt rules for Googlebot are the control (AI features and your website).
  • Apple: “Applebot-Extended does not crawl webpages”, and pages that block it “can still be included in search results” (About Applebot).
  • Perplexity-User “generally ignores robots.txt rules” because a user asked for the page (Perplexity's crawlers). Meta says meta-externalfetcher “may bypass robots.txt rules” and lists meta-externalagent and Meta-WebIndexer (Meta's web crawlers), and Common Crawl documents CCBot (CCBot).
  • Google reads only user-agent, allow, disallow and sitemap, and “other fields such as crawl-delay aren't supported” (How Google interprets the robots.txt specification).

The read-back uses Google's matching rules for every crawler. Most crawlers follow the same standard (RFC 9309), but each company's page has the final word on its crawler. Companies add and rename crawlers; we re-check these pages and update the list.

Pages worth crawling

Serpwright reads your Search Console data, finds the searches your site shows up for but has no page for, then writes and publishes an article for each one you approve.

  • Articles go to WordPress, Ghost, Webflow or Shopify, with their title, description and schema markup.
  • With Search Console connected, it checks each article a day after it goes live, again after a week if it isn't indexed yet, and tells you in the app if it still isn't.
  • Nothing goes live until you approve it.
Start free trial Test a robots.txt file

3 days free, no card.

Questions

How do I block AI crawlers with robots.txt?
Give each AI crawler its own group: "User-agent: GPTBot" and then "Disallow: /" on the next line, and the same for ClaudeBot, CCBot and the others. Tick the ones you want blocked in the generator and it writes those groups. Each company names its own crawlers, so a crawler that isn't named in your file follows your * rules.
Can I block AI training but still show up in ChatGPT and other AI search?
Yes, for the companies that split the two. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search, and says each setting is independent. Anthropic splits ClaudeBot (training) from Claude-SearchBot (search). Pick "Block AI training, keep AI search" to block the training crawlers and Common Crawl's CCBot and leave the search ones alone. Google is the exception: Google-Extended covers both training Gemini and grounding Gemini's answers, so blocking it does both, while Google Search itself is unaffected.
Does blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." It only decides whether your pages may train Gemini and ground its answers. Google's AI Overviews and AI Mode use Googlebot, so the only robots.txt control for those is Googlebot itself, which also removes you from Search.
Will every AI crawler obey my robots.txt?
Not every request. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person asked for the page, Perplexity says Perplexity-User generally ignores robots.txt, and Meta says meta-externalfetcher may bypass it. robots.txt is a request crawlers choose to follow. To truly keep a page private, put it behind a login.
Does Disallow keep a page out of Google?
No. Disallow stops crawling, not indexing. Google can still list a blocked URL that other pages link to, without a description. To keep a page out of search, let Google crawl it and add a noindex robots meta tag or an X-Robots-Tag header.
Should I add Crawl-delay?
Google ignores it: its robots.txt page says it supports user-agent, allow, disallow and sitemap, and that other fields such as crawl-delay aren't supported. Apple says Applebot doesn't follow it either. Some crawlers do read it (Anthropic says it supports Crawl-delay), so add a line by hand under a crawler's group if one of them is hitting your server too hard. To slow Googlebot down, return 503 or 429 for a short time.
Where do I put the file?
At the root of each host: https://example.com/robots.txt. A robots.txt in a folder is ignored, and a file only covers the protocol, host and port it's served from, so shop.example.com needs its own. WordPress and Shopify make one for you; WordPress lets SEO plugins edit it, and Shopify uses a robots.txt.liquid theme file.
Does this tool save what I type?
No. The file is built in your browser and nothing is sent to us or fetched. We count one anonymous use when you copy or download a file, recording only whether the whole site is open or blocked and which AI preset you picked.