Free robots.txt tester
Paste your robots.txt and the URLs you want to check, then pick Googlebot, Bingbot or an AI crawler such as GPTBot. This robots.txt checker shows which URLs are blocked, the line that decides, and lines Google ignores. Free, no sign-up, and nothing you paste leaves your browser.
Short answer
A robots.txt tester tells you whether a crawler may fetch a URL under your robots.txt rules. Paste the file and the URLs below and pick a crawler. Google follows the group that names it (or * if none does), then the longest matching rule; if an Allow and a Disallow rule are the same length, Allow wins.
Open yoursite.com/robots.txt, select all and copy, or paste a draft before you upload it. Nothing is uploaded.
Google's example file. Replace it with yours.
One per line, full URLs or paths starting with /, up to 100. Rules are case-sensitive and cover the query string too.
Google Search, on phones and computers.
Googlebot may crawl all 2 URLs.
Results for Googlebot
Follows the "Googlebot" group; the * group is ignored.
https://example.com/includes/site.css
Allowed · Line 6: Allow: /includes/
Path tested: /includes/site.css
https://example.com/blog/first-post
Allowed · No rule matches, so it's allowed.
Path tested: /blog/first-post
Every crawler on the first URL
- GooglebotAllowed
- Googlebot-ImageAllowed
- Googlebot-NewsAllowed
- Storebot-GoogleBlocked
- AdsBot-GoogleAllowed
- BingbotBlocked
- OAI-SearchBotBlocked
- GPTBotBlocked
- ClaudeBotBlocked
- PerplexityBotBlocked
https://example.com/includes/site.css
Sitemap: https://example.com/sitemap.xml
Serpwright finds the searches your site shows up for but has no page for, then writes and publishes the articles you approve, then checks in Search Console whether Google has indexed each one.
3 days free, no card.
How Google reads robots.txt
From Google's robots.txt documentation, checked November 12, 2026.
One group per crawler
A group is one or more User-agent lines and the rules under them. A crawler follows every group that names it, merged into one, and ignores the * group when any group names it, even if that group has no rule for the URL.
The longest rule wins
Of the Allow and Disallow rules that match a URL, Google uses the one with the longest path. When an Allow and a Disallow rule are the same length, Allow wins.
* and $
* matches any run of characters, and $ at the end of a rule means the URL ends there. Rules match from the start of the path, are case-sensitive, and include the query string.
Only four fields count
Google reads user-agent, allow, disallow and sitemap. It ignores crawl-delay, noindex, host and anything else, and skips lines it can't read.
Names stop at a slash
A user-agent value is read up to the first character that isn't a letter, - or _, and case doesn't matter. Googlebot/2.1 and googlebot* both mean Googlebot.
Size limit
Google reads the first 500 KiB of the file and ignores the rest.
Which rule wins: Google's examples
When an Allow and a Disallow rule both match, the longer one decides. The tester gives the same answer for each of these.
| URL path | Rules | Rule that applies |
|---|---|---|
| /page | allow: /p disallow: / | allow: /pthe longer rule is more specific |
| /folder/page | allow: /folder disallow: /folder | allow: /foldera tie goes to the less restrictive rule |
| /page.htm | allow: /page disallow: /*.htm | disallow: /*.htmthe longer rule matches more of the URL |
| /page.php5 | allow: /page disallow: /*.ph | allow: /pagesame length, so the less restrictive rule wins |
| / | allow: /$ disallow: / | allow: /$the longer rule is more specific |
| /page.htm | allow: /$ disallow: / | disallow: /allow: /$ only matches the home page |
What * and $ match
Google's path examples. Matching is case-sensitive, so /fish doesn't match /Fish.asp.
- /fish
- Matches /fish /fish.html /fish/salmon.html /fishheads /fish.php?id=anything
- Doesn't match /Fish.asp /catfish /?id=fish /desert/fish
- /fish/
- Matches /fish/ /fish/?id=anything /fish/salmon.htm
- Doesn't match /fish /fish.html /animals/fish/ /Fish/Salmon.asp
- /*.php
- Matches /index.php /filename.php /folder/filename.php?parameters /folder/any.php.file.html /filename.php/
- Doesn't match / /windows.PHP
- /*.php$
- Matches /filename.php /folder/filename.php
- Doesn't match /filename.php?parameters /filename.php/ /filename.php5 /windows.PHP
- /fish*.php
- Matches /fish.php /fishheads/catfish.php?parameters
- Doesn't match /Fish.PHP
What Google says
Checked November 12, 2026.
- “Crawlers use the most specific rule based on the length of the rule path. In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule” (How Google interprets the robots.txt specification, updated October 23, 2026).
- The same page: Google “can't index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet.” It reads only user-agent, allow, disallow and sitemap, and a file only applies to the protocol, host and port it is served from.
- Some Google crawlers answer to two names. Googlebot-Image and Googlebot-News also follow rules for Googlebot, and “you need to match only one crawler token for a rule to apply” (Google's common crawlers). AdsBot ignores the * group (special-case crawlers).
- Search Console's robots.txt report shows the files Google found, when it last fetched them and any parse problems, and lets you ask for a fresh fetch after a fix.
- The details the documentation leaves out (accepted misspellings, a missing colon, the index.html rule) come from Google's open-source robots.txt parser. The standard itself is RFC 9309.
The tester reads the text you paste. It doesn't fetch your live file, so it can't see redirects, server errors or a file served on another host. Crawlers outside Google are tested with Google's rules; most follow the same standard, but check each company's own page for its details. To write a new file, with AI crawler blocks, use the robots.txt generator.
New pages Google can crawl from day one
Serpwright reads your Search Console data, finds the searches your site shows up for but has no page for, then writes and publishes an article for each one you approve.
- Articles go to WordPress, Ghost, Webflow or Shopify, with their title, description and schema markup.
- With Search Console connected, it checks each article a day after it goes live, again after a week if it isn't indexed yet, and tells you in the app if it still isn't.
- Nothing goes live until you approve it.
3 days free, no card.