Fetch the robots.txt of any website, find syntax errors, and test whether a URL is allowed or blocked for Googlebot, Bingbot and AI crawlers like GPTBot and ClaudeBot. Edit the rules and re-test instantly — nothing changes on your site.
robots.txt is a plain text file at the root of a website (https://example.com/robots.txt)
that tells crawlers which parts of the site they may visit. Search engines like Google and Bing and
AI crawlers like GPTBot check it before crawling. One wrong line can stop Google from crawling your
whole site, so it is worth testing every change.
User-agent lines above them. A crawler uses only the most specific group that names it, or the * group if none does.* matches any characters and $ marks the end of the URL. Disallow: /*.pdf$ blocks every PDF./Admin/ and /admin/ are different.
Blocking a URL in robots.txt stops crawling, not indexing. If other sites link to a blocked page,
Google can still list it — without a description. To keep a page out of search results, allow
crawling and add a noindex meta tag instead.
User-agent: GPTBot followed by Disallow: /. Common AI tokens are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini training) and CCBot (Common Crawl). Blocking Google-Extended does not affect Google Search.Noindex: in robots.txt in 2019. Use a robots meta tag or the X-Robots-Tag header instead.blog.example.com/robots.txt is separate from example.com/robots.txt, and http and https are treated as different hosts too.