Home / SEO Tools / Page Crawl Test

Page Crawl Test

Find out if Googlebot can crawl and index a page. We fetch the URL as Googlebot and as a normal browser, check robots.txt, noindex tags, canonical, redirects and status codes, and tell you exactly what blocks the page.

Fetched as Googlebot robots.txt rules noindex & canonical
Try:
Enter a page URL above to test if Google can crawl and index it.

Crawling vs. indexing

Crawling means Googlebot downloads your page. Indexing means Google stores it so it can appear in search results. A page must be crawlable before it can be indexed, and even a crawlable page stays out of the index if it says noindex or points its canonical tag at another URL.

What can stop Google from crawling a page

  • A Disallow rule in robots.txt that matches the URL.
  • A server error (5xx), a 404/410, or a redirect loop.
  • A firewall or bot protection that blocks Googlebot.
  • robots.txt itself returning a server error — Google then pauses crawling the whole site.

What can stop a crawled page from being indexed

  • A noindex robots meta tag or X-Robots-Tag header.
  • A canonical tag that points to a different URL.
  • Thin or duplicate content, or content that only appears after JavaScript runs.

Frequently Asked Questions

This test checks the technical signals. Google also decides based on quality, duplicates and how important the page looks. Make sure the page has unique, useful content, is linked from other pages and is in your sitemap, then request indexing in Google Search Console.

No. Search Console’s URL Inspection uses Google’s own servers and index data and only works for sites you own. This tool works for any URL and shows the same technical blockers, but it cannot see Google’s index.

Some sites show bots a different page, redirect them, or block them with a firewall. If real Googlebot is blocked, your pages disappear from Google. Note that our request uses Googlebot’s user agent but not Google’s IP address, so firewalls that verify Googlebot may treat it differently.

No. A blocked page can still be indexed from links, just without a description. To remove a page, allow crawling and add a noindex tag.

Yes, but rendering happens later and costs more resources, and many other crawlers (including AI bots) do not run JavaScript. Important content and links should be in the HTML the server sends.

More free SEO tools