robots.txt blocks the page but it’s still in Google. Why?

Google can index a blocked URL from links without crawling it. To remove it, allow crawling and add noindex, so Google can actually see the tag.

Does this check nosnippet and AI Overview rules?

No. A page can be indexable and still barred from snippets and AI features. Run the Snippet Eligibility Checker for that.

Do you check googlebot-specific meta tags?

This check reads and the X-Robots-Tag header. Bot-specific tags like are covered by the Snippet Eligibility Checker.

Yes. Every tool is free with no signup.

Do you store what I enter?

Browser tools never leave your device. Live tools fetch the URL you enter and don’t save the result.

How is this different from the site audit?

This tool checks one thing in depth. The site audit checks everything at once and scores SEO, AI Search and Agent Readiness.

Tools / Crawling & indexing

Can this page be indexed?

Status code, noindex, canonical, robots.txt and X-Robots-Tag in one check, with exactly what’s blocking the page.

Free, no signup.

Page URL

Example result: smoothrecorder.com · checked 2026-09-30

✓ Indexable

URL	https://smoothrecorder.com/
HTTP status	200
Meta robots	none (index,follow)
X-Robots-Tag	none
Verdict	Search + AI crawlers can index this page

Why it matters

Indexing is the entry ticket

Google says a page must be indexed and eligible for a snippet to appear in AI Overviews and AI Mode. The same goes for regular results.

Blocks hide in five places

The status code, meta robots, the X-Robots-Tag header, the canonical and robots.txt can each keep a page out. Checking them one by one is slow and easy to get wrong.

Mistakes survive launches

A noindex left over from staging, or a robots.txt rule meant for another folder, can hide pages for months without any visible error.

What it checks

HTTP status

The final status after redirects. Pages returning 4xx or 5xx can’t be indexed.

Meta robots

noindex in <meta name="robots">.

X-Robots-Tag

noindex in the HTTP header, often set at the server or CDN and used for PDFs.

Canonical

Whether rel=canonical points to a different URL, which asks Google to index that one instead.

robots.txt

Whether Googlebot may crawl the path, matched per RFC 9309: the longest rule wins and Allow wins a tie.

How it works

Enter a URL

A domain or page address. No install, nothing to add to your site.

We fetch it live

Our servers request the page read-only, the way a crawler would.

Read and fix

Get the results in seconds, with what each one means.

Questions

More in Crawling & indexing

robots.txt CheckerFetch robots.txt and test any path against Googlebot and AI bots.XML Sitemap CheckerParse a sitemap or sitemap index, count URLs and list entries.Googlebot SimulatorFetch a page as Googlebot and as a browser, and compare what each gets.

Check everything at once

The site audit checks your whole site and scores SEO, AI Search and Agent Readiness in about 30 seconds.

Audit my site