Does Google-Extended affect AI Overviews?

No. AI Overviews and AI Mode use Googlebot. Google-Extended only controls use for Gemini training and grounding outside Search.

Do AI bots obey robots.txt?

The search crawlers do. OpenAI and Perplexity say robots.txt may not apply to ChatGPT-User and Perplexity-User, because a person asked for the page. Anthropic says all three of its bots honour it.

Why is a path “allowed” with no rule?

If no rule in the matching group covers the path, it’s allowed by default. The same applies when no group matches the bot at all.

Yes. Every tool is free with no signup.

Do you store what I enter?

Browser tools never leave your device. Live tools fetch the URL you enter and don’t save the result.

How is this different from the site audit?

This tool checks one thing in depth. The site audit checks everything at once and scores SEO, AI Search and Agent Readiness.

Tools / Crawling & indexing

Test any path against Googlebot and AI bots

Fetch a robots.txt and see exactly which crawlers can reach a given path, plus every sitemap it declares.

Free, no signup.

Site or page URL

Path to test (optional)

Example result: github.com · checked 2026-09-30

robots.txt

Location	https://github.com/robots.txt
Lines	271
Sitemaps declared	— none (add one!)

Access to /search

Googlebot	🚫 blocked · Longest matching rule: Disallow: /search$
Bingbot	✓ allowed · No rule in the matched group covers /search
OAI-SearchBot	🚫 blocked · Longest matching rule: Disallow: /search$
Claude-SearchBot	🚫 blocked · Longest matching rule: Disallow: /search$
PerplexityBot	🚫 blocked · Longest matching rule: Disallow: /search$
GPTBot	🚫 blocked · Longest matching rule: Disallow: /search$

Why it matters

Blocking the wrong bot hides you

Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those AI answers. Blocking GPTBot or ClaudeBot only opts you out of training.

robots.txt doesn’t remove pages

Google says a URL blocked by robots.txt can still be indexed from links, just without its content. Use noindex to keep a page out.

Specific groups override *

A crawler follows only the most specific User-agent group that names it. A bot with its own group ignores your * rules entirely.

What it checks

File and location

Fetches /robots.txt from the site’s origin and counts its lines.

Sitemaps

Every Sitemap: line the file declares.

Search bots

Googlebot and Bingbot, tested against your path.

AI search and user bots

OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot.

AI training bots

GPTBot, ClaudeBot and Google-Extended.

Deciding rule

The exact rule behind each result, using RFC 9309 matching: the longest rule wins and Allow wins a tie.

How it works

Enter a URL

A domain or page address. No install, nothing to add to your site.

We fetch it live

Our servers request the page read-only, the way a crawler would.

Read and fix

Get the results in seconds, with what each one means.

Questions

More in Crawling & indexing

Indexability CheckerFind out if a page can be indexed, and exactly what's blocking it.XML Sitemap CheckerParse a sitemap or sitemap index, count URLs and list entries.Googlebot SimulatorFetch a page as Googlebot and as a browser, and compare what each gets.

Check everything at once

The site audit checks your whole site and scores SEO, AI Search and Agent Readiness in about 30 seconds.

Audit my site