Should I block AI training bots?

That’s your call. Blocking GPTBot, ClaudeBot, Google-Extended or CCBot keeps your content out of training data but doesn’t remove you from AI search answers. Blocking the search and user bots does.

Why does robots.txt say allowed but the live test fails?

Almost always a CDN or firewall rule, like Cloudflare’s AI bot blocking or a WAF rule that challenges unknown user-agents.

Is the live test exact?

Close, not exact. We send each bot’s user-agent from our servers. Real bots come from their own IP ranges, which some firewalls treat differently. Treat a live block as “likely”.

Yes. Every tool is free with no signup.

Do you store what I enter?

Browser tools never leave your device. Live tools fetch the URL you enter and don’t save the result.

How is this different from the site audit?

This tool checks one thing in depth. The site audit checks everything at once and scores SEO, AI Search and Agent Readiness.

Tools / AI crawlers & agents

Can ChatGPT, Claude and Perplexity reach your site?

Test robots.txt and a live request for every major AI bot. Find the firewall rule that quietly hides you from AI answers.

Free, no signup.

Domain

Example result: smoothrecorder.com · checked 2026-09-30

AI bot access for smoothrecorder.com

✓ OAI-SearchBot	ChatGPT search · robots.txt: allowed · live: HTTP 200
✓ ChatGPT-User	ChatGPT browsing & agent · robots.txt: allowed · live: HTTP 200
✓ Claude-SearchBot	Claude search · robots.txt: allowed · live: HTTP 200
✓ Claude-User	Claude fetching for a user · robots.txt: allowed · live: HTTP 200
✓ PerplexityBot	Perplexity search · robots.txt: allowed · live: HTTP 200
✓ Google-Extended	Gemini training (robots.txt only) · robots.txt: allowed

Every AI bot we tested can reach your homepage.

Why it matters

Robots.txt isn’t the whole story

Your robots.txt can say “allow” while your CDN answers 403. Only a live request shows what the bot actually gets.

Cloudflare blocks AI by default

Since July 2025, new Cloudflare sites block AI crawlers unless you opt in. Many owners never notice.

Search bots are not training bots

Blocking GPTBot stops training. Blocking OAI-SearchBot or ChatGPT-User removes you from ChatGPT answers.

What it checks

OpenAI

OAI-SearchBot (ChatGPT search), ChatGPT-User (browsing and agent) and GPTBot (training).

Anthropic

Claude-SearchBot, Claude-User and ClaudeBot, each tested separately.

Perplexity

PerplexityBot for search results and Perplexity-User for live answers.

Google

Google-Extended, the robots.txt switch for Gemini training.

Common Crawl

CCBot, which feeds the datasets many models are trained on.

Firewalls and paywalls

HTTP 403, 429, 503, pay-per-crawl 402 and challenge pages like “Just a moment…”.

How it works

Enter a URL

A domain or page address. No install, nothing to add to your site.

We fetch it live

Our servers request the page read-only, the way a crawler would.

Read and fix

Get the results in seconds, with what each one means.

Questions

More in AI crawlers & agents

AI Crawler ViewSee the text an AI crawler gets without running JavaScript, and whether your page is an empty shell.Agent Discovery CheckerLook for markdown pages, MCP server cards, API catalogs, agent cards, UCP and signed-agent keys.llms.txt CheckerFetch and validate /llms.txt and /llms-full.txt against the llmstxt.org format.

Check everything at once

The site audit checks your whole site and scores SEO, AI Search and Agent Readiness in about 30 seconds.

Audit my site