Does robots.txt stop a page being indexed?

No, it stops crawling. A blocked URL can still appear in results if other pages link to it. Use noindex instead, and leave the page crawlable so Google can see the tag.

Will blocking GPTBot remove me from ChatGPT?

No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot and ChatGPT-User, which this generator leaves on your general rules.

Do I need a separate group for Googlebot?

No. Googlebot follows the User-agent: * group unless you give it its own. A bot with its own group ignores the * rules entirely, so repeat any rules it needs.

Yes. Every tool is free with no signup.

Do you store what I enter?

Browser tools never leave your device. Live tools fetch the URL you enter and don’t save the result.

How is this different from the site audit?

This tool checks one thing in depth. The site audit checks everything at once and scores SEO, AI Search and Agent Readiness.

Tools / Generators

Generate a robots.txt that doesn’t hide you from AI

Block the paths you want, point crawlers at your sitemap, and choose whether to block AI training bots, without blocking AI search.

Free, no signup.

Disallow paths (one per line)

Sitemap URL

Block AI training bots?

Content Signals (optional)

Example result: example input

robots.txt

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /admin
Disallow: /cart

# Block AI training crawlers (AI search stays allowed)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /

Sitemap: https://example.com/sitemap.xml

Content-Signal states how you want your content used after it is crawled: search (search results and links), ai-input (grounding live AI answers) and ai-train (model training). It is a preference, not a block. Pair it with bot rules if you need enforcement.

Why it matters

Training and search are separate bots

Blocking GPTBot, ClaudeBot or Google-Extended keeps you out of training data. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from AI answers. This generator only blocks the first kind.

Point crawlers at your sitemap

A Sitemap: line helps Google, Bing and other crawlers find the URLs you want indexed.

State your AI preferences

Content Signals let you say search=yes, ai-input=yes, ai-train=no in plain terms. It’s a preference, not a block, and about 4% of top domains already publish one.

What it checks

Disallow rules

One Disallow line per path you enter, under User-agent: *.

Training-bot block

Optional groups that block GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent.

AI search left open

Search and user bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot get no group of their own, so they follow your general rules.

Content Signals

An optional Content-Signal line with your search, ai-input and ai-train choices.

Sitemap

A Sitemap: line with your sitemap URL.

How it works

Fill in the fields

Or hit “Try example” to see how it works.

Generate instantly

Everything runs in your browser. Nothing is sent to a server.

Copy and ship

Copy the output straight into your site.

Questions

More in Generators

Title Tag GeneratorTitle patterns with length hints and an optional brand suffix.Meta Description GeneratorDescription drafts sized for search snippets.Schema GeneratorJSON-LD for Article, Product, FAQ, Organization and more.

Check everything at once

The site audit checks your whole site and scores SEO, AI Search and Agent Readiness in about 30 seconds.

Audit my site