robots.txt Generator
Generate a robots.txt file with sensible defaults, sitemap declaration, AI crawler controls and per-bot rules — without accidentally deindexing your site.
What robots.txt does and doesn’t do
It asks well-behaved crawlers not to fetch certain paths. That’s all.
Three things people expect it to do that it doesn’t:
It doesn’t keep pages out of search results. A blocked page can still be indexed from links pointing at it — it just gets indexed without content. To keep a page out of results, let crawlers fetch it and serve <meta name="robots" content="noindex">. Blocking in robots.txt actually prevents Google seeing the noindex tag.
It doesn’t provide security. The file is public, and listing a path advertises it. Never put admin URLs or private directories in robots.txt. Use real authentication.
It doesn’t bind anything. It’s a convention. Well-behaved crawlers follow it; scrapers ignore it entirely. Blocking a bad bot here does nothing — block it in .htaccess instead.
The one rule that matters most
User-agent: *
Disallow: /
Those two lines block your entire site from every crawler. They appear on staging sites constantly and get copied to production during launch, and the site quietly vanishes from search. If your traffic dropped off a cliff after a redesign, check this file first.
Declare your sitemap
Sitemap: https://example.com/sitemap.xml
An absolute URL, and it can point at a different host from the robots.txt itself. This is the cheapest discovery win available — it costs one line and tells every crawler where your page list is.
AI crawlers
A newer set of user agents crawl for training data and for answering questions in AI assistants. They are worth thinking about separately, because they are not the same trade.
GPTBot, ClaudeBot, Google-Extended and CCBot are largely training crawlers — blocking them keeps your content out of model training and costs you nothing in traffic. OAI-SearchBot and PerplexityBot fetch pages to cite in answers, so blocking those removes you from AI search results the way blocking Googlebot removes you from Google.
Decide which of those two things you want. The generator lets you pick either separately.