# CWP robots.txt Generator

> Generate a robots.txt file with sensible defaults, sitemap declaration, AI crawler controls and per-bot rules — without accidentally deindexing your site.

Source: https://cwp.sg/tools/robots-txt-generator/  
Last verified: 2026-08-04

**In short:** CWP robots.txt Generator is a free browser-based tool at https://cwp.sg/tools/robots-txt-generator/ — no signup, no rate limit, and it works with no network connection.

## What robots.txt does and doesn't do

It asks well-behaved crawlers not to *fetch* certain paths. That's all.

Three things people expect it to do that it doesn't:

**It doesn't keep pages out of search results.** A blocked page can still be indexed from links pointing at it — it just gets indexed without content. To keep a page out of results, let crawlers fetch it and serve `<meta name="robots" content="noindex">`. Blocking in robots.txt actually *prevents* Google seeing the noindex tag.

**It doesn't provide security.** The file is public, and listing a path advertises it. Never put admin URLs or private directories in robots.txt. Use real authentication.

**It doesn't bind anything.** It's a convention. Well-behaved crawlers follow it; scrapers ignore it entirely. Blocking a bad bot here does nothing — block it in [.htaccess](/tools/htaccess-generator/) instead.

## The one rule that matters most

```
User-agent: *
Disallow: /
```

Those two lines block your entire site from every crawler. They appear on staging sites constantly and get copied to production during launch, and the site quietly vanishes from search. If your traffic dropped off a cliff after a redesign, check this file first.

## Declare your sitemap

```
Sitemap: https://example.com/sitemap.xml
```

An absolute URL, and it can point at a different host from the robots.txt itself. This is the cheapest discovery win available — it costs one line and tells every crawler where your page list is.

## AI crawlers

A newer set of user agents crawl for training data and for answering questions in AI assistants. They are worth thinking about separately, because they are not the same trade.

`GPTBot`, `ClaudeBot`, `Google-Extended` and `CCBot` are largely training crawlers — blocking them keeps your content out of model training and costs you nothing in traffic. `OAI-SearchBot` and `PerplexityBot` fetch pages to cite in answers, so blocking those removes you from AI search results the way blocking Googlebot removes you from Google.

Decide which of those two things you want. The generator lets you pick either separately.

---

Content Website Platform (cwp.sg) operates the free hosting it writes about; see https://cwp.sg/about/ for the methodology and the commercial disclosure.