Generate robots.txt files to control search engine crawling. 100% client-side processing.
SEO control
The robots.txt file tells search engine crawlers which pages or files they can or cannot request from your site. Place the generated file at the root of your domain (e.g., example.com/robots.txt).
User-agent: *
The Robots.txt Generator builds a valid robots.txt — the small text file at the root of your domain that tells crawlers which parts of your site they may request. Pick a user-agent, add the paths to Allow or Disallow, list your sitemaps, then copy the result or download the file. Alongside Googlebot and Bingbot, the user-agent list covers the modern AI crawlers — GPTBot, ChatGPT-User and CCBot — so you can decide what AI training and retrieval bots are allowed to fetch. Everything is generated in your browser; nothing you type is uploaded.
A plain-text file at the root of your domain (https://example.com/robots.txt) that tells crawlers which URLs they may request. It's usually the first thing a crawler fetches. Importantly, it's a set of instructions that well-behaved crawlers follow voluntarily — it is not a security control, and it won't stop anyone determined from reading a page.
No — this is the most common and most costly misunderstanding. Disallow stops a compliant crawler from fetching the page, but Google can still index the URL if other pages link to it, listing it without a description. To keep a page out of results, let it be crawled and add a noindex robots meta tag, or put it behind authentication. Blocking it in robots.txt is actually counterproductive: Google can't see the noindex tag if it isn't allowed to fetch the page.
Add a rule group and set the user-agent to GPTBot (OpenAI's training crawler), ChatGPT-User (which fetches pages on a user's behalf), or CCBot (Common Crawl), then Disallow /. A crawler follows the most specific group that matches it, so a User-agent: * group already applies to these bots unless you give them their own group — a named group is how you treat them differently from Google. Blocking them is a genuine business decision: it can keep your content out of model training, but it can also reduce how often your brand gets surfaced by AI assistants.
At the root of the host it governs: https://example.com/robots.txt. It only applies to that exact host and protocol, so a file on example.com does not cover blog.example.com — each subdomain needs its own. A robots.txt in a subfolder is ignored entirely.
Yes. It's the simplest way to point every crawler at your sitemap without registering it in each search engine's tools, and you can list several. The Sitemap directive applies to the whole file rather than to one user-agent group, which is why this tool writes sitemaps at the top.
Yes. The file is assembled entirely in your browser. Nothing you type is uploaded, stored, or logged.
We use minimal, privacy-focused cookies to improve your experience. Our developer tools process all data locally — we never store your content. Learn more