Robots.txt Generator

A robots.txt generator that produces a valid robots.txt for the three common situations — allow all crawlers, block all crawlers, or block specific paths — with an optional crawl-delay and sitemap URL.

robots.txt is a plain-text file at the root of your domain — example.com/robots.txt — that tells search engine crawlers which parts of the site they may fetch. It's the first thing Googlebot requests when it visits, and because a single misplaced character can deindex an entire site, it deserves to be generated carefully rather than hand-typed from memory. This generator produces a valid file for the three situations that cover almost every site.

Allow all is the right choice for most public sites: it emits an empty Disallow directive, the standard way to say 'crawl everything'. Block all (Disallow: /) shuts crawlers out entirely — correct for staging servers, pre-launch sites and internal tools, and catastrophic if it ever ships to production, which is exactly the accident this generator helps you avoid by making the two modes explicit. Custom mode blocks only the paths you list, one per line — admin panels, cart and checkout pages, internal search results and other pages that waste crawl budget without deserving to rank. Paths are normalized to start with a slash. You can also add a Crawl-delay for aggressive bots and a Sitemap line pointing crawlers at your sitemap.xml — the one line every robots.txt should carry.

One misconception is worth clearing up: robots.txt controls crawling, not privacy. A disallowed URL can still appear in Google's index if other sites link to it, and the file itself is public — listing /secret-admin-panel in it advertises the path to anyone curious. Use noindex tags to keep pages out of results, and real authentication to keep people out of pages.