Free tools

Robots.txt Generator

A robots.txt that says what you meant, with a warning on the two lines that quietly remove a site from search.

  • No account
  • No email
  • Free, every time
Try one

Result

User-agent: *
Disallow:

Robots.txt does not hide a page

It asks a crawler not to fetch it. A blocked page that other sites link to can still appear in results, with no description, because Google knows it exists and is not allowed to read it. To keep a page out of search, use a noindex tag and leave it crawlable.

One wrong line empties a site

Disallow: / under User-agent: * is valid, takes effect within a day or two, and is the commonest way a site disappears from Google overnight. Anything here that would do it is flagged above the file rather than below it.

What the generator does and how to read its answer

The generator turns your crawl rules into a clean robots.txt file. You choose which paths to allow and which to block, and it writes the directives in the order search engines expect. The answer comes back as plain text you can copy. Read it top to bottom: each User-agent line names the crawler, and the Allow and Disallow lines under it set the rules for that crawler. A Disallow line with a path blocks that path. A Disallow line with just a slash blocks the whole site. The Sitemap line points crawlers to your sitemap file. Check that every path in the answer matches the folders you meant to protect.

The mistake it is built to catch

The common mistake is a single stray character that blocks everything. A Disallow line written as a bare slash tells every crawler to stay out of the whole site. Another version is a rule placed under the wrong User-agent line, so it applies to a crawler you did not mean to target. A third is a blocked folder that holds pages you want found, such as a blog or a product directory. The generator checks your paths against the rules you set and shows the finished file so you can see the effect before it goes live. The cost of a bad file is real: crawlers follow it, pages drop out of the index, and the work you put into content stops reaching search.

What to do with the answer

Copy the text and save it as robots.txt in the root of your site, so it loads at your domain followed by /robots.txt. Upload it and open that address in a browser to confirm it loads. If you run a content platform, check whether it manages robots.txt for you before you overwrite anything. Keep a copy of the previous file so you can restore it. When you add new sections to your site, revisit the file and add or remove paths. A sitemap line helps crawlers find your pages faster, so include the full address of your sitemap.

Who the generator is for

The generator suits site owners who want control over what search engines crawl. It fits teams that run a blog, a store or a documentation site and need to keep admin pages, staging folders or duplicate URLs out of the index. It fits anyone who has inherited a robots.txt file and wants to rebuild it with rules they understand. It fits people who want a correct file in minutes without learning the syntax by heart. Copy the answer, save it, and your crawl rules are set.

Pair it with the rest of HaloSEO

A clean robots.txt lets crawlers reach the pages you publish. Connect your website to HaloSEO and the platform researches your keywords, writes the posts, publishes them, links them up and builds the backlinks. Your crawl rules stay in your hands, and the publishing runs on its own. Use the generator whenever you change your site structure, then let HaloSEO keep the content flowing to the pages you have opened up.

Related

How it runs, once it is connected

You do the first step. HaloSEO does the other six, on its own, every week.

  1. 01Connect your website
  2. 02HaloSEO researches keywords
  3. 03Writes blog posts
  4. 04Publishes automatically
  5. 05Creates internal links
  6. 06Builds backlinks
  7. 07Improves rankings on Google and AI search engines