Robots.txt generator

Generate a simple robots.txt file with sitemap and crawl rules. Nothing leaves your browser.

Runs in your browser -- no data sent
robots.txt
User-agent: *
Disallow: /admin

Sitemap: https://example.com/sitemap.xml

Guide

Robots.txt Generator: what it does and how to use it

What this SEO tool does

This generator creates a simple robots.txt file for websites. Robots.txt tells crawlers which areas they may access and where the XML sitemap is located. For most public sites, simpler rules are better than complex crawler-specific blocks.

How it works

Enter a sitemap URL, choose whether to disallow an admin path, and optionally add a crawl delay. The generated file can be copied into /robots.txt. You can keep the default broad Allow rule so tools, security pages, and blog content remain crawlable.

Examples

  • Generate a simple Allow: / robots file.
  • Add Sitemap: https://example.com/sitemap.xml.
  • Disallow /admin/ while keeping /tools/ and /blog/ open.

Security considerations

Robots.txt is public and should not be used to hide secrets. Sensitive pages need authentication, not only a Disallow rule.

FAQ

Should I block /tools/?

No. Tool pages are SEO landing pages and should stay crawlable.

Does Disallow remove pages from Google?

Not always. Use noindex or authentication for content that must not appear.

How simple should robots.txt be?

For SAVR-style sites, Allow: / plus a Sitemap line is usually enough.