Robots.txt generator
Control which crawlers can enter before search engines, AI bots, or staging leaks discover the wrong paths.
Rule editor
Use * for every crawler, or enter a specific crawler name such as Googlebot.
One path per line. Start each path with /.
Use allow rules to override a broader disallow rule.
Point crawlers to your sitemap so public URLs are easier to discover.
File preview: robots.txt
Robots.txt reference
User-agent: *
Matches every crawler and is typically used as the default rule group.
Disallow: /private/
Asks compliant crawlers not to request paths inside the /private/ directory.
Allow: /private/image.jpg
Lets crawlers access a specific file even when a parent path is disallowed.
Sitemap: URL
Points crawlers to the sitemap so they can discover your public URLs more efficiently.
Frequently asked questions
Do I need a robots.txt file?
It is not mandatory, but it is useful for giving crawlers clear crawl rules and a sitemap location. Without one, compliant crawlers can request any public URL they discover.
Can I upload this file directly even if I am not technical?
Yes. The generator follows the standard robots exclusion protocol. For a typical marketing site, the default allow-all preset plus a sitemap URL is often enough.
How do I block ChatGPT or other AI crawlers?
Use the Block AI crawlers preset. It adds disallow rules for common AI-related crawlers, but robots.txt is voluntary. It controls polite crawlers, not access security.
Does robots.txt keep private pages secure?
No. robots.txt is public and should not be used as protection. Private dashboards, staging builds, and customer files still need authentication, noindex controls, or network-level restrictions.
What is the difference between noindex and Disallow?
Disallow tells crawlers not to fetch a path. noindex tells search engines not to keep a page in search results. If a crawler cannot fetch a page because it is disallowed, it may not see a page-level noindex tag.