Client-Side Generator • 100% Free

Robots.txt Generator & Syntax Validator

Generate standard-compliant robots.txt directives for Googlebot, Bingbot, and AI crawlers. Live validation, syntax checks, copy, and instant download.

1. Quick Presets

Choose a pre-configured template or customize your own rules below.

2. Directives Configuration

Tip: Leave blank to allow everything for this bot.
Must be an absolute URL starting with http:// or https://.

Robots.txt Output Preview

/robots.txt (RFC 9309 compliant)Syntax Valid
⚠ Google Search Central Security Reminder
Robots.txt is not access control. Disallowed files remain publicly viewable in a browser if someone navigates directly to the URL. Never use robots.txt to hide sensitive information. Protect private resources using authentication (passwords, JWT, firewall).
Understanding Robots Exclusion Protocol
The robots.txt file is a standardized text document located in the root directory of your website. It guides automated search engine crawlers (such as Googlebot and Bingbot) on which URLs they are allowed to crawl.
User-agent
Specifies which crawler the rule applies to. Using * targets all compliant robots, while specific agents like Googlebot apply only to Google's primary web crawler.
Disallow Directive
Tells bots not to request specified URL paths. For instance, Disallow: /admin/ prevents crawling of anything inside the admin directory.
Allow Directive
Overrides a disallow directive for a specific subdirectory or file. For example, you can disallow /admin/ but allow /admin/public-preview.html.
Sitemap Reference
Points crawlers directly to your XML Sitemap index, enabling rapid discovery of new and updated canonical URLs across your site.
Crucial Google Search Central Gotchas
  • Do not block CSS or JS files: If Googlebot cannot access external CSS, JavaScript, or images necessary for page rendering, it may misinterpret your layout or fail mobile-friendly rendering checks.
  • Crawling vs. Indexing: Blocking a page in robots.txt does not guarantee it won't appear in Google search results. If external links point to that URL, Google can index the URL with anchor text snippet. Use a <meta name="robots" content="noindex"> tag on a crawlable page instead.
  • Case Sensitivity: Robots.txt path matching is case-sensitive. /Private/ is completely different from /private/.
Frequently Asked Questions
Does robots.txt stop search engines from indexing a webpage?
No. As documented in Google Search Central, robots.txt tells search engines where they cannot crawl, but not where they cannot index. To prevent indexing, use the noindex directive or require login authentication.
Does Googlebot honor the Crawl-delay directive in robots.txt?
No. Googlebot ignores the Crawl-delay directive in robots.txt. Google dynamically tunes crawl speed automatically based on your web server response times and health.
Where should the robots.txt file be uploaded?
The robots.txt file must reside in the top-level directory (root) of your domain, accessed at https://yourdomain.com/robots.txt. It will not be honored if placed in subdirectories.