Robots Txt Generator
Robots Txt Generator tool.
Quick presets
Site & global directives
/robots.txt from your site (via server proxy) into the editor groups.User-agent rule groups
Common paths — click to add to the active group’s Disallow (Shift-click → Allow):
Live robots.txt
Path tester
Simplified longest-match check against your generated file (not a full crawler substitute).
Stats
Tips
Disallow:with an empty path blocks the whole site for that bot.- More specific (longer) Allow/Disallow usually wins in modern crawlers.
- Place
robots.txtat your site root:https://yoursite.com/robots.txt. - Crawl-delay is ignored by Googlebot; some other bots still honor it.
- Always validate in Google Search Console after publishing.