Free tool
Robots.txt generator
Configure your crawler rules on the left — the robots.txt file is generated instantly on the right.
Crawlers
All crawlers are allowed. Add specific paths to block below.
Press Enter or click Add. Leading / is added automatically.
Optional
Seconds between crawler requests. Not supported by all crawlers.
Generated robots.txt
User-agent: * Disallow:
Allow all — all crawlers can access every page. Add disallow paths above to block specific sections.
How to use
Save as robots.txt and upload to your site's root directory so it's accessible at yourdomain.com/robots.txt.
Want a full site audit?
Check your SEO, speed, security and more — free.