Free Robots.txt Generator

Generate robots.txt for any website. Download and upload to your site if it’s not pagespeed.co.in.

Used to auto-fill sitemap if empty.
Bot Rules
Set custom rules for specific bots or keep “Default”.
Google
Googlebot
Google Image
Googlebot-Image
Google Mobile
Googlebot-Mobile
Bing
bingbot
Yahoo
Slurp
Yandex
Yandex
Baidu
Baiduspider
DuckDuckGo
DuckDuckBot
Apple
Applebot
Facebook
facebookexternalhit
Twitter
Twitterbot
Pinterest
Pinterestbot
Ahrefs
AhrefsBot
Semrush
SemrushBot
MJ12
MJ12bot
Naver
NaverBot
Sogou
Sogou
DotBot
DotBot
Gigablast
Gigabot
Restricted Directories
Example: /cgi-bin/ , /wp-admin/
Custom Rules
Only writes to pagespeed.co.in. Other domains must download and upload.
View robots.txt
Generated Robots.txt
User-agent: *
Disallow:
Sitemap: https://pagespeed.co.in/sitemap.xml