Generate robots.txt
Create a robots.txt file for your website
Generate a robots.txt file to control search engine crawling.
Generate robots.txt →robots.txt Generator Features
- Allow/disallow all bots
- Set crawl delay
- Add sitemap URL
- Download as file
How to Generate robots.txt
- Set default policy
- Add sitemap URL
- Click Generate
- Download and upload to your site
Frequently Asked Questions
What is robots.txt?
robots.txt is a file that tells search engine crawlers which pages to crawl or not crawl.