Robots.txt Generator
Robots.txt generator
Download robots.txtGuide: Robots.txt Generator
Topics in this category: On-page SEO and metadata
Robots.txt role in SEO
Robots.txt is a crawl directive file at your site root. It tells compliant bots which paths they should not fetch—it does not enforce security or hide private URLs from determined actors.
Do not block CSS, JS, or assets required to render pages if you want Google to understand layout and content. Include your sitemap URL as a hint for discovery.