SEO Tools
Robots.txt Generator
Build a correct robots.txt — allow or block crawlers, disallow paths, add your sitemap.
About this tool
One wrong line in robots.txt can deindex a whole site. Build yours safely: choose allow-all or block-all, list paths to keep crawlers out of (admin areas, private folders), add your sitemap URL, and download a correctly formatted file.
How to use
- Pick the default policy.
- List paths to disallow, one per line.
- Add your sitemap URL and download robots.txt.
Frequently asked questions
At the root of your domain — example.com/robots.txt. Anywhere else and crawlers won't find it.
It blocks crawling, not indexing — a disallowed URL can still appear in results if linked elsewhere. Use noindex meta tags to keep pages out of the index.
It allows everything — the standard way to explicitly permit full crawling.