Robots.txt Checker
Enter a domain to read its robots.txt file and check it for blocking rules, missing sitemap lines and syntax errors.
The checker reads the live robots.txt file. Rules are checked for common mistakes, not for every crawler's own quirks.
About the Robots.txt Checker
A robots.txt file tells search engines which parts of your site they may crawl. One wrong line, such as "Disallow: /", can keep your whole site out of Google. This checker finds those mistakes.
How to use the Robots.txt Checker
- Enter your domain, for example example.com.
- Press Check robots.txt.
- Read the checks for blocked pages, sitemap lines and syntax.
- Fix the file and check again.
Frequently asked questions
Where is the robots.txt file?
It always sits at the root of the site, for example example.com/robots.txt.
Does robots.txt hide a page from Google?
Not reliably. It stops crawling, but a blocked page can still appear in results if other sites link to it. Use a noindex tag to keep a page out of search.
What if my site has no robots.txt?
Crawlers then assume everything is allowed. That is fine, but adding one with a sitemap line is good practice.
