Robots.txt tester
Paste any URL. We fetch that site's robots.txt, parse it the way a crawler does, and tell you whether the URL is allowed - for Googlebot, for our own bot, and for everything else.
Free, no account, no card. One URL at a time.
What this checks
- The file itself
- That it exists, returns 200, and how many rules it holds. A robots.txt that 5xxes is read as 'block everything' until it recovers.
- Your URL, three ways
- Whether the path you gave is allowed for Googlebot, for SeodarBot, and for the wildcard agent every other crawler falls back to.
- The site-wide mistake
- Disallow: / under User-agent: * - the staging rule that reaches production and removes a whole site from search.
- Sitemap directives
- Whether the file tells crawlers where the sitemap is, which is the one line that saves them guessing.
Why it matters
robots.txt controls crawling, not indexing, and the difference is where most of the damage happens. A blocked URL can still appear in results - Google can learn it exists from links pointing at it - but it appears with no description, because nothing was ever read.
The same rule makes a noindex tag useless: a crawler that is not allowed to fetch the page never reads the tag asking it to stay out. If you want a page gone from search, it has to be crawlable and carry noindex, or be removed properly.
So the useful question is never 'what does my robots.txt say'. It is 'is this particular URL allowed', which is what this answers.
Questions
- Does a missing robots.txt hurt?
- No. A 404 means everything is allowed, which is a valid configuration and the most common one on small sites. A 500 is the one to fix - crawlers treat a server error as a temporary block on the whole site.
- Why test Googlebot and your bot separately?
- Because rules are per user-agent, and a file that allows the wildcard can still name a specific crawler and block it. Sites often block everything except a short list without meaning to.
- Does this check whether the page is indexed?
- No - it checks whether it may be crawled. Indexing depends on the meta robots tag, the canonical, and Google's own judgement. A full audit checks all of them together.
One URL is a spot check
The full audit covers robots.txt reachability, blocked-but-linked URLs, and blocked pages that still take traffic - across every page it crawls, with a score and a ranked list of what to fix first.