Seodar

The crawler

What our crawler does to your site

How many requests, how fast, what it fetches, when it runs a browser, and how to keep it out - written for whoever has to approve letting it in.

Written for whoever has to approve letting a crawler onto a site, which is usually not the person who signed up.

How it identifies itself

It sends a User-Agent naming SeodarBot and a URL explaining what it is. It is not disguised as a browser, and it does not rotate identities - if you want to see what it did, it is in your access log under one name.

What it obeys

  • robots.txt, including Crawl-delay. A path you disallow is not fetched, and the report says the pages were blocked rather than pretending they do not exist.
  • noindex. It still reads the page - that is how it knows - but it reports the page as excluded rather than scoring it as a normal one.
  • Your page limit. The crawl stops at the number you set, and the report says how many pages it reached.

How hard it hits

It is a small number of requests in parallel, not a flood, and Crawl-delay slows it further where you set one. A crawl of a few hundred pages is minutes rather than hours - measured on a real site, 200 pages in about 100 seconds end to end, including the checks.

If that is still more than you want, the page limit is yours to set, and scheduled crawls can be turned off entirely.

When it runs a browser

Most pages are read as HTML, which is what a search engine reads first and is far cheaper for your server. A browser is only started where the HTML alone is not enough to judge the page - and when that happens it is reported, because a page that needs JavaScript to say anything is itself a finding.

What it does not do

  • It does not submit forms, follow links that change anything, or fetch anything behind a login.
  • It does not store your page content beyond what a check needs; the report holds findings and page metadata, not copies of your pages.
  • It does not follow links off your host. A crawl of one site stays on that site.

How to keep it out

The same way you keep any crawler out - a robots.txt rule for SeodarBot. There is no separate mechanism and no way for an account holder to override it: if your robots.txt says no, the answer is no, and their report will tell them so.

Last changed 2026-08-10. Every check this refers to is listed on the checks page.