SeodarBot

SeodarBot fetches pages so their owner can see what a search engine would find. It only runs when somebody asks it to - either the site's own operator from their dashboard, or a visitor who typed a URL into the free scanner on our home page. It does not crawl the web on its own.

User agent

SeodarBot/0.1 (+https://seodar.io/bot)

How it behaves

• Reads robots.txt before anything else and obeys it, including Crawl-delay.
• One request at a time per site, with a pause between them, and it backs off further if your server slows down or answers 429.
• Stays on the host it was given and its www/apex twin. It does not wander onto other domains.
• Requests GET on HTML only; it does not submit forms, follow logout links, or send cookies.
• Stops at the page limit of the plan that asked - at most 25 pages for an anonymous scan.

Proving it is really us

Every request SeodarBot makes is signed, so you never have to take a user-agent string’s word for it - anyone can forge a name, and nobody can forge a signature. It carries the three headers from RFC 9421 with the web-bot-auth tag, over the host being requested and the address of our key directory:

Signature-Agent: "https://api.seodar.io/.well-known/http-message-signatures-directory"
Signature-Input: sig2=("@authority" "signature-agent");created=…;keyid=…
                 ;alg="ed25519";expires=…;nonce=…;tag="web-bot-auth"
Signature: sig2=:…:

The public keys are published as a JWKS at https://api.seodar.io/.well-known/http-message-signatures-directory. There is more than one because the set is how a key is rotated without a gap: the next key is published before it is used, and the previous one stays until nothing can still be verifying against it.

If your firewall is blocking it

A scan that stops with “nothing on this site could be measured” usually means a bot defence answered instead of your site. Allowing the name is not enough on its own: a managed challenge is passed by running JavaScript, which a crawler does not do, so the rule has to skip the check rather than permit the agent.

In Cloudflare: Security → WAF → Custom rules → Create rule, with this expression, and the action Skip → all remaining custom rules, plus Managed Challenge and Bot Fight Modeunder “More components to skip”:

(http.user_agent contains "SeodarBot" and ip.src eq 85.10.199.27)

Both halves matter. The address on its own would let anything from this server through; the name on its own is a string anybody can send. Other firewalls use different words for the same thing - an allow rule, a skip rule, an exception - and the pair to enter is always the same:

• User agent contains SeodarBot
• Source IP 85.10.199.27

That address is this service’s only outbound address, and this page is the place it is published - if it ever changes, it changes here first.

Blocking it

Add this to your robots.txt and it will stop on the next request:

User-agent: SeodarBot
Disallow: /

If SeodarBot is causing you trouble, or you think it ignored a rule, write to [email protected] with a timestamp and we will look at the logs.