Integrations
The only place that knows what the crawler was handed
A sitemap says what you offered; Cloudflare knows what was actually returned, including the page that is a 404 to Googlebot.
Crawl budget, measured rather than guessed
Every tool in this market talks about crawl budget and estimates it. Cloudflare already counted the requests, so the question stops being theoretical: how often did Googlebot come, and how much of what it fetched was worth fetching?
Joined against your own page inventory, that becomes a share - requests spent on URLs marked indexable = false, absent from the sitemap, or answering with a redirect. Waste, in requests, not in adjectives.
- Pages you want indexed61%
- Noindex, redirects, parameters26%
- Assets and URLs the crawl never found13%
An illustration of the shape. The denominator is deliberately not every request a crawler makes - most of those are assets, and counting them would make any site look wasteful.
Arriving and being refused is not reading
A crawler that comes as often as ever and is handed 4xx looks healthy in any tool that counts visits. The request count hides the failure completely.
This product has already scored a site 88 out of 100 while it sat behind a Cloudflare interstitial - the crawl saw a page, and what Googlebot saw was a challenge. The response code is the difference, and it lives here.
What a visit count says
1,480 requests, steady
What the response codes say
286 of them answered 404
The first number is in every tool. The second is in one.
Whether the AI crawlers can read you at all
GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended and the rest arrive under their own names and get their own answers. Whether an assistant can cite you starts with whether it was allowed to fetch you, and that question has a factual answer sitting in your own edge logs rather than an opinion.
Which matters more each month, and is the half of AI visibility that is actually measurable - see AI visibility for the other half.
We keep the history Cloudflare does not
A free zone keeps eight days of request analytics, and the two findings worth having - Googlebot getting errors on pages you crawl cleanly, and the share of its budget spent on URLs you know are worthless - need more history than that to be trustworthy.
So a nightly sample accumulates per-path history from the day you connect. The findings stay quiet until there are fourteen days of it, because a rate computed over three days of sample and twenty-eight days of traffic is not a rate, it is an artefact - and that particular mistake was made once, in this exact place, before the gate existed.
What connecting it adds
Crawler traffic by day
Which bots arrived and how often, named rather than lumped into 'bot traffic' - Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, AhrefsBot.
Response codes per crawler
What each one was handed. Arriving and being refused is not the same as reading, and only one of those is visible in a request count.
Crawl budget waste
The share of Googlebot's requests spent on URLs your own crawl already knows are not worth indexing.
AI crawler access
Whether the assistants' crawlers reached you and what they got, which is where any claim about AI visibility has to start.
Questions
- One permission - Zone · Analytics · Read - on one zone, from a token you create yourself in your own Cloudflare dashboard. It cannot change anything, cannot see other zones, and is encrypted before it is stored. The connect screen shows the exact path with pictures, because Cloudflare's templates do not include analytics and that is where people get stuck.
- Eight days on a free Cloudflare plan and more on paid ones. We ask one day at a time and stop where your zone stops, so you get whatever your plan keeps without configuring anything - and we keep the per-path history from then on, which Cloudflare itself does not.
- Yes. This reads your zone's own request analytics, so it only works for sites served through Cloudflare. Nothing else in the product depends on it.
- It is the same class of answer without the log file. Nothing to export, nothing to parse, and no sampling decision made by whoever set up the logging - the numbers come from the edge that served the request.
Start free
One site and 100 credits - up to 200 pages a scan, about 5 scans, and the whole report. No card. Paid plans add more sites, bigger crawls and the AI rewrites.