An Allow result establishes only the applicable robots rule. Prove delivery separately with the observed status, headers, response body, request identity evidence, and the security event that handled the request.
Start with the apparent contradiction
Suppose robots.txt has no matching Disallow rule for Googlebot, OAI-SearchBot, or another product token. A local robots tester therefore returns Allow. The live request can still receive 403, 429, a login page, or an HTML challenge instead of the intended document.
RFC 9309 explains why: robots rules are not access authorisation. They tell participating crawlers which URI paths the site owner asks them to avoid. They do not bypass authentication, a web application firewall, bot scoring, geo rules, rate limits, or an unavailable origin.
Four decisions happen before useful HTML reaches a crawler
A green result at the first layer says nothing certain about the next three. Cloudflare's current documentation, for example, describes separate rules for verified bots, likely automated traffic, managed challenges, and skip actions. Other providers use different products and fields, but the diagnostic principle is the same: inspect the layer that made the decision.
- robots.txt matching: which rule applies to the declared product token and target path;
- request classification: how the CDN or firewall classifies the address, headers, TLS fingerprint, rate, and other request properties;
- security action: allow, log, challenge, rate limit, or block;
- origin response: what the application returns if the request reaches it.
A browser success is useful, but it is not the crawler result
Your browser and a crawler rarely send the same request. Their addresses, cookies, JavaScript support, user agents, connection fingerprints, and request rates differ. A browser may solve a challenge or carry a clearance cookie that a simple HTTP client never receives.
Keep both observations. Write “browser returned the page” and “this crawler-shaped request returned a challenge” instead of allowing one result to erase the other.
Do not solve the discrepancy by allowlisting a user-agent string. The requester supplies that string, so it is easy to copy. Use the operator's documented IP or DNS verification when identity changes an access decision.
Capture the response before changing a rule
- Record the exact URL, time, status, redirect chain, content type, relevant headers, and a hash or safe excerpt of the returned body.
- Fetch robots.txt separately and reproduce the decision for the intended product token and path.
- Classify the response: intended page, login gate, managed challenge, explicit block, rate limit, or upstream failure.
- Find the matching CDN, WAF, reverse-proxy, or origin event at the same time and path.
- Verify the crawler identity before adding a consequential exception.
- Change the narrowest responsible rule, then repeat the same request and retain both captures.
Read status codes as evidence, not as a motive
A 403 establishes that the responding server understood the request and refused it; it does not identify which rule or product caused the refusal. A 429 establishes rate limiting, not permanent exclusion. A 200 can still be a challenge page, so status must be read together with content and headers.
When Googlebot is genuinely requesting too quickly, Google recommends 429 or a temporary 5xx response rather than using 403 or 404 as an improvised throttle. That advice is specific to signalling Googlebot and does not replace capacity planning or abuse controls for other clients.
What the current AnalyseSpider comparison can prove
The crawler view can compare one neutral AnalyseSpider request with one second request carrying an allowlisted crawler product token. A different status, final URL, canonical, noindex value, or body hash is direct evidence that the two observed responses differed.
It is still a header simulation. It does not originate from the operator's crawler network, reproduce its full request fingerprint, or authenticate the product identity. The first published control baseline found no response difference across six local static fixtures; that negative result is a baseline, not a universal claim about CDNs.
A useful report ends with the unresolved layer: “robots allows the token; the simulated request received a challenge; the responsible security event and genuine crawler identity remain to be verified.”