A crawler comparison needs the same test site, settings and repeated runs.
This page publishes the fixture, the first controlled request baseline and the raw evidence before any product ranking. No crawler is called a winner from a single run.
The fixture is available before the claims
The controlled site contains ordinary HTML links, a one-hop redirect, a canonical pointing elsewhere, a noindex page, one robots-disallowed path, a link added by JavaScript and an orphan URL. Every page is excluded from the public sitemap and carries noindex, so the laboratory does not create search-result clutter.
Open the fixture start URL. The expected behaviour is also listed here so another person can challenge the protocol before any vendor result is published.
| Case | Test URL | Expected observation |
|---|---|---|
| Ordinary HTML link | /fixtures/crawler-lab/static-200 | Discover a linked 200 page in server-rendered HTML. |
| One-hop redirect | /fixtures/crawler-lab/redirect-one-hop | Record a permanent redirect and its final 200 target. |
| Canonical elsewhere | /fixtures/crawler-lab/canonical-source | Record the source URL and its canonical target without silently merging rows. |
| Meta noindex | /fixtures/crawler-lab/noindex | Discover the URL and report its noindex directive. |
| robots.txt disallow | /fixtures/crawler-lab/disallowed | Report the disallow rule and avoid fetching when configured to obey robots.txt. |
| JavaScript-added link | /fixtures/crawler-lab/js-only | Discover the target only when JavaScript rendering and rendered-link extraction are enabled. |
| Orphan URL | /fixtures/crawler-lab/orphan | Remain undiscovered from the start URL; appear only when supplied through another source. |
First baseline: does the server vary the HTML by crawler token?
On 28 August 2026, six directly served fixture cases were requested once with the normal AnalyseSpider user agent and once with a header containing the OAI-SearchBot product token. The Vercel-owned redirect case was kept outside this static-preview baseline. The run did not render JavaScript or enforce robots.txt.
The two request identities received the same status, canonical, meta-robots value and HTML body hash for all six tested fixture pages. That is a useful control result: these fixtures did not vary their response for that simulated token. It does not prove how OpenAI crawls, renders, indexes or selects sources.
| Measure | Observed difference | Meaning |
|---|---|---|
| HTTP status | 0 of 12 requests | No token-dependent status response in this baseline. |
| Canonical and noindex | 0 of 12 requests | The delivered indexability signals were identical. |
| HTML SHA-256 | 0 of 12 responses | The directly served HTML bodies were byte-equivalent after decoding. |
| Crawler identity | Not verified | The second header was a simulation, not an authenticated OpenAI request. |
Download the raw baseline JSON. The file includes the identities, hashes, environment, cases and explicit limits.
Run contract
- Record product name, exact version, licence tier, user agent and configuration export.
- Use only the published start URL and the same host scope. Clear each product's project cache before a run.
- Run each configuration at least three times without changing the fixture deployment.
- Export discovered URLs, status, redirect target, canonical, robots status, indexability and discovery source.
- Publish the raw export or a lossless redacted equivalent, run time, machine or service limits, and every manual correction.
Measures, not a synthetic score
The first comparison will report fixture coverage, correct response classification, canonical and noindex recognition, JavaScript discovery, robots handling, repeatability, elapsed time and peak resource use when the product exposes it. Price and workflow notes will be kept separate from technical correctness.
Crawl Foundry is operated by Matthias Ramahi, who also operates AnalyseSpider. That conflict will be repeated beside any Crawl Foundry result. Screaming Frog and Sitebulb are not affiliated with this site.
Current result status
The controlled header-response baseline is verified; vendor crawler comparisons remain not proven. The protocol and fixture are published so later Crawl Foundry, Screaming Frog, Sitebulb or other runs cannot silently change the test after a preferred result appears. The parser tests remain available as a separate robots-rule fixture report.