Compare crawlers without mixing different jobs.
Search bots, AI crawlers and site-audit tools do different things. We separate those groups and show a source for every changing fact.
Search and AI crawlers
The table describes published functions. It does not prove that one request really came from the named operator.
| Crawler | What it is used for | robots.txt | How to verify it | Checked |
|---|---|---|---|---|
| GooglebotSource | Builds Google's search index. | Automated common crawlers obey robots.txt. | Published IP ranges or confirmed reverse and forward DNS. | 22 Aug 2026 |
| BingbotSource | Builds Bing's search index. | Bing publishes a dedicated robots.txt token. | Bingbot verification tool and Bing's documented DNS check. | 22 Aug 2026 |
| OAI-SearchBotSource | May collect pages for search features in ChatGPT. | Has its own token, separate from GPTBot. | User agent plus IP ranges published by OpenAI. | 22 Aug 2026 |
| GPTBotSource | Collects content that may be used to train generative models. | Its own token; blocking signals exclusion from training. | User agent plus IP ranges published by OpenAI. | 22 Aug 2026 |
| ClaudeBot / Claude-SearchBotSource | Anthropic separates model training and AI search. | Anthropic says these bots honor robots.txt. | Bot token plus the IP list linked by Anthropic. | 22 Aug 2026 |
How this comparison works
We use current operator documentation, record the check date and separate a published statement from our own tests. A user-agent string alone does not prove bot identity.
Read the full methodWhat about site-audit crawlers?
Tools such as Contextter, Screaming Frog or Sitebulb need a different comparison. Useful measures include discovered URLs, redirect handling, canonicals, repeatability, cost and access to raw evidence.
We will publish that comparison after several runs on the same unchanged fixture site. One green crawl is not enough to name a fair winner.
View the current test fixtures