Googlebot and Bingbot
They discover pages for search indexes. Access is necessary, but does not guarantee indexing or ranking.
Which crawler visits your site — and why?
Google, Bing, ChatGPT and Claude use different bots for search, training and individual user requests. Find the right name, the relevant robots.txt control and the evidence needed to verify a real request.
Start with the job, not the bot name.
A label such as “AI crawler” is too broad. What matters is which system triggered the fetch.
OAI-SearchBot and Claude-SearchBot
They support their operators’ search products. Google search experiences still use Google Search controls; Google-Extended is not a search inclusion control.
GPTBot and ClaudeBot
These bots can be controlled separately from search bots. Blocking training therefore need not block AI search automatically.
ChatGPT-User and Claude-User
They may fetch a page because a person asks for it now. That request is not proof of automatic search inclusion.
1. FetchA client requests a URL and receives a status, headers and HTML.
2. ProcessingA search or AI system may store, render or discard the content.
3. SelectionA later search or answer may select the page — or may not.
Crawler roles and controls, side by side
Roles come from current operator documentation. A user-agent is copyable text and cannot prove identity on its own.
Last checked against primary sources: 28 August 2026| Crawler or token | Operator and job | Control | Verification | Preset |
|---|---|---|---|---|
| GooglebotOfficial source | Google · Search index | Googlebot robots.txt token | Google IP list or reverse and forward DNS | Test rule |
| BingbotOfficial source | Microsoft · Bing search index | Bingbot robots.txt token | Verify Bingbot or reverse and forward DNS | Test rule |
| OAI-SearchBotOfficial source | OpenAI · ChatGPT search | Controlled separately from GPTBot | User agent plus published OpenAI IP ranges | Test rule |
| GPTBotOfficial source | OpenAI · possible model training | Dedicated robots.txt token | User agent plus published OpenAI IP ranges | Test rule |
| ChatGPT-UserOfficial source | OpenAI · user-triggered fetch | Not a search inclusion control | User agent plus published OpenAI IP ranges | Test rule |
| Claude-SearchBotOfficial source | Anthropic · Claude search | Dedicated robots.txt token | User agent plus published Anthropic IP list | Test rule |
| ClaudeBotOfficial source | Anthropic · possible model training | Controlled separately from search | User agent plus published Anthropic IP list | Test rule |
| Claude-UserOfficial source | Anthropic · user-triggered fetch | Dedicated robots.txt token | User agent plus published Anthropic IP list | Test rule |
| Google-ExtendedOfficial source | Google · Gemini training and certain grounding uses | Product token; does not affect Google Search | No separate HTTP user agent | Test rule |
How strong is your bot evidence?
Use evidence proportionate to the decision. A labelled estimate may be enough for a report. It is not enough for a firewall allowlist.
- Claimed
The user-agent contains Googlebot, GPTBot or ClaudeBot.
Useful for filtering. Easy to spoof. - Observed
Server log, IP, timestamp, URL and status belong to one request.
Proves the request — not yet the operator. - Supported
The IP falls within a current range published by the operator.
Stronger support. Lists can change. - Verified
The operator's documented IP or reverse-and-forward DNS check passes.
Appropriate for consequential access decisions.
What a log entry really tells you
A verified request with status 200 shows that this bot fetched this URL at that time. It does not show whether JavaScript was fully processed, the page was indexed or the page was later cited. A 403 or 429, however, is concrete evidence of a firewall, bot-protection or capacity problem.
- Mark the user-agent as a candidate.
- Keep the IP and timestamp from the same line.
- Verify the IP using the operator's method.
- Read the status, URL and response size.
From one URL to a defensible finding
Each tool answers a different part of the diagnosis. Start with the evidence already in front of you.
Crawler Analysis Tool
Checks one public URL, robots.txt, indexing directives and the initial HTML.
Open toolRobots.txt Tester
Shows which rule wins for Googlebot, OAI-SearchBot, GPTBot or Claude.
Open toolCrawler Log Analyzer
Finds bot candidates, status codes and busy paths in Apache or Nginx logs.
Open toolWhat search and AI systems need from a page
This creates technical eligibility. Indexing, ranking and citation remain decisions made by each system.
Read the AI search guide- A stable public HTTPS URL with a useful 200 response.
- A robots.txt rule that does not block the bot you intend to allow.
- No accidental denial by a CDN, firewall or bot-protection layer.
- A clear core answer and important links in server-rendered HTML.
- Consistent internal links, canonicals and sitemap entries.
- Original, verifiable content rather than interchangeable summaries.
What about Screaming Frog, Sitebulb and site-audit crawlers?
They solve a different job: auditing a website for its operator. A fair comparison needs the same unchanged fixture, the same settings and several runs. The first controlled header baseline and its raw evidence are published. Vendor results remain open until repeatable runs exist, so there is no artificial winner yet.
View the benchmark and raw evidenceResolve two common contradictions
Robots policy, CDN response and JavaScript rendering are separate evidence. These two guides show how to capture each layer without turning a browser screenshot into a crawler claim.
Primary sources and maintenance
Crawler names, IP ranges and product roles change. This page therefore links to operator documentation and records its review date.
Last checked against primary sources: 28 August 2026