Use the user agent to find candidates; use operator-published IP or DNS evidence when identity changes the decision; keep the original log row and verification time together.
Comparison matrix
No single field establishes every part of crawler identity. Treat the methods as evidence layers with different failure modes.
- User-agent token — fast candidate filter; supplied by the requester and easy to spoof; never sufficient for a genuine-bot claim.
- Published IP range — strong when the operator publishes current machine-readable ranges; requires fresh range data and exact address matching.
- Forward-confirmed reverse DNS — strong for operators that document hostname masks; requires both reverse lookup and a forward lookup back to the original IP.
- Operator verification tool — useful operator verdict for a specific address, such as Verify Bingbot; preserve the lookup time and input address.
- Access-log row — primary evidence that your logging layer observed a request, status, path, address, and user agent; it does not by itself verify the requester.
- Search Console crawl or indexing report — platform evidence about Google processing; it is not a row-level identity check for arbitrary server traffic.
Choose the minimum evidence for the decision
Exploratory log analysis can label user-agent matches as claimed crawlers. Blocking an address, attributing an outage, publishing crawler shares, or escalating abuse needs stronger verification because the consequence is larger.
Google documents both published range files and a manual DNS workflow. Bing publishes crawler strings, warns that they can be spoofed, and provides an address-verification tool. Apply the method documented by the claimed operator rather than one universal hostname rule.
- Preserve the original request time, IP, full user agent, method, path, and status.
- Extract a product token only as a candidate label.
- Select the operator's current verification method and record its source version or lookup time.
- Store the result as claimed, verified, failed verification, or unknown; do not collapse unknown into human.
- Repeat verification when the decision is delayed because IP allocations and documentation can change.
Failure modes worth preserving
- Reverse DNS name matches an expected suffix, but forward resolution does not return the original address.
- The address was checked against an old local copy of a published range file.
- A proxy or load balancer log recorded its own address because the trusted client-IP chain was not configured.
- A generic token grouped product fetchers, preview bots, advertising crawlers, and search crawlers into one family.
- An operator tool returned no verdict or was unavailable, and the report silently promoted the request to verified.
A failed or unavailable verification is not proof that a request is malicious. Record the evidence state and choose a proportionate next check.
A portable evidence record
A useful handoff keeps observation and verification separate: observed request fields, claimed product token, verification method, source URL or range version, verification timestamp, result, and unresolved proxy or retention caveats.