Store the full string, extracted token, source IP, identity state, verification method, and timestamp as separate fields.
Five fields that should not collapse into one
- Full user-agent string: the exact request header value.
- Product token: the crawler name used for matching or robots rules.
- Source address: the IP recorded by the logging layer.
- Identity state: claimed, verified, or unknown.
- Verification evidence: range file or DNS procedure plus timestamp.
Google crawler evidence
Google publishes common crawler strings and tokens as well as machine-readable IP ranges. It also documents a forward-confirmed reverse-DNS method for manual verification.
Different Google products use different crawlers and fetchers. Do not normalise every string containing “Google” into Googlebot.
Bing crawler evidence
Bing publishes examples for bingbot and other crawler products. Its documentation explicitly warns that user-agent strings are easy to spoof and provides a verification workflow for IP addresses.
A classification rule that survives change
Keep the operator documentation URL and the date you checked it. Crawler strings, rendering-engine versions, IP ranges, and product names can change.
A historical log should retain the evidence available at analysis time rather than silently receiving a present-day identity label.