Label unverified user agents as claimed crawlers. Reserve verified crawler labels for requests that pass the operator's published checks.
Build a candidate set
Filter the user-agent field for known product tokens, but preserve the original string. Normalising too early can erase distinctions between search crawlers, product fetchers, ad crawlers, and user-triggered fetchers.
- Keep request time, IP address, method, path, status, bytes, and full user agent together.
- Group by token and IP only after recording the untouched row.
- Treat missing or custom user-agent fields as unknown, not human traffic.
Verify requests when identity matters
Google publishes crawler IP ranges and a forward-confirmed reverse-DNS process. Bing provides its own verification tool and warns that user-agent strings are easy to spoof.
Verification may be unnecessary for a quick exploratory count. It becomes important before blocking traffic, attributing load, reporting crawl shares, or calling a request genuine search-engine activity.
- Match the request IP against the operator's current published ranges when available.
- If using DNS, perform the documented reverse lookup and then resolve the returned hostname forward.
- Require the forward result to contain the original IP.
- Store the verification timestamp and method because ranges and hostnames can change.
Measure paths and outcomes, not vanity volume
Total crawler hits are rarely the decision. Group verified or claimed requests by page type and returned status. Compare important 200 pages, redirected URLs, 404s, parameter traps, and repeated 5xx responses.
A spike can reflect a deployment, sitemap change, new internal links, retries, or unwanted URL expansion. Logs alone do not identify the cause.
Do not turn crawl into index status
Crawling, rendering, canonical selection, and indexing are different stages. A request proves only that the recorded layer saw a request and produced the logged response fields.
Use Search Console or the relevant engine's tools for indexing evidence. Use rendered HTML and response inspection for what a crawler could receive. Keep these evidence types in separate columns.