A useful report can contain both counts: claimed crawler requests and the verified subset.
Start with the field your server logged
Combined Apache and Nginx formats commonly include the User-Agent request header. The client supplies that value, so it is evidence about what the requester claimed to be.
Filter for the operator's current tokens and retain the original string. A broad search for “bot” is useful for discovery but mixes search crawlers, monitoring systems, scrapers, and arbitrary labels.
Use three identity states
This small distinction prevents a spoofed user agent from inflating a crawl report or triggering a production block rule.
- Claimed: the string matches a known crawler token, but the source is not verified.
- Verified: the IP or DNS check passed the operator's current published method.
- Unknown: the field is absent, custom, ambiguous, or failed verification.
Count what helps you decide
Group the candidate rows by URL class, status, and date. A useful table shows successful content requests, redirects, 404s, and server errors separately.
If a crawler never appears on an important page, the log does not tell you why. Check links, sitemaps, robots controls, response behavior, and the engine's own reporting before choosing a cause.