A 12-line crawler log: what the inspector finds and misses

This is not another general log-analysis checklist. Download the same 12-line file used below, run it through the browser-local inspector and compare the output with the hand-checked expected result.

Observed in this case

The inspector finds five crawler claims, three 4xx responses and one 5xx response. It cannot tell that one Googlebot claim comes from a documentation-only address rather than a verified Google range.

The file is deliberately small and slightly deceptive

All addresses use IANA documentation ranges, all paths are invented and no person or production system appears in the sample. Eleven rows match the Apache Combined-style parser. One malformed line is intentional and must be reported as rejected.

The file contains two ordinary Googlebot strings, one Bingbot string, one GPTBot string and one copied Googlebot string attached to 203.0.113.77. The parser groups all five as crawler claims because it has no DNS or published-range verification step.

203.0.113.77 - - [28/Aug/2026:08:01:07 +0000] "GET /pricing HTTP/1.1" 200 1450 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

Expected local result

Run the sample without editing it. These counts are a deterministic check of the current parser, not an estimate.

Accepted rows
11
Rejected rows
1
Unique addresses
8
Claimed crawler requests
5
4xx responses
3
5xx responses
1

Download the sample from the next-step panel and use “Analyse locally”. Nothing in the file needs a live network request.

The useful finding is the mismatch, not the bot total

One Googlebot claim receives a 301 on /old-guide and the following request reaches /guides/new-guide with 200. That pair is evidence of two server responses. It does not prove that Google consolidated the URLs or indexed the destination.

Bingbot receives a 404 on /missing-product. GPTBot receives 429 on /api/export. Those are specific investigation leads: confirm whether the URLs should exist and whether the throttling rule is intended for that verified requester.

What the local parser cannot settle

The 203.0.113.77 row looks like Googlebot to a text classifier and stays labelled as a claim. To attribute it to Google, follow Google's current verification method using published ranges or reverse and forward DNS. The address in this synthetic file is reserved for documentation, so it is intentionally not a real Google crawler address.

The sample also cannot answer whether JavaScript rendered, whether a URL entered an index, which canonical an engine selected or why the 500 occurred. Each of those needs different evidence.

Reproduce the check

  1. Download the synthetic Combined-style access log.
  2. Open the Log File Inspector and load or paste the file.
  3. Run the local analysis and compare the six counts with the expected result above.
  4. Open the crawler table and keep all five rows labelled as claims.
  5. Use the operator-specific verification method before changing a firewall rule or publishing an attributed bot count.

Primary sources

Checked for this edition on Aug 28, 2026. External documentation can change.

  1. 01
    Apache HTTP Server — Log files

    Official access-log configuration and Common and Combined Log Format fields.

  2. 02
    NGINX — HTTP log module

    Official access_log and log_format behavior, variables, buffering, and escaping.

  3. 03
    Google — Common crawlers

    Published crawler tokens, user-agent strings, products, and IP-range context.

  4. 04
    Google — Verify crawler requests

    Official IP-range and forward-confirmed reverse-DNS verification methods.

  5. 05
    Bing Webmaster Tools — Bing crawlers

    Published Bing crawler user agents and an explicit warning that strings can be spoofed.