Trace crawl and response evidence before opening another audit ticket

Use AnalyseSpider for the small evidence step before a larger crawl or platform investigation: isolate the requests, responses, directives, and unknowns that justify the next check.

Decision rule

The output is a diagnostic handoff, not an automated verdict about indexing or ranking.

01

Useful starting jobs

Parse a bounded log sample, classify redirect and error patterns, inspect pasted response headers and HTML, or check whether an IP value is public, private, or reserved.

  • Crawler candidate inventory
  • Status and path distribution
  • Redirect and directive capture
  • Observed, inferred, and unknown handoff
02

Where this stops

AnalyseSpider does not currently fetch arbitrary URLs, crawl sites, query Search Console, verify bot IPs, or decide canonical selection. Those need separate evidence and controls.

Client logs can contain personal or confidential data. Minimise the sample before pasting it into any tool, including a local one.

03

A practical sequence

  1. Reduce the sample to the affected time and route set.
  2. Inspect local observations and rejected rows.
  3. Write the strongest supported hypothesis.
  4. Verify it in the relevant server, crawler, search-platform, or application surface.