AnalyseSpider articleshttps://analysespider.com/2026-08-28T00:00:00ZFree browser tools and plain-language guides for checking crawler traffic, redirects, robots rules, HTTP responses and server logs.A 12-line crawler log: what the inspector finds and misseshttps://analysespider.com/blog/how-to-find-search-bots-in-server-logs2026-08-22T00:00:00Z2026-08-28T00:00:00ZA reproducible field note using one downloadable synthetic access log, exact local results and a spoofed Googlebot case the parser cannot verify.Matthias Ramahi What a 301 response does not provehttps://analysespider.com/blog/what-a-301-response-does-not-prove2026-08-22T00:00:00Z2026-08-28T00:00:00ZA 301 states that a resource has moved permanently. It does not prove destination quality, intent equivalence, or search-engine selection.Matthias Ramahi Private data can hide inside ordinary access logshttps://analysespider.com/blog/private-data-in-access-logs2026-08-22T00:00:00Z2026-08-28T00:00:00ZAccess logs may contain addresses, identifiers, query values, referrers, and user agents. Minimise the sample before sharing or analysis.Matthias Ramahi robots.txt allows the bot. Why can the CDN still block it?https://analysespider.com/blog/robots-txt-allows-bot-cdn-blocks-it2026-08-28T00:00:00Z2026-08-28T00:00:00ZA practical diagnosis for pages that allow a crawler in robots.txt but return a firewall block, rate limit, login gate, or bot challenge.Matthias Ramahi Initial HTML and rendered DOM are different evidencehttps://analysespider.com/blog/initial-html-vs-rendered-dom2026-08-28T00:00:00Z2026-08-28T00:00:00ZLearn what an HTTP crawler can prove from the initial response, what appears only after JavaScript runs, and how to compare the two without guessing.Matthias Ramahi Server log analysis without invented certaintyhttps://analysespider.com/guides/log-file-analysis2026-08-22T00:00:00Z2026-08-22T00:00:00ZA practical workflow for turning Apache or Nginx access logs into a bounded request inventory before drawing SEO conclusions.Matthias Ramahi Analyse crawler traffic in access logshttps://analysespider.com/guides/crawler-log-analysis2026-08-22T00:00:00Z2026-08-22T00:00:00ZFind declared search crawlers in server logs, verify identity where needed, and avoid confusing crawl activity with indexing.Matthias Ramahi Debug HTTP responses in evidence orderhttps://analysespider.com/guides/http-response-debugging2026-08-22T00:00:00Z2026-08-22T00:00:00ZInspect status, Location, canonical, robots directives, and rendered HTML without letting one signal stand in for the whole response.Matthias Ramahi Analyse redirect chains hop by hophttps://analysespider.com/guides/redirect-chain-analysis2026-08-22T00:00:00Z2026-08-22T00:00:00ZA bounded workflow for inspecting 301, 302, 307, and 308 responses, resolving Location fields, and identifying the check that comes next.Matthias Ramahi HTTP status codes for web diagnosticshttps://analysespider.com/reference/http-status-codes2026-08-22T00:00:00Z2026-08-22T00:00:00ZA compact diagnostic reference for common 2xx, 3xx, 4xx, and 5xx responses, with protocol meaning separated from SEO inference.Matthias Ramahi Crawler user agents: claim, token, and verificationhttps://analysespider.com/reference/crawler-user-agents2026-08-22T00:00:00Z2026-08-22T00:00:00ZA reference for classifying crawler strings without treating an easily spoofed header as verified identity.Matthias Ramahi Robots directives in headers and HTMLhttps://analysespider.com/reference/robots-directives2026-08-22T00:00:00Z2026-08-22T00:00:00ZRead robots meta elements and X-Robots-Tag response headers as page-level controls, with crawler access and conflicts kept visible.Matthias Ramahi Crawler verification methods comparedhttps://analysespider.com/reference/crawler-verification-methods2026-08-22T00:00:00Z2026-08-22T00:00:00ZCompare user-agent matching, published IP ranges, forward-confirmed reverse DNS, operator tools, and request-log evidence without overstating identity.Matthias Ramahi