Decision rule
Inspect response headers and HTML separately, record targeted user agents, and resolve conflicts with the relevant crawler documentation.
Where directives appear
- Robots meta element: an HTML element, normally in the document head.
- X-Robots-Tag: an HTTP response header that can also apply to non-HTML resources.
- robots.txt: a crawl-control file, not a page-level noindex mechanism.
Common rules
Support and interpretation can differ by crawler. Recheck the operator's current documentation before relying on a rule outside Google Search.
- noindex asks a supporting search engine not to show the resource in results.
- nofollow asks a supporting crawler not to follow links on the resource.
- nosnippet prevents a text or video preview in supporting Google results.
- max-snippet, max-image-preview, and max-video-preview set supported preview limits.
Crawler access comes first
Google states that a crawler must be allowed to access a page to discover its meta or X-Robots-Tag rules. Disallowing a URL in robots.txt can prevent those page-level rules from being seen.
This is why crawl control and index control should be tested as a combined system, not as interchangeable files.
What an inspector should report
- Every robots meta element and its name attribute.
- Every X-Robots-Tag field and any targeted crawler token.
- Duplicate or conflicting rules without silently discarding one.
- Whether the HTML, response headers, or original fetch context are missing.