A rule match is valid only for the observed robots.txt representation, the named product token, and the exact path and query tested.
Capture delivery before parsing
Request the lowercase /robots.txt URL on the exact scheme and host the crawler uses. Preserve the status, redirect chain, final content type, response body, observation time, and request identity. A pasted file cannot establish any of those delivery facts.
RFC 9309 treats successful access, 4xx unavailability, 5xx or network unreachability, and redirect chains differently. A syntactically perfect body attached to the wrong status is not equivalent to a successfully delivered robots.txt file.
Select one product token
Use the product token declared by the crawler operator, not an arbitrary full user-agent string. Exact matching is case-insensitive. When the same exact token appears in more than one group, combine those rules. Use the wildcard group only when no exact-token group exists.
A declared token still does not authenticate a request. If crawler identity matters, follow the operator's documented IP or DNS verification method separately.
Test the exact path and query
Match from the first octet of the path. The most specific matching rule wins; specificity is measured by matched octets. When equally specific Allow and Disallow rules conflict, Allow should win. Wildcards and the terminal dollar sign change the match and should remain visible in the evidence.
Test the production URL, not a shortened example. Case, percent encoding, query parameters, and a trailing slash can change the result.
Keep crawl, index, and security decisions separate
robots.txt asks conforming crawlers not to request matching resources. It is not authentication, and a disallowed URL may still be discovered or represented in search from external links. Use access controls for confidential content and an appropriate indexing control when the job is exclusion from search.
Record the local rule decision as supported until delivery and crawler-specific behavior are verified. The Robots.txt Rule Tester produces that bounded local decision; the fixture register shows the exact cases exercised by the implementation.