LOCAL ONLYNO UPLOADNO NETWORK REQUESTRobots testing guide
1. TEST INPUTRFC 9309NO FETCH

Robots rule input

Input is capped at 500 KiB and stays in this browser tab.

Use the product token from the robots.txt user-agent line, not a full browser user-agent string.

Waiting for a robots.txt rule set.

What this result establishes

The tester applies RFC 9309 group selection and path matching to the exact text, token, and target you entered. It combines repeated exact-token groups, falls back to the wildcard group only when no exact group exists, and chooses the most specific matching path rule.

It does not fetch /robots.txt, verify its HTTP status or content type, model a crawler vendor's undocumented behavior, or prove that a crawler will obey the file. Robots rules are crawl controls, not authentication and not a reliable way to keep a URL out of search results.

Follow the complete robots.txt testing workflow or inspect the versioned fixture evidence.