Robots rule input
Choose the job first. The matching robots.txt token is filled in automatically.
OAI-SearchBot controls ChatGPT Search access. GPTBot controls possible model-development access.
The live check uses the protected crawler gateway and the selected crawler token. It does not guarantee indexing or a future visit.
Input is capped at 500 KiB and stays in this browser tab.
Use the product token from the robots.txt user-agent line, not a full browser user-agent string.
Waiting for a robots.txt rule set.
Robots rule decision
DECISION
MATCHED EVIDENCE
BOUNDARIES
Which preset should I choose?
Choose ChatGPT search when you care about possible inclusion in ChatGPT search. GPTBot is a separate decision about model training. Claude publishes a separate search token as well. Googlebot remains the relevant crawler for Google Search and its AI search features.
What this result establishes
The live check reads the crawler decision returned for the public page. The advanced local test applies RFC 9309 group selection and path matching to the exact rules, token and target you pasted. It combines repeated exact-token groups, falls back to the wildcard group only when no exact group exists, and chooses the most specific matching path rule.
Starting from scratch? Use the robots.txt Generator to create a clean draft, then return here to test a specific path.
Follow the complete robots.txt testing workflow
The local mode does not fetch /robots.txt or verify its HTTP status and content type. Neither mode models undocumented vendor behaviour or proves that a crawler will obey the file. Robots rules are crawl controls, not authentication and not a reliable way to keep a URL out of search results.
Follow the complete robots.txt testing workflow or inspect the versioned fixture evidence.