robots.txt inputs
Start open and block only paths that genuinely should not be crawled. robots.txt does not protect confidential content.
Use * for all crawlers or a published product token such as Googlebot.
One path per line. Each rule must start with /.
Leave empty when nothing should be blocked.
One complete HTTP or HTTPS URL per line.
Allow all is loaded as a safe starting point.
Generated robots.txt
What the robots.txt generator creates
The tool writes one clean group with User-agent, Allow and Disallow directives. Complete Sitemap URLs appear below it. Blank lines and duplicate entries are removed.
Publish the file at /robots.txt on the relevant host, then open that exact URL to confirm delivery. Test important paths afterwards with the Robots.txt Tester.
What the file cannot do
robots.txt is a crawl instruction for cooperating bots. It is neither access control nor a reliable noindex mechanism. Confidential content needs authentication; an indexing diagnosis must consider robots.txt, response headers, HTML and Search Console separately.
Use “Block all” only deliberately, such as on a non-public staging environment that also has access control. On a public site, that rule can exclude search crawlers from every path.