What does this tool do?
The Robots.txt Generator lets you define crawler allow and disallow rules visually and outputs a valid robots.txt file. You add Allow/Disallow lines per User-agent, an optional Crawl-delay and a sitemap URL, then copy the finished file in one click.
How to use it
- Enter your configuration details.
- Review the generated output.
- Copy the ready-to-use code or file.
Managing multiple user-agent rules
The tool lets you define separate blocks for different bots, such as a wildcard (*) block for all crawlers and a dedicated Googlebot block. In the Allow and Disallow fields you write one path per line, and the tool converts each into Allow: and Disallow: directives automatically. If you leave Crawl-delay empty it is omitted entirely, since not every crawler supports that directive and Google ignores it altogether.
Placing robots.txt correctly
The file only works at the root of your domain, at example.com/robots.txt; putting it in a subfolder has no effect. Remember that robots.txt is a crawling directive, not a security control: a page blocked with Disallow can still be indexed if other sites link to it. To truly hide content, use authentication or a noindex meta tag. Before going live, copy the output and save it to your server's web root.
Frequently asked questions
When should I add a Crawl-delay line?
Crawl-delay asks bots to wait a set number of seconds between requests, easing server load. Bing and Yandex honor it, but Googlebot ignores it; set Google's crawl rate in Search Console instead. Leave the field blank and the line is omitted.
What do I put in the Allow and Disallow fields?
Write one path per line starting from the site root, such as /admin/ or /api/. Disallow tells bots not to crawl that path, while Allow lets a specific path inside an otherwise blocked directory still be crawled.
Is the sitemap URL required?
It is optional but recommended. The Sitemap line points crawlers to a full list of your important pages and can speed up indexing. If you leave it blank, no Sitemap line is added to the output.
Is robots.txt Generator free?
Yes. robots.txt Generator and all 300+ tools on MagmaNex are completely free with no signup or account required.
Is my data safe?
Yes. Everything runs locally in your browser. The data you enter is never sent to a server, stored, or shared.
Does it work on mobile?
Yes. robots.txt Generator works on phones, tablets and desktops — no app download needed.
🔒 Privacy: everything runs in your browser, your data is never sent to a server.