Robots.txt Generator
Avoid accidentally blocking CSS, tools, or the sitemap
Your input is processed on your device for this tool. Only a daily usage count is sent.
Result will appear here.
What this tool does
Build a robots.txt file that allows the public site, blocks private routes, and points to the sitemap. Review the output before uploading it.
Avoid accidentally blocking CSS, tools, or the sitemap. Everything runs in your browser; your text is not uploaded.
A new site, a staging domain, or a site with a clear private route list.
Blocking CSS, JavaScript, or the sitemap by accident; test the file before deploying.
Private routes first
Use the disallow list for /admin, /api, and account pages.
Sitemap reference
A sitemap line makes the canonical URL discoverable.
Browser-first
Everything runs in your browser; your text is not uploaded.
No install, no lock-in
Open the page, finish the job, and copy the result. No desktop software.
How to use
- 1
Paste your text
Add the text you want to work on.
- 2
Run the tool
Adjust the few options that matter, then start the job.
- 3
Take the result
Copy, download, or keep editing. Nothing is saved to an account.
Example
A blank robots.txt.
User-agent: * Allow: / Disallow: /admin Sitemap: https://aibynow.com/sitemap.xml
FAQ
Robots.txt Generator questions
Will robots.txt remove a page from Google?
No. It can stop crawling, but a page already indexed may remain. Use noindex for removal.
Is my text stored?
No. The text is processed in your browser.
What is the limit?
A handful of rules
Do I need an account?
No for the guest allowance. A free account gives 20 runs per tool per day; Pro gives 50.