Try the tool

Robots.txt Generator

Avoid accidentally blocking CSS, tools, or the sitemap

Open tool

A robots.txt file tells crawlers which paths they may request. It is a crawl directive, not an access control system, and it is not a reliable way to remove a page that is already indexed.

The basic structure

User-agent: *
Allow: /
Disallow: /admin
Disallow: /api/
Sitemap: https://aibynow.com/sitemap.xml
  • User-agent: * applies to all crawlers.
  • Allow: / permits the public site.
  • Disallow lists paths that should not be crawled.
  • Sitemap points to the XML sitemap.

Only one Sitemap line is needed. It can be included in a robots.txt at the root of the host.

Crawling is not indexing

If a page is already indexed, adding Disallow can stop Google from crawling it, but the URL may remain in results without a fresh snippet. To remove a page from search results, use a noindex meta tag or the Search Console removal tool. Do not rely on robots.txt for that job.

Conversely, blocking CSS, JavaScript, or the sitemap can prevent Google from understanding the page. Keep those resources allowed.

What to disallow

  • Private admin routes such as /admin.
  • Internal APIs such as /api/.
  • Account and session pages such as /account.
  • Temporary staging paths, if they are publicly reachable.

Do not disallow a path just because it has query parameters. Use canonical tags and noindex where appropriate.

Test the file

  1. Fetch https://your-domain.com/robots.txt and confirm it returns 200 with text/plain.
  2. Check that a public tool page is still allowed.
  3. Check that a private route is disallowed.
  4. Submit the sitemap URL in Search Console.
  5. Recheck after changing a rule; crawlers cache robots.txt.

Robots.txt Generator builds the file from your allow/disallow list and sitemap URL. The output still needs a human review before upload.

Common mistakes

  • Disallow: / by accident. This blocks the whole site.
  • Blocking CSS or JS. Google may render the page incorrectly.
  • Assuming robots.txt hides a page. It does not.
  • Using a relative sitemap URL. Use the full HTTPS URL.
  • Different rules for every bot without a reason. Start with *; add a specific user agent only when necessary.

For page-level indexing, Meta Tag Generator is the relevant tool. For the canonical URL list, see the sitemap.

The short version

Allow public content, disallow private routes, point to the sitemap, and remember that robots.txt controls crawling — not removal from search.

A
AIbyNow

We build small, practical tools for everyday work and write about the tasks behind them.

All guides Try Robots.txt Generator