robots.txt is a plain text file at your website's root that tells search engine crawlers which URLs they can and cannot access. It uses Allow and Disallow directives for specific user agents. It's a suggestion, not a security measure — well-behaved bots follow it, but malicious bots may ignore it.

What should I block in robots.txt?

Block admin pages, staging environments, internal search results, duplicate content paths, and API endpoints. Don't block CSS, JavaScript, or images — Google needs these to render your pages. Never block pages you want indexed.

Does robots.txt affect SEO?

Yes — incorrectly configured robots.txt can prevent Google from crawling and indexing your pages. Common mistakes include accidentally blocking your entire site with Disallow: /, blocking CSS/JS files needed for rendering, or blocking pages that should be indexed.

robots.txt Generator — Free Tool

What Is the Robots.txt Generator?

Create valid robots.txt files for your website. Configure crawl rules for different user agents, add sitemap references, and validate syntax.

Why Use This Tool?

The robots.txt file tells search engine crawlers which parts of your site to access. An incorrect robots.txt can accidentally block Google from indexing your entire site — a common and costly mistake.

How to Use This Robots Txt Generator

Select your user-agents — Choose which crawlers to configure — Googlebot, Bingbot, or all user-agents (*). You can create separate rules for each.
Set allow/disallow rules — Specify which paths to allow or block. Disallow /admin/ to block admin pages, or Disallow /api/ to prevent API endpoint indexing.
Add your sitemap URL — Include your XML sitemap location so search engines can find your complete page list.
Set a crawl delay — Optionally add a crawl-delay directive for bots that respect it (Bing does, Google ignores it).
Copy or download — Grab the complete robots.txt content and save it to your site's root directory.

Tips and Best Practices

→ robots.txt must be at the root. The file must live at example.com/robots.txt — search engines won't find it in subdirectories.
→ Disallow doesn't mean deindex. robots.txt prevents crawling, not indexing. A page blocked by robots.txt can still appear in search results (without a snippet) if other sites link to it. Use a noindex meta tag to prevent indexing.
→ Don't block your CSS and JS files. Google needs to render your pages. Blocking CSS and JavaScript files prevents Googlebot from seeing your page as users do, which can hurt rankings.
→ Always include a sitemap reference. Adding Sitemap: https://example.com/sitemap.xml ensures crawlers can discover all your pages, even if some aren't linked from your navigation.

Frequently Asked Questions

Does robots.txt prevent indexing?

No. Disallow only prevents crawling, not indexing. If other sites link to a disallowed URL, search engines may still index it. Use the noindex meta tag to prevent indexing.

Should I block /admin/ paths?

Generally yes — there's no SEO benefit to having admin pages crawled, and it can expose your CMS structure. But remember this doesn't provide security; use proper authentication for that.

Where does robots.txt go?

It must be at the root of your domain: https://yoursite.com/robots.txt. Search engines only look for it at this exact location.

What is a robots.txt file?+

robots.txt is a plain text file placed at the root of your website that tells search engine crawlers which pages or sections they're allowed or not allowed to access. It follows the Robots Exclusion Protocol and is one of the first things crawlers check when visiting a site.

Does Google follow robots.txt?+

Yes, Googlebot respects robots.txt directives for crawling. However, if Google finds a URL through external links, it may still index the URL (showing it in results without a snippet) even if crawling is blocked. To fully prevent indexing, use a noindex meta tag or X-Robots-Tag header.

What happens if I don't have a robots.txt file?+

If no robots.txt file exists, search engines assume they can crawl all pages on your site. This is fine for most websites. Having a robots.txt is most important when you need to block specific sections (admin panels, staging areas, duplicate content) from being crawled.

📖 Learn More

Built by Derek Giordano · Part of Ultimate Design Tools

Related Tools