A text file at your domain root that tells search engine crawlers which URLs they should and shouldn't crawl. It controls crawl behavior, not indexing — use noindex meta tags to prevent indexing.

Does robots.txt prevent pages from being indexed?

No. It prevents crawling, not indexing. If other sites link to a disallowed URL, Google may still index it (without the page content). Use a noindex meta tag to prevent indexing.

Where do I put robots.txt?

At the root of your domain: https://example.com/robots.txt. It must be at the root — subdirectory placement (example.com/blog/robots.txt) won't work.

How to Create a robots.txt File (With Examples)

🤖

Try Robots.txt Generator

Free, no signup →