Robots Meta Tag Builder
Toggle directives, copy the tag. The builder covers every robots directive Google and Bing recognize — index/noindex, follow/nofollow, noarchive, nosnippet, noimageindex, max-snippet, max-image-preview, max-video-preview, unavailable_after — and emits a clean, minimized tag with only the directives you have changed from default. No defaults are emitted, because emitting them is redundant and clutters the head.
Why Page-Level Beats Site-Level Crawling Control
The robots meta tag is the precise instrument; the robots.txt file is the blunt one. robots.txt tells crawlers what URLs to fetch; the meta tag tells them what to do with the page once they have fetched it (index it? follow its links? show a snippet? store an image preview?). For sites where individual pages need different crawling behavior — a draft page that should be crawled but not indexed, a paginated archive where deep pages should be followed but not surfaced — the meta tag is the only correct control surface. robots.txt simply cannot express that granularity.
How the Builder Handles Conflicts
Some directive combinations conflict or are redundant. noindex,follow is valid and useful for paginated archives, but noindex,nofollow means Google will not follow any links on a page it will not index — usually wasted crawl budget. The builder warns on combinations like this without blocking them (you might have a reason). It also strips redundant defaults: if you toggle index without changing follow, the output is just the unchanged-from-default form, not a noisy tag with every directive spelled out. The goal is the minimum tag that expresses what you actually want.
Frequently Asked Questions
Where does the robots meta tag go?+
In the head of the HTML page, before the closing head tag. It must be in the served HTML — adding it via JavaScript after page load is unreliable; Google may have already decided before the JS runs.
What is the difference between noindex and disallow in robots.txt?+
robots.txt disallow tells crawlers not to fetch the URL at all. Noindex tells them they can fetch it but not include it in the index. If you both disallow and noindex, Google may never see the noindex directive (because it is not fetching the page), defeating the purpose.
When should I use noarchive?+
When you want the page indexed and findable, but do not want Google to serve a cached version. Common for stock-quote pages, news that updates frequently, or content that is licensed only for current display.
Does nofollow on a page affect external links to it?+
No — nofollow in the meta tag affects how your page's outbound links are treated, not how inbound links to your page are weighted. It is a per-page outbound link policy.
What is max-snippet?+
max-snippet:N caps the length of the text snippet Google shows in the SERP at N characters. Common values: 0 (no snippet — same as nosnippet), -1 (unlimited), or a specific cap like 160. Most sites do not set this; the default snippet length is fine.
Can I target specific crawlers?+
Yes — replace 'robots' with the user agent: a meta tag with name='googlebot' and content='noindex' targets only Google's crawler. Bingbot, Yandex, and others have their own user-agent strings. The builder defaults to robots which applies to all.
What does unavailable_after do?+
Tells Google to drop the page from the index after the date you specify (ISO 8601 format). Useful for time-bound content like event pages or limited-run promotions, where you want it indexed during the run and gone after.
Does the meta tag override robots.txt?+
Neither overrides — they answer different questions. robots.txt controls fetching; the meta tag controls indexing of fetched pages. The combination of both is the full picture.
Built by Derek Giordano · Part of Ultimate Design Tools
Privacy Policy · Terms of Service