Use noindex, or password protection, to keep a page out of search results. Robots.txt controls crawling, not indexing, and Google calls it "not a mechanism for keeping a web page out of Google." A blocked URL can still appear in results, just without a description.
What robots.txt does
Google says robots.txt tells crawlers which URLs they may access, mainly to avoid overloading your site. It can manage crawl traffic and keep media files out of Search results. It can also block unimportant resource files, but only if the page works without them. Blocking resources Google needs to understand a page can hurt how it analyzes that page.
What it does not do
- It does not hide a page. If other sites link to the URL, it can still be indexed.
- It does not enforce anything. Reputable crawlers obey it. Others may not.
- It is public, so listing a secret path in it advertises the path.
Sensitive information should be protected with authentication.
What noindex does
Noindex, set in a meta tag or HTTP header, tells Google not to show the page in results. Google recommends it, along with password protection, for keeping pages out of search. Removing the page is another option.
<meta name="robots" content="noindex">
For non HTML files, use the X-Robots-Tag response header.
The trap: combining them
Google warns that combining crawling and indexing rules can make one counteract another. If robots.txt blocks a page, Googlebot cannot fetch it, so it never sees the noindex. The URL may then stay in results. To remove a page, let it be crawled and serve noindex.
A quick decision table
- Reduce crawl load from search result pages or sort parameters: robots.txt.
- Keep a thin or private page out of results: noindex, or authentication for truly private content.
- Retire a page permanently: remove it and return 404 or 410, or redirect. See redirects and status codes.
- Choose between duplicates: canonical or redirect, not noindex or robots.txt. See canonical URLs for versioned docs.
Related controls
For snippets, Google documents nosnippet, data-nosnippet and max-snippet. For AI training crawlers, vendors publish separate user agents. See robots.txt for AI crawlers.
A note on staging sites
Put staging behind authentication. A disallow rule alone will not stop a linked staging URL from being indexed.
Quick check
My page shows in results with no description. Why?
Google says that happens when robots.txt blocks the URL but it is still indexed. Remove the block, or use another method if you want it hidden.
Can I use noindex inside robots.txt?
I did not find that in Google's documentation. Use the meta tag or header.
Sources
- Introduction to robots.txt, Google Search Central
- Consolidate duplicate URLs, Google Search Central
- Snippets and meta descriptions, Google Search Central
Last reviewed: 19 October 2026.
