robots.txt
A robots.txt file provides crawling instructions for automated crawlers visiting a website.
It can tell supported crawlers which areas of the site they are allowed or not allowed to request.
What It Controls
robots.txt primarily controls crawling.
It can be used to prevent supported crawlers from requesting URL patterns that do not need to be crawled, such as certain filtered pages, internal search results, or other low-value URL variations.
It does not function as a reliable way to remove ordinary web pages from search results.
For that, page-level indexing controls such as noindex are more appropriate.
How It Works
The file normally lives at the root of a hostname:
https://example.com/robots.txt
Rules can target specific crawlers and specify URL paths they may or may not request.
A crawler checks these rules before requesting affected URLs.
Why It Matters
A well-configured robots.txt file can reduce unnecessary crawling and keep crawlers away from parts of a site that do not need to be explored.
A badly configured file can block important pages, assets, or sections of a website.
Because it operates at the crawling stage, mistakes can also make other instructions harder for search engines to see.
Common Problems
Common issues include:
- Accidentally blocking the entire website
- Blocking important content sections
- Assuming
Disallowmeansnoindex - Blocking a page that contains a noindex directive
- Leaving staging rules in place after launch
- Using broad wildcard rules without checking which URLs they affect
- Trying to protect sensitive information with robots.txt
A robots.txt file is publicly accessible and should not be treated as a security mechanism.
How to Improve
Keep rules as simple as possible.
Review them during launches, migrations, and troubleshooting, and verify that important pages remain crawlable.
If the goal is to keep a page out of search results rather than prevent crawling, use the appropriate indexing control instead.