Robots.txt

How it works

A robots.txt file uses the Robots Exclusion Protocol to give instructions to compliant crawlers. It lists one or more user-agent directives (e.g., User-agent: *) followed by rules such as Disallow or Allow. Each rule targets a specific URL path. For example, Disallow: /private/ tells crawlers not to request any URL beginning with /private/. The file can also specify a crawl delay (Crawl-delay: 10) to slow down aggressive bots, though Google ignores this directive. Rules are evaluated from top to bottom, and the most specific user-agent match takes precedence. If no rule matches, crawling is implicitly allowed.

The file must be placed in the root directory of the host, e.g., https://www.example.com/robots.txt. Google states that robots.txt is mainly used to avoid overloading a site with requests. If no rule is specified for a URL, it is implicitly allowed for crawling. For more context on how robots.txt works alongside other files, see what is a sitemap.

Why it matters

Using robots.txt correctly helps direct search engine crawl traffic toward valuable pages and away from low-value or resource-heavy URLs. This can improve crawl budget allocation, especially for large sites. Conversely, blocking important resources like CSS or JavaScript can prevent Google from rendering pages correctly, harming visibility. For example, blocking a CSS file may cause Google to see a page as broken, leading to lower rankings.

Many SEO mistakes arise from treating robots.txt as an indexing tool. To ensure your robots.txt is not damaging your SEO, performing an seo audit can reveal common errors such as blocking critical pages or misplacing the file. Additionally, using a canonical tag on blocked pages does not work because Google cannot crawl them to see the canonical directive.

Where it changes your decision

• Reducing crawl waste on admin, duplicate, or thin content pages: block paths like /admin/ or /tag/ to preserve crawl budget for important pages. For large e-commerce sites with many filter combinations, disallowing query parameters can save significant crawl capacity.
• Blocking resource-heavy files that are not needed for indexing, such as large PDFs or images, to reduce server load. This is particularly useful for sites with limited server resources or high traffic.
• Allowing access to a specific crawler while blocking others, e.g., letting Googlebot crawl but blocking a lesser-known bot. This can be done by using multiple user-agent directives, with the specific user-agent preceding the general one.

What it is not

• Robots.txt is not a reliable way to keep a page out of search results. Google explicitly says it is not a mechanism for controlling indexing; use noindex or other indexing controls instead. If you block a page in robots.txt, Google may still index it if it finds links to it from other sites.
• It is not a security measure. The file only provides guidelines for compliant bots; non-compliant bots, such as those used for web scraping, may ignore it entirely. To protect sensitive data, use authentication or server-side blocking.
• It is not a substitute for meta robots tags or canonical tags when you need to prevent indexing or consolidate duplicate content. For example, a noindex meta tag works even if the page is crawled, while robots.txt prevents crawling entirely.
• It is not a guarantee that all bots will obey the rules. Only well-behaved crawlers follow robots.txt. Malicious bots often ignore the file, so never rely on it for security.

Worked example

An e-commerce site with 10,000 product pages also has a filter system that generates hundreds of thousands of near-duplicate URLs (e.g., /products?color=red&size=m). The site also has an admin area at /admin/ and a shopping cart at /cart/ that is not indexable. Without a robots.txt file, Googlebot may waste a significant portion of its crawl budget on the filter URLs, admin pages, and cart pages, leaving fewer resources for actual product pages. In practice, this could mean that only a fraction of the product pages get crawled each week.

By adding a robots.txt file that disallows /admin/, /cart/, and any URL containing a query parameter for filtering, the site can reduce crawl waste significantly. For instance, Googlebot might now spend 80% of its crawl budget on product pages instead of 20%. This helps Google focus on the most important pages, improving the chance that new products are discovered quickly. The key is to test the file in Search Console after changes, as Google's crawling behaviour can evolve over time.