robots.txt Explained: What to Block and What Never to Block

robots.txt is a small text file at the root of your site (example.com/robots.txt) that tells crawlers which parts of the site they may request. It is useful for keeping crawlers away from areas with no search value, but it is also behind some of the most damaging SEO mistakes, such as accidentally blocking an entire site.

How the rules work

A robots.txt file contains groups. Each group starts with one or more User-agent lines naming a crawler, followed by Allow and Disallow rules for URL paths. Google’s robots.txt introduction is the reference.

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

• User-agent: * applies to all crawlers that do not have their own group.
• Disallow: /cart/ blocks every URL starting with /cart/.
• Disallow: with nothing after it allows everything.
• Rules are case-sensitive and match from the start of the path.
• The Sitemap line tells crawlers where your sitemap is.

What robots.txt cannot do

• It does not remove pages from Google. A blocked URL can still appear in results (without a description) if other pages link to it. To keep a page out of the index, allow crawling and use a noindex tag, or protect it with a password.
• It is not security. Anyone can read the file, and badly behaved bots ignore it. Never list secret paths there.
• It does not stop Google from seeing a `noindex` you added. If a page is blocked, Google cannot crawl it to see the noindex tag. Do not combine both on the same page.

Reasonable things to block

• Cart, checkout and account pages.
• Internal search result pages that create endless URL combinations.
• Filter or sort parameters that generate huge numbers of near-duplicate URLs, if they are causing crawling problems.
• Staging or test sections, though password protection is better.

Never block these by accident

• The whole site: Disallow: / under User-agent: *. Common after launching a site that was hidden during development.
• CSS and JavaScript files Google needs to render your pages.
• Image folders, if you want images to appear in search.
• Pages you want indexed, including blog and product directories.

How to check and test

1. Open yourdomain.com/robots.txt in a browser and read it line by line.
2. In Google Search Console, open Settings → robots.txt report to see the version Google fetched and any errors.
3. Use URL Inspection on an important page; it states if the URL is blocked by robots.txt.
4. After any change, re-check a few key pages.

Platform notes

Shopify, Wix, Squarespace and many WordPress setups generate robots.txt automatically. Some let you edit it; others only through settings such as “hide site from search engines”. Check those settings whenever pages unexpectedly disappear from Google.

robots.txt works alongside your sitemap and canonical tags. See canonical tags and duplicate content and the indexing checklist.