Launch mistakes

robots.txt still blocking the live site

robots.txt is a plain text file at the root of a site that tells crawlers which paths they may fetch. Staging sites often carry “User-agent: *” with “Disallow: /”, and if that file is deployed to production, search engine crawlers are told not to fetch any page of the live site. Open /robots.txt on the live domain right after launch and confirm it only blocks what should stay private.

Last reviewed

← All launch mistakes

Google's documentation says robots.txt is mainly for managing crawler traffic and is not a mechanism for keeping a web page out of Google: a disallowed page can still be indexed if other sites link to it. (Source: Google Search Central: Introduction to robots.txt)

To keep a page out of search, use noindex and leave the page crawlable: a crawler that may not fetch the page never sees the rule. (Source: Google Search Central: Block Search indexing with noindex)

How it happens

Many teams block crawlers on staging with a robots.txt file kept in the repository or generated by the CMS. When the same file ships with the production build, or the CMS setting that generates it is copied with the database, the live site serves the staging rule.

Narrower rules cause the same problem on a smaller scale: a Disallow for /blog/ or /products/ written during development, or a rule that blocks a folder the site's pages are served from.

How to spot it yourself

  1. Open /robots.txt on the live domain in a browser. Read the group for “User-agent: *” and any group for a named search crawler.
  2. A line “Disallow: /” or “Disallow: /*” in that group blocks the whole site. Ask of every other Disallow line whether it should be there.
  3. Compare the file with the staging robots.txt. If they are identical, the staging file probably shipped.
  4. Check that the pages listed in the sitemap are not under a disallowed path.

How to fix it

  • Replace the staging file with a production robots.txt that allows crawling and points at the sitemap.
  • Generate robots.txt per environment, so staging keeps its block and production never receives it.
  • Take disallowed pages out of the sitemap, or allow them.

What Launch Ready checks

Each finding below shows the title, “Why it matters” and “How to fix” text the report uses. Severities are the defaults; a project rule override can change or silence a rule.

Rule robots_blocked

What the scan looks at: Runs only when the project's environment is set to Production. It reads /robots.txt at the root of the scanned address and reports the site when the group for all crawlers (User-agent: *) contains Disallow: / or Disallow: /*.

Default severity: Block Launch.

Production site is blocked by robots.txt

Why it matters: robots.txt disallows all crawlers. Search engines will not index this launch.

How to fix: Remove the site-wide Disallow: / before launch (keep it only on staging).

Rule robots_blocked_page

What the scan looks at: Runs only on Production projects, and only when the site as a whole is not blocked. For each crawled page it applies robots.txt the way Google reads it: the Googlebot group when there is one, otherwise the group for all crawlers; the longest matching rule wins and Allow wins a tie. Search, cart, login and similar pages, and pages that are also noindexed, are skipped unless the sitemap lists them.

Default severity: Important when the sitemap lists the page; Improvement otherwise.

Page not listed in the sitemap

Page is blocked by robots.txt

Why it matters: Crawlers are told not to fetch this page, so it will not be indexed even if it is linked.

How to fix: Allow the path in robots.txt if this page should be public.

Page listed in the sitemap

Page in the sitemap is blocked by robots.txt

Why it matters: The sitemap lists this page, but robots.txt tells crawlers not to fetch it, so it will not be indexed.

How to fix: Allow the path in robots.txt, or take the page out of the sitemap if it should stay private.

What Launch Ready does not check here

  • A site-wide block aimed at one named crawler only: the site-wide finding reads the group for all crawlers.
  • Projects set to Staging or Preview. Blocking crawlers is expected there, so these findings run only on Production projects.
  • robots.txt on other hostnames, such as the www or non-www version of the domain. Only the scanned address's file is read.
  • Whether search engines have dropped pages or index a blocked page anyway. Launch Ready does not read any search engine account.
  • Launch Ready does not guarantee a bug-free website.
  • It does not make the launch decision. People make it, with the report in hand.

How we scan describes what the crawler fetches, renders and stores.

Related checklist items

From the free 72-item website launch checklist, with each item's coverage label: Scan check, Partly checked or Manual.

Check your next client site before handoff

Add the staging or live URL, read the fix list, and send the client a report when it is ready.

Create a Launch Ready account to run a scan.