Launch mistakes

A noindex tag left on after launch

A noindex rule, set in a robots meta tag or an X-Robots-Tag HTTP header, tells search engines not to index the page it is on. Agencies add it to staging on purpose, and when it travels to the live site, the launched pages stay out of search results for as long as it is there. Check the homepage and a few key templates for noindex on launch day, before the client starts asking about traffic.

Last reviewed

← All launch mistakes

Google documents two ways to set noindex: a <meta> tag and an HTTP response header. A page blocked by robots.txt is not crawled, so the crawler never sees its noindex rule. (Source: Google Search Central: Block Search indexing with noindex)

How it happens

Staging sites are kept out of search with noindex, often through a CMS visibility setting, an SEO plugin option or a header rule in the hosting configuration. When the site moves to production, the database, the configuration file or the server rules move with it, and the setting comes along.

A header-based noindex is easy to miss because nothing on the page shows it. A template-level tag can also come back after launch, when a staging database is copied over production again.

How to spot it yourself

  1. Open the live homepage, view the page source and search for “noindex”. Check the robots meta tag in the <head>.
  2. Check the response headers for an X-Robots-Tag header: in the browser's developer tools (Network tab, the document request), or with curl -I followed by the page URL.
  3. Repeat on one page of each template: a service page, a blog post, the contact page.

How to fix it

  • Remove the noindex meta tag and the X-Robots-Tag header from production templates and server configuration. Keep them on staging.
  • Leave noindex only on pages that should stay out of search, such as thank-you pages, and take those pages out of the sitemap.
  • Make the setting environment-specific (an environment variable or a config file that is not deployed), so the next database copy does not bring it back.

What Launch Ready checks

Each finding below shows the title, “Why it matters” and “How to fix” text the report uses. Severities are the defaults; a project rule override can change or silence a rule.

Rule production_noindex

What the scan looks at: Runs only when the project's environment is set to Production. It reads the robots meta tag (<meta name="robots">) and the X-Robots-Tag header of every crawled page. It reports the whole site when the homepage has noindex, or when at least two public pages have it and they make up half or more of the public pages crawled. Search, cart, checkout, account, login and thank-you pages, and URLs with a query string, do not count as public pages.

Default severity: Block Launch.

Production site is set to noindex

Why it matters: Search engines are told not to index this site. A globally noindexed production launch will not appear in Google.

How to fix: Remove noindex from production templates and HTTP headers before launch. Keep noindex on staging only.

Rule noindex_page

What the scan looks at: When the site-wide finding is not raised, which is always the case on a project set to Staging or Preview, each crawled page with noindex is reported on its own. Pages kept out of search on purpose (search, cart, login, thank-you and query-string URLs) are skipped, and so, on Production projects, are pages also blocked in robots.txt, unless the sitemap lists them.

Default severity: Important when the project is set to Production and the sitemap lists the page; Improvement otherwise.

Page not listed in the sitemap

Page is marked noindex

Why it matters: This page will not appear in search results. Confirm that is intentional.

How to fix: Remove noindex unless the page should stay out of search (thank-you pages, private tools).

Page listed in the sitemap

Page in the sitemap is marked noindex

Why it matters: The sitemap asks search engines to index this page, but the page tells them not to. One of the two is a mistake.

How to fix: Remove noindex if the page should be found in search, or take it out of the sitemap.

What Launch Ready does not check here

  • Meta tags aimed at one crawler, such as <meta name="googlebot">: the check reads the generic robots meta tag and the X-Robots-Tag header.
  • Pages the crawl did not reach within its page limit.
  • Whether a page is already indexed or how it appears in search results. Launch Ready does not connect to any search engine account.
  • Launch Ready does not guarantee a bug-free website.
  • It does not make the launch decision. People make it, with the report in hand.

How we scan describes what the crawler fetches, renders and stores.

Related checklist items

From the free 72-item website launch checklist, with each item's coverage label: Scan check, Partly checked or Manual.

Check your next client site before handoff

Add the staging or live URL, read the fix list, and send the client a report when it is ready.

Create a Launch Ready account to run a scan.