Have you ever noticed a spike in daily traffic, say of the magnitude of 100-200%? It’s the sort of thing you’re unlikely to miss (unless you don’t watch your stats, but of course you watch your stats, right?). Sometimes the cause of that spike has nothing to do with your site at all. Sometimes it has much more to do with problems other sites are having. And the good news? You can very often learn from other’s mistakes to your advantage.

Be Careful of Robots META Tag

You can include a META tag with your HTML that contains instructions for Search Engine Crawlers. It looks like this:

<meta content="noindex,nofollow" name="ROBOTS" />

Here’s what happens when you include the above instruction in visual terms:

Vodafone.ie Homepage with NoFollow links Highlighted in Pink

Not only is the page de-indexed, but also all links on the page are set to NoFollow – they cease to exist in the eyes of the Search Engines. The Robots META is like the big brother of the rel attribute on HTML elements. While rel applies only to the element it is an attribute of, the Robots META is applied to the entire page it appears on. Therefore it’s far more blunt than rel, and also far more powerful.

In fact the META tag I block-quoted above is the actual Robots META tag from www.vodafone.ie (I wrote previously about Vodafone.ie homepage here). I first noticed this last Thursday, and all images etc. were taken last Saturday Feb 7 by the way.

How Google Reacts to Robots=”noindex,nofollow”

Here’s the outcome once Google sees this tag on your pages:

Google Search Result for [vodafone.ie]

You can clearly see that WWW is no longer in the results, except for some search results pages (which Google doesn’t want sites to include BTW) which appear to be sitting on a GSA (probably the reason they don’t have the same Robots META).

Moral Of The Story?

Be extremely careful with the Robots META tag. You should occasionally check your site, especially your homepage, to ensure nothing has gone awry. You should also check your analytics regularly, I would say at least daily for large corporate sites that run e-commerce modules, as you’ll likely see issues such as this instantly.

How did I notice this?

Well I login to GA each morning and spend about 30-40 minutes digging through the stats for redcardinal.ie and all other client sites I have access to. When you’re traffic jumps 2.5x over the past few days it tends to make you dig a little deeper. I imagine the guys in Vodafone’s mobile email dept might also be wondering why Google is sending them so much traffic.

[Sidenote: Interesting to see Vodafone.ie are using Woopra rather than GA now.]

How To Avoid This?

You can easily avoid this issue, and some of the solutions are quite simple:

• Build some sanity-checking into your CMS. Make it alert you when the Robots tag contains “noidex”.
• Utilise a diagnosis product, such as Maximine, to alert you when it finds this META tag.
• Build a simple manual SEO-checkpoint into your QA process. You could simply use a plugin to highlight NoFollow (as I have) which would instantly show the issue during QA.
• Check your Search Engine referral logs daily to watch for spikes (negative and positive).
• Check your Google Webmaster Console for data and stats. In particular Statistics->Crawl Stats contains some very useful info for both marketing and network engineering folk. (NB: make sure you set up accounts for all relevant domains and subdomains.)
• Create a Google Alert for “site:[mydomain.tld]” and add it to your feed reader. (I haven’t tested this yet.)

I’d love to hear if you have any other ideas for checking that your site is indexed by Google, or if you know of any service offering this ability? Oh, and is anyone else using Woopra – any feedback to offer?