The X-Robots-Tag Header

The idea behind X-Robots-Tag is that it allows robots directives normally found in a meta element to be sent as part of the server’s HTTP response headers. In other words the instructions are sent with the file rather than within the file, with the main advantage that this can be used with any type of content. So an HTTP response might look like this (using the Live HTTP Headers plugin for Firefox):

HTTP/1.x 200 OK
Date: Thu, 27 Aug 2009 09:21:23 GMT
Server: Apache/2.0.52 (Red Hat)
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html
X-Robots-Tag: noindex


A ‘noindex’ for Images

As an example consider the case of a webmaster wishing to prevent images being indexed and appearing in search results. One approach might be to add the entire ‘images’ directory to the robots.txt file (i.e. Disallow: /images). The problem with this is that the robots.txt file provides crawling directives rather than indexing directives – that is, the search engine is instructed not to visit the ‘images’ directory when crawling your site, but could still end up at one of your images if it is embedded in someone else’s site. Furthermore, what if we only want to block crawlers from certain images – the robots.txt file would quickly become large and unmanageable.

X-Robots-Tag in Apache (htaccess)

A combination of ‘Header set’ and the FilesMatch directive allows us to add the robots tag in Apache. Here’s a couple of examples which could be in Apache’s httpd.conf or .htaccess files.

Add ‘noindex’ header to all image files:

<FilesMatch "\.(gif|jpe?g|png)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

Add noindex to image files matching a particular pattern – in this case those with ‘thumbnail’ in the filename (i.e. /images/product41-thumbnail.jpg would be served with a noindex header while /images/product41-large.jpg would not):

<FilesMatch "images/.+-thumbnail\.jpg$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

X-Robots-Tag in PHP

PHP’s header function allows us to send any HTTP header, as follows:

header('X-Robots-Tag: noindex,nofollow');

This supports more complex scenarios than using Apache alone. Rather than applying the same rule to all images, we could use some custom logic such as a database lookup to determine whether to add the X-Robots-Tag header. The first step would be to route requests for images to a php script using Apache’s .htaccess file (e.g. requests for jpeg files within the images directory will be handled by ‘image-handler.php’).

RewriteEngine On
RewriteRule ^images/.*.jpg$ image-handler.php

Then in image-handler.php, we can perform our custom logic (defined in the allowImageIndexing() function) and set the appropriate header:

$filename = basename($_SERVER['REQUEST_URI']); // extract image filename 
header('Content-Type: image/jpg');             // set content type (otherwise it 
                                               // will be the dafault text/html)
if (!allowImageIndexing($filename)) {          // perform lookup
    header('X-Robots-Tag: noindex');           // set the x-robots-tag accordingly
}
readfile('images/' . $filename);               // stream image file in response

X-Robots-Tag Search Engine Support

Fortunately the three major search engines all now support the X-Robots-Tag

• Google – supports any value which can be used in the robots <meta> tag
• Yahoo – supports noindex, noarchive, nofollow, nosnippet
• Bing – as Yahoo plus noodp

The answer is quite simple, it’s got to be millions of times a day. Hasn’t it?

There was a time when Google would visit even the biggest sites once a week, once a month or only 6 times a year. Now Google visits certain sites such as www.bbc.co.uk, www.sky.com and www.guardian.co.uk every 15 mins.

Test Google Crawl speed

Try it now. Take a very recent headline from the Guardian news page, then search for it. If it doesn’t appear, wait five minutes, then search again. Bingo – it’ll be in the top 5 searches (depending on how many words you used).

Now copy a handful of words from the article and paste them into google search. Again the result will be the new page.

Millions of Keywords and amendments

Think about how many keywords have therefore changed in the last five minutes and how this has therefore effected 1000s of SERPS results with ten of thousands of derivatives of keywords and permutations on the results. (Read the first blog post and I will be back to tell you how quickly the article was crawled.)

SERPS are therefore changing all the time. As one result goes to the top of the Google SERPS, others are slipping down. And as the new article gets less credit or another usurps it, then that too will slip down, thus changing the SERPS again.

This IP-SEO URL tracker tool shows just that.

Buy toys online

This is a chart for “Buy Toys Online” on google, over the Christmas period and beyond – a big search at a busy time. The SERPS are changing every day throughout the top 100 results.

This Chart (one of our SEO tools) also only takes a snap-shot of a daily search, not one done every 15 mins or half an hour.

To find out more give us a bell or drop a line below with your comments.

So why is Live search still returning an illegal movie website at the top of their rankings for a popular keyword term?

www.watch-movies.net was apparently closed down and reopened under another domain – which is a different argument altogether – and still returns on page one in Live.com for the search term ‘Download Movies Online‘, so why is this? Why does a search engine that wants to compete with Google still returns a broken link to an apparent illegal website within their SERP’s? There are many legitimate Movies available online – so why is a perfectly normal search query returning poor results?

Take a look at the screenshot below:

This is appearing in your search results today for a highly competitive keyword term. So why is it still there? Does Live continue to reward the huge volumes of backlinks pointing to the website? For some reason the webmaster didn’t place a 301 redirect on the website – which of course would have passed over page value and history to the new domain.

Take a look at the performance of the website within Google, Yahoo! and Live over the past five months:

So Yahoo! didn’t even rank the website highly in the first place, Google did but immediately the ranking dropped once the content was removed (even though there were a vast number of backlinks pointing to the website) and then there is Live, which continues to rank a website highly (amidst some fluctuations it has to be said) that hasn’t be live for over four months……and people wonder why Google has such a stronghold online.

Sir Stuart Rose, chairman of Marks & Spencer Group, said, “While the outlook remains uncertain, we’re confident that we’re doing the right things for our customers and for our business.

“Online sales were up 20% with further growth in market share as we continue to broaden the product offer,” he added.

SEO Recommendations

There is so much that can be done for the M & S website, and for a company that craves consistency and continued growth it is surprising they don’t seem to look at it.
Until then good luck with continued online growth.

Due to the nature of Bartz’s background, some cynics have questioned the motives of appointing her in such a role.  Many believe that her appointment could rekindle the interest of Microsoft, however the software giants have stated they aren’t interested in an outright acquisition anymore.

Bartz has an uphill task on her hands with kickstarting the World’s second favourite search engine fortunes as Google continues to grow.