x-robots-tag

We’ve all got used to being able to control how the major search engines index our sites using a combination of robots.txt and the robots meta tag to add attributes like ‘noindex’ to individual pages. While this works great for the pages themselves, it’s not so good for non-HTML, indexable content such as PDFs or embedded media, as we have no HTML <meta> tag in which to insert the meta-information. In this article we take a look at a potential solution to this problem: the X-Robots-Tag HTTP Header.
(more…)