For any of this to work you’ll need an Apache server with mod_rewrite enabled. To get started, add the following to the top of your .htaccess file. If this results in a 500 server error, your host probably doesn’t support mod_rewrite, and there is likely not much you can do about it.

Options +FollowSymLinks
RewriteEngine On

Domain Canonicalization

While many web users view www.example.com and example.com as functionally equivalent addresses, search engines cannot make such an assumption since, technically, the URLs are different and a webserver could return different content for each. In reality, hosting packages often automatically create a ‘www’ subdomain to serve the same content as the main domain, leading to many pages appearing twice in search results – both with and without the ‘www’ prefix.

The simple fix for this is to pick one domain to use as the canonical hostname and redirect any requests for other domains to that one:

RewriteCond %{HTTP_HOST} !^www.example.com [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [L,R=301]

HTTP/HTTPS Canonicalization

Websites handling sensitive information (e.g. ecommerce) will often need to make use of secure connections via the https protocol. Typically, this is only required for certain parts of the site, such as those accessed via a login, but links to the https versions of normal pages often creep into search results.

The following example ensures that any requests for paths beginning with ‘/checkout’ will be redirected to https if they are not using the secure protocol, but attempts to access any other parts of the site via https will be redirected to their non-https equivalents.

# redirect non-https requests for /checkout to https
RewriteCond %{HTTPS} off
RewriteRule ^checkout/ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
# redirect all other https requests to http
RewriteCond %{HTTPS} on
RewriteCond $1 !^checkout/
RewriteRule ^(.*) http://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

Directory Index Canonicalization

Apache allows us to specify a list of pages which will be returned if a directory page is requested (i.e. a url ending with a slash), such as http://www.example.com/. Typically, index.html or index.php are used. However, direct access to these files is not prevented and careless creation of links can easily lead to http://www.example.com/ and http://www.example.com/index.php appearing in search results separately.

The following rule ensures that any requests ending in /index.php will be redirected to the parent directory.

# matches original request header
# (to avoid infinite loop with apache internal rewriting)
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*index\.php\ HTTP/
RewriteRule ^(.*)index\.php$ http://%{HTTP_HOST}/$1 [R=301,L]

Search engines love text

So if you write a document with some special formatting, that doesn’t fit well into a web page or contains some graphics that must be preserved, you can publish over the web converting it to a PDF file, and let it accessible to the entire world.

Search engines are smart enough to crawl your web pages and index (normally) all the link and documents contained. Google started to index PDF documents later in 2001 so they are not completely new to this kind of stuff, but recently they enhanced the quality and the user experience introducing the “Quick View” PDFs feature.

The reason why Google developed “Quick View” was due to poor quality of the “View as HTML” feature, originally developed to “translate” a file into a document readable directly into the browser (unless searchers weren’t interested in opening it into different applications after downloading it).

Unfortunately the “View as HTML” feature isn’t perfect and often the layout proposed doesn’t respect the original one. These kind of problems no longer exist thanks to Quick View which has changed its approach to opening PDF files, opening the documents directly into the browser whilst keeping the formatting intact.

But, well formatted or not, a PDF document – as any other web page – should be optimized before being shown in the SERP.

How can I optimize my PDF file for ranking?

Having looked into research papers, that contained possible indicators on how to properly optimize a PDF document, I was unable to find anything really useful. Since I’m aware (are you?) Google is particularly interested in details and quality, I decided to spend some time to create a test case to evaluate many different combinations of the same PDF document to understand which factors really influences the PDF ranking.

The test has been published on a recently registered domain, so I can entirely appreciate the results that I will collect from this test.

To stay updated about this test, or to simply have a look at it, point your web browser to my “PDF ranking and indexing test”.

The X-Robots-Tag Header

The idea behind X-Robots-Tag is that it allows robots directives normally found in a meta element to be sent as part of the server’s HTTP response headers. In other words the instructions are sent with the file rather than within the file, with the main advantage that this can be used with any type of content. So an HTTP response might look like this (using the Live HTTP Headers plugin for Firefox):

HTTP/1.x 200 OK
Date: Thu, 27 Aug 2009 09:21:23 GMT
Server: Apache/2.0.52 (Red Hat)
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html
X-Robots-Tag: noindex


A ‘noindex’ for Images

As an example consider the case of a webmaster wishing to prevent images being indexed and appearing in search results. One approach might be to add the entire ‘images’ directory to the robots.txt file (i.e. Disallow: /images). The problem with this is that the robots.txt file provides crawling directives rather than indexing directives – that is, the search engine is instructed not to visit the ‘images’ directory when crawling your site, but could still end up at one of your images if it is embedded in someone else’s site. Furthermore, what if we only want to block crawlers from certain images – the robots.txt file would quickly become large and unmanageable.

X-Robots-Tag in Apache (htaccess)

A combination of ‘Header set’ and the FilesMatch directive allows us to add the robots tag in Apache. Here’s a couple of examples which could be in Apache’s httpd.conf or .htaccess files.

Add ‘noindex’ header to all image files:

<FilesMatch "\.(gif|jpe?g|png)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

Add noindex to image files matching a particular pattern – in this case those with ‘thumbnail’ in the filename (i.e. /images/product41-thumbnail.jpg would be served with a noindex header while /images/product41-large.jpg would not):

<FilesMatch "images/.+-thumbnail\.jpg$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

X-Robots-Tag in PHP

PHP’s header function allows us to send any HTTP header, as follows:

header('X-Robots-Tag: noindex,nofollow');

This supports more complex scenarios than using Apache alone. Rather than applying the same rule to all images, we could use some custom logic such as a database lookup to determine whether to add the X-Robots-Tag header. The first step would be to route requests for images to a php script using Apache’s .htaccess file (e.g. requests for jpeg files within the images directory will be handled by ‘image-handler.php’).

RewriteEngine On
RewriteRule ^images/.*.jpg$ image-handler.php

Then in image-handler.php, we can perform our custom logic (defined in the allowImageIndexing() function) and set the appropriate header:

$filename = basename($_SERVER['REQUEST_URI']); // extract image filename 
header('Content-Type: image/jpg');             // set content type (otherwise it 
                                               // will be the dafault text/html)
if (!allowImageIndexing($filename)) {          // perform lookup
    header('X-Robots-Tag: noindex');           // set the x-robots-tag accordingly
}
readfile('images/' . $filename);               // stream image file in response

X-Robots-Tag Search Engine Support

Fortunately the three major search engines all now support the X-Robots-Tag

• Google – supports any value which can be used in the robots <meta> tag
• Yahoo – supports noindex, noarchive, nofollow, nosnippet
• Bing – as Yahoo plus noodp

The process of indexing flash is more challenging than for web pages for two reasons. With its clearly defined structure, XHTML provides a rich description of content – it identifies headings, paragraphs, and lists and provides a way to assign metadata such as title and description – it is by definition a markup language. Flash is not: flash is a multimedia platform focussed on interaction and experience and providing fewer semantic cues to search engine robots. In addition, Flash supports storing entire applications within a single file, allowing multiple pages of content to share a single URL and thus presenting as a single indexable resource.

According to a Google announcement last year..

“We’ve developed an algorithm that explores Flash files in the same way that a person would, by clicking buttons, entering input, and so on. Our algorithm remembers all of the text that it encounters along the way, and that content is then available to be indexed.”

Unsurprisingly, the details of this algorithm are not public, although Google suggest that it is based on Adobe’s Searchable SWF library. While Adobe says that content developers do not need to do anything in order to benefit from the improved indexing of swf files, they do provide a utility called the Search Engine SDK which was originally designed by Macromedia to “provide search engines with the means to search and index Macromedia Flash movies”.

The tool is available for free, but it’s quite hidden on the Adobe website and you do need to sign up to the Adobe Player Licensing Program. The best way to get it is via this link: https://www.adobe.com/cfusion/entitlement/index.cfm?e=search_sdk. Follow the prompts to sign up and verify your email and you’ll receive a download link.

For anyone not familiar with command-line applications, here’s how to use the tool on windows:

1. Extract the flash_search_sdk.zip file somewhere on your computer.
2. Open the extracted folder and copy the swf2html.exe file from the ‘windows’ directory to a new directory such as C:\swf
3. Copy the swf file you want to convert into the same directory.
4. Fire up a terminal window (in Vista, click the start button, type ‘cmd’ in the search field and press enter)
5. Switch to the directory containing the tool

   cd c:\swf

6. enter the following (where myswf is the name of your flash file):

   swf2html myswf.swf

This will output the html to the screen, but if you’d like to save the html to a file (for viewing in your browser) then simply redirect the output as follows:

swf2html myswf.swf > myhtml.html

While the tool won’t show you exactly what will appear in search engines, a quick look at the output should give you some idea whether all of the important content is visible as text, and conversely whether unimportant text could be hidden (for example, by converting it to images within the Flash).