What is an XML Sitemap?

An XML sitemap is an XML document which contains a list of all of the pages (or at least the one’s you would like to be indexed!) on a website.

Your XML sitemap should look something like:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd" xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url>
<loc>http://www.rutley.co.uk/</loc>
<lastmod>2010-12-07T13:39:54+00:00</lastmod>
<changefreq>daily</changefreq>
<priority>1.0</priority>
</url>
<url>
<loc>http://www.rutley.co.uk/blog/misc/footfeed-send-an-email-ccing-all-footfeed-beta-testers</loc>
<lastmod>2010-12-07T13:39:54+00:00</lastmod>
<changefreq>monthly</changefreq>
<priority>0.2</priority>
</url>
...
</urlset>


What is the purpose of an XML sitemap?

Providing an XML sitemap allows you to inform search engines of the location of content on your website. Search engines usually spider your website in order to find new content/pages and can easily miss new content.

How do I get an XML sitemap?

If your site has been built on a CMS you may be able to install a plugin/extension/module which will automatically generate an XML sitemap for your website when you add new content. For WordPress Yoast’s WordPress SEO plugin will allow you to generate an XML sitemap for your website (amongst other things).

If your website has not been built on a CMS or the process of installing a plugin is too costly/time consuming (or a plugin doesn’t exist) then you can generate an XML sitemap using a tool such as Xenu Link Sleuth.

Ok, now I have an XML sitemap, what next?

Now you have an XML sitemap you need to upload it to your website, the most common place to place it is on /sitemap.xml (e.g. http://www.rutley.co.uk/sitemap.xml). Once you’ve done this you’ll want to tell Google (and the other search engines) about your new XML sitemap – to do this add the sitemap protocol to your robots.txt file (click the link for instructions).

Sitemap: http://www.rutley.co.uk/sitemap.xml

And submit the sitemap to Google Webmaster Tools and Bing Webmaster Tools.

When you mouse over a search result with a magnifying glass icon next to it a preview appears to the right of the result.

What do you think?

“Successful search marketing today is all about collaboration: with clients to make sure that their websites are written and organized in a way that search engines understand; with clients’ customers to make sure engagement is happening at all levels; and with other marketing channels to get the best out of every penny spent. The winning campaign here used solid SEO techniques, social media and tactical PPC to achieve that collaboration with great success.”

Read more on the PushON blog.

How to find an SEO company

• Search for “SEO + Your Town/City” (for example “SEO Manchester“) and look at the highest ranking SEO companies. Ignore the paid advertising along the top and right-hand side of the page and also ignore any Google Local Listing/Google News items that appear.
• Get a recommendation from other business owners. If you know someone who has worked with a search marketing company ask them about their experience using that particular agency.

Questions to ask a potential SEO company

• “Can I be number one on the Google for some competitive term? By some unrealistic period of time” – if the answer is yes steer clear of them. Many cowboy agencies claim they can get you to number one for X terms but they will actually mean through pay-per-click advertising.
• “How much will you charge?” – if the answer is less than a few hundred pounds per month avoid them.
• “How do you go about building links to my website?” – if the answer mentions any of the following avoid them; buying links, link farms, content networks, bulk directory submissions, link exchanges.

Choosing an SEO company is hard and you may not get it right the first time so make sure you ask the above questions, keep asking what you’re getting for your money each month and monitor the results closely.

• Events
• Housing
• Jobs
• News
• Products
• Services
• Travel Packages
• Vehicles

Google then pumps this information into a variety of services including Google Product Search, Google Base Search and most importantly Universal Search (this is the name for when Google search includes news, images, products and other items in the results page).

Here is an example of a Universal Search result for the query garden shed:

As you can see from the example above the Google Product Search results have been inserted into the SERP (search engine results page) above all organic listings and directly below any PPC adverts- this is valuable space. It is also worth noting that Shedstore appears twice in the Google Product Search listings and is ranked #3 in the organic listings and that Screwfix.com appears in the Google Product Search listings but does not appear in the top 100 search results.

This example illustrates the value of adding items to Google Base. Over the next few weeks I am planning to post about each of the different Google Base formats and how to optimise Google Base.

• Contact related websites and see if they will link to your/your clients site. For example if you sell model aeroplanes, try to contact fan forums for links.
• Submit to directories, there are several quality directories you should try and list your site on the most important directory (in my opinion) is Dmoz, however you should also look for industry directories. You can search for industry directories using a Google search for “industry directory”.
• Check the backlinks of your competitors, ensure that if they have a link on a quality site you have tried to get one on too.
• Create link bait, these are pages on your site that will attract linkage. To create link bait your site should be humorous, informative, controversial, useful or provide something no one else does.
• Use dofollow blog search engines, these are search engines for blogs which have nofollow disabled meaning when you comment on the blog it will pass link juice back to your site. There are many dofollow blog search engines to choose from but you should make sure you leave good quality comments otherwise they will probably be removed during moderation.

There are of course many other ways of link building, however these are the main ones we deploy when working on a clients site. If you think I’ve missed a technique or you have another suggestion please leave a comment!

What is a robots.txt file?

A robots.txt file is usually located in the root of a website for example http://www.rutley.co.uk/robots.txt. A robots.txt file tells search engines what parts of the site they should visit, and which ones they should not include in their index.

What does a robots.txt file look like?

A robots.txt file must be laid out in a specific format. It should first say how much access each of the search engines have.

As there are many search engines you can specify a general policy for all search engines, and also set policies for individual search engines.

To set a general policy:

User-agent: *
Disallow:


The * means the rule applies to all search engines and the Disallow: means that no directories/files are disallowed.

To disallow the entire domain from all search engines you can use:

User-agent: *
Disallow: /


To disallow all the search engines from a specific directory you can use:

User-agent: *
Disallow: /admin/


To specify a rule for a specific search engine you can add:

User-agent: googlebot
Disallow: /articles/

This will stop googlebot indexing the articles folder.

You can also specify the location of your sites XML sitemap. To do this you simply add:

Sitemap: http://www.rutley.co.uk/sitemap.xml

This is great for SEO, as it tells the search engines where your sitemap is located, which in turn lets the search engine know what pages are available for it to index.

Summary

Robots.txt files are useful for several reasons. They allow you to stop directories and files from being indexed and can help more of your pages to be indexed.

1. Choose a domain name that is avaliable with your countrys TLD. For example if you’re in the UK then .uk, in spain .es and italy .it
2. Choose a domain name that has the .com equivalent available (you should then 301 redirect this to your other domain).
3. Include a keyword in your domain- for example if you sell children’s toys a domain like samstoystore.co.uk is much better for SEO than samsonline.co.uk
4. Don’t make your domain too long- the domain should be memorable but not long, customers simply won’t type it and the search engines will truncate it.
5. Don’t use hyphens or numbers- just use letters, hyphens don’t make a difference for SEO and make the domain hard for people to remember.

Thankfully if your website is on an Apache server it is easy to fix.

Simply add the following code to your .htaccess file in the root directory:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^domain\.com$ [NC]
RewriteRule ^(.*)$ http://www.domain.com/$1 [R=301,L]

Another canonicalisation problem is when you can access both http://www.domain.com and http://www.domain.com/index.php, again the search engines can see these pages as duplicate.

This can also be fixed easily by adding the following rewrite rule to your .htaccess file in the root directory.

RewriteEngine On
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index\.php
RewriteRule ^index\.php$ http://www.domain.com/ [R=301,L]