It is important to have robots txt on your site, this allows you a degree of control over where the search engine spiders visit and what they report and store in their index about your site, this allows you to keep them away from commercially sensitive material or out of member only information if you have a membership area on the site.

Matt Cutts a leading engineer at Google has produced a video that explains the robots txt in more detail and and answes some questions such as

Why is my url showing up in Google when I blocked it in robots.txt? Did you fetch that url?

How do I make that url disappear from Google?

The video can be seen on Matts Blog at http://www.mattcutts.com/blog/robots-txt-remove-url/

This is going to be be of interest to any one who needs to know how to sculpt the page rank of their sites to ensure they are getting the most from the search engines.

This is an important part of search engine optimisation and implementing these changes on your site in the appropriate manor can help your rankings in the search engines.

One word of advice, do not go too over board with the robots txt file or you might find you have stopped the search engines indexing some important parts of your website and this will see you fall in the rankings rather than rise.

No we are not talking about the Caffeine that fuels so many offices and comes in cups of lovely warm coffee, instead this is Googles latest change to there algorithm, the program that helps them rank websites on the internet and allows them to pull the correct results from the data base when you search for a term.

Caffeine, which is new technology that improves the indexing infrastructure is according to Matt Cutts of Google due to go live some time after the holidays, I have consulted my American to English dictionary and I believe this means in January.

Matt Cutts comments that this is so web developers can enjoy there holidays with out worrying about what effect the changes will have on their sites, although from the initial testing we have seen a well optomised and site with content that adds value to the internet as a whole will benefit rather than suffer when these changes take full effect.

The changes are designed to give a better experience to the user of Google and so if you have a search engine optimisation plan that is not spammy and that adds value to both your site and the whole of the web then you should be looking forward to the introduction of Caffeine.

Keyword research is not something that should ever stop, it needs to be something that you continue through out the life of your website.

The key words are the life blood of your website and ensuring you have enough relevant and commercially economic phrases to look at in the future will ensure you do not get to a point here you are unable to grow and your site becomes static.

There are a number of tools available to help with your keyword research, among them is Googles free keyword research tool, it gives good general information about suggested words to target, the levels of visitors and some idea of what you would expect to pay for a Pay Per Click campaign.

The number of keyword research tools is growing and the only limits are your budget, most good Search Engine Optimisation (SEO) companies subscribe to 3 or 4 paid for services that can cost several hundred pounds a month, but as the research is important this is not a cost that should be scrimped on. >

Looking at the keywords and phrases that your competitors are ranked for is a good way of learning the phrases you need to consider and using some of the more advanced features of the paid software you can drill right down into this information to give you the competitive advantage you need.