Search engines like Google, Yahoo! and MSN perform four basic tasks. They use programmes called ‘bots’ or ‘spiders’ to ‘crawl’ the internet looking for pages. When they find and crawl the page, they can then store the details in the search engine’s database. This process is known as ‘indexing’. The search engines also process queries, returning results based on what has been typed in by internet users. (The words people type in to search for information are known as ‘keywords’). The search engines rank the results, based on how relevant each website is to the enquiry. Search Engine Optimization is all about creating pages that search engines find to be the most relevant matches to the query.

Danny Sullivan of www.SearchEngineWatch.com says crawler-based search engines determine relevancy by following a set of rules known as ‘algorithms’. Exactly how a particular search engine’s algorithm works is a closely-kept trade secret.

One of the main rules in a ranking algorithm involves the location and frequency of keywords on a web page, says Sullivan. Pages with the search terms appearing in the HTML title tag are often assumed to be more relevant than others to the topic.

Search engines will also check to see if the search keywords appear near the top of a web page, such as in the headline or in the first few paragraphs of text. They assume that any page relevant to the topic will mention those words right from the beginning.

Frequency is the other major factor in how search engines determine relevancy. A search engine will analyse how often keywords appear in relation to other words in a web page. Those with a higher frequency are often deemed more relevant than other web pages.

Search engines may also penalise pages or exclude them from the index if they detect search engine “spamming” (if a word is repeated hundreds of times on a page, to increase the frequency and propel the page higher in the listings). Search engines watch for common spamming methods in a variety of ways, including following up on complaints from their users.

Crawler-based search engines have plenty of experience now with webmasters who constantly rewrite their web pages in an attempt to gain better rankings. All major search engines now also make use of “off the page” ranking criteria.

Off the page factors are those that a webmaster cannot easily influence. Chief among these is link analysis. By analysing how pages link to each other, a search engine can both determine what a page is about and whether that page is deemed to be “important” and thus deserving of a ranking boost. In addition, sophisticated techniques are used to screen out attempts by webmasters to build “artificial” links designed to boost their rankings.

Another off the page factor is click through measurement. In short, this means that a search engine may watch what results someone selects for a particular search, and then eventually drop high-ranking pages that aren’t attracting clicks, while promoting lower-ranking pages that do pull in visitors. As with link analysis, systems are used to compensate for artificial links generated by eager webmasters.

A query on a crawler-based search engine often turns up thousands or even millions of matching web pages. In many cases, only the 10 most “relevant” matches are displayed on the first page.
Naturally, anyone who runs a website wants to be in the “top 10” results. This is because most users will find a result they like in the top 10. Being listed 11 or beyond means that many people may miss your website.