Exploiting the long tail for search

The 'Long Tail', the name given by Chris Andersen in his Wired Magazine article to the phenomenon  on the Net where the majority of people (esp. in the ecommerce arena) search for completely unique items, has a similar corresponding phenomen in global and enterprise search. The concept basically identifies that on large web shops the majority of customers buy unique items. This is evidenced by shops such as Amazon.com and Netflix where the most popular titles are not the majority of purchases. In actual fact, the majority of customers buy single unique items that few or no other users purchase. So, for example, the number of customers who bought some odd specialized book adds up to more than the number of customers who bought one of the most popular books like the Da Vinci Code or similar.

In Enterprise Search this concept is well known and Mondosoft in fact identified this behavior with its 'Expectation Map' released in 2001. The 'Expectation Map' wll take this data and show you what kind of site your users are expecting and what kind of information (popular or diverse) these users are interested in. For most web, local or enterprise sites, the 'long tail' is not so long because internal users, employees, or corporate visitors have much more similar interests than Amazon shoppers. Some sites, however, especially large public facing site can have a pretty long tail. One of my customers who has a public site with almost 100,000 documents and a wide user base had 55% of their users searching for one unique terms, whereas 33% searched for just 25 common terms.

Global search is the complete opposite and has an incredibly long tail, having to cater to the greatest possible diversity of web users.

What does this mean for a site owner? Well, for local or site search you can tell very little about this behavior but you can certainly see how the site is functioning if you watch users' expectations through this data. You can take those popular terms and do something about it but catering to those other 'one off' searches is nearly impossible.

There is, however, an element of Search Engine Optimization that can be done with this data in respect to how users go from global to local search. Although many sites will never reach the top of ranking on Global search engines for their most popular searches, there are thousands, perhaps millions, of possible searches that may lead users to their site - searches that are being done every day. We can look at the referrals with these searches to identify searches that can help users get to your site and optimize for those.

A new service called Hittail caters to exactly this kind of analysis. I've been using it for my own site now for a few months and find the information it gives very interesting. I can see what terms people are searching for when they find my site and I can also see what kinds of terms are in the long tail and could help my small site perform better.

If you have your own site, I suggest you check it out.

Read the complete post at http://blog.mondosoft.com/art-of-search/archive/2007/02/12/12.aspx