Innovator's Dilemma, the search engine version

Blekko is soon opening its new search engine, and getting some good coverage. Microsoft's Bing is spending hundreds of millions (billions?) trying to dog chase Google in search. Countless companies tried competing in search before - Cuil, Wolfram Alpha, etc. 

It would be fantastic for the internet to have a real #2 player in this space, but unfortunately I'm afraid none of these efforts will be a real contender to Google in the long run, and not because they fall short on product quality. In fact, most of these products (and Bing is the best example) offer a far superior experience to Google's in many ways. 

But search is a strange category to compete in. I think it has a special dynamic to it -- sort of a weird variation on the "Innovator's Dilemma" -- that makes it so tricky to take any meaningful market share from Google...

The Search Innovator's Dilemma

The trouble is that search companies (and reviewers) evaluate new search engines based on what they do well. Bing does travel search great (as well as images, and many other categories), Wolfram does data-intensive search great, Blekko apparently does vertical search great, etc, etc. It's common wisdom to focus on the things you do well, and do them better than anyone else in the world. But what works well for mostly any other category of company in the world is, I think, precisely the achilles heel of all non-Google search engines. 

Because users, subconsciously, place less value on the queries a search engine handles exceptionally well, than they do on queries the search engine doesn't. In other words - a great search result for a travel query on Bing will matter less to users than getting no good results on a query for say "italian vespa collection poster" (compare to Google...)

Google might lose in the areas where its competition shines, but it doesn't matter. Our habits as users don't form there... they form on those multiple-keyword, 1-off, "non-important" queries, and that's where Google absolutely kills its competition. 

Hence the 'Search Innovator's Dilemma'... to win search market share from Google, it hardly matters to one-up Google on quality, or UI, in any specific type of search. The search habits don't form there. Where you'd need to focus is on consistently delivering OK results on all those billions of queries that "don't matter", and that's a task that, it seems, is nearly impossible for a company to take on. 

Comments

[Yaron Galai]

One comment to myself - I fell for the exact trap I described above when I posted about Mahalo a while back - http://www.webx0.com/2007/07/mahalo.html

I loved the concept behind Mahalo - taking the 20% search queries that "matter" and using human editors - making them great. But I now realize my mistake in thinking this could take significant market share from Google... it hardly mattered for people that some results were great. It's the 80% of not-so-great search results that matter most when trying to form new search habits.

[Jay Meydad]

Yaron,

You are correct that Google is very good at the tail queries compared to other search engines. Personally, I gave Bing a try but that lasted about a month until I realized that I was going back to Google to perform more than 50% of the queries due to poor results on Bing.

However, the strategy for becoming a strong #2 player in search requires more than offering a product that returns relevant results for tail queries (+ the head of course). It requires changing habits that were formed among most users between 5-10 years ago. Today, to get someone to switch to a new search engine is pretty much like asking that person to start speaking a new language and stop using his first language because the new one offers a richer vocabulary. It is impossible.

In addition to that, the Google logo by itself, builds an enormous level trust and confidence in the quality and relevance of the results. I witnessed it myself shortly after we launched Snap.com, our next generation search engine. We held several focus groups in our usability labs. We ran a “blind taste test” where the Snap, Yahoo and Google feeds were hooked to several UIs: plain vanilla SERP with no logo, a SERP with the Google logo at the upper left corner, and the Snap Enhanced UI. No matter which feed was used behind the scene, anytime the Google logo appeared on the document, users ranked the result set as the most relevant one.

Last thing, distribution. A new player will have to pay a lot for distribution. Then they need to count on getting enough users and make offering such a wow experience that these users will become loyal repeaters rather than 1-3 time users. Once this happens they will have to good problems of building a sales force and start signing up advertisers. Ask.com and Bing are showing all of us how expensive and difficult it is.

IMHO, the search game is over.

[Hershberg]

Good post, Yaron -- I agree with your POV. Coincidentally, I also wrote about the 80/20 rule when Mahalo initially launched, although I was confident that focusing on the 20% of searches that "matter" was a flawed approach. Of course, I had the benefit of having worked for a company that tried the same thing years earlier (and failed), so it was a lesson learned the hard way: http://bit.ly/a512oP

[Yaron Galai]

Hey Jay - thanks for the great comment. Agreed - it's going to take more than improving results for the long tail queries to change our Google habit... Also - I wrote in the past that catching up on paid search is nearly impossible as well (http://www.webx0.com/2007/05/microsoft_final.html). Microsoft is dog chasing Google on 2 fronts that are nearly impossible to win. Ugh...

[Yaron Galai]

Thanks Pete - wish I had read your post back in 2007. You were spot on, and I totally missed this paradox - search matters not on those queries that "matter", but rather on all those queries that don't... ;-)

[Sharel]

Great post, thanks.

I loved the "innovators dilemma" and Clayton M. Christensen newer book "The Innovator's Solution: Creating and Sustaining Successful Growth" where he talks about the importance of a scalable business model.

When you look at search engines and think of their buisness model, then all of them are the same... i think that the one that will have a different biz model would have a chance to win some market share... Clayton wrote a great article about it : http://hbr.org/2008/12/reinventing-your-business-model/ar/1 where he talks of apple and how the app store made the difference not the iPhone itself..

[SEOGeneration]

I'm just loving blekko and outbrain - Just had a small local publisher try it out. Proven to be great so far.