[Sigia-l] ROI/Value of Search Engine Design - Resources?

[Richard Wiggins]

Bravely or foolishly launching into a week-old thread that got a tad
contentious:

1) I hope everyone testing the Amazon DVD player search is comparing apples
to apples.  Go to Amazon.com -- the home page, not the Electronics store --
and type into the search box:

 dvd player

Do NOT select an area on the drop-down list; leave it set to: "All
Products". (Most customers will behave likewise.) Click "Go".  Observe the
results.  Amazon is presenting a federated list that's trying to be all
things to all people.  I claim that in this example the federated list is
too complicated.  The percent of people who search for "dvd player" who want
to buy The Godfather on DVD or who want to buy a book on DVD players has got
to be very small.

Now, do the exact same search, only do it for:

 dvd players

Observe the results.  Note that the federated hit list is quite different! 
I claim you can't distinguish user intent between a search for "dvd player"
and "dvd players".  

2) Amazon is one of the most challenging examples you could choose to pick
apart. If the thesis is that search engines can't deliver and that site
designers should put all their energy into crafting links -- a thesis that
is demonstrably wrong -- it's easy to find examples of failure from the
world's most complicated online retailer.

3) What matters in all this isn't the end cases that we can arbitrarily
imagine; it's real world behavior. Amazon should be looking at search logs
and finding out how many people are typing "dvd players" -- and they should
tailor the federated hit list to match most common user intent.  (They
should also consider training their search engine to do plurals and
stemming.)

4) Measuring searches in a lab by arbitrarily picking tasks to test is
interesting but not necessarily indicative of anything.  If you want to
measure search effectiveness, you MUST consider what people search for the
most.  You have got to analyze search logs and test what people search for
the most.  The top 500 search terms may account for 40% of all searches
performed. The Zipf curve is almost a law of physics in search
distributions. (See
http://www.netfact.com/rww/write/searcher/rww-searcher-aug2002d.pdf ) OK, so
you identify that a randomly-selected search fails compared to browsing. 
What percent of real world customers ever type that search? 

5) The observation that local search often sucks is not news. For years I've
used external search engines for local site searches.  First I used
AltaVista (host:nytimes.com) and now of course Google (site:nytimes.com). 
Many end users do the same.  Many sites pick inferior search tools and pay
no attention to search performance.  You can easily prove that searching is
inferior to browsing if the tools are inferior.  Re-do your end user testing
with the free Google Web services site index and you'll get far better
results for searches.

6) It is a simple fact that about half of all users, upon landing on a Web
site with a purpose in mind, will click, and about half will search.

Those who click may convert to searchers, though, when their clicking proves
fruitless.  

Those who search are unlikely to convert to browsing if the search fails. 
They are likely to just give up.

So this says over half of your customers are going to depend on search in
one way or another.  I don't care what a lab experiment purports to show;
you can't ignore search.  You need a good search engine; you need to take
good care of it; you need to build a Best Bets service matching the most
popular searches with the best hits. (See previous thread a few months back.)

/rich

On Sat, 15 Feb 2003 07:41:00 -0800 (PST), "Boniface Lau" wrote:

> 
> > From: sigia-l-admin at asis.org [mailto:sigia-l-admin at asis.org]On
> > Behalf Of Christopher Fahey [askrom]
> >  
> [...]
> > 
> > Your two screenshots are superficially different, but they are
> > functionally the same thing (from the perspective of the search
> > engine program).
> 
> The issue was about the user perspective. Thus, how the results were
> presented was significant. The UIE article took shot at search engines
> by arguing that they did not help users to locate relevant items. But
> the cited example did.

____________________________________________________
Richard Wiggins
Writing, Speaking, and Consulting on Internet Topics
rich at richardwiggins.com       www.richardwiggins.com