[Sigia-l] ROI/Value of Search Engine Design - Resources?

[Jared M. Spool]

rich at richardwiggins.com wrote:

>Last year we reached a point where Google outstripped Yahoo in the number of
>referrals to other Web sites worldwide.  People who trust Google first
>arrive at your Web site expecting to trust local search first.

It's an interesting theory and I see how you could infer that users have 
these expectations. However, our studies show that how people behave on 
Google or Yahoo is very different than they behave with on-site Search.

Users searching on Google expect that their destination page (the target 
content) will be found on a site *other* than Google. It's a different ball 
game when they expect to find the content on the site they are on.

>It's silly to assume that there is one ratio of searchers to browsers for
>all sites or for all purposes.

I agree with you here.

However, would you go so far as to say that, if we observed the same user 
use different sites in the same genre of content (such as office supplies, 
apparel, or medical information), we should see them use Search the same 
across each site? If the Search Dominance Theory (which says that some 
percentage of users always uses Search) is true, then, while observing 
users use different sites, we should see some percentage of those users 
always using Search.

Except we didn't. When we watched 30 users each use between three and six 
sites, not a single one of them always used Search. 18% always used 
categories, the rest were mixed usage -- some sites they used Search, some 
sites they used categories.

However, we did see *sites* where every user used Search. In fact, on 21% 
of the sites, every user who came to it clicked on Search immediately. In 
addition, on 32% of the sites, every user who came to it clicked on one of 
the category links immediately and didn't use Search. Our inference from 
this is that it is the design of the site that encourages users to utilize 
Search, not some natural tendency of the user. (More details can be found 
in this article: http://www.uie.com/Articles/always_search.htm and this 
report: http://www.uie.com/what_they_want.htm)

>If I go to Nytimes.com to read the news,
>obviously I scan the headlines and click on the ones of interest.  If I go
>to the Times to find Thomas Friedman's last column, I always search.  I
>don't even try to figure out if he's under Columnists or whatever random
>hierarchical label someone has chosen.  I don't care if the Times has hired
>the Leonardo da Vinci of information architects to design its navigation;
>with a good search engine, it's faster to search.

Faster in what sense? When we time users, it isn't faster by clock time. 
Search takes 15% to 20% longer. When we count clicks, the average on-site 
Search takes 5.3 clicks, whereas, when using categories, the average is 4.8 
clicks. Longer time and more clicks. (Sounds sorta like a beer commercial.)

If we look at perceived time, we know that users rate sites as faster when 
they complete their tasks on the site. (See http://www.uie.com/truth.htm.) 
Since they are more likely to complete their tasks with the categories than 
with Search, they actually rate sites faster when they *don't* use Search.

I've heard the "faster to search" argument many times. I just don't see it 
reflected when in any of our research. Does anybody else?

>If I'm looking to buy a Linksys WiFi router, I'm going to type that phrase
>into the search box, whether I choose Amazon or PC Connection.  The site
>whose search engine gets me to the right item wins my business.

That's interesting. When I typed "Linksys WiFi" into Google, the first page 
that came up was this one: 
http://www.tcwo.com/cgi-bin/webc.cgi/st_main.html?catid=309. If I 
understand you correctly, you're telling me that you'd click Search in the 
upper left and wouldn't click on any of the Linksys WiFi product links. Is 
that correct? If so, you'd definitely be in the minority of the users we've 
tested.

It's interesting that, for your example, you chose the product genre of 
computer accessories, because it's one of the worst performing genres 
overall that we've measured. Almost all users go to Search in that genre 
because the category links are usually dismal. (By the way, PC Connection 
is one of the 21% of sites where every user in our study went to Search 
immediately. I'm not surprised that you go straight to Search there 
yourself. I know I do.)

Apparel and Pets are two of the best performing genres we've measured. I 
wonder if you'd naturally go to Search in each of those. (In a recent study 
of 44 users shopping for apparel, the average user who used Search actually 
clicked on 4 pages before trying Search. And two of the top three 
performing sites in the study didn't have a Search engine at all--and users 
didn't seem to notice. It seems that the nature of the content makes a huge 
difference in the desire to use Search.)

>I object strenuously to the notion that clickable navigational elements are
>part of information architecture and to be celebrated, and search is a
>robotic straitjacket, to be deprecated.  Search is part of your site's
>information architecture.  If "only" 20% of your customers start with
>search, it's worth delivering the best search experience you can give them.

You won't hear any disagreement from me here. In fact, at the upcoming UIE 
Research Forum (on Thursday of the User Interface 7 West conference: 
http://www.uiconf.com), we'll be discussing the results of a recent study 
we've just completed that compares the search results from 76 different 
on-site search engines in 7 different content genres. It's fascinating to 
see the range of experiences that these sites deliver to users and what we 
can learn by comparing them side-by-side.

That being the case, the question always is one of resource management. If 
your development resources are limited, is it easier to completely retrofit 
your site's Search capabilities or to redo the categories? My guess is that 
it's a little bit of both.

As someone recently mentioned (maybe you--I can't remember), most content 
requests fall into a standard Zipf distribution. Given that, designers 
should take the 1%-10% of most sought after content and make sure the 
category structure is flawless for delivering that content to the users. 
Then you focus on the Search for the remaining 90% of the content that less 
than 10% of the users will request.

(An interesting side note: We recently discovered that, on Microsoft.com, 
90% of the visits to the site requested less than 1% of the available 
content. That's far more dramatic than a standard Zipf distribution, 
considering the number of visitors to the site.)

BTW, I agree that we'd all benefit from more studies published in 
peer-reviewed journals. At UIE, we currently don't pursue that route 
because it's extremely time consuming and our funding sources don't require 
it. However, I'd like to encourage anyone interested in getting publishing 
credit to go forward with it. Over the years, we've worked with quite a few 
students on projects of this nature and I expect we'll continue to do so in 
the future.

[I think it's a nice change of pace to have a discussion where people are 
interacting with civility and decorum. This is something that we should all 
encourage. I don't know about anyone else, but I'm happy to participate 
when everyone is treating each other in a professional manner, showing 
consideration for differing viewpoints and respect for a diversity of 
opinions.]

Jared


Jared M. Spool
User Interface Engineering
http://www.uie.com   jspool at uie.com

Don't miss User Interface 7 West, March 23-27, Burlingame, CA. 
http://www.uiconf.com