[Sigia-l] ROI/Value of Search Engine Design - Resources?

[Jared M. Spool]

rich at richardwiggins.com wrote in response to my comments:

>I certainly don't believe that the same ratio of searchers versus browsers
>applies at all sites, or even within a category of sites.   I may believe,
>for instance, that PC Connection's search engine is great, and the Amazon's
>sucks (example only).  Once I know that, I'll tend to use the engine that
>works more, and use other approaches at the site where search sucks.

Excellent point.

However, our users had never previously visited most of the sites in our 
study. So, they wouldn't have that information up front.

Therefore, how did, on 21% of the sites, every user always know to use 
Search? Why did every user choose to use Search on PC Connection (without 
knowing that it's engine is great, as you say), whereas, on 
SmarterKids.com, they always used the categories, even when they were 
shopping for something specific, like something from the Life On Mars line 
of Lego toys?

Your contention, if I understand your point, is that some users actively 
choose to use Search. My contention is that, more often than not, designers 
force users into using Search.

I'm not saying that designers shouldn't work hard to make an effective 
Search tool -- on the contrary -- I'd love to see one that really worked 
well. (Actually, we're about to announce several that did well in our 
recent study, so I actually have seen them. But, they are the exception, 
not the rule.)

However, designers need to also realize that it is their action that causes 
users to go to Search and if their Search tool is 'less than stellar', they 
have alternatives to improving the user's experience than just working on 
improving Search. Again, I'm not saying they *shouldn't* work on improving 
Search -- I'm just saying they have options.


> > If the Search Dominance Theory (which says that some
> > percentage of users always uses Search) is true, then, while observing
> > users use different sites, we should see some percentage of those users
> > always using Search.
>
> > Except we didn't. When we watched 30 users each use between three and six
> > sites, not a single one of them always used Search. 18% always used
> > categories, the rest were mixed usage -- some sites they used Search, some
> > sites they used categories.
>
>Again, I'd expect variations among users, tasks those users had in mind, and
>the sites they are trying to use.

Well, there was variation. Users were different. Their tasks were 
different. (The sites were chosen by us, so, they weren't different.)

There wasn't correlation, however. At least, none that we could see.


>Heck, I'd expect a huge variation just depending on whether the site has a
>Search box, as opposed to a Search link, on the home page.

I'd expect that too. But, alas, no such luck. We've never seen anything to 
support the notion that a Search Box on the home page increases the 
likelihood of using Search over Search links. (Jakob has reported to see 
this difference, but we've looked for it in all of our data and can't find it.)


> > Faster in what sense? When we time users, it isn't faster by clock time.
> > Search takes 15% to 20% longer. When we count clicks, the average on-site
> > Search takes 5.3 clicks, whereas, when using categories, the average is
>4.8
> > clicks. Longer time and more clicks. (Sounds sorta like a beer commercial.)
>
>I challenge you to a race.  You just overheard someone say "Thomas Friedman
>had a really great piece in the Times on Peking Duct Tape."  I just went to
>nytimes.com and found the piece in about 7 seconds. It took me two clicks.
>You find it that fast by examining the zillion links on the Times' home page
>and clicking, I'll buy you a beer.  :-)

This would be fun and we could play this game all night. We both can create 
tasks which would naturally drive the other player into Search or into the 
categories. At the end of the night, we'd have both bought each other a lot 
of beer. (And I'm guessing our searching would get slower as the number of 
beers increased.)

The question isn't "Is it faster for you or I to find something with 
Search?" Instead, it's "Do users actually find things faster with Search?" 
They are different questions.

I readily admit that a trained LIS or IA professional would have no trouble 
making Search look lightning fast in demonstrating specific tasks on 
specific sites. That's why reference librarians can find things that 
patrons can't. That's why my wife could always find things in the fridge 
that I claimed weren't there. (She always referred to this specific 
phenomena as "Male Refrigerator Blindness".)

However, when real users visit sites, what is actually happening. That's 
what I'm trying to understand.

> > That's interesting. When I typed "Linksys WiFi" into Google, the first
>page
> > that came up was this one: ....If I
> > understand you correctly, you're telling me that you'd click Search in the
> > upper left and wouldn't click on any of the Linksys WiFi product links. Is
> > that correct? If so, you'd definitely be in the minority of the users
>we've
> > tested.
>
>No no no, I might start with froogle.google.com to buy something, but I'd
>never start a purchase at generic Google.  I was referring to local search.
>The comparison was that if PC Connection makes it easier to buy the same
>known item at competitive prices due to superior search, then the one with
>the better local search engine wins my business.

Ok. That's not what I thought you were talking about. I thought you were 
talking about how you would always use Search to find your "Linksys WiFi" 
content. My point was that, if there was a link on whatever landing page 
you ended up on (such as PC Connection's homepage) that blatantly said 
"Linksys WiFi", would you still go to Search?

In our studies, when users find the trigger words (in this case, "Linksys 
WiFi") on the page, they *don't* go to Search. It's only when those trigger 
words are absent, that they end up in Search.

> > That being the case, the question always is one of resource management. If
> > your development resources are limited, is it easier to completely
>retrofit
> > your site's Search capabilities or to redo the categories? My guess is
>that
> > it's a little bit of both.
>
>You don't have to completely retro-fit the search service.  You build a Best
>Bets service.  At the upcoming IA Summit in Portland, I will present a paper
>on how easy it is to do, and ask why it is that everyone doesn't do it.

Excellent. I look forward to the paper.

Jared



Jared M. Spool
User Interface Engineering
http://www.uie.com    jspool at uie.com

Don't miss User Interface 7 West, March 23-27, Burlingame, CA. 
http://www.uiconf.com