[Sigia-l] ROI/Value of Search Engine Design - Resources?

[Jared M. Spool]

Boniface Lau wrote:
> > I come from a school that there isn't ever a problem with users --
> > it's always a problem with the design.
>
>Extreme statement like that is good for generating debates. I can even
>see academic holding on to such belief. But seeing it coming from
>someone who is supposed to have industry experience is a bit
>surprising, to say the least. Are you making the above statement just
>to be provocative?

As I said, I have this belief. I do not say it *just* to be provocative. It 
is what I believe. I probably have a lot of surprising beliefs. It is not 
my goal to partake in a public dissection of my beliefs, particularly on 
this list which doesn't have a recent history of being a 'friendly, safe 
place' for people to share their views. So, we're just going to leave this 
conversation at this. I know you can respect that.

>It is also possible that the data have been mis-interpreted. Care to
>publish details of the study behind the UIE article "Why Amazon
>Succeeds -- And Why It Won't Help You"
>(http://www.uie.com/Articles/why_amazon_succeeds.htm)?
>
>Regarding the UIE article "Why On-Site Searching Stinks"
>(http://www.uie.com/searchar.htm), I am sure many people would like to
>see the details you have for coming to the following extreme
>conclusion:
>
>WOSSS> Our data showed that today's on-site search engines are worse
>WOSSS> than nothing -- significantly worse.
>
>Will you publish the study details to support the above extreme
>statement?

Because I have a substantial research budget to fund, I have no problem 
telling you that the study details (and many more like them) are available 
to clients who pay to have access to that information. Feel free to contact 
me if you'd like pricing information.

> > I question each of these things regularly and we're constantly
> > making improvements.
> >
> > But, in this case, to the best of my knowledge and our research
> > abilities, I believe this user was the right user and their behavior
> > was indicative of what we often see when trying to perform similar
> > tasks.
>
>If you are so convinced, then may be the problem is indeed with
>data interpretation.

Yes it may. However, other institutions that have duplicated our results 
have come to the same interpretation.

>For example, the UIE article "Why On-Site Searching Stinks" presented
>user interface problems and then claimed that "on-site search engines
>are worse than nothing". To me, that was data mis-interpretation.
>
>Would you mind explaining the logic behind the statement "on-site
>search engines are worse than nothing"?

The logic is simple: When users don't use on-site search, they are twice as 
likely to find their content than when they do. This finding has been 
duplicated several times by us and others.

It would be easy for you to duplicate it also. All you have to do is ask 
users to find content (it works best with content they personally find 
important instead of randomly assigned tasks) on a variety of sites. Track 
their usage of categories versus search. Measure whether they actually get 
to the content they were seeking. If your data matches ours, you'll see 
about 30% with Search and about 53% without Search.

> > As we do more testing, we'll have more results, from which we'll
> > draw more inferences and opinions. We'll continue to present the
> > data so that everyone can draw their own inferences and opinions.
>
>Does that mean you will publish details of the studies behind the
>articles "Why Amazon Succeeds -- And Why It Won't Help You" and
>"Why On-Site Searching Stinks"?

It is available. Contact me for pricing. (I'm sorry, but it is not 
economically feasible for us to make it available for free. We invest more 
than $1 million annually in our research. We need to recoup our expenses 
and continue to fund future research.)

Remember, if we haven't satisfied you that what we're saying is accurate or 
believable, simply don't believe it. I can attest that the Sun will 
continue to rise in the east every day, whether you agree with our findings 
or not.

Our clients are happy with our results. They are making informed decisions 
using our data. And they keep asking for more research. We must be doing 
something right.

Jared

Jared M. Spool
User Interface Engineering
http://www.uie.com    jspool at uie.com

Don't miss User Interface 7 West, March 23-27, Burlingame, CA. 
http://www.uiconf.com