[Sigia-l] ROI/Value of Search Engine Design - Resources?

[Boniface Lau]

> From: sigia-l-admin at asis.org [mailto:sigia-l-admin at asis.org]On
> Behalf Of Jared M. Spool
> 
> > > Sure, 'laptops' works as a query. It didn't work for our user
> > > because they didn't get what they needed.
> >
> >As my above reply indicated, just because some people did a search
> >and not getting what they needed is NOT necessary a problem of the
> >search engine or the site. It could very well be a problem of the
> >users.
> 
> Maybe at this point we can agree to disagree.  I have drawn my
> inferences from our data, you can draw different inferences. That's
> how it's supposed to work.

Nonetheless, I thank you for trying to explain your perspective.


> 
> I come from a school that there isn't ever a problem with users --
> it's always a problem with the design. 

Extreme statement like that is good for generating debates. I can even
see academic holding on to such belief. But seeing it coming from
someone who is supposed to have industry experience is a bit
surprising, to say the least. Are you making the above statement just
to be provocative?

In theory, one can come up with a design solution for every possible
user problems. But are you aware that individual solutions when
combined together can become conflicting forces within a system?
Combining the solutions for every possible user problems will create
an un-usable mess, even if cost is not an issue. Of course, when no
one uses the mess, it has no problem with users.

So, what do people use? They use systems which do not pretend that
there is no problem with users. People do their part by learning the
system specifics to avoid causing problems.


> I believe that there is no such thing as 'user error' -- I believe
> that there always is a design that could've prevented such errors

How could you prevent something without acknowledging its existence?
By saying that there is a way to prevent 'user error', you are
implying that there is indeed "such thing as 'user error'".

'User error' is a reminder of the reality of being human. As long as
there are humans, there will be user errors.


> and it's part of our research agenda to understand what such a
> design would entail.
> 
> If you believe that the problem was that our user was somehow
> incapable of or shouldn't be using Amazon,

"Users having problems" does not mean they are incapable of or should
not use the system. Users learn from experience.


> I can accept that. I have a different opinion.
> 
> It is possible that the user selection process we use in our
> research is finding the wrong users. It is possible that the tasks
> we've asked users to perform are the wrong tasks. It is possible
> that our test method and our measurement techniques are somehow
> faulty. 

It is also possible that the data have been mis-interpreted. Care to
publish details of the study behind the UIE article "Why Amazon
Succeeds -- And Why It Won't Help You"
(http://www.uie.com/Articles/why_amazon_succeeds.htm)?

Regarding the UIE article "Why On-Site Searching Stinks"
(http://www.uie.com/searchar.htm), I am sure many people would like to
see the details you have for coming to the following extreme
conclusion:

WOSSS> Our data showed that today's on-site search engines are worse
WOSSS> than nothing -- significantly worse.

Will you publish the study details to support the above extreme
statement?


> I question each of these things regularly and we're constantly
> making improvements.
> 
> But, in this case, to the best of my knowledge and our research
> abilities, I believe this user was the right user and their behavior
> was indicative of what we often see when trying to perform similar
> tasks.

If you are so convinced, then may be the problem is indeed with 
data interpretation. 

For example, the UIE article "Why On-Site Searching Stinks" presented
user interface problems and then claimed that "on-site search engines
are worse than nothing". To me, that was data mis-interpretation.

Would you mind explaining the logic behind the statement "on-site
search engines are worse than nothing"?


> 
> As we do more testing, we'll have more results, from which we'll
> draw more inferences and opinions. We'll continue to present the
> data so that everyone can draw their own inferences and opinions.

Does that mean you will publish details of the studies behind the
articles "Why Amazon Succeeds -- And Why It Won't Help You" and 
"Why On-Site Searching Stinks"?


> Then we'll have more healthy debate,

I look forward to the healthy debate triggered by your publication of
the above mentioned study details.


> which will help us refine our research and move our knowledge
> forward. That's how it is supposed to work.
> 
> Thanks for encouraging our behavior.

Extreme statements get people's attention and are therefore quite a
marketing tool. But making extreme statements without the supporting
details often undermines people's confidence in the person's
judgement.


Boniface