[Sigia-l] ROI/Value of Search Engine Design - Resources?

[Jared M. Spool]

scottmcd at cognetics.com wrote:

>I could easily see the genre being a significant factor in a person's 
>general approach to finding things.  Likewise with a web site's 
>design.  Are you able to tease apart the effect of genre from the effect 
>of design in your research?
>
>Does a person's mental model of something include assumptions or 
>perceptions that lead them toward finding it in a particular way (search 
>or browse)?

Very astute! Excellent questions.

To answer the first question: When working with correlations (which is the 
basis of our resource), it's very hard to separate the effect of genre from 
the effect of design because sites within a genre tend to be designed very 
similarly. For example, a site displaying medical information is far more 
likely to have similar design features to the other sites displaying 
medical information than sites displaying office supplies. Even within 
e-commerce, sites displaying one type of product often share similar design 
qualities to the other sites within that product genre.

Given that the design attributes travel with the genres, it's hard to say 
that something is specifically the effect of the genre or the effect of the 
attributes of the designs.

Having said that, let me address the second question:

At the IA Summit a few years back, I talked about a recent study we'd done 
where we had collected the product descriptions for items users had shopped 
for in our studies. We placed them on forms where we asked several hundred 
survey respondents to tell us which major categories they'd expect to find 
these products.

For example, one shopper had looked for 'Iguana Food'. We'd asked the 
respondents to tell us where, on a pet supply site, they'd expect to find 
'Iguana Food'. We didn't give them any description of the site or any 
preliminary categories to start with -- just a blank box to write in 
whatever category came to mind.

When we compared the answers, we found that for some categories of 
products, such as apparel or pet supplies, the respondents were generally 
in agreement as to where they'd find the products. In our 'Iguana Food' 
example, 75% of respondents said either 'Reptile' (53%) or 'Food' (22%) as 
the top-level category.

For other categories, such as computer accessories, there was no such 
agreement. Users were all over the board as to where they would expect to 
find the various products, often with no single category getting more than 
20% agreement.

We found that, in our user studies, the users were far less likely to use 
Search for those genres where we saw lots of agreement in our survey. There 
was a strong negative correlation between Search use in the user studies 
and the agreement in the surveys.

Now, the best performing sites in the genres where we had strong agreement 
happened to always use the same terms our users did. (The poor performing 
sites often used marketing jargon like "lifestyles" or "jeanology" for 
major categories -- terms users rarely understood.)

In the genres where we didn't have strong agreement, the sites that did the 
best were those that had lots of categories on the home page, much like you 
see on http://www.Staples.com today.

So, yes, we do see a combination of site design effect and the user's 
knowledge of the genre playing a role in the strategy that users take.

Jared


Jared M. Spool
User Interface Engineering
http://www.uie.com    jspool at uie.com

Don't miss User Interface 7 West, March 23-27, Burlingame, CA. 
http://www.uiconf.com