[Sigia-l] Summary: card sort order effects

[Martin, Chris C.]

The following is a list of all the responses I got to my initial e-mail
which was

> Hello all,
>  
> I'm involved with a usability test and card sorting exercise. 
> For both we
> have to use the same set of nine participants, so we have to 
> decide whether
> to do the card sort before or after the usability test.
>  
> We know of one case where card sorting was done after 
> usability testing.  In
> that case, the participants did not sort the cards in a manner that
> reflected the current site structure. This could mean, however, that
> participants felt under pressure to improve the site structure.
>  
> Apart from one source that says card sorting should be done before
> participants have seen the site at all, we haven't found any 
> guidelines or
> research on this issue. We realize that some order effects 
> are going to bias
> our results regardless. But if anyone here has experience 
> with this issue
> and found that a particular order works better, please e-mail me. I'll
> summarize the results and post them to the list.
>  
> Thanks,
> Chris

---

There's a fine line between biasing results and confounding them.  If these
tests are related 
(as is inferred), you would be stepping over that line as if shot from a
cannon.  The very 
basis of reliable testing is participant selection – get it wrong, and
the results will be 
invalid and most likely incorrect, which can cost the business dearly in the
long run.

I realise business and time constraints probably apply here, but it would be
better to produce 
reliable results in one test, than to get misleading results that would
destroy the basis of 
project decisions in another.

Just my two cents.

Best regards,

Ash Donaldson
User Experience Designer 
infoarchitect at ourbrisbane.com
---

Hi Chris,

In my view card sorting is part of the user research phase possibly after 
the first few site maps have been drawn up but before interactive 
prototypes have been created. I presume this is what you meant by the 
testing of the existing site, that is researching how the new site should 
be by asking users what they felt of the existing site. Have you also 
considered using similar sites to the existing sites - that is using 
competitor sites during the research phase?  I have done this kind of 
research before drawing up an initial IA that,in turn, could be used as a 
basis of card sorting (how you card sort is another whole argument).

So in short I would do this after the initial user research and capturing 
of user requirements of which the exploring of existing sites is part of.

Regards

Stewart Dean
User Experience Freelance
Stewart Dean [stewart at webslave.dircon.co.uk]

---

I don't understand what you are working on that you would be both card 
sorting and usability testing at once (assuming that when you say 'at once' 
you mean on the same day not a few months apart.)

It sounds vaguely like you have an existing site and you are trying to 
determine whether the site structure is usable. If this is the case, I have 
found that usability testing (eg. traditional at computer, scenario-based, 
speak aloud testing) is not a great way to test site structure. In many
sites 
there are so many information-retrieval scenarios that it is difficult to
test 
them in this type of usability test. I have, however, found that this type
of 
usability test is great for testing that participants can get around the
site 
well, and that the screen elements are usable.

So, assuming this is the right scenario, I'm not sure what you would gain 
out of card sorting at the same time. Are you giving participants content 
that is on the existing site, and asking them to sort it into groups and
label 
them?

Again, assuming I have your scenario right:
- if you do it before testing, people will have their own groupings and
labels 
in mind which will impact on how they navigate the site. If their groupings 
are different to yours they are likely to feel inadequate - after all, they
will be 
thinking of you as at least some type of expert and will assume that your 
structure is right, which means their grouping must be wrong.
- if you do it after the usability test (not forgetting that they will be at
least a 
little tired), they will be either influenced by the structure and labelling
that 
they have just used and will reflect it, or assume that you are asking for 
improvements so will try not to do what the site already does.

Either way, I can't imagine that the results will be useful. The bias will
be 
too strong.

If I absolutely had to do something like this, I'd somehow make sure that 
my usability test scenarios and my content in the card sort did not overlap 
at all. Perhaps you can tell them that you only want to test a small part of

the site, and get them to organise content for the other part. And I'd still

split them into 2 groups, change the sequence and watch for impacts (but 
with such small groups it will be difficult to detect anyway)

Hope this makes sense. I may have completely misinterpreted ;)

Donna
donna at maadmob.net
---
You might even split the participants into three groups: 1 group that
sorts before the usability test, a second group that sorts after the
usability test and a third group that sorts both before and after the
usability test.

With the third group, you could determine whether using the site had any
influence on their mental map--something you wouldn't get from just
having only the first two groups. Of course, you risk participants
trying to re-create their original structure.

-Rick
Rick Cecil [rick at hesketh.com]

---
chris,
i find card sorts super useful to reveal alternate versions of a scheme -
not duplication of what i've proposed. what i'm looking for is alternative
interpretations of organization & nomenclature, to help anticipate use cases
and possible errors. so i do my card sorts before to factor out any
bias/learning from the test p-type.

if you do the card sort after, consider positioning it as a design
exercise...build it the way "you" want to see it based on your experience of
the test p-type.  compare these card sort results with the those conducted
before the test.
susan  SusanC at ZAAZ.com

---
I would mix it up. For half of the participants do the card sorting first.
For the second half do it last. That way you'll "even out" any type of
bias.

-Chiara chiara at chiarafox.com