[Sigia-l] SUMMARY: Any help on EZ sort working with over 100 cards?

[Chris Rourke]

A long overdue summary of a question I posed a few months ago on the
problem of analysing the results of very large card sort studies.
Apologies for the delay - the good news was that a solution was devised
in the end, and some good learnings to share.  Original question and
responses are below:

___________ORIGINAL QUESTION_________
We are doing some card sorting work for a client involving over 100
cards.  This  is by far the most we have had to sort in any session (the
client actually had over 500 topics they wanted to include!). Anyway,
the problem is that IBM EZ sort (consisting of Usort and EZCalc) only
seems to handle 100 cards maximum.  If I try to run the cluster analysis
& create a tree diagram in EZ Calc, it crashes.  (anyone reading this
from IBM - maybe something to work on for the next Beta release 1.4?)

Are there any programs out there that allow over 100 topics in the
cluster analysis, ideally leading to a tree diagram like the EZ sort?  I
know there are several others out there (WebSort, Web Cat, CardZort) -
do any of these do the trick? If you want to send your answers directly
to me, I can summarise for the list.

____________ SOLUTION / ANSWER _________

The easiest solution: Don't test with over 100 cards.  It was quite
exhausting for the subject and at that number there is a real risk of
weaker results just through the difficulty of people trying to get their
heads around 100 + topics.  

The solution that we arrived ended up going back to EZ Sort, and one of
its developers Jianming Dong who changed the settings to allow over 100
topics and produced the usual, very useful tree diagram and other
analysis.  I don't think Jianming wants to make a regular practice out
of it though.

Another solution was offered by Joe Lamantia in a method recently
presented on Boxes and arrows.  It involved a spreadsheet and a bit of
preliminary setting up, but it is well worth it.  It does not give a
tree diagram although the other analysis output are very useful, in a
more quantitative method.  A very good methods we will consider for the
future.   Have a look at the article on this:
http://www.boxesandarrows.com/archives/analyzing_card_sort_results_with_
a_spreadsheet_template.php

We also could have done this with Jorge Toro's useful CardZort.  It does
handle > 100, and gives a good output.  However with so many cards it
means the cards start off as a random stack on the computer screen, and
replicating the groups (done with real physical cards) was quite time
consuming, eventually abandoned.

Other individual solutions offered are mentioned below

___________RESPONSES__________

Respondent 1
This might not be the answer you're looking for, but typically, we do 
this by hand in excel. Not as efficient as USort and EZCalc, but it 
does work.

Cheers!

Todd R. Warfel
_//message first [method second]
-=========================-
User Experience Architect
message first
[P] (607) 339-9640
[E] twarfel at messagefirst.com
[w] http://messagefirst.com
-=========================-
Respondent 2
It has been a while since I have done any card sorting, but some of the
high end statistics packages out there will do cluster analysis over
large sets.  I have never had access to any of them when I needed to
work with a large set, so I don't have any recommendations.  I've always
just done it in Excel, which is time consuming, but does the job.
The bigger problem that I have encountered is that a pile of more than
about 70 cards seems to be too much for test subjects.  The subjects
either don't want to do the test (once they see the card deck), or wear
out part way through the task, or speed through the task so fast that
the results aren't reliable.  I always found it necessary to break
something like this down into smaller tasks. I did multiple rounds
(i.e., find the top layers of navigation and do separate testing to
refine the lower layers).  Also, since I always used card sorts as one
of a several tests in developing site navigation, I would be refining
navigation to cover those other situations as I went through the rest of
the protocol.
Dale Mead



Chris Rourke
Director
User Vision - Focusing on the user experience
http://www.uservision.co.uk