[Sigia-l] Usability testing into the dustbin?

[Mitchell Gass]

At 04:03 PM 5/1/2003 -0400, Listera forwarded from Usability News:
>...usability testing now appears to be a highly variable art in which the 
>results depend on who is testing what by which protocol with which 
>particular subjects. It is quite possible that for some systems being 
>evaluated by some procedures, no matter how many subjects you test, you 
>will continue to uncover new and significant problems...

When something better than current diagnostic usability testing comes 
along, I hope that all design teams embrace it. But what should we do until 
then? How many of you who are currently conducting usability tests think 
you should abandon them?

When diagnostic usability tests find problems, and the value of solving the 
problems outweighs the costs of conducting the tests and making the 
changes, the tests are worth doing. Should we improve testing methods to 
make them more reliable and efficient? Yes. Should we work harder to 
understand how much testing is optimal? Of course. Should we work with 
users earlier in the design process and use what we learn to prevent 
problems? Without question. But the fact that usability testing methods 
aren't perfect doesn't mean they're not valuable.

Diagnostic usability testing is not science. It is a practical, 
cost-effective technique for improving designs. In my own work, I've found 
that Nielsen's recommendations for the numbers of test participants you need at

   http://www.useit.com/alertbox/20000319.html

to be very good. The important messages are

- If your user population includes highly distinct groups of users, you 
need more participants.

- If each test participant can't attempt all of the important tasks that a 
product is designed to support in a single test session, you need either 
more participants or more or longer test sessions with the same participants.

- Iterative testing is the most efficient way to test. The goal is to find 
problems; bring in too many participants and you waste your time seeing the 
same problems again and again. Test quickly, address the problems you find, 
and test again.

You can't test Yahoo! with five participants, and the recommendations make 
that clear.

Mitchell Gass
uLab | PDA: Learning from Users | Designing with Users
Berkeley, CA 94707 USA
+1 510 525-6864 voice
+1 510 525-4246 fax
http://www.participatorydesign.com/