[Sigia-l] User Test Cost - Does this sound reasonable?

[Jared M. Spool]

At 10:30 AM 5/21/2004 -0400, Knyal, Jarrett wrote:
>What we're getting:
>- recruiting of 10 users
>- two days of a room at their facility, equipment supplied including 
>videotaping
>- summary report based on videotapes of sessions
>
>We are coming up with the test plans and will moderate the sessions.
>
>Their charge for this is 7K.  Does this sound reasonable?

I think you're getting quite a bargain at $700 per user. Our most recent 
project, the per-user cost was $8,750. But, we deliver a lot more for each 
study. (A *lot* more!)

If these guys are going to produce the results you want, go for it. 
However, if they aren't, you've just wasted $7,000, plus *your* time (which 
is probably worth a lot more), and, possibly, will make some bad decisions 
in your design based on poor information.

How will you know if these guys do a good job for you?

Interestingly, when we coach our clients on conducting tests, we actually 
have these tips:

1) Do your own recruiting. The recruiting process is *very* educational. 
You learn a lot about your users when you talk to them. More importantly, 
you learn a lot about people who you don't want as users. That's just as 
educational.

Plus, with some simple tricks, you end up with better qualified users who 
are more reliable. User-hours are the most expensive part of the project. 
Wasted user-hours, because of no-shows or poorly qualified users, throw 
away a lot of money.

2) Don't use a lab. Conference room testing is more effective, less 
expensive, and can get the development team closer to the users. There is 
no evidence that labs are anything more than a waste of nice furniture and 
glass.

We try to get the testing as close to the development team as possible. The 
nearest conference room is best. Been doing it this way for years and it 
works great.

For recording, you can use Camtasia, which works wonderfully. If you really 
want video, any consumer-grade video camera and tripod will do the trick. 
(We're currently experimenting with Morae. After some initial setup 
headaches and bugs, we've got it working and it's growing on us. But, 
unless you need the sophisticated logging features for later analysis, 
Camtasia, which is 1/10 the cost, is almost as good.)

3) Avoid reports. We have a rule of thumb in our work: If the team reads 
anything in our report that they didn't *already* know, we've failed.

The team should know everything that happens in the test long before any 
report tells them. Reports are good for archives and the few people who 
can't be at the test for really good reasons, like they are off having a 
baby or they weren't hired yet. Other than that, they should get the 
results directly from the tests themselves or from quick summaries and 
email discussions that follow each test session.

A meeting to discuss testing results within 24 hours of the last test will 
trump the benefits of any report, any day.

Given this, what are you paying your $7,000 for? Stuff that you should be 
avoiding...

I'm wondering if it is *really* what you want to do.

Of course, what I've just written is extremely controversial and most 
people will probably disagree. But, I'm practicing to be a crumudgeon (or 
maybe I should say "guru") and it's been a long week of hard hours, so 
maybe I'm really just a bit cranky.

Either way, my opinions are worth what you paid for them. :) If, on the off 
chance you found these comments intriguing and not just irritating, we go 
into a lot more detail on these (and many other) techniques in our UIE 
Roadshow. (Look at the Day 2 program at http://www.uie.com/events/roadshow 
for more details.)

Jared


Jared M. Spool        User Interface Engineering
http://www.uie.com    jspool at uie.com

Join us for UIE's extremely popular Roadshow event
UIE Advanced Techniques: http://www.uie.com/events/roadshow