[Sigia-l] Questioning common test scripting

[Jeff Lash]

In Listera's 7th post in this thread, "Listera" wrote:

> Your intentions or even your delivery are unimportant here.
> The fact remains that by declaring the design faulty you've
> given them implicit/explicit permission and, perhaps,
> even encouragement to find faults.

That's exactly the point of usability testing -- to uncover issues in a
system that may impact usability. As someone mentioned earlier, many
evaluators don't want to mention things that annoy them, due to the
"politeness" factor, so they need encouragement.

In a usability test, evaluators verbalize issues that they would encounter
using the software on their own. They hopefully speak aloud all their
observations, whether it's a small issue or a huge one. Yes, maybe we're
encouraging them to mention things that normally wouldn't annoy them, but
it's up to the usability specialist facilitating the test to know what is a
minor annoyance and what is a major problem. You don't just take everything
that an evaluator says as gospel and follow it to a T (hopefully).

> Like it or not, you are now testing the user (as opposed to or
> in addition to the product/design).

No, if we were comparing the issues that they found to the issues that we
know existed, then we'd be testing them. Many times, early on in a usability
test, people will ask, "Was that good? Did I get everything?" I nicely
assure them that they are doing a great job, thinking aloud, just doing what
they would would normally do.

No matter what you say at the beginning of the test, evaluators will:
- ask if they're doing it right
- say that they did something stupid and it's their fault
- say that something was easy to do when they really had problems doing it

> Now, this may be exactly what you want, but let's not call it neutral,
> inert, nonbiased or 'scientific'.

Who said it was "scientific"? For someone who's always railing on about
white lab coats, I don't know why you would care?

To paraphrase Jared Spool, of course usability tests are biased! We're
bringing someone into a strange room, giving them money, watching them
behind a one way mirror, videotaping their every move, watching over their
shoulder, having them use a computer they haven't ever used, telling them to
use a system that they might not ever use, asking them to do things on the
system that they might not ever do... that sounds pretty biased to me!

You can harp on the details of biased/nonbiased, or you can accept the fact
that there are biases introduced and focus on the real goal -- making the
system easier to use. I know that saying "We're testing the system, not you"
does infinitesimally more good towards reaching that goal (by getting people
to talk out loud) than it works against it (by a long chain of reverse
psychology that says that we really are testing them).

Jeff
--
http://jefflash.com