[Sigia-l] Automated User Testing

[Lyle Kantrovich]

Michael,

Thanks for clarifying, although you didn't mention why you want/need
to test MORE users.  Can you talk about that a bit?    My assumption
is that you are doing formative, not summative testing.  Summative
testing usually requires more users...and is therefore done much more
infrequently.

I'm happy that your just trying to augment your "toolbox" rather than
replace it.  Frankly, I'm glad folks like you are around, pushing the
envelope.  I'm happy to let others try new things - if it works, I'll
be right behind you.  Personally, I don't think "automated" testing is
likely to work, so I'll not be on the bleeding edge of this
exploration.  Here's a bit of my rationale:

Observation requires an observer...of some type.  That's a basic
premise of most usability testing.  Even with remote u-tests, there is
an observer who usually can observe what's going on "on the screen"
and listen/probe for what the user is thinking/feeling (what you
called intent).

So-called "automated" methods try (almost by definition) to remove the
observer (or replace them with a non-human "observer").  They then
often will try to augment the data-gathering with things like
questionnaires throughout or after a task to probe for more info
(Vividence did this last I checked).  Doing that relies on users
self-reporting - which has inherent flaws.  So while you end up with
data from 100 users instead of 8, you get less rich data, and likely
only around things you thought to ask about ahead of time.  With
"live" testing, the facilitator (usually) can go off-script and delve
into surprising "ah-ha" kinds of observations.  Often we learn about
things we wouldn't have thought to ask about.  Would you like a
mountain of superficial comments to sift through, or eight rich user
sessions to glean many different kinds of insights from?

I think the reason we don't have (and probably never will have)
automated usability testing tools is that people don't come with a
brain implant that allows you to export what they are thinking and
feeling.  (That's something one of my ex-girlfriends used to wish for,
but that's a whole other story.  :)

Vividence (now Keynote) and Netraker were two companies working on
this kind of tool in the past.  There was another that I saw at CHI
2001 (called iSomething...maybe iRise?) that went belly-up last I
checked.  Keynote (in my opinion) is an engineering driven company,
and is likely to go more toward IT kinds of solutions vs. improving
the psychology and social science aspects of their tool.  (I hope I'm
wrong on that though.) I've looked at these vendors in the past, but
like you, determined that they just aren't worth what they cost. 
"Automated" does not mean "cheaper."

Like imitation vanilla flavoring, "automated" usability testing just
isn't as good as the real thing.

Lyle

--------------------------
Lyle Kantrovich
Blog: Croc O' Lyle
http://crocolyle.blogspot.com