[Sigia-l] Questioning common test scripting

[InfoArchitect]

Jeff wrote:
To paraphrase Jared Spool, of course usability tests are biased! We're
bringing someone into a strange room, giving them money, watching them
behind a one way mirror, videotaping their every move, watching over 
their shoulder, having them use a computer they haven't ever used, 
telling them to use a system that they might not ever use, asking them 
to do things on the system that they might not ever do... that sounds 
pretty biased to me!

Again, it depends on the type of test conducted and the goal of the 
test.  If I want to analyse someone's natural interactions with a 
product (especially where environmental factors and equipment set-up 
are key to the user, such as testing products with users that have 
physical impairments), there's no way I will perform a Think Aloud 
Protocol test in a lab.  A field study of scripted tasks 
(where 'observation' is the order of the day) is more appropriate.

If I wanted to collect data for metrics or to validate a GOMS 
analysis, I wouldn't want to add to the cognitive load of the 
participant by having them verbalise what they were doing (which often 
leads to not only slower performance times of incalculable variance, 
but more contrived actions).  However, tests with such goals could be 
performed on standard equipment in a lab with negligible impact on the 
results (as I would be comparing the data collected at one point in 
time to data collected at another point in time under the exact same 
conditions).

If I wanted to know what users thought of a product, I'd happily take 
two participants, give them a task and leave them in a lab for a Co-
Discovery Method test - listening to what they had to say to each 
other.

The testing of an obviously incomplete interface or system does not 
require a natural setting or context to produce the high-level results 
it is usually aimed at achieving.  So, if I wanted a cheap way of 
getting a lot of good qualitative feedback during the early stages of 
a design, then the Think Aloud Protocol would be an attractive 
option.  

Best regards,

Ash Donaldson
User Experience Designer

----------------
Get your free email address from www.ourbrisbane.com