[Sigia-l] Questioning common test scripting

[Listera]

"InfoArchitect" wrote:

> "Today we'll be looking at a new design for <'x' type of product>.  Of
> course, there's going to be some flaws in that design, so the purpose
> of today’s exercise is to find any areas that may cause confusion or
> difficulty for people."

So you have just pre-conditioned them:

a) You told them that the design *is* (of course) flawed. You cannot escape
the conscious or subconscious inference in the participant's mind that they
*must* find faults (unless they are shown to be retarded).

b) You have taken away their innate tendency to question themselves when
faced with a task (to find a better/more efficient way to accomplish it)
since you just urged them to question the design (for confusion and
difficulty) before anything else.

Your intentions or even your delivery are unimportant here. The fact remains
that by declaring the design faulty you've given them implicit/explicit
permission and, perhaps, even encouragement to find faults.

Like it or not, you are now testing the user (as opposed to or in addition
to the product/design). This is somewhat equivalent to: "Is this a trick
question?"

Now, this may be exactly what you want, but let's not call it neutral,
inert, nonbiased or 'scientific'.

> To enforce scientific rigor, we try not to introduce extraneous variables or
> confounding factors in any methods, even subjective ones (such as Likert-scale
> questionnaires), thus the origin of this thread.

If this were true, I might entertain the notion of believing it. See the
variables/problems introduced by the pre-conditioning above.

> Fair enough, it's a fledgling industry and these graphic designers and
> business analysts will eventually find their way to the cognitive or library
> sciences.  

Having studied both design and engineering, I'm continually baffled by the
utter lack of knowledge regarding (formal) design education by those who
equate "Skip Intro" concoctors with designers. There are also plenty of
voodoo research/studies by people wearing white coats.

> To borrow one of Ziya's metaphors, it's like having your best friend operate
> on your Mother because he's a personal trainer that has read a couple of books
> on surgery, and then claim the surgical methods don't work because she died
> during the operation. ;)

He may be a quack but, ultimately, you are the one who foolishly allowed him
to operate on your Mother! Not nice :-)

Ziya
Nullius in Verba