[Sigia-l] Usability Testing comments from Giga

[Lyle_Kantrovich at cargill.com]

Too many people here try to argue for black or white, yet all I see is 
grey.

Product Design (and hence IA), Medicine, Nautical Navigation, and 
Artillery are inexact practices.  Some may be more scientific than 
others, but still there are many variables to consider in being 
effective in these inexact fields.

Usability testing (as Boniface said) is about risk management.  Test 
plans should be driven by risk factors, business goals, user goals, and 
past experience.  How you test a small workgroup intranet web site is 
very different from the way you test a medical device (like a 
resuscitator) or a NASA shuttle navigation system.  In reality, you 
can't test everything - anyone who's done much testing knows that you 
set priorities and create a test plan around the most important areas 
of the product based on highest risk and most important goals.  (e.g. 
if a user can't open the box, then they can't read the instruction 
manual or actually use the product - you may need high confidence that 
the box is easy enough to open.  Working on an online product?  Think 
'login' instead of opening boxes.)

To Marios' question about what is a good test for an Information 
Architecture, I'd point out that while HCI is focused primarily on 
computer-based products, "usability" and User-Centered Design (UCD) are 
not.  I know usability folks who test many other kinds of products with 
basically the same kinds of methods I use for testing computer 
applications.  Check out http://jthom.best.vwh.net/usability/general.htm

It seems obvious, but usability testing is primarily meant to assess 
the USABILITY of a product.  There are lots of other kinds of testing 
that can and should be done on different products.  Hopefully 
Information Architects want their output/products to be usable - hence 
usability tests should be very relevant to practicing IAs.  It's also 
worth noting that usability testing is only one method of evaluating 
the usability of a product.  See: 
http://www.userdesign.com/usability_uem.html

There are lots of variables that can change over time: user experience 
level, technology (OS, browser, etc.), language and semantics.  Hence 
most products will benefit from testing - if nothing else, you validate 
that what you "know" is still accurate and timely.

A product designer who doesn't test their product is like an artillery 
crew without a spotter.  Unless someone can tell them where the round 
from their gun falls, they don't know if they've hit their target or 
not, and furthermore, they don't know if or how to adjust their aim.  
They don't know if they need to take another shot at it.  Artillery is 
very mathematical and 'scientific', yet they also lots of variables to 
contend with.  Try this short article for an example.  
http://www.kdhnews.com/military_art302.html  

IA is much less exact and scientific than medicine or artillery.  
Without testing how do you know if you're hitting the mark or how to 
improve your aim?

Rolf Molich reminded me at CHI 2002 that there's a difference between 
your opinion (theory) and what you've proven/tested within reason 
(theory with direct evidence to support it).  Unless your theories 
about your product's usability are grounded in some type of real 
feedback from users actually using it, you're probably not standing on 
very solid ground.  Usability testing is one form of feedback - no 
more, no less.  How do you get your feedback if not through usability 
testing?

Regards,

Lyle

----
Lyle Kantrovich
User Experience Architect
Cargill 
http://www.cargill.com 

Croc O' Lyle: personal commentary on usability, Information 
Architecture, and web design
http://crocolyle.blogspot.com

"Simplicity is the ultimate sophistication." 
	- Leonardo da Vinci


-----Original Message-----
From: listera at rcn.com [mailto:listera at rcn.com]
Sent: Sunday, March 30, 2003 11:03 PM
To: sigia-l at asis.org
Subject: Re: [Sigia-l] Usability Testing comments from Giga


"Jared M. Spool" wrote:

> While an experienced navigator can trust his gut to say that the ship 
is on
> course, having instruments to verify position, speed, and conditions 
gives
> everyone on board the confidence needed.

Speaking of one of my favorite subjects, for millennia perhaps the most
significant barrier to the discovery of our planet, the oceans, the
continents, etc., was not ignorance but the delusion of knowledge. The
Church gave us the 'ideal' map, the working seamen gave us portolanos,
harbor guides. Guess which one you'd consult if you were a captain of a
caravel off the cost of East Africa? I'm decidedly not against testing.
However...

> This is all about confidence -- trusting that we have the underlying
> knowledge, practices, and methodologies to produce reliably effective
> results every time.

...IA is not a profession where every result can be reliably and 
predictably
'engineered' from a codified set of rules. I don't think that's even 
needed.
In an incredibly fast moving platform of technology and development, we 
(try
to) deliver efficient (and hopefully enjoyable) results.

Furthermore, I predict IA will never attain the 'scientific' status some
hope to achieve. IA is just as much a craft as medicine has been and is
today. As I provided earlier, we've come from days of autopsies being 
done
by barbers to fantastic technological medical achievements. Medicine is 
one
of the most highly regulated, licensed and tested professions around. 
And
yet 50% of what doctors know today are invalid. So white lab coats are 
not
enough and to the extent that they may obscure the forest for the trees 
they
may even be detrimental in certain ways.

> Ziya asked an important question: does every decision need to be 
tested.

The context I uttered what you paraphrased above was in reaction to an
assertion that someone would test what he was sure of, as well as what 
he
was not. That's simply not attainable in the commercial world: there 
just
isn't the money, resources, time and personnel for that. You'd have to 
be in
the testing business to advocate that :-) That's why I gave the example 
of a
doctor not testing everything during his practice, otherwise he'd have 
no
practice in short order.

> Today, our confidence is weak.

I don't find that.

To me, the basic tenets of IA have been in practice for centuries. The 
use
of the title may have gained currency with the birth of the web in 
general,
but as I pointed out previously, what Mercator did with cartography or 
what
Ray and Linnaeus did with the taxonomy of species was nothing less than 
IA.
If you take a historical view, I think there's less reason for panic or
gloom. :-)

Ziya
Nullius in Verba 


------------
When replying, please *trim your post* as much as possible.
*Plain text, please; NO Attachments

ASIST IA 03 Summit: Making Connections
http://www.asist-events.org/IASummit2003/

Searchable list archive:   http://www.info-arch.org/lists/sigia-l/
________________________________________
Sigia-l mailing list -- post to: Sigia-l at asis.org
Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l