No subject

Usability can sometimes be more about belief than about evidence or
engineering, with usability testing heading the list as a central tenet of
the dogma of modern practice.

Far from being a scientific and reproducible procedure, as it is touted by
many professionals and regarded by many managers, usability testing now
appears to be a highly variable art in which the results depend on who is
testing what by which protocol with which particular subjects. It is quite
possible that for some systems being evaluated by some procedures, no matter
how many subjects you test, you will continue to uncover new and significant
problems. The problems you find will be different from those you would find
with other users and different from those that another tester would uncover
with the same number of users.

One major conclusion from the panel and the conference might be that
usability testing is so entrenched in the canon of usability practice that
no amount of counter-evidence will shake the faith of its true believers. An
unbiased reading of the research results would suggest that no amount of
testing is enough, a conclusion already well-established a quarter century
ago within the software quality movement. The focus of software quality
improvement is now on reducing the so-called injection rate, that is,
avoiding problems in the first place. There is a limit to how many defects
can be uncovered, cataloged, and analysed in a given number of sessions no
matter what protocols one follows or how rapidly one iterates. The more
problems lurking in the system to be tested, the more hopeless the fate of
those who put their faith in testing.

Among the panellists, only Rolf Molich took that next logical step to
question the very role of usability testing. He shared a hopeful vision of
the future in which robust and disciplined design processes avoided most
usability problems from the outset. In his vision, usability testing
facilities would gradually fall into disuse and ultimately be abandoned to
gather dust. 

Usability News - CHI 2003 Feature: Testing... 1 2 3 4 5 ... Testing...
<http://www.usabilitynews.com/news/article1058.asp>

Ziya
Nullius in Verba