[Sigia-l] Usability Testing

[Will Parker]

On Jan 20, 2007, at 9:56 PM, Listera wrote:
> Bill Killam:
>
>> No, the person who asked the question is the one who framed it,  
>> not you.
>> And he asked: "is there an advantage to using an external vendor".
>
> And the answer is, no. The advantage is in INTEGRATING it into  
> design, not
> debating if external or internal is better. Trees. Forest.

My industry experience regarding usability testing practices is  
admittedly limited, but I've sat in on several dozen usability tests  
and related meetings as a stakeholder. I've also been the liaison  
between internal management and external UxD firms, and up to my  
elbows in the decision-making process around the findings of the  
usability teams.

It's fairly obvious that usability people need to educate the people  
who use their services, whether those are co-workers or external  
clients. The question is whether the usability team comes with a "How  
to Use Your New Usability Team" manual, and whether the client/user  
bothers to read that manual. In my opinion, the real and unalterable  
key to whether *any* usability testing is effective is the attitude  
and ability of the product management, not the usability team.

I've seen PMs who literally move into the usability office during  
usability tests. These folks tend to do the hard work of triaging the  
proposed changes, modifying the product as far as their schedule and  
budget allows, and where more is required than they can force into  
the current version, they fight for changes in the future product  
roadmap.

In short, they act _very_ much as if they were the owner of the  
usability team, not the product team.

I've seen other PMs who treat 'usability testing' as a checkbox  
requirement. "OK, we've done that." Insert one of:

  - Nothing found we {need, want, can afford} to change for this  
release. We'll consider this data for the product roadmap.
  - OK, I've filed N usability bugs. The devs will have solutions for  
those by next week.
  - Thanks. Marketing will use this as input for further research.
  - The (IT managers of ) our five largest corporate customers are  
absolutely opposed to the proposed changes.

I've seen both types of managers working with the same internal and  
external usability teams. I've seen the resulting changes, and I've  
been been at least partially responsible for seeing that the changes  
work as desired.

In my opinion and experience, if the people responsible for pushing  
the product development see usability testing as **their own valuable  
tool**, even a mediocre usability team can be a benefit. If product  
management hasn't a clue about the value of usability, they're likely  
to ignore the clearest of red flags, even if it's being waved  
frantically by the chief usability engineer.

======================================
My bottom-line recommendations for Dan's study:

  - Include surveys of product management's general assumptions and  
attitudes about usability before, during and after the primary  
usability testing phase is complete. Include discussion of desired  
vs. actual communication styles and deliverables.

- Ask for PM's _stated_ plan regarding how the findings of the  
usability are to be used, when the testing will be performed relative  
to the rest of the development and testing process, and how PM will  
decide, individually or collectively, how to triage suggested design  
changes. If possible, follow up with what actually occurred during  
the process.

My bet is that you'll find that in the absence of a truly Jobsian or  
Tog-esque level of charisma on the usability team, PM's communication  
skills will have a stronger influence on the perceived success of the  
usability team than their location internal or external to the company.

- Will

Will Parker
wparker at ChannelingDesign.com

"The only people who value your specialist knowledge are the ones who  
already have it." - William Tozier