[Sigia-l] User Test Cost - Does this sound reasonable?

[George Olsen]

On 5/24/04 4:31 AM, "Jared M. Spool" <jspool at uie.com> wrote:
> The problem with traditional usability reports is that they don't really do
> a very good job of informing the team. Even when written clearly and
> concisely, (which they often aren't,) there is a large burden on the author
> to fully understand the needs and concerns of the readers, to predict what
> questions the reader will have, and to clearly communicate every
> alternative interpretation of the observations. Most reports rarely meet
> this standard and, therefore, fail the team miserably.

I think it's important to distinguish between feedback needed by the
immediate team and learnings that are intended for a wider audience over
time.

For the former, I agree that it's ideal for the team to actually observe the
testing, in which case a report is usually pretty redundant if there's been
a mutual debriefing. Likewise, the team gets a couple days headstart on
problem-solving, since they're not waiting for the analysis.

(Incidentally, having the team observe is beneficial in other significant
ways. 1) More eyes means more coverage. For example, I recently had a good
insight that occurred from observing a particular eye movement among test
users that our researcher couldn't see from where she was sitting. 2) A
greater variety of eyes widens the scope of the analysis. For example,
visual designers will pick up on things user researchers might miss, simply
due to their greater sensitivity to layout, color, typography, etc.)

But I still think it's often critical to capture the learnings in a report
so that they can be shared with others -- or even the same team at a later
date.

At Yahoo! there's simply too many teams for everyone to observe everyone
else's testing. But there are also design issues that overlap between teams.
So a written report -- while imperfect -- is better than no shared knowledge
at all.

Likewise, reports are good at capturing the rationales behind design
decisions, decisions that might not be obvious a year or two later,
particularly if there's been turnover on the team. And particularly as
sites/products mature, it's likely you'll end up tweaking some the same
parts of them again. That's when it's nice to have a memory aid about what
worked, what didn't and why.

Solving the problems that Jared raises can be solved by not having the
researcher create the test and write the report single-handedly. Instead, it
should be a team effort. Obviously the team won't write the report, but the
team can help define the needs and concerns the report needs to address, and
joint observation/mutual debriefing helps identify alternate
interpretations.

George