[Sigia-l] SUMMARY: Distance Usability Labs? [x-post]

[David Heller]

Original question:
"With the advent of video conferencing and shared applications
and desktops over highspeed networks has anyone successfully 
conducted a usability test with customers remotely?"

SUMMARY of Replies:
There are some people who are doing remote usability testing and that
the cost benefit ratio seems to support the practice. One person pointed
out that a big loss occurs when you can't see the person as so much of
usability testing is around inferring emotional responses that are not
vocalized.

Websites worth mentioning:
http://boltpeters.com/bpvideo.html (20mb, QuickTime 5+)
http://www.boltpeters.com/remote-howto1.html
http://www.vividence.com/

Actually clipped replies:

Julian Orr-julian.orr at documentum.com:
http://www.vividence.com/public/products/vividence+xms+enterprise/vivide
nce+xms+enterprise.htm 

Again more "marketing-y" than pure usability, but when I talked this
lady she swore that it yieled good usability feedback in addition to the
mushy stuff

--------------------
Mitchell Gass-mitchell at participatorydesign.com:
I've conducted many remote usability tests, all with good success. I use
a sophisticated ASP for remote technical support for the tests; users
anywhere in the world can try websites or applications running on my
computer and talk with me with a speakerphone. After adapting how I 
facilitate to compensate for the limitations of remote testing, I feel
I'm getting at least 80% of the data I'd get from in-person tests, with 
substantial cost and time savings over conducting in-person tests at
other locations. Remote testing also makes it easier to recruit
hard-to-get participants, such as highly paid professionals, who don't
have to leave their workplaces to participate.

Other languages and cultures pose the same problems for remote testing
that they do for in-person testing. I've conducted tests only in
English, but I've had participants from many countries. For example, in
one recent study, of an intranet for a large US-based company, we had
several participants in countries other than the US who use English
regularly in their work. Not all the sessions went well; a participant
in China was hard to reach, his English was not nearly as good as the
others, and his slow Internet connection was a problem for remote
viewing of the site, but even here we got data that was valuable for the
design. The alternatives - going to China with a native speaker or
arranging for a local usability specialist to conduct the tests - would
have been too expensive for the project. The choice was between getting
some data about users in other countries at reasonable cost and getting
no data at all.

An interesting note: I've conducted a number of remote studies over the 
last several months for one of my main clients, a Fortune 500 company,
and during that time our only face-to-face meeting was not directly
related to the studies. We've been able to do all the work for the
studies by phone and e-mail, and we're planning more studies for the
coming months.

-----------------------------
Jon Antin-jantin at att.com:
Remote usability testing has become the primary way we do usability
testing.  We are located in Greensboro, NC, but most of our users are
elsewhere (e.g., New Jersey, Atlanta, etc.).

I believe we have found that we lose little or nothing compared with
in-house usability testing.  We just set up a NetMeeting and a speaker
phone and it's almost as if the person were in our lab.

-----------------------------
Jennifer Healy-jhealy at book.com:
I recently conducted usability testing via PlaceWare and was pleased
with the results. Since it was impossible to gather our respondents --
users of our B2B sites -- in one location, this was our only option. I
don't think this is the ideal set up, as non-verbals are a key
component. However, it worked well for showing static gifs and enabled
us to obtain useful feedback.

Based on my previous experience with distance education, however, I do
think that usability can be effectively conducted via this medium. With
the proper video set up that allows the user and moderator to see one
another and communicate in real time, I suspect that the experience is
likely to be quite similar to an in-person experience. In fact, research
I conducted for my master's thesis a few years ago showed that, in a
distance learning environment, students in the remote and live
classrooms reported similar levels of satisfaction with their
interactions with the instructor as well as with students at the other
location(s).

However, it seems like there are limited cases in which distance
usability would be necessary. Regardless of the technology's viability
and effectiveness, live interaction is always preferrable.

------------------------------------
Nate Bolt-nate at boltpeters.com:

We have conducted over 40 remote studies involving 180 users over the
last 2 years. In three parallel remote and laboratory studies we've
done, where we used the same test plan and very similar users, there
were near duplicate results from the two types of studies.

We started out using Timbuktu back in the day, and now we use WebEx and
have developed a proprietary plug-in that captures user's desktop along
with logging information and sync info for the digital audio we capture
on the recording end... 

You can see some highlights that show our lab and remote clips side by
side here: http://boltpeters.com/bpvideo.html (20mb, QuickTime 5+)

-------------------------------

Thanx to everyone who replied!

-- dave

David Heller
Sr. User Interface Designer
Documentum: The Leader in Enterprise Content Management
925.600.5636
 
david.heller at documentum.com
http://www.documentum.com/
AIM: bolinhanyc  //  Yahoo: dave_ux  //  MSN: hippiefunk at hotmail.com
 
--"If it isn't useful, it will never be usable."