[Sigia-l] user survey questionnaire

[Lyle_Kantrovich at cargill.com]

Patrick,

A survey is just a tool.  You have to consider what the questions are 
that you really need answers to.  Are they closed ended questions, or 
open ended questions?  E.g. do you want to know something like "Do the 
users like this representation of the brand?" or something like "What do 
they like or dislike about this design, and why?"

I've had pretty good luck with the SUS survey as a post-usability test 
survey.  It helps measure test participants' *perception* of the 
system's usability.  It's a great example of a well designed survey 
based on closed-ended questions.

SUS - A quick and dirty usability scale
http://www.usability.serco.com/trump/documents/Suschapt.doc

I often will add a few open-ended questions to the SUS survey, but these 
don't factor into the SUS score.  And of course, the SUS score is 
nothing close to a good measure of a system's usability - but it's a 
good measure of perceived usability.  I've had users fail at many tasks 
in a test and still give a very high rating of usability in the survey.  
This is just further evidence that users often blame themselves rather 
than the system/design.

One client recently did a pretty extensive survey - many of the 
open-ended comments were difficult to decipher and just opened up 
further questions.  Luckily, it wasn't an anonymous survey, so they 
could follow up with the person who made the comments to understand why 
they blasted the design or to clarify comments that didn't make sense to 
the team.

The risk with surveys is that they may not really be measuring what you 
want them to measure.  If you show a few screenshots, and do a survey 
asking what users think of them, is that a good measure of the design's 
completeness?  of the visual design or branding?  of the design's 
usability?  of the IA?  Probably not.

BUT - sometimes a survey can be used as a communications tool or as a 
way to get buy-in.

Regards,

Lyle

----
Lyle Kantrovich
User Experience Architect

Croc O' Lyle - Personal Commentary on usability, information 
architecture and design.
http://crocolyle.blogspot.com/

"Simplicity is the ultimate sophistication." 
	- Leonardo da Vinci
 

-----Original Message-----
From: gamutant at earthlink.net [mailto:gamutant at earthlink.net]
Sent: Monday, March 29, 2004 1:53 PM
To: sigia-l at asis.org
Subject: Re: [Sigia-l] user survey questionnaire


Thanks everyone for your responses. I should have mentioned that we 
initially requested to conduct telephone interviews with international 
users but were told that would be too difficult to organize. The 
questionnaire was intended as a back-up solution to provide us with at 
least _some_ feedback from the trenches in addition to the 'top-down' 
goals mandated by the various stakeholders. As far as the survey is 
concerned, I think we have struck a reasonable compromise between 
open-ended questions and more concise multiple choice queries with room 
for additional comments in text fields. But, I guess the big picture 
problem is a general intransigence on the client's part toward 
conducting user research. As Ziya suggested though, I think we might 
have better luck getting input once we are actually closer to launching. 
Since there is a bit of a time crunch, perhaps the solution is to go 
ahead with whatever feedback we can get now and then push a bit harder 
for access to users when we are ready with a prototype? Just thinking 
out loud. Thanks again for everyone's input.

:P


 
-----Original Message-----
From: Eric Reiss <elr at e-reiss.com>
Sent: Mar 29, 2004 10:40 AM
To: sigia-l at asis.org
Subject: Re: [Sigia-l] user survey questionnaire

Dear Patrick,

Keep in mind that all volunteer questionnaires are self-segmenting.
In other words, you only get results from people who like to take
surveys. This has a tendency to skew the results - quite a lot in
many cases.

Also, getting a questionnaire right is very tricky (if you insist,
check out Don A. Dillman's fat but practical book - "Mail and
Internet Surveys, the tailored design method").

As Jesse suggested, if you're new to this game, get someone with good
interviewing skills to call people up in person. Much better ROI. And
if you're doing this yourself, get a couple of clocks to tell you the
local time. This is particularly useful if you're dealng with
anything more than three timezones away (the mere fact that CET went
on summertime yesterday has screwed up my own "in-the-head"
calculation throughout the day).

Cheers,
Eric

Eric Reiss
Principal
e-reiss aps
copenhagen, denmark

www.e-reiss.com

------------
When replying, please *trim your post* as much as possible.
*Plain text, please; NO Attachments

Searchable list archive:   http://www.info-arch.org/lists/sigia-l/
________________________________________
Sigia-l mailing list -- post to: Sigia-l at asis.org
Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l


Patrick Grizzard
140 Jackson St.
Apt. 3A
Brooklyn, NY 11211
t:646.522.9667
http://www.gneumatic.net

"And yet, it moves."
------------
When replying, please *trim your post* as much as possible.
*Plain text, please; NO Attachments

Searchable list archive:   http://www.info-arch.org/lists/sigia-l/
________________________________________
Sigia-l mailing list -- post to: Sigia-l at asis.org
Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l