[Sigia-l] Measuring user satisfaction and intuitivenessofasystem

Thu Apr 28 09:12:11 EDT 2011

Hi Darin,

You're right. I agree that usability testing is about more than just task efficiency and usability. There is definitely a significant amount of subjective information that can be garnered from usability testing.  

Your comments have definitely helped and I've enjoyed our discussion. I appreciate it. 

Thanks,

Ron Polka
User Interface Design Engineer

Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipients and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail, delete this message and destroy all copies thereof.

-----Original Message-----
From: sigia-l-bounces at asis.org [mailto:sigia-l-bounces at asis.org] On Behalf Of darin sullivan
Sent: Wednesday, April 27, 2011 5:31 PM
To: SIG Information Architecture
Subject: Re: [Sigia-l] Measuring user satisfaction and intuitivenessofasystem

Hi Ron... Yes, level of satisfaction is a completely different metric,
and when used alone still gives insight into a user's experience with the
system. Doing a survey is certainly better than not doing a survey, IMO.

If testing were only about task completion, then I would agree that my
points speak only to efficiency and usability. But it has been my experience
that users that participate in testing tend to offer loads of personal
observations and judgements about what they feel while interacting with the
system. We once had a test participant clutch closed her blouse at her neck
and state that "she felt violated," in response to being presented a page
requesting her credit card information when having started a task that was
to "sign up for a free trial." The marketing folks observing in the next
room really took notice of that I can tell you!

Plus, at the end of each testing session, our facilitator does a post-test
interview that incorporates questions about satisfaction with the existing
service, whether or not they think new functionality is useful, if they
would use it, if it's worth paying more to have and if so, how much, etc.

I hope that my comments have helped. Cheers!

On Wed, Apr 27, 2011 at 12:51 PM, Polka, Ronald <Ron.Polka at mckesson.com>wrote:

> Hi Darrin,
>
> I understand your points but I think they are all going back to task
> efficiency, design usability, and other aspects which are necessarily more
> objective and will necessarily have more practical impact on making the
> system more usable. I don't dispute the absolute value of those tests and I
> agree they are more useful in the process.
>
> However neither usability testing nor field studies will get into the
> user's own perspective of the system. They were able to complete their tasks
> efficiently but did they find their experience doing so satisfying? If they
> had two systems that allowed them to complete the same tasks relatively
> efficiently with little difference would they come away perceiving one
> system as more satisfying than the other?
>
> All of the usability efforts focused on task efficiency, validating the
> intuitiveness of the user interface, etc are absolutely indispensible in
> creating a usable system. But isn't there still a different aspect that
> focuses on the user's perceived satisfaction with the system? That seems to
> be a different metric that while when used alone is quite limited still
> gives insight into the user's experience with the system.
>
> Thanks,
>
> Ron Polka
> User Interface Design Engineer
>
> Confidentiality Notice: This e-mail message, including any attachments, is
> for the sole use of the intended recipients and may contain confidential
> and/or privileged information. Any unauthorized review, use, disclosure or
> distribution is prohibited. If you are not the intended recipient, please
> contact the sender by reply e-mail, delete this message and destroy all
> copies thereof.
>
>
> -----Original Message-----
> From: sigia-l-bounces at asis.org [mailto:sigia-l-bounces at asis.org] On Behalf
> Of darin sullivan
> Sent: Wednesday, April 27, 2011 3:35 PM
> To: SIG Information Architecture
> Subject: Re: [Sigia-l] Measuring user satisfaction and intuitiveness
> ofasystem
>
> Hi Ron... Measuring user satisfaction only (through the use of surveys)
> will
> mean that the discussion is structured in a way that will limit what can be
> learned. Usability testing is necessarily open ended, in that the user is
> allowed to wander and to puzzle things out (or not), and usually they are
> encouraged to talk aloud. Much is learned through this activity, and in
> more
> cases than I care to admit, it has resulted in findings that show our team
> is concerned about things that are
> of less concern to users and that we are unaware of things that are of
> great
> importance to our users. Now, if we had only our survey results to review,
> we would be missing this very important user input.
>
> More to this, field studies will gather important information that
> usability
> testing will miss. Because studies are done in the user's own, actual
> environment, we can see all of the tasks that the user is engaged in, at
> the
> same time that they are using our service. This becomes very important for
> understanding ambient cognitive load and identifying adjacent tasks and
> areas of the user's role and function, within the context of their
> enterprise. We've created very important changes and additional
> functionality to support needs that could have only been identified through
> field studies.
>
> As to UI efficiency, your business customer would love to hear how you have
> streamlined a key task that their end user performs frequently, such that
> it
> requires 20% less time for them to complete :)
>
>
> On Wed, Apr 27, 2011 at 11:15 AM, Polka, Ronald <Ron.Polka at mckesson.com
> >wrote:
>
> > Hello Darin,
> >
> > Thank you for your interesting response and for sharing your thoughts.
> >
> > I understand that measuring user satisfaction is subjective and limited
> as
> > well that the wording of such a questionnaire need be carefully worded. I
> > agree on all of those points.
> >
> > To me usability testing and a user satisfaction questionnaire get at two
> > different metrics - the former more objective and the latter more
> > subjective. There is definite, indisputable evidence for the use of both
> > together and I won't argue that point. I suppose I don't see, though, why
> > the subjective metric cannot be measured on its own for the purpose of
> > gauging the user experience from the subject's own opinion. We can
> > definitely reference it on various releases of the application to see if
> > we're making progress or not.
> >
> > Right now we do not have a grasp on the user experience qua user
> experience
> > - not the efficiency of the system, whether certain screens are less than
> > usable, etc. This information may be more important to our team right now
> as
> > a way to gain stakeholder buy-in for further UCD efforts going forward.
> If
> > we can say that user satisfaction is low then we know we need to drill
> into
> > it to find out why: trouble points, where we can improve, and what
> > enhancements we need.
> >
> > We do have customer satisfaction metrics but that is the business
> customer
> > who is rating their experience with our company and their satisfaction
> with
> > the functionality our system offers compared to their needs. To me that
> > seems quite different than the actual user who works on the system all
> day
> > and who may rate the system differently in their experience. For our
> clients
> > we have the layers of the business customer and the actual system user.
> >
> > I suppose I just struggle to understand why measuring user satisfaction
> > alone isn't valuable. I've come across that idea a few times. Maybe I am
> > missing something.
> >
> > Thanks,
> >
> > Ron Polka
> > User Interface Design Engineer
> >
> >
> > Confidentiality Notice: This e-mail message, including any attachments,
> is
> > for the sole use of the intended recipients and may contain confidential
> > and/or privileged information. Any unauthorized review, use, disclosure
> or
> > distribution is prohibited. If you are not the intended recipient, please
> > contact the sender by reply e-mail, delete this message and destroy all
> > copies thereof.
> >
> >
> > -----Original Message-----
> > From: sigia-l-bounces at asis.org [mailto:sigia-l-bounces at asis.org] On
> Behalf
> > Of darin sullivan
> > Sent: Wednesday, April 27, 2011 1:54 PM
> > To: SIG Information Architecture
> > Subject: Re: [Sigia-l] Measuring user satisfaction and intuitiveness of
> > asystem
> >
> > Hi Ron... At least quantitatively speaking, validation of your UI can be
> > accomplished utilizing standard techniques such as GOMS
> > Analysis<http://en.wikipedia.org/wiki/GOMS>, which
> > will measure the efficiency of a user's interaction with a system. A
> survey
> > of your user base can be used to help establish their level of
> > satisfaction,
> > but questions must be carefully worded so as to minimize bias in the
> > response. In my experience, only usability testing will effectively
> measure
> > the quality of an interaction with a system, or how users generally feel
> > about their interaction. Usually analysis is done in parallel with
> > usability
> > testing, to best establish both the efficiency and quality of the UI.
> >
> > A simple example of why both measures are important: When users move
> > through
> > open applications using keystroke combinations, such as Alt-Tab on
> Windows
> > and Command-Tab on Mac, it is often much more efficient to combine a
> third
> > key, Alt-Shift-Tab, to move to the target application. However, most
> users
> > prefer to simply repeat the Alt-Tab combination to move through the list
> of
> > applications rather than add the third key, because it's easier. There
> are
> > many examples of highly efficient systems that are used very little
> because
> > users don't like them (not easy or intuitive), and well used systems that
> > are very inefficient. Analysis and testing combine to provide important
> > guideposts to engineers, designers and stakeholders and can help to
> > establish the value of a user centered process.
> >
> > In terms of measuring the system as a whole, most businesses in my
> > experience are unwilling to analyze or test a system in its entirety and
> > prefer to look at trends in the size of their customer base and customer
> > satisfaction before and after large implementations-are we now adding
> more
> > customers than we're losing (churn rate) and how are they rating our
> > product
> > or service? These are both quantitative and qualitative measures too, and
> > can help to identify where analysis and testing can provide the greatest
> > benefit.
> >
> > Good luck, and keep us posted :)
> >
> >
> > On Wed, Apr 27, 2011 at 8:32 AM, Polka, Ronald <Ron.Polka at mckesson.com
> > >wrote:
> >
> > > All,
> > >
> > >
> > >
> > > We have a system that was designed largely without utilizing a
> > > user-centered design process. We're constantly facing challenges on how
> > > to validate our designs and the UI of the system. Some stakeholders
> > > within the organization think the user experience of the system is
> > > intuitive and satisfactory while others think the exact opposite.
> > >
> > >
> > >
> > > My question: is there a standard measure I can use to validate whether
> > > or not our user base finds their experience with the system in general
> > > as satisfactory and intuitive.
> > >
> > >
> > >
> > > I've come across the SUS, Microsoft's Desirability Toolkit, QUIS, USE,
> > > and other measures out there but it seems these measures are designed
> to
> > > be used in conjunction with a usability testing session focused on
> > > specific screens in the system. I'd like to measure the system as a
> > > whole so as to gauge where we stand today with our users.
> > >
> > >
> > >
> > > Any thoughts or comments? Has anyone run into a similar scenario or
> > > situation - if so what did you do?
> > >
> > >
> > >
> > > Thanks,
> > >
> > >
> > >
> > > Ron Polka
> > >
> > > User Interface Design Engineer
> > >
> > >
> > >
> > > McKesson Corporation
> > >
> > >
> > >
> > > Confidentiality Notice: This e-mail message, including any attachments,
> > > is for the sole use of the intended recipients and may contain
> > > confidential and/or privileged information. Any unauthorized review,
> > > use, disclosure or distribution is prohibited. If you are not the
> > > intended recipient, please contact the sender by reply e-mail, delete
> > > this message and destroy all copies thereof.
> > >
> > >
> > >
> > > ------------
> > > 2012  IA Summit
> > > March 21 - 25, 2012
> > > Hyatt Regency New Orleans, LA
> > > -----
> > > When replying, please *trim your post* as much as possible.
> > > *Plain text, please; NO Attachments
> > >
> > > Searchable Archive at http://www.info-arch.org/lists/sigia-l/
> > > ________________________________________
> > > Sigia-l mailing list -- post to: Sigia-l at asis.org
> > > Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l
> > >
> > ------------
> > 2012  IA Summit
> > March 21 - 25, 2012
> > Hyatt Regency New Orleans, LA
> > -----
> > When replying, please *trim your post* as much as possible.
> > *Plain text, please; NO Attachments
> >
> > Searchable Archive at http://www.info-arch.org/lists/sigia-l/
> > ________________________________________
> > Sigia-l mailing list -- post to: Sigia-l at asis.org
> > Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l
> >
> > ------------
> > 2012  IA Summit
> > March 21 - 25, 2012
> > Hyatt Regency New Orleans, LA
> > -----
> > When replying, please *trim your post* as much as possible.
> > *Plain text, please; NO Attachments
> >
> > Searchable Archive at http://www.info-arch.org/lists/sigia-l/
> > ________________________________________
> > Sigia-l mailing list -- post to: Sigia-l at asis.org
> > Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l
> >
> ------------
> 2012  IA Summit
> March 21 - 25, 2012
> Hyatt Regency New Orleans, LA
> -----
> When replying, please *trim your post* as much as possible.
> *Plain text, please; NO Attachments
>
> Searchable Archive at http://www.info-arch.org/lists/sigia-l/
> ________________________________________
> Sigia-l mailing list -- post to: Sigia-l at asis.org
> Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l
>
> ------------
> 2012  IA Summit
> March 21 - 25, 2012
> Hyatt Regency New Orleans, LA
> -----
> When replying, please *trim your post* as much as possible.
> *Plain text, please; NO Attachments
>
> Searchable Archive at http://www.info-arch.org/lists/sigia-l/
> ________________________________________
> Sigia-l mailing list -- post to: Sigia-l at asis.org
> Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l
>
------------
2012  IA Summit
March 21 - 25, 2012
Hyatt Regency New Orleans, LA
-----
When replying, please *trim your post* as much as possible.
*Plain text, please; NO Attachments

Searchable Archive at http://www.info-arch.org/lists/sigia-l/
________________________________________
Sigia-l mailing list -- post to: Sigia-l at asis.org
Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l