[Sigia-l] Measuring user satisfaction and intuitiveness ofasystem

Wed Apr 27 15:51:43 EDT 2011

Hi Darrin,

I understand your points but I think they are all going back to task efficiency, design usability, and other aspects which are necessarily more objective and will necessarily have more practical impact on making the system more usable. I don't dispute the absolute value of those tests and I agree they are more useful in the process. 

However neither usability testing nor field studies will get into the user's own perspective of the system. They were able to complete their tasks efficiently but did they find their experience doing so satisfying? If they had two systems that allowed them to complete the same tasks relatively efficiently with little difference would they come away perceiving one system as more satisfying than the other? 

All of the usability efforts focused on task efficiency, validating the intuitiveness of the user interface, etc are absolutely indispensible in creating a usable system. But isn't there still a different aspect that focuses on the user's perceived satisfaction with the system? That seems to be a different metric that while when used alone is quite limited still gives insight into the user's experience with the system. 

Thanks,

Ron Polka
User Interface Design Engineer 

Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipients and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail, delete this message and destroy all copies thereof.

-----Original Message-----
From: sigia-l-bounces at asis.org [mailto:sigia-l-bounces at asis.org] On Behalf Of darin sullivan
Sent: Wednesday, April 27, 2011 3:35 PM
To: SIG Information Architecture
Subject: Re: [Sigia-l] Measuring user satisfaction and intuitiveness ofasystem

Hi Ron... Measuring user satisfaction only (through the use of surveys) will
mean that the discussion is structured in a way that will limit what can be
learned. Usability testing is necessarily open ended, in that the user is
allowed to wander and to puzzle things out (or not), and usually they are
encouraged to talk aloud. Much is learned through this activity, and in more
cases than I care to admit, it has resulted in findings that show our team
is concerned about things that are
of less concern to users and that we are unaware of things that are of great
importance to our users. Now, if we had only our survey results to review,
we would be missing this very important user input.

More to this, field studies will gather important information that usability
testing will miss. Because studies are done in the user's own, actual
environment, we can see all of the tasks that the user is engaged in, at the
same time that they are using our service. This becomes very important for
understanding ambient cognitive load and identifying adjacent tasks and
areas of the user's role and function, within the context of their
enterprise. We've created very important changes and additional
functionality to support needs that could have only been identified through
field studies.

As to UI efficiency, your business customer would love to hear how you have
streamlined a key task that their end user performs frequently, such that it
requires 20% less time for them to complete :)

On Wed, Apr 27, 2011 at 11:15 AM, Polka, Ronald <Ron.Polka at mckesson.com>wrote:

> Hello Darin,
>
> Thank you for your interesting response and for sharing your thoughts.
>
> I understand that measuring user satisfaction is subjective and limited as
> well that the wording of such a questionnaire need be carefully worded. I
> agree on all of those points.
>
> To me usability testing and a user satisfaction questionnaire get at two
> different metrics - the former more objective and the latter more
> subjective. There is definite, indisputable evidence for the use of both
> together and I won't argue that point. I suppose I don't see, though, why
> the subjective metric cannot be measured on its own for the purpose of
> gauging the user experience from the subject's own opinion. We can
> definitely reference it on various releases of the application to see if
> we're making progress or not.
>
> Right now we do not have a grasp on the user experience qua user experience
> - not the efficiency of the system, whether certain screens are less than
> usable, etc. This information may be more important to our team right now as
> a way to gain stakeholder buy-in for further UCD efforts going forward. If
> we can say that user satisfaction is low then we know we need to drill into
> it to find out why: trouble points, where we can improve, and what
> enhancements we need.
>
> We do have customer satisfaction metrics but that is the business customer
> who is rating their experience with our company and their satisfaction with
> the functionality our system offers compared to their needs. To me that
> seems quite different than the actual user who works on the system all day
> and who may rate the system differently in their experience. For our clients
> we have the layers of the business customer and the actual system user.
>
> I suppose I just struggle to understand why measuring user satisfaction
> alone isn't valuable. I've come across that idea a few times. Maybe I am
> missing something.
>
> Thanks,
>
> Ron Polka
> User Interface Design Engineer
>
>
> Confidentiality Notice: This e-mail message, including any attachments, is
> for the sole use of the intended recipients and may contain confidential
> and/or privileged information. Any unauthorized review, use, disclosure or
> distribution is prohibited. If you are not the intended recipient, please
> contact the sender by reply e-mail, delete this message and destroy all
> copies thereof.
>
>
> -----Original Message-----
> From: sigia-l-bounces at asis.org [mailto:sigia-l-bounces at asis.org] On Behalf
> Of darin sullivan
> Sent: Wednesday, April 27, 2011 1:54 PM
> To: SIG Information Architecture
> Subject: Re: [Sigia-l] Measuring user satisfaction and intuitiveness of
> asystem
>
> Hi Ron... At least quantitatively speaking, validation of your UI can be
> accomplished utilizing standard techniques such as GOMS
> Analysis<http://en.wikipedia.org/wiki/GOMS>, which
> will measure the efficiency of a user's interaction with a system. A survey
> of your user base can be used to help establish their level of
> satisfaction,
> but questions must be carefully worded so as to minimize bias in the
> response. In my experience, only usability testing will effectively measure
> the quality of an interaction with a system, or how users generally feel
> about their interaction. Usually analysis is done in parallel with
> usability
> testing, to best establish both the efficiency and quality of the UI.
>
> A simple example of why both measures are important: When users move
> through
> open applications using keystroke combinations, such as Alt-Tab on Windows
> and Command-Tab on Mac, it is often much more efficient to combine a third
> key, Alt-Shift-Tab, to move to the target application. However, most users
> prefer to simply repeat the Alt-Tab combination to move through the list of
> applications rather than add the third key, because it's easier. There are
> many examples of highly efficient systems that are used very little because
> users don't like them (not easy or intuitive), and well used systems that
> are very inefficient. Analysis and testing combine to provide important
> guideposts to engineers, designers and stakeholders and can help to
> establish the value of a user centered process.
>
> In terms of measuring the system as a whole, most businesses in my
> experience are unwilling to analyze or test a system in its entirety and
> prefer to look at trends in the size of their customer base and customer
> satisfaction before and after large implementations-are we now adding more
> customers than we're losing (churn rate) and how are they rating our
> product
> or service? These are both quantitative and qualitative measures too, and
> can help to identify where analysis and testing can provide the greatest
> benefit.
>
> Good luck, and keep us posted :)
>
>
> On Wed, Apr 27, 2011 at 8:32 AM, Polka, Ronald <Ron.Polka at mckesson.com
> >wrote:
>
> > All,
> >
> >
> >
> > We have a system that was designed largely without utilizing a
> > user-centered design process. We're constantly facing challenges on how
> > to validate our designs and the UI of the system. Some stakeholders
> > within the organization think the user experience of the system is
> > intuitive and satisfactory while others think the exact opposite.
> >
> >
> >
> > My question: is there a standard measure I can use to validate whether
> > or not our user base finds their experience with the system in general
> > as satisfactory and intuitive.
> >
> >
> >
> > I've come across the SUS, Microsoft's Desirability Toolkit, QUIS, USE,
> > and other measures out there but it seems these measures are designed to
> > be used in conjunction with a usability testing session focused on
> > specific screens in the system. I'd like to measure the system as a
> > whole so as to gauge where we stand today with our users.
> >
> >
> >
> > Any thoughts or comments? Has anyone run into a similar scenario or
> > situation - if so what did you do?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Ron Polka
> >
> > User Interface Design Engineer
> >
> >
> >
> > McKesson Corporation
> >
> >
> >
> > Confidentiality Notice: This e-mail message, including any attachments,
> > is for the sole use of the intended recipients and may contain
> > confidential and/or privileged information. Any unauthorized review,
> > use, disclosure or distribution is prohibited. If you are not the
> > intended recipient, please contact the sender by reply e-mail, delete
> > this message and destroy all copies thereof.
> >
> >
> >
> > ------------
> > 2012  IA Summit
> > March 21 - 25, 2012
> > Hyatt Regency New Orleans, LA
> > -----
> > When replying, please *trim your post* as much as possible.
> > *Plain text, please; NO Attachments
> >
> > Searchable Archive at http://www.info-arch.org/lists/sigia-l/
> > ________________________________________
> > Sigia-l mailing list -- post to: Sigia-l at asis.org
> > Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l
> >
> ------------
> 2012  IA Summit
> March 21 - 25, 2012
> Hyatt Regency New Orleans, LA
> -----
> When replying, please *trim your post* as much as possible.
> *Plain text, please; NO Attachments
>
> Searchable Archive at http://www.info-arch.org/lists/sigia-l/
> ________________________________________
> Sigia-l mailing list -- post to: Sigia-l at asis.org
> Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l
>
> ------------
> 2012  IA Summit
> March 21 - 25, 2012
> Hyatt Regency New Orleans, LA
> -----
> When replying, please *trim your post* as much as possible.
> *Plain text, please; NO Attachments
>
> Searchable Archive at http://www.info-arch.org/lists/sigia-l/
> ________________________________________
> Sigia-l mailing list -- post to: Sigia-l at asis.org
> Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l
>
------------
2012  IA Summit
March 21 - 25, 2012
Hyatt Regency New Orleans, LA
-----
When replying, please *trim your post* as much as possible.
*Plain text, please; NO Attachments

Searchable Archive at http://www.info-arch.org/lists/sigia-l/
________________________________________
Sigia-l mailing list -- post to: Sigia-l at asis.org
Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l