[Sigia-l] Usability Profession Increasingly Under Scrutiny?

[Peter Merholz]

Jon--

It frustrates me that no one bothered to really address your question, 
and instead banged on the same limp drums that they always bang on.

Your question is a good one, and a hard one -- it's difficult to 
measure the impact of any one element on a process. What I find 
interesting is that other departments are rarely required to "prove" 
their value in such a concrete way... There's something about the UCD 
methodology that suggests we ought to have more clearly delineated 
concrete outcomes.

Anyway, at BayCHI in December, there was a panel on this very topic, 
moderated by Rashmi Sinha. The panelists included people from Ebay, 
Yahoo!, and Adaptive Path. Here are the notes from that talk, and I 
hope you find it helpful:

> ==========
>
>                     BayCHI December Program Meeting
>                  Panel: The Business of User Experience
>            Klaus Kaasgaard of Yahoo!, Jeffrey Herman of eBay,
>                 and Scott Hirsch, Independent Consultant
>              Moderated by Rashmi Sinha of Uzanto Consulting
>
> by Nerija Sinkeviciute-Titus ntitus at baychi.org
>
> The three speakers talked about the business of User Experience (UE)
> from somewhat different angles and touched different layers of the
> problem.  Scott Hirsch approached the topic from a macro, conceptual
> level, Klaus Kaasgaard of Yahoo! described how UE can add value to
> companies, and Jeff Herman presented the process of UE design as well 
> as
> specific practices at eBay.
>
> Scott Hirsch of Adaptive Path started with a quotation from Harvard
> Business Review (May 2003) by Nicholas G. Carr: "IT doesn't matter."
> The underlying premise, according to Scott, is that "investing in fancy
> new technology is not where the competitive advantage is.  Rather you
> should do more with the IT you have."  If that is the case, User
> Experience should be an important area of focus.
>
> He defined User Experience as a design approach centered on user needs
> and behavior.  The tools of UE are Information Architecture (IA),
> usability, interaction design, functionality, etc.  He compared this
> field to the field of Customer Experience, a design approach that
> addresses user needs and behavior.  The tools of this area are brand,
> advertising, public relations, corporate identity, architecture, etc.
> He pointed to The Gap clothing store as a good example of a holistic
> experience: You know what it feels like, what their music is...
>
> Next, Scott Hirsch described the variables of UE: user needs, which are
> uncontrollable, and user behavior, which is controllable.  In order to
> address the former, one must consider users tasks, do research, and use
> intuition and high-level vision.  The latter variable involves such
> issues as project finance and accountability.
>
> Scott pointed out that in the context of UE the traditional 
> cost/benefit
> analysis assumes a cost perspective rather than investment perspective.
> The product development process should also be more interdisciplinary
> and product-focused.
>
> When it comes to ROI, Scott Hirsch prefers a detailed approach:
>
>  - Project A returns $X next quarter, Project B returns $X next year.
>
> Rather than a macro approach:
>
>  - Investing $X in Project will yield a return of $Y.
>
> To him the latter generalization is meaningless because ROI is NOT
> empirical.  It is rather a comparative financial tool.  The micro
> approach emphasizes ROI comparisons that help prioritize projects.  
> They
> also help separate long-term problems from quick wins.
>
> So how do we use ROI to get more resources for UE?  Here Scott brings 
> up
> a topic of budgets and headcount, where UE lives in another department
> and doesn't have as much pull.  Total budget and headcount tend to be
> fixed in the short-term.  Scott Hirsch suggests advocating UE beyond 
> the
> quick win, i.e. build a business case that the finance people would
> understand, think of it as an investment, just as valuable as
> advertisement or brand investment.  BUT, money should not be the only
> criterion.  According to Scott Hirsch, it is extremely important to
> develop good business cases.  Here are the necessary actions:
>
> 1. Designate accountability.  The business owner should be someone
>    outside the department.
>
> 2. Define selection criteria and consider returns (revenue cost 
> savings,
>    competitive position).  Investment cost (technical feasibility, time
>    to market, opportunity cost) and other factors (politics, strategic
>    importance, real options).
>
> 3. Determine performance metrics.
>
> 4. Establish continuing ownership and ongoing accountability.
>
> In short, here's what Scott Hirsch wanted his audience to take away:
>
> 1. Think about projects as interventions that can change customer
>    behavior.  (Persuasive Technology, BJ Fogg)
>
> 2. Attach business value ($) to changes in behavior--try writing a
>    business case.
>
> 3. Develop criteria for the evaluation of projects.
>
> 4. Assign ongoing accountability for performance metrics.
>
> Klaus Kaasgaard of Yahoo! started by describing the reality of user
> research and design teams as less than glamorous.  In our daily lives,
> he said, we have to constantly deal with fuzzy goals, no 
> accountability,
> and lack of sharing in success.  The visibility and influence isn't
> great; we are thought of as "hands," not "heads."  But we are also
> guilty ourselves of being clueless about the business objectives as 
> well
> as about principles of business management and strategy.
>
> "There is a lot of focus on ROI these days, but this tool has been
> misused, misunderstood, and used in unreliable ways, thus damaging the
> credibility of our profession.  It is clear that we need to become
> better at defining our value, pursuing it and articulating it--whether
> that value can be quantified or not."
>
> Klaus Kaasgaard pointed out several ways to look at the value of User
> Experience Design:
>
> 1. Value relative to giving significant directional feedback for 
> design.
>    He showed how enhancing comparison features in Yahoo! Shopping made
>    users recognize the product more.  Comparison-shopping turned out to
>    be a key feature adding value to the user during the on-line 
> shopping
>    experience.  As a result, Yahoo!'s redesigned comparison shopping
>    feature was highlighted by industry commentators as a key
>    differentiator.
>
> 2. Value relative to key business metrics, including UE metrics (lab
>    benchmarking, Vividence, in-house metrics).
>
> 3. Value relative to informing strategic product decisions.
>
> 4. Value, relative to organizational design (processes, culture,
>    leadership).
>
> Klaus Kaasgaard cited Peter Senge: "In a learning organization ...
> leaders are designers, teachers, and stewards.  These roles require new
> skills: the ability to build shared vision, to bring to the surface and
> challenge prevailing mental models, and to foster more systemic 
> patterns
> of thinking." (Peter Senge: "Building Learning Organizations," 1990.)
>
> Klaus Kaasgaard pointed out that the most interesting thing about our
> field is that we are in the business of changing behaviors.  Persuasive
> design is key to providing value.  He presented some examples from
> popular Web sites making use of the principle of suggestion, where
> company's interests are being tied in with the users interests.  For
> example, Amazon.com might suggest a list of other books that people who
> bought your selected book also liked.  Hotmail might suggest you buy
> more storage so you could send larger attachments, and so on.
>
> Jeff Herman of eBay shared his UI design group's seven-step process,
> which ensures that the projects the group proposes are approved and
> successful.  The steps are as follows:
>
> 1. Understand project approval process.
>
> The questions to ask: Who approves projects, what criteria do they use
> to approve projects?
>
> At eBay this process is open and criteria for approval are standard and
> objective.  Executive staff approves projects and their approval is
> comprehensive.  It comes with a promise that the project won't get
> stopped at a later time.
>
> 2. Know levers that drive the business.
>
> At eBay, levers include bids, listings, registered users, and costs
> (customer service, engineering, etc.).  It is important to know the
> value of those levers, e.g.  "What's the dollar value of each new
> registered user?"  The finance department is the place to go in order 
> to
> find out a standard value for each of these levers.
>
> It is also crucial to understand how the UE can affect those levers.
> For example, "Improving the registration user experience will increase
> the percentage of users who complete the registration process."
>
> 3. Determine current issues and future opportunities.
>
> Proactively research UE issues in your product today, i.e. look at site
> statistics, customer support incidents, baseline usability studies,
> heuristic analyses, best practices, field studies, surveys, and focus
> groups.
>
> Determine future opportunities and challenges by performing product
> metric projections and interviews with stakeholders (users, company
> leaders).
>
> 4. Select a product area to improve.
>
> Select a product area where no other group is focusing, where the 
> impact
> will be significant, measurable, and attributed back to your project,
> and where the cost and risks are relatively low, benefits are high, and
> chances of success are high.
>
> 5. Estimate the ROI for your project.
>
> Estimate both the cost and the benefit.  When identifying the
> advantages, one should identify business levers your project will
> affect, get the baseline statistics, consult with local experts, and
> understand the impact of previous projects.  Be conservative.
>
> 6. Present proposal on equal footing.
>
> Justify the project according to standard criteria (ROI).  It is
> important not to position the project as "special" or different.  You
> have to justify results, not process.  The ends (e.g. increased
> registered users) will justify the means.  Know your audience, speak to
> their issues, and speak their language.
>
> 7. Follow up after the project launches.
>
> Get the metrics that will help you understand what worked and what
> didn't work.
>
> Market your success within the company.  Project success can create a
> virtuous cycle of initiative, responsibility, credibility, and 
> increased
> investment (job openings, money, time for new projects).
>
> To illustrate this seven-step process, Jeff walked us through examples
> of several of their redesign projects, including registration flow and
> password recovery.
>
> According to Jeff, this process is effective at eBay for several
> reasons:
>
> 1. User goals and business goals are in synch.
>
> 2. Metrics and levers are understood and pervasive.
>
> 3. Even incremental improvements have significant monetary benefits.
>
> 4. Switching costs are low.
>
> 5. The user experience is the product.
>
> The questions that these presentations raised dealt with the challenges
> of aligning business goals and UE goals, conflicts of different
> interests (marketing vs. engineering), the power of numbers (ROI), and
> more.


--peter