User Experience Evaluation Framework

for Human-Centered Design

Hiroyuki Miki

Oki Electric Ind. Co., Ltd., R&D Center

1-16-8 Chuou, Warabi-shi, Saitama 335-8510, Japan
hmiki@cf.netyou.jp

Abstract. Recently, the word “User Experience (UX)” has been often used in
usability-related areas such as web design and system design. Although it was
defined in ISO 9241-210 and its importance has been growing, details of the
notion and results of introduction of it have not been well clarified yet. In the
previous paper, a UX evaluation framework based on ISO 9241-11 and ACSI
(American Customer Satisfaction Index) was proposed. Following the previous
paper, this paper proposes an integrated new evaluation framework of usability
and UX, explains its usage, and discusses its application.

Keywords: User Experience, Usability, ISO 9241, ISO 13407, ISO/IEC 25010,
Evaluation framework, American Customer Satisfaction Index.

1 Introduction

Usability is a notion, for example, that addresses a degree of how easy one can use
products, systems, or services. As products, systems, or services become more com-
plex and provide high-level functions to the user, designing and evaluating usability
become more difficult. In addition, as business competitions go worldwide and be-
come fiercer, conditions of successful products, systems, or services become more
complex [4].
Norman considered this kind of changes and claimed that broader scope than usa-
bility should be considered [16]. He claimed that the user wants not only good usa-
bility but also high UX to be truly pleased with good products, systems, or services.
To consider UX, one needs to consider user’s good/ bad feelings and responses,
namely results of relating products, systems, or services more than usability. It was
the first time that UX was mentioned by a leading expert of usability.
Since international standards provide common bases for international businesses, it
is quite reasonable that the word UX is introduced in them. However, details of the
notion and results of the introduction of it have not been well clarified yet.
In the previous paper [15], a UX evaluation framework based on ISO 9241-11 [9]
and ACSI (American Customer Satisfaction Index) [1] was proposed. Following the
previous paper, this paper proposes an integrated new evaluation framework of usa-
bility and UX. After brief explanations of usability, UX, and Human-Centered Design

S. Yamamoto (Ed.): HIMI 2014, Part I, LNCS 8521, pp. 602–612, 2014.
© Springer International Publishing Switzerland 2014
 User Experience Evaluation Framework for Human-Centered Design 603

in international standards in section two, the UX evaluation framework of the previ-

ous paper is explained in section three. Then the integrated evaluation framework of
usability and UX is newly proposed in section four, followed by its example in sec-
tion five, and discussions in section six.

2 Usability, UX, and Human-Centered Design in International

Standards

2.1 Current Coverage of Usability and UX in International Standards

International Standards have the specific role to provide business organizations with
technical standards. Currently, UX related standards are built mainly by the ergonom-
ics committee (ISO/TC159/SC4) in ISO 9241 series and by the software quality
committee (ISO/IEC JTC1/SC7/WG6) in ISO/IEC 25000 series.
ISO 9241-210 [8] defines Human-Centered Design as an iterative process of user
research (understanding and specifying the context of use, and specifying the user
requirements), design, and evaluation.
Current coverage of usability and UX by international standards is shown in Table
1. User research is covered for example by ISO/TR 16982 [11] which explains gen-
eral methods for it and is applicable to both usability and UX.

Table 1. Design and evaluation covered by ISO

Usability User Experience

User research Yes
Design Yes Slightly yes
Evaluation Yes Slightly yes

On the other hand, coverage of usability and UX is different in design and evalua-
tion. Firstly about usability, ISO 9241 series provide principles and guidelines as a
design framework, and ISO 9241-11 provides an evaluation framework. ISO/IEC
25010 [10] also provides product quality model and quality in use model about usabil-
ity. Product quality model can be used for both design and evaluation, and quality in
use model can be used for evaluation. Quality in use model is almost same with ISO
9241-11 evaluation framework.
Secondly about UX, there has been no specific design framework in international
standards so far. In addition, there have been claims that UX cannot be designed by
leading experts. For example, Kim Goodwin says as follows.

“We can design every aspect of the environment to encourage an optimal

experience, but since each person brings her own attitudes, behaviors, and per-
ceptions to any situation, no designer can determine exactly what experience
someone has.” (pp.5 in [5])
604 H. Miki

Since there have still been disputes about this matter, creation of design framework of
UX is premature.
Meanwhile, there are some descriptions related to evaluation of UX in international
standards: definition of UX in ISO 9241-210 and similar descriptions to UX in
sub-notions of satisfaction in ISO/IEC 25010. However, there has been no specific
evaluation framework of UX yet in international standards. Since there were several
evaluation frameworks of UX proposed by an ad-hoc work group [17], it is expected
for international standard communities to create an evaluation framework of UX
which goes along with related international standards.

2.2 Definitions of Usability and UX, and Evaluation Framework of Usability

in International Standards

Before proposal of an evaluation framework of UX, differences between usability and

UX are described.

Fig. 1. Usability (evaluation) framework of ISO 9241-11

Fig.1 shows the usability evaluation framework of ISO 9241-11. There are three
big boxes: Intended Goals, Context of Use, and Usability Measures. This figure
shows that when a product or service is used by a user under the context of use and
intended goals, usability is measured by effectiveness, efficiency, and satisfaction. By
this figure, usability is described as it depends on a specific context of use, thus you
cannot measure usability without determining a specific context of use. Definition of
usability is specified as the following in ISO 9241-11.

• Definition of usability in ISO 9241-11:1998 [9]:

Extent to which a product can be used by specified users to achieve specified goals
with effectiveness, efficiency and satisfaction in a specified context of use.
 User Experience Evaluation Framework for Human-Centered Design 605

In the meantime, there is no such evaluation framework of UX in international stand-

ards. Only definition of UX is described as the following in ISO 9241-210.

• Definition of User Experience in ISO 9241-210:2010 [8]:

Person’s perceptions and responses resulting from the use and/or anticipated use of
a product, system or service．

Definition of UX is very different from that of usability. Firstly, time span is different.
While usability only deals with “during usage”, UX deals with “before usage”, “dur-
ing usage”, “after usage”, and “over time” [17]. UX deals with a longer time span
since subjective feelings are affected by many related things. Secondly, measures are
different in subjective and objective measures. While effective and efficiency of
usability are objective measures, UX is evaluated by subjective measures as with
satisfaction of usability. These differences must be considered in a UX evaluation
framework.

3 UX Evaluation Framework in the Previous Paper

Considering arguments of section 2, UX evaluation framework was proposed in the

previous paper. After explaining ACSI (American Customer Satisfaction Index)
framework [1] which is referenced in the framework, the proposed framework is ex-
plained in section 3.2.

3.1 Inclusion of Concepts of American Customer Satisfaction Index (ACSI)

The American Customer Satisfaction Index (ACSI) (Fig.2) is the national cross-
industry benchmark of customer satisfaction in the United States. Since 1994, ACSI is
widely used to evaluate customer satisfaction across government systems and ser-
vices, industries and their services, and so on. Based on ACSI, similar indices have
been created in other countries as well, including ECSI (European Customer Satisfac-
tion Index) and JCSI (Japanese Customer Satisfaction Index).

Fig. 2. American Customer Satisfaction Index (Arrows represent influence. This is created
based on http://www.theacsi.org/index.php)
606 H. Miki

While core questions of ACSI are three questions on Customer Satisfaction (“over-
all satisfaction”, “confirm/ disconfirm expectation”, and “comparison with ideal”),
standard model has about total 14 questions on 1-10 scale about six indices: customer
expectation, perceived quality, perceived value, customer satisfaction, customer com-
plaint, and customer loyalty. Six indices are ordered from left to right chronologically
in Fig.2.
Starting from Customer Expectation, the index model shows causal relations of
indices which are important to evaluate products and services in terms of customer
satisfaction. While Customer Expectation evaluates the customer’s anticipation of
products and services before an actual usage, Perceived Quality and Perceived Value
evaluate feelings during usage followed by Customer Complaints and Customer Roy-
alty for the evaluation after usage.
From ACSI questionnaire, two kinds of results are obtained. One is score of each
of six indices, which is used to compare different products and services. The other is
weight of each arrow in Fig. 2, from which reason of high/ low score of each of six
indices will be figured out.
Not surprisingly, since ACSI focuses on subjective evaluations of products and
services, it does not include evaluations of design elements or objective measures.

3.2 UX Evaluation Framework in the Previous Paper

Fig.3 shows the result of the previous paper, namely an evaluation framework of UX
based on ISO 9241-11 and ACSI. Three major components and relationship among

Intended/Expected

User
Outcome
Intended/Expected
Goals
Task UX

Equipment
Perceived Quality
Outcome of

Environment Use/Expectation

Perceived Value Complaints

Context of use
Satisfaction Loyalty
Product/ Short term measures Long term measures
Service
UX measures

Fig. 3. UX evaluation framework based on ISO 9241-11 and ACSI of the previous paper
 User Experience Evaluation Framework for Human-Centered Design 607

them are deployed from ISO 9241-11: Goals, Context of Use, and UX Measures. In
the UX Measure component, components of ACSI are deployed. Along with ACSI,
each component is supposed to be broken down into sub-indices (sub-questions).
Meanings of small components such as Goals, Perceived Quality, and Perceived
Value are slightly changed from ISO 9241-11 and ACSI. First, Goals are specified as
composed of Intended Goals of Fig. 1, and Expected Goals which correspond to Cus-
tomer Expectation of ACSI.
Second, meaning of Perceived Quality is extended as composed of various quali-
ties. Although Perceived Quality is calculated as a total score of desired and undesired
degrees against needs in ACSI, since quality measures of UX other than satisfaction
and long term measures vary a lot, appropriate measures should be selected for a sys-
tem and a service.
Third, meaning of Perceived Value is changed as relative quality against input
compared with the relative quality against price in ACSI. Examples of Perceived Val-
ue include relative pleasure against stress in game, relative relief against anxiety in
public machine usage, and so on. Although usability international standards do not
treat a value as a measure, it should be added to consider UX.
Long term measures, namely Complaints and Loyalty, are not changed from ACSI.
Advantages of the proposed framework are as the following. First, since ACSI has
been widely applied to many products and services, the proposed framework are
expected to be applied to many products and services as well. Second, since the
proposed framework is based on ISO9241-11 framework (Fig.1), it will be easily
integrated with it. In fact, the integration will be conducted in the next section.

4 Integrated Evaluation Framework of Usability and UX

Fig.4 shows the integrated framework based on the previous paper’s framework
(Fig.3) and the ISO 9241-11 framework (Fig.1). Large change of Fig.4 from Fig.1 is
that Satisfaction of Fig.1 is replaced by UX Measures of Fig.3. While objective
measures are still represented by Effectiveness and Efficiency, subjective measures
are represented by UX Measures derived from ACSI. Fig.4 represents both objective
measures and subjective measures of UX.
While objective measures are what designers want to measure, subjective measures
of UX are supposed to represent as close user’s subjective evaluations as possible.
Basically, there is no direct connection between Effectiveness and Efficiency, and the
subjective measures of UX. However, if measures of Effectiveness and Efficiency are
well designed enough to represent user’s subjective evaluation of Perceived Quality,
Perceived Value, and possibly other measures of UX, the connection will be tighter.
When considering UX, Effectiveness and Efficiency need to be reevaluated by Per-
ceived Quality and Perceived Value toward satisfaction.
608 H. Miki

Fig. 4. Integrated evaluation framework of usability and UX

5 Example

Fig.5 shows an example of the right half of new framework, Fig.4. It is about a racing
game. It does not show details of each element which is represented by a box, but
illustrates causal relations of the framework.
Top part is about objective measures of usability consisting of Goal, Effectiveness,
and Efficiency and bottom part is about subjective measures of UX consisting of
ACSI indices. In this example, Goal is “getting to goal fast.” When, as Expectation
before usage, this person says “Since previous version of this game was fun, this new
one should be fun,” Expectation should be high. Although Effectiveness and Efficien-
cy are high when this person plays this game, this person says “This version is a bit
lazy than the previous version.” Then result of Perceived Quality should be med. This
result influences Perceived Value, and this person feels “Acquired fun level is just so-
so compared with given complexity.” Then evaluation result of Perceived Value
should be also med. As a result, this person’s Satisfaction should be most likely me-
dium. After some time, considering results of Satisfaction, Complaint, and Loyalty,
this person thinks to buy the next version as well as a result of med loyalty. These
results will formulates this person’s next Expectation as medium, so lower than the
previous expectation.
 User Experience Evaluation Framework for Human-Centered Design 609

Fig. 5. Example of integrated evaluation framework of usability and UX

As you see in this example, when you consider UX, this level of causal analyses of
subjective matters should be analyzed.

6 Discussion on the Proposed Framework

In this section, three points are discussed concerning the proposed integrated frame-
work.

6.1 Comparison with Usability Questionnaires

As usability questionnaires, there are some well-known questionnaires such as QUIS
[2], SUMI [11], PSSUQ [13], and SUS [1]. For example, SUS consists of 10 ques-
tions: disposition to use, complexity, easiness to use, support, functional integration,
inconsistency, learnability, irritation, confidence to use, and volume to learn. Subjects
are asked to answer each question with one of five responses that range from Strongly
Agree to Strongly Disagree. These questions correspond to subjective measures of
Fig.4. While irritation and confidence to use are about feelings and related to satisfac-
tion, other questions are related to perceived quality and perceived value.
610 H. Miki

It is quite understandable that most usability questionnaires are on Perceived

Quality since usability questionnaires focus on validity of usability design from de-
signer’s point of view and focus on “during usage”.

6.2 Use of the Proposed Framework

Integrated evaluation framework of Fig.4 will be used in a variety of ways. Firstly,
usage of usability and UX measures of Fig.4 will be different between the develop-
ment phase and the after-development phase. In the development phase, questions of
Expectation, Complaint, and Loyalty are asked before or after an experiment, such as
the usability evaluation experiment, as background information of development.
Meanwhile, questions of Perceived Quality, Perceived Value, and Satisfaction are
asked to improve design.
After the development phase, following ACSI, scores of six indices and strengths
of links are calculated from a questionnaire by the factor analysis and so on. For ex-
ample, when the calculated customer satisfaction index is low, calculated strengths of
links are used to figure out which line of links is a main cause of it. Considering this
result, one can improve the customer satisfaction index.
Secondly, when considering “over time,” products and services are repeatedly used
and usability evaluation data of Effectiveness and Efficiency are taken as well as sub-
jective measurement data. Such data will be used to keep track of changes of usability
and UX.
Thirdly, when objective measures of usability, namely Effectiveness and Efficien-
cy, are broken down into sub measures, Perceived Quality and Perceived Value will
also be broken down into sub measures. In ACSI, it is often the case that Perceived
Quality is broken down into sub measures in a specific domain like Fig.6 of govern-
ment system since Perceived Quality has large impact on Satisfaction. Following
ACSI, Perceived Quality of Fig.4 should be broken down when necessary.

Fig. 6. ACSI for government systems (This is created based on http://www.

theacsi.org/acsi-model-for-most-government-agencies)
 User Experience Evaluation Framework for Human-Centered Design 611

6.3 Core Three Questions on Satisfaction

In ACSI, although there are six measures in the standard model, core three questions
of satisfaction measure, “overall satisfaction”, “confirm/ disconfirm expectation”,
and “comparison with ideal”, are most important to compare different products and
services. By this reason, ACSI model can be consolidated into one measure, namely
satisfaction with the three questions.
When using the three questions to evaluate satisfaction, degree of “overall satisfac-
tion” will be clearer since “confirm/ disconfirm expectation” and “comparison with
ideal” will contribute to adjusting degree of satisfaction to compare with other similar
products or services. In this sense, when evaluating satisfaction for UX, the three
questions are important in not only the consolidated version but also the non-
consolidated version just like ACSI.
When subjective measures of UX are consolidated into one measure of satisfaction,
the proposed framework is almost same with ISO 9241-11 usability framework. Dif-
ference is that satisfaction is asked by the three questions and comparisons with ex-
pectation and ideal are characteristic compared with ISO 9241-11. In such a case,
however, it will be difficult to figure out reasons of low/ high customer satisfaction
since strength of links cannot be calculated from a questionnaire. Hence, either con-
solidated version or non-consolidated version of subjective measures of UX should be
selected properly in the proposed framework as well with ACSI by considering usage.
It is especially so when we consider internet questionnaire like Japanese Customer
Satisfaction Index since large amount of questionnaire results will contribute to the
calculation of link strengths of subjective measures of UX well.

7 Concluding Remarks

In the previous paper, a UX evaluation framework based on ISO 9241-11 and ACSI
was proposed. Following the previous paper, this paper proposed an integrated new
evaluation framework of usability and UX, explained its usage, and discussed its
application.
Since UX is a complex notion [6, 13, 18, 19], the integrated new framework is
expected to be applied to and examined against real applications. Results of this paper
are also expected to be considered in the creation of UX related international stand-
ards.

References
1. ACSI: The American Customer Satisfaction Index, http://www.theacsi.org/the-american-
customer-satisfaction-index
2. Brooke, J.: SUS: a “quick and dirty” usability scale. In: Jordan, P.W., et al. (eds.) Usability
Evaluation in Industry, Taylor & Francis (1996)
3. Chin, J.P., et al.: Development of an instrument measuring user satisfaction of the human-
computer interface. In: Proc. CHI 1988, pp. 213–218. ACM, New York (1988)
612 H. Miki

4. Christensen, C.M.: The Innovator’s Dilemma: The revolutionary book that will change the
way you do business. Harvard Business Review Press (1997)
5. Goodwin, K.: Designing for the digital age: how to create human-centered products and
services. Wiley (2009)
6. Hartson, R., Pyla, P.S.: The UX Book: Process and Guidelines for Ensuring a Quality User
Experience. Morgan Kaufmann (2012)
7. ISO 13407: Human-centred design processes for interactive systems. ISO (1999)
8. ISO 9241-210: Ergonomics of human-system interaction – Part 210: Human-centred de-
sign for interactive systems. ISO (2010)
9. ISO 9241-11: Ergonomic requirements for office work with visual display terminals
(VDTs) – Part 11: Guidance on usability. ISO (1998)
10. ISO/IEC 25010: Systems and software engineering – Systems and software Quality Re-
quirements and Evaluation (SQuaRE) – System and software quality models. ISO/IEC
(2011)
11. ISO/TR 16982: Usability methods supporting human-centred design. ISO (2002)
12. Kirakowski, J., Corbett, M.: SUMI: The Software Usability Measurement Inventory. Brit-
ish J. of Educational Technology 24(3), 210–214 (1993)
13. Kurosu, M.: The conceptual model of Experience Engineering (XE). In: Kurosu, M. (ed.)
HCII/HCI 2013, Part I. LNCS, vol. 8004, pp. 95–102. Springer, Heidelberg (2013)
14. Lewis, J.R.: Psychometric evaluation of the PSSUQ using data from five years of usability
studies. Int. J. of Human–Computer Interaction 14, 463–488 (2002)
15. Miki, H.: Reconsidering the Notion of User Experience for Human-Centered Design. In:
Yamamoto, S. (ed.) HCI 2013, Part I. LNCS, vol. 8016, pp. 329–337. Springer, Heidelberg
(2013)
16. Norman, D.A.: Invisible Computer: Why good products can fail, the personal computer is
so complex and information appliances are the solution. MIT, Cambridge (1998)
17. Roto, V., et al.: User Experience White Paper (2011),
http://www.allaboutux.org/uxwhitepaper (electronic version)
18. Sauro, J., Lewis, J.R.: Quantifying the user experience. Morgan Kaufmann (2012)
19. Tullis, T., Albert, B.: Measuring the User Experience. Morgan Kaufmann (2008)