ESCUELA DE ARQUITECTURA Y TECNOLOGÍA

Human-Machine Interaction

Lecture 3.3
Evaluation
 User-centered design
process
3nd
step: evaluate how good is our design and
interaction.

Understand the needs Explore Evaluation

from the user's point of solutions
view.

3.3. Evaluation 2
 Agenda
• What and when to evaluate
– Formative vs. summative evaluation
– Analytical vs empirical
• Evaluation approaches
– Cognitive Walkthrough
– Card Sorting
– Think Aloud
– Prototype Evaluation

3.3. Evaluation 3
 What and when to evaluate
• Original title: Evaluation and user testing
• ...we‘ll talk about testing products, not users, right?
• general mindset: accept the user as the reference!

• "Evaluation" somehow sounds as if it only happens

at the end
• evaluation methods can be used throughout the entire
development process!

Tests Development Tests Process Tests Test

3.3. Evaluation 4
 What can be evaluated?
• The usability of a system!
… it depends on the stage of a project
– Ideas and concepts
– Designs
– (paper and functional) Prototypes
– Implementations
– Products in use
… it also depends on the goals
• Approaches:
– Formative vs. summative evaluation
– Analytical vs. empirical evaluation

3.3. Evaluation 5
 Formative vs. summative
evaluation
Formative: what and
how to (re)design

Design Construction

Summative: how
did we do?

• M. Scriven: The methodology of evaluation, 1967

3.3. Evaluation 6
 Analytical vs. empirical
evaluation

Scriven, 1967: “If you want to evaluate a

tool, say an axe, you might study the
design of the bit, the weight distribution,
the steel alloy used, the grade of hickory in
the handle, etc., or you may just study the
kind and speed of the cuts it makes in the
hands of a good axeman.”

3.3. Evaluation 7
 Analytical vs. empirical
evaluation
Both methods are complementary
Analytical evaluation identifies the crucial characteristics
– Why does the axe have a special-shaped handle?
– Analytical evaluation produces facts which need to be interpreted

Empirical evaluation helps to understand the context for

object properties
– If the axe does not cut well, what do we have to change?
– Empirical evaluation produces facts which need to be interpreted

3.3. Evaluation 8
 Evaluation approaches:
types

Formative Summative

Without users Without users

Analytical Evaluates design Evaluates the
choices implementation

With users With users

Empirical Evaluates design Evaluates the
choices implementation

3.3. Evaluation 9
 Evaluation approaches:
methods

Formative Summative

Analytical • Cognitive • Heuristic

walktrough evaluation

Empirical • Card sorting • Controlled

• Thinking experiments
aloud • Usability lab
• Prototype tests
test • Field study

3.3. Evaluation 10
 Cognitive Walkthrough
• One or more evaluators going through a set of tasks
– Evaluating understandability and ease of learning
• Procedure:
– Defining the input:
• Who will be the users of the system?
• What task(s) will be analyzed?
• What is the correct action sequence for each task?
• How is the interface defined?
– During the walkthrough, answer four questions at each step:
• Will the users try to achieve the right effect?
• Will the user notice that the correct action is available?
• Will the user associate the correct action with the effect to be achieved?
• If the correct action is performed, will the user see that progress is being
made toward solution of the task?
From www.usabilityhome.com

3.3. Evaluation 11
 Card Sorting
• Is a technique used in UX in which a person tests a group
of subject experts or users to generate a category tree.
• Can be used to find out categories (open) or to judge
whether a given set of category names provides an
effective way to organize a given collection of content
(closed).

3.3. Evaluation 12
 Activity: Card Sorting
• Perform a card sorting using this topics for a cinema app:

– Privacy policy – Ranking Films

– Buy tickets – Premium rooms
– Account settings – Favourite list of films
– Select cinema – Register user
– Select seat – Forget password
– Watch trailer – Discounts
– Events – Help
– Work with us – Contact
– Legal notice
– Login

3.3. Evaluation 13
 Think Aloud Evaluation
• Evaluation and feedback are central to interaction design
• In think aloud a task is given to a user and then he needs
to perform it using the prototype.
– He start with easy to accomplish tasks and then switch to more
complex ones.
• During the evaluation, we try to ask him to tell us what is
he doing and why, we want to understand his mental
model.

3.3. Evaluation 14
 Roles for evaluation:
• Computer
– Simulates prototype, makes it alive.
– Doesn’t give any feedback that the computer wouldn’t.
• Facilitator
– Presents interface and tasks to the user
– Encourages user to think aloud by asking questions
– Keeps user test from getting off track
• Observer
– Keeps mouth shut, only takes notes

3.3. Evaluation 15
 What to test with a
prototype?
• Conceptual model
– Do users understand it?
• Functionality
– Does it do whats needed? Missing features?
• Navigation & task flow
– Can users find their way around?
• Terminology
– Do users understand labels?
• Screen contents
– What needs to go on the screen?

3.3. Evaluation 16
 Think Aloud

3.3. Evaluation 17
 More examples:
• Guidelines on prototyping and testing from Google:
– https://www.youtube.com/watch?v=JMjozqJS44M
• Even games can be paper prototyped:
– https://www.youtube.com/watch?v=zJqiZrnJSuw&t=79s

3.3. Evaluation 18
 Hands On! On groups.
Test your prototype on one of your partners! (like in the
video)
– Depending on what you want to evaluate (conceptual model,
navigation, terms, etc.), write the list of questions that the
facilitator is going to ask to the users during the evaluation,
– Decide what are the tasks will be done by the users (at least one
task).
– Take notes on the issues identified (user thinks something
different to what you expected).
– Given your notes, what would you modify in your prototype?

3.3. Evaluation 19