Chapter 4

COGNITIVE ASPECTS
 Overview
• What is cognition?
• Why it is important to understand in HCI
• Describe how cognition has been applied
to interaction design
• Explain what are mental models and how
to elicit them
• Cover relevant theories of cognition
www.id-book.com
 What is cognition?
• Thinking, remembering, learning,
daydreaming, decision-making, seeing,
reading, talking, writing…
• Ways of classifying cognition at a higher
level:
▪ Experiential vs. reflective cognition (Norman,
1993)
▪ Fast vs slow thinking (Kahneman, 2011)

www.id-book.com
 Which involves fast vs slow
thinking?
• 2+2=
• 21 × 29 =
• What color eyes do you have?
• How many colors are there in the rainbow?
• How many months in the year have 31 days?
• What is the name of the first school you
attended?
www.id-book.com
How can understanding cognition help?

• Provides knowledge about what users can

and cannot be expected to do
• Identifies and explains the nature and
causes of problems that users encounter
• Provides theories, modeling tools,
guidance, and methods that can lead to
the design of better interactive products

www.id-book.com
 Cognitive processes
• Attention
• Perception
• Memory
• Learning
• Reading, speaking and listening
• Problem-solving, planning, reasoning and
decision-making
www.id-book.com
 Attention
• Selecting things on which to concentrate at a point in
time from the mass of stimuli around us
• Allows us to focus on information that is relevant to
what we are doing
• Involves audio and/or visual senses
• Focused and divided attention
▪ Enables us to be selective in terms of the mass of competing
stimuli, but limits our ability to keep track of all events
• Design recommendation
▪ Information at the interface should be structured to capture
users’ attention, for example, use perceptual boundaries
(windows), color, reverse video, sound, and flashing lights
www.id-book.com
Activity: Find the price for a double room
at the Quality Inn in Pennsylvania

www.id-book.com
Activity: Find the price of a double room at the
Holiday Inn in Columbia

www.id-book.com
 Activity
• Tullis (1987) found that the two screens
produced quite different results
▪ 1st screen: Took an average of 5.5 seconds to search
▪ 2nd screen: Took 3.2 seconds to search
• Why, since both displays have the same density of
information (31percent)?
• Spacing
▪ In the 1st screen, the information is bunched up
together, making it hard to search
▪ In the 2nd screen, the characters are grouped into
vertical categories of information making it easier

www.id-book.com
 Multitasking and attention
• Is it possible to perform multiple tasks without
one or more of them being detrimentally
affected?
• Multitasking can cause people to lose their train
of thought, make errors, and need to start over
• Ophir et al. (2009) compared heavy vs light
multitaskers
▪ Heavy multitaskers were more prone to being
distracted than those who infrequently multitask
▪ Heavy multitaskers are easily distracted and find it
difficult to filter irrelevant information
www.id-book.com
 Multitasking experiment
• Lotteridge et al. (2015) conducted another
study involving writing an essay under two
conditions: relevant or irrelevant
information
▪ Heavy multitaskers were easily distracted but
able to put this to good use if the distracting
sources were relevant to the task in hand
▪ Irrelevant information was found to impact
task performance negatively

www.id-book.com
 Multitasking at work
It is increasingly common for workers to
multitask
• For example, hospital workers have to attend
to multiple screens in an operating room that
provide new kinds of real-time information
• This requires clinician’s constant attention to
check if any data is unusual or anomalous
• Need to develop new attention and scanning
strategies

www.id-book.com
Is it OK to use a phone when
driving?

www.id-book.com
 No!
• Driving is very demanding
• Drivers are prone to being distracted
• There is a significant chance of causing accidents
• Drivers’ reaction times are longer to external events when talking
on the phone in a car (Caird et al., 2018)
• Drivers using their phones rely more on their expectations about
what is likely to happen next as conducting a conversation takes
up their attention
• Response time is slower to unexpected events (Briggs et al.,
2018)
• Drivers often try to imagine what the other person’s face is like−
the person to whom they are speaking
▪ Doing so competes with the processing resources needed to enable
them to notice and react to what is in front of them

www.id-book.com
Are hands-free phones safer to use
when driving?
• No, as same type of cognitive processing is
happening when talking
• The same thing happens when talking with front
seat passenger
▪ But both can stop in mid-sentence if a hazard is spotted
allowing the driver to switch immediately to the road
▪ So, it’s less dangerous talking to a front seat passenger
than a remote person
▪ A remote person on the end of a phone is not privy to
what the driver is seeing and will carry on the
conversation when there is a hazard
▪ This makes it difficult for the driver to switch all their
attention to the road

www.id-book.com
Design implications for attention
• Context: Make information salient when it needs
to be attended to at a given stage of a task
• Use techniques to achieve this:
▪ For example, color, ordering, spacing, underlining,
sequencing, and animation
• Avoid cluttering visual interfaces with too much
information
• Consider designing different ways to support
effective switching and returning to an interface
www.id-book.com
 Perception
• How information is acquired from the
world and transformed into experiences
• Obvious implication is to design
representations that are readily
perceivable, for instance:
▪ Text should be legible
▪ Icons should be easy to distinguish and
read

www.id-book.com
Is color contrast good? Find Italian

www.id-book.com
Are borders and white space better?
Find French

www.id-book.com
 Activity
• Weller (2004) found people took less time to
locate items for information that was grouped
▪ Using a border (2nd screen) compared with using
color contrast (1st screen)

• Some argue that too much white space on

web pages is detrimental to search process
▪ Makes it hard to find information

• Do you agree?

www.id-book.com
Activity: Which is the easiest to
read and why?

What is the time? What is the time?

What is the time? What is the time?

What is the time?

www.id-book.com
 Design implications
• Icons should enable users to distinguish their
meaning readily
• Bordering and spacing are effective visual ways
of grouping information
• Sounds should be audible and distinguishable
• Research proper color contrast techniques
when designing an interface:
▪ Yellow on black or blue is fine
▪ Yellow on green or white is a no-no
• Haptic feedback should be used judiciously
www.id-book.com
 Memory
• Involves recalling various kinds of knowledge that allow
people to act appropriately
▪ For example, recognizing someone’s face or remembering
someone’s name
• First encode and then retrieve knowledge
• We don’t remember everything−it involves filtering and
processing what is attended to
• Context is important as to how we remember (that is,
where, when, how, and so on)
• We recognize things much better than being able to recall
things
• We remember less about objects that we have
photographed than when we observe them with the naked
eye (Henkel, 2014)
www.id-book.com
 Processing in memory
• Encoding is first stage of memory
▪ Determines which information is attended to in the environment
and how it is interpreted

• The more attention paid to something…

• The more it is processed in terms of thinking
about it and comparing it with other knowledge…
• The more likely it is to be remembered
▪ For example, when learning about HCI, it is much better to
reflect upon it, carry out exercises, have discussions with others
about it, and write notes than just passively read a book, listen to
a lecture or watch a video about it

www.id-book.com
 Context is important
• Context affects the extent to which information
can be subsequently retrieved
• Sometimes it can be difficult for people to recall
information that was encoded in a different
context:
▪ “You are on a train and someone comes up to you
and says hello. You don’t recognize him for a few
moments, but then realize it is one of your neighbors.
You are only used to seeing your neighbor in the
hallway of your apartment building, and seeing him
out of context makes him difficult to recognize initially”

www.id-book.com
 Activity
• Try to remember the dates of your grandparents’
birthday
• Try to remember the cover of the last two books you
read
• Which was easiest? Why?
• People are very good at remembering visual cues
about things
▪ For instance, the color of items, the location of objects and
marks on an object
• They find it more difficult to learn and remember
arbitrary material
▪ For example, birthdays and phone numbers
www.id-book.com
 Recognition versus recall
• Command-based interfaces require users to
recall from memory a name from a possible
set of 100s of names
• Graphical interfaces provide visually-based
options (menus, icons) that users need only
browse through until they recognize one
• Web browsers provide tabs and history lists of
visited URLs that support recognition memory

www.id-book.com
The problem with the classic ‘7,+
or − 2’
• George Miller’s (1956) theory of how much
information people can remember
• People’s immediate memory capacity is very
limited to 7, + or − 2
• Has been applied in interaction design when
considering how many options to display
• But is it a good use of a theory in HCI?
• Is it helpful?
www.id-book.com
 When creating an interface, should the
designer…
• Present only 7 options on a menu
• Display only 7 icons on a tool bar
• Have no more than 7 bullets in a list
• Place only 7 items on a pull down menu
• Place only 7 tabs on the top of a website page?
• Not necessarily…

www.id-book.com
 The reason is…
• People can scan lists of bullets, tabs, and menu
items for the one they want
• They don’t have to recall them from memory,
having only briefly heard or seen them
• So you can have more than nine at the interface
▪ For instance, history lists of websites visited
• Sometimes a small number of items is good
▪ For example, smart watch displays
• Depends on task and available screen estate

www.id-book.com
 Personal Information management
Is a growing problem for many users:
• They accumulate a vast numbers of documents,
images, music files, video clips, emails, attachments,
bookmarks, and so forth
• Where and how to save them all; then remembering
what they were called and where to find them again
• Naming most common means of encoding them
• But can be difficult to remember, especially when
you have 10,000s
• How might such a process be facilitated taking into
account people’s memory abilities?

www.id-book.com
Personal Information management
• Bergman and Whittaker, three interdependent
processes model (2016) to help people manage their
stuff:
I. How to decide what stuff to keep
II. How to organize it when storing
III. Which strategies to use to retrieve it later
• Most common approach is to use folders and
naming
• Strong preference for scanning across and
within folders when looking for something
• Search engines only helpful if you know the
name of the file
• Smart search engines help with listing relevant
files for partial name or when type in first letter
www.id-book.com
Apple’s Spotlight search tool

www.id-book.com
 Memory load
• Online/mobile and phone banking now require users to
provide multiple pieces of information to access their
account
▪ For instance, ZIP code, birthplace, a memorable date, first
school attended
▪ Known as multifactor authentication (MFA)
• Why?
▪ Increased security concerns
• Password managers, such as LastPass, have been
developed that require only one master password
▪ Reduces stress and memory load on users
• Passwords could become extinct with the widespread
use of biometrics and computer vision algorithms

www.id-book.com
 Digital Forgetting
• When might you wish to forget something that
is online?
▪ When you break up with a partner
▪ Emotionally painful to be reminded of them
through shared photos, social media, and so on.
• Sas and Whittaker (2013) suggest ways of
harvesting and deleting digital content
▪ For example, making photos of ex into an abstract
collage
▪ Helps with closure

www.id-book.com
 Memory aids
• SenseCam, developed by Microsoft Research
Labs (now Autographer)
▪ A wearable device that intermittently takes photos
without any user intervention while worn
▪ Digital images taken are stored and revisited using
special software
▪ Has been found to improve people’s memory,
especially those suffering from dementia
• Other aids include RemArc, which triggers
long-term memory using old BBC materials

www.id-book.com
SenseCam

www.id-book.com
 Design implications
• Reduce cognitive load by avoiding long
and complicated procedures for carrying
out tasks
• Design interfaces that promote recognition
rather than recall
• Provide users with various ways of
labelling digital information to help them
easily identify it again
▪ For example, folders, categories, color,
flagging, and time stamping
www.id-book.com
 Learning
• Involves the accumulation of skills and
knowledge involving memory
• Two main types:
▪ Incidental learning (for example, recognizing people’s faces, what
you did today)
▪ Intentional learning (for instance, studying for an exam, learning
to cook)
▪ Intentional learning is much harder!
▪ Many technologies have been developed to help (for example,
multimedia, animations, VR)

• People find it hard to learn by following

instructions in a manual
• People prefer to learn by doing

www.id-book.com
 Design implications
• Design interfaces that encourage
exploration
• Design interfaces that constrain and guide
learners
• Dynamically linking concepts and
representations can facilitate the learning
of complex material

www.id-book.com
 Reading, speaking, and listening
The ease with which people can read, listen,
or speak differs:
• Many prefer listening to reading
• Reading can be quicker than speaking or
listening
• Listening requires less cognitive effort than
reading or speaking
• Dyslexics have difficulties understanding
and recognizing written words
www.id-book.com
 Applications
• Voice user interfaces allow users to interact with
them by asking questions
▪ For example, Google Voice, Siri, and Alexa

• Speech-output systems use artificially-generated

speech
▪ For instance, written text-to-speech systems for the
visually impaired
• Natural-language systems enable users to type
in questions and give text-based responses
▪ Such as, chatbots

www.id-book.com
 Design implications
• Speech-based menus and instructions
should be short
• Accentuate the intonation of artificially
generated speech voices
▪ They are harder to understand than human
voices

• Provide opportunities for making text large

on a screen
www.id-book.com
 Problem-solving, planning,
reasoning, and decision-making
• All these processes involve reflective
cognition
▪ For example, thinking about what to do, what the options
are, and the consequences
• Often involves conscious processes,
discussion with others (or oneself), and the
use of artifacts
▪ Such as maps, books, pen and paper
• May involve working through different
scenarios and deciding which is best option
• Weighing up alternatives
www.id-book.com
 Design implications
• Provide information and help pages that
are easy to access for people who wish to
understand more about how to carry out
an activity more effectively (for example,
web searching)
• Use simple and memorable functions to
support rapid decision-making and
planning

www.id-book.com
 Dilemma
• The app mentality is making it worse for people to
make their own decisions because they are
becoming risk averse (Gardner and Davis, 2013)
▪ Instead, they now rely on a multitude of apps
▪ This makes them increasingly anxious
▪ They are unable to make decisions by themselves
▪ They need to resort to looking up info, getting other’s
opinions on social media, and comparing notes
• Do you agree?
• Did it happen to you when deciding which
university/school to attend?
www.id-book.com
 Cognitive frameworks
• These are used to explain and predict user
behavior at the interface
▪ Based on theories of behavior
▪ Focus is on mental processes that take place
▪ Also use of artifacts and representations
• Most well known are:
▪ Mental models
▪ Gulfs of execution and evaluation
▪ Distributed cognition
▪ External and embodied cognition

www.id-book.com
 Mental models
• Users develop an understanding of a
system through learning about and using it
• Knowledge is sometimes described as a
mental model:
• How to use the system (what to do next)
• What to do with unfamiliar systems or
unexpected situations (how the system works)
• People make inferences using mental
models of how to carry out tasks
www.id-book.com
 More mental models
• Craik (1943) described mental models as:
▪ Internal constructions of some aspect of the
external world enabling predictions to be made
• Involves unconscious and conscious
processes
▪ Imagery and analogies are activated
• Deep versus shallow models
▪ For example, how to drive a car and how it works

www.id-book.com
 Everyday reasoning and mental models

(a) You arrive home on a cold winter’s night to a cold

house. How do you get the house to warm up as
quickly as possible? Set the thermostat to be at
its highest or to the desired temperature?
(b) You arrive home starving hungry. You look in the
fridge and find all that is left is an uncooked
pizza. You have an electric oven. Do you warm it
up to 375 degrees first and then put it in (as
specified by the instructions) or turn the oven up
higher to try to warm it up quicker?

www.id-book.com
 Heating up a room or oven that is
thermostat-controlled
• Many people when asked (a) choose the first option
• Why?
▪ They think it will heat the room up quicker
▪ General valve theory, where ‘more is more’ principle is
generalized to different settings (for instance, gas pedal, gas
cooker, tap, radio volume)
▪ But it is a wrong mental model for thermostats based on on-off
switch model
• Many people when asked (b) choose the first option
▪ Electric ovens work on the same principle as thermostats
• Most of us have erroneous mental models (Kempton,
1996)

www.id-book.com
 Erroneous mental models
• Lots of people hit the button for elevators
and pedestrian crossings at least twice
▪ Why? Think it will make the lights change faster or
ensure that the elevator arrives!

• What kinds of mental models do users have

for understanding how interactive devices
work?
▪ Poor, often incomplete, easily confusable, based on
inappropriate analogies and superstition (Norman,
1983)

www.id-book.com
 How can UX be designed to help
people build better mental models?
• Clear and easy to use instructions
• Appropriate tutorials and contextual
sensitive guidance
• Provide online videos and chatbot
windows when needing help
• Transparency: to make interfaces intuitive
to use
• Affordances of what actions an interface
allows
▪ For example, swiping, clicking, or selecting

www.id-book.com
 Gulfs of execution and evaluation
• The ‘gulfs’ explicate the gaps that exist between the
user and the interface
• The gulf of execution
▪ The distance from the user to the physical system
• The gulf of evaluation
▪ The distance from the physical system to the user

• Bridging the gulfs can reduce cognitive effort required

to perform tasks
• Can reveal whether interface increases or decreases
cognitive load and whether it is obvious what to do
next (Norman, 1986; Hutchins et al, 1986)

www.id-book.com
Bridging the gulfs

www.id-book.com
 Information processing
• Conceptualizes human performance in
metaphorical terms of information
processing stages

www.id-book.com
 Limitations
• Based on modeling mental activities that
happen exclusively inside the head
• Do not adequately account for how people
interact with computers and other devices
in real world

www.id-book.com
 Distributed cognition
• Concerned with the nature of cognitive
phenomena across individuals, artifacts,
and internal and external representations
(Hutchins, 1995)
• Describes these in terms of propagation
across representational state
• Information is transformed through
different media (computers, displays,
paper, heads)

www.id-book.com
A cognitive system for ATC

www.id-book.com
 What’s involved
• The distributed problem-solving that takes
place
• The role of verbal and non-verbal behavior
• The various coordinating mechanisms that are
used (for example, rules and procedures)
• The communication that takes place as the
collaborative activity progresses
• How knowledge is shared and accessed

www.id-book.com
 External cognition
• Concerned with explaining how we interact with
external representations (such as maps, notes,
and diagrams)

• What are the cognitive benefits and what

processes involved

• How they extend cognition

• What technologies can we develop to help

people carry out complex tasks (for example,
learning, problem solving, and
decision-making)?
www.id-book.com
 Externalizing to reduce
memory load
• Examples include the use of diaries, reminders,
calendars, notes, shopping lists, to-do lists
▪ Written to remind us of what to do
• Post-its, piles, marked emails are used to:
▪ Where placed indicates priority of what to do
• External representations:
▪ Remind us that we need to do something (for example,
to buy something for mother’s day)
▪ Remind us of what to do (for instance, buy a card)
▪ Remind us when to do something (for example, send a
card by a certain date)

www.id-book.com
 Computational offloading
• When a tool is used in conjunction with an
external representation to carry out a
computation (for instance, pen and paper)
• Try doing the two sums below (a) in your
head, (b) on a piece of paper, and (c) with a
calculator.
234 ×456 = ??
CCXXXIIII×CCCCXXXXXVI = ???
• Which is easiest and why? Both are identical
sums

www.id-book.com
 Annotation and cognitive tracing
• Annotation involves modifying existing
representations through making marks
▪ For example, crossing off, ticking, and underlining

• Cognitive tracing involves externally

manipulating items into different orders or
structures
▪ For instance, playing Scrabble or cards

www.id-book.com
 Design implication
• Provide external representations at the
interface that can reduce memory load
and facilitate computational offloading
▪ For example, information visualizations
have been designed to allow people to
make sense and rapid decisions about
masses of data

www.id-book.com
 Embodied Interaction
• The practical engagement with the social and physical
environment (Dourish, 2001)
• Creating, manipulating and making meaning through our
interaction with things
• How our bodies and active experiences shape how we
perceive, feel, and think (Hornecker et al., 2017)
• They enable us to develop a sense of the world at both a
concrete and abstract level
• Can provide new ideas about interaction and better design
principles
▪ For example, we think with our bodies not through them (Kirsh,
2013)

www.id-book.com
 Summary
• Cognition involves many processes including
attention, memory, perception, and learning
• The way an interface is designed can greatly affect
how well users can perceive, attend, learn, and
remember how to do their tasks
• Theoretical frameworks, such as mental models
and external cognition, provide ways of
understanding how and why people interact with
products
• This can lead to thinking about how to design
better products

www.id-book.com
 In-depth activity
Write down how you think a contactless card or
smartphone app like Apple Pay works
• What information is sent between the card/smartphone
and the card reader when it is placed in front of it?
• What is the maximum amount you can pay for something
using a contactless card, Apple Pay or Google Pay?
• Why is there an upper limit?
• How many times can you use a contactless card or
Apple/Google Pay in a day?
• What happens if you have two contactless cards in the
same wallet/purse?
• What happens when your contactless card is stolen and
you report it to the bank? What does the bank do?
www.id-book.com