Chapter 7
INTERFACES
20 interface types covered
1. Command
2. Graphical
3. Multimedia
4. Virtual reality
5. Web
6. Mobile
7. Appliance
8. Voice
9. Pen
10. Touch
11. Gesture
12. Haptic
13. Multimodal
14. Shareable
15. Tangible
16. Augmented Reality
17. Wearables
18. Robots and drones
19. Brain–computer interaction
20. Smart
www.id-book.com 2
11. Gesture-based systems
• Gestures involve moving arms and hands to communicate
• Uses camera recognition, sensor, and computer vision
techniques
▪ Recognize people’s arm and hand gestures in a room
▪ Gestures need to be presented sequentially to be understood
(compare with the way sentences are constructed)
• Inspirations of gesture-based systems: cricket umpires, live
concert signers for the deaf, rappers, Charlie Chaplin, mime
artists, Italians, more : https://vimeo.com/224522900
• HCI Application inspired by driving hand gestures :
https://www.youtube.com/watch?v=Ir3ViXwtvyI
www.id-book.com 3
Gestures used in the operating theater
Recognizes core gestures for manipulating MRI or CT images using Microsoft Kinect
www.id-book.com 4
Research and design considerations
• How does computer recognize and
delineate user’s gestures?
▪ Start and end points?
▪ Difference between deictic (a deliberate
pointing movement) and hand waving (an
unconscious gesticulation)
• How realistic must the mirrored graphical
representation of the user be in order for
them to be believable?
www.id-book.com 5
12. Haptic interfaces
• Provide tactile feedback
▪ By applying vibration and forces to a person’s body,
using actuators that are embedded in their clothing or
a device they are carrying, such as a smartphone
• Vibrotactile feedback can be used to simulate the
sense of touch between remote people who want
to communicate
• Ultrahaptics creates the illusion of touch in midair
using ultrasound to make the illusion of 3D shapes
www.id-book.com 6
Realtime vibrotactile feedback from
MusicJacket
• Provides nudges when
playing violin incorrectly
• Uses motion capture to
sense arm movements that
deviate from model
• Nudges are short vibrations
on arms and hands
www.id-book.com 7
Exoskeleton with artificial muscles that
uses bubble haptic feedback
www.id-book.com 8
Research and design considerations
• Where best to place actuators on body
• Whether to use single or sequence of ‘touches’
• When to buzz and how intense - continuous and
intermittent buzzing.
• How does the wearer feel it in different contexts?
• What kind of new smartphone/smartwatch apps
can use vibrotactile creatively?
▪ For example, slow tapping to feel like water drops
meant to indicate that it is about to rain, and heavy
tapping to indicate a thunderstorm is looming
www.id-book.com 9
13. Multimodal Interfaces
• Provide enriched user experiences
▪ By multiplying how information is experienced and detected
using different modalities, such as touch, sight, sound, and
speech
▪ Support more flexible, efficient, and expressive means of
human-computer interaction
▪ Most common is speech and vision
• Can be combined with multi-sensor input to
enable other aspects of the human body to be
tracked
▪ For example, eye gaze, facial expression, and lip movements
▪ Provides input for customizing user interfaces
www.id-book.com 10
Tracking a person’s movements
• Kinect camera can detect multimodal input in real time using RGB camera for facial
recognition and gestures, depth camera for movement tracking, and microphones
for voice recognition
• Used to build model of person and represented as avatar on display programmed to
move just like them
www.id-book.com 11
Research and design considerations
• Need to recognize and analyze user behavior,
for example, speech, gesture, handwriting, or
eye gaze
• Much harder to calibrate these than single
modality systems
• What is gained from combining different input
and outputs
• Is talking and gesturing, as humans do with
other humans, a natural way of interacting
with a computer?
www.id-book.com 12
14. Shareable interfaces
Designed for more than one person to use:
• Provide multiple inputs and sometimes allow
simultaneous input by co-located groups
• Large wall displays where people use their own
pens or gestures
• Interactive tabletops where small groups interact
with information using their fingertips
• For example, DiamondTouch, Smart Table, and Surface
DiamondTouch : https://www.youtube.com/watch?v=S9QRdXlTndU
www.id-book.com 13
A smartboard and an interactive
tabletop interface
DiamondTouch :
https://www.youtube.com/wat
ch?v=S9QRdXlTndU
Mitsubishi’s interactive tabletop
www.id-book.com 14
Benefits
• Provide a large interactional space that can
support flexible group working
• Can be used by multiple users
▪ Can point to and touch information being
displayed
▪ Simultaneously view the interactions and have the
same shared point of reference as others
• Can support more equitable participation
compared with groups using single PC
www.id-book.com 15
Research and design considerations
• Core design concerns include whether size, orientation, and
shape of the display have an effect on collaboration
• Horizontal surfaces compared with vertical ones support
more turn-taking and collaborative working in co-located
groups
• Providing larger-sized tabletops does not improve group
working but encourages more division of labor
• Having both personal and shared spaces enables groups to
work on their own and in a group
▪ Cross-device systems have been developed to support seamless
switching between these, for example, SurfaceConstellations
www.id-book.com 16
15. Tangible Interfaces
• Type of sensor-based interaction, where physical
objects, for example, bricks, are coupled with digital
representations
• When a person manipulates the physical object/s, it
causes a digital effect to occur, for example, an
animation
• Digital effects can take place in a number of media and
places, or they can be embedded in the physical object
Video:
https://www.youtube.com/watch?v=guhhcg3f25U&list=PLO15NdIKIIJX7
RHrSihVcTA--K_HIcstD
www.id-book.com 17
Examples
• Flow Blocks
▪ Depict changing numbers and lights embedded in the
blocks
▪ Vary depending on how they are connected together
• Urp
▪ Physical models of buildings moved around on tabletop
▪ Used in combination with tokens for wind and shadows
Digital shadows surrounding them to change over time
• MagicCubes
▪ Connect physical electronic components and sensors to
make digital events occur (for example, change color
depending on how much shaken)
www.id-book.com 18
Learning to code and create with the
tangible MagicCubes
www.id-book.com 19
Benefits
• Can be held in one or both hands and combined and
manipulated in ways not possible using other interfaces
▪ Allows for more than one person to explore the interface
together
▪ Objects can be placed on top of each other, beside each other,
and inside each other
▪ Encourages different ways of representing and exploring a
problem space
• People are able to see and understand situations
differently
▪ Can lead to greater insight, learning, and problem-solving than
with other kinds of interfaces
▪ Can facilitate creativity and reflection
www.id-book.com 20
VoxBox
A tangible system that gathers opinions at events through playful
and engaging interaction (Goldsteijn et al., 2015)
https://dl.acm.org/doi/10.1145/2677199.2680588
www.id-book.com 21
Research and design considerations
• What kinds of conceptual frameworks to use to help identify novel
and specific features
• What kind of coupling to use between the physical action and
digital effect
▪ If it is to support learning, then an explicit mapping between action and
effect is critical
▪ If it is for entertainment, then it can be better to design it to be more implicit
and unexpected
• What kind of physical artifact to use
▪ Bricks, cubes, and other component sets are most commonly used because
of flexibility and simplicity
▪ Stickies and cardboard tokens can also be used for placing material onto a
surface
• With what kinds of digital outputs should tangible interfaces be
combined?
www.id-book.com 22
16. Augmented Reality
• Augmented reality: Virtual representations are
superimposed on physical devices and objects
• Pokémon Go made it a household game
▪ Used smartphone camera and GPS to place virtual
characters onto objects in the environment as if
they really are there
• Many other applications including medicine,
navigation, air traffic control, games, and
everyday exploring
www.id-book.com 23
Other examples
In medicine
• Virtual objects, for example, x-rays and scans, are
overlaid on part of a patient’s body
• Aid the physician’s understanding of what is being
examined or operated
In air traffic control
• Dynamic information about aircraft overlaid on a
video screen showing the real planes, and so on
landing, taking off, and taxiing
• Helps identify planes difficult to make out
• https://www.youtube.com/watch?v=92YKZVlsrzg
www.id-book.com 24
Augmented reality overlay
on a car windshield
www.id-book.com 25
AR that uses forward facing camera
• Enables virtual try-ons (for example, Snapchat filters)
• AR mirrors set up in retail stores for trying on make-up,
sunglasses, jewelry
▪ Convenient, engaging, and easy to compare more choices
▪ But cannot feel the weight, texture, or smell of what is being
tried on
• Can be used to enable users to step into a character
(for example, David Bowie, Queen Victoria)
www.id-book.com 26
Singers trying on the virtual look of two
characters from the opera Akhnaten
www.id-book.com 27
Research and design considerations
• What kind of digital augmentation?
▪ When and where in physical environment?
▪ Needs to stand out but not distract from ongoing
task
▪ Needs to be able to align with real world objects
▪ What happens if the AR is slightly off?
• What kind of device?
▪ Smartphone, tablet, head up display or other?
www.id-book.com 28
17. Wearables
• First developments were head- and
eyewear-mounted cameras that enabled user
to record what was seen and to access digital
information
• Since then, jewelry, head-mounted caps,
smart fabrics, glasses, shoes, and jackets have
all been used
▪ Provides the user with a means of interacting with
digital information while on the move
• Applications include automatic diaries, tour
guides, cycle indicators, and fashion clothing
(electroluminescent embroidery)
www.id-book.com 29
Google Glass (2014)
Why was there so much excitement and concern about people filming what they
could see right in front of them?
https://www.youtube.com/watch?v=4EvNxWhskf8
www.id-book.com 30
Research and design considerations
• Comfort
▪ Needs to be light, small, not get in the way, fashionable,
and preferably hidden in the clothing
• Hygiene
▪ Is it possible to wash or clean the clothing once worn?
• Ease of wear
▪ How easy is it to remove the electronic gadgetry and
replace it?
• Usability
▪ How does the user control the devices that are embedded
in the clothing?
www.id-book.com 31
18. Robots
Main types
• Remote robots used in hazardous settings
▪ Can be controlled to investigate bombs and other dangerous
materials
• Domestic robots helping around the house
▪ Can pick up objects and do daily chores like vacuuming
• Pet robots as human companions
▪ Have therapeutic qualities, helping to reduce stress and
loneliness
• Sociable robots that work collaboratively with humans
▪ Encourage social behaviors
www.id-book.com 32
Social robots: Mel and Paro
• Cute and cuddly
• Can open and close eyes and make sounds and
movements
Source: Images courtesy of Mitsubishi Electric Research Labs (Mel and Paro).
www.id-book.com 33
Drones
• Unmanned aircraft that are controlled remotely and
used in a number of contexts
▪ For example, entertainment, such as carrying drinks and
food to people at festivals and parties
▪ Agricultural applications, such as flying them over
vineyards and fields to collect data about crops, which is
useful to farmers
▪ Helping to track poachers in wildlife parks in Africa
• Can fly low and and stream photos to a ground station
where images can be stitched together into maps
• Can be used to determine the health of a crop, or
when it is the best time to harvest the crop
www.id-book.com 34
Drone being used to survey
the state of a vineyard
www.id-book.com 35
Research and design considerations
• How do humans react to physical robots designed to exhibit
behaviors (for example, making facial expressions) compared
with virtual ones?
• Should robots be designed to be human-like or look like and
behave like robots that serve a clearly-defined purpose?
• Should the interaction be designed to enable people to
interact with the robot as if it was another human being or
more human-computer-like (for example, pressing buttons to
issue commands)?
• Is it acceptable to use unmanned drones to take a series of
images or videos of fields, towns, and private property
without permission or people knowing what is happening?
www.id-book.com 36
19. Brain-computer interfaces
• Brain-computer interfaces (BCI) provide a communication
pathway between a person’s brain waves and an external
device, such as a cursor on a screen
• Person is trained to concentrate on the task, for example,
moving the cursor
• BCIs work through detecting changes in the neural
functioning in the brain
• BCIs apps:
▪ Games (for example, Brain Ball)
▪ Enable people who are paralyzed to control robots
www.id-book.com 37
A brain-computer interface being used by a
woman who is paralyzed to select letters
on the screen
www.id-book.com 38
20. Smart interfaces
• Smart: phones, speakers, watches, cars, buildings, cites
• Smart refers to having some intelligence and connected
to the internet and other devices
• Context-aware
▪ Understand what is happening around them and
execute appropriate actions, for example, a Nest
thermostat
• Human-building interaction
▪ Buildings are designed to sense and act on behalf of
the inhabitants but also allow them to have some
control and interaction with the automated systems
www.id-book.com 39
Which interface?
• Which interface to use will depend on task, users, context,
cost, robustness, and so on
• Is multimedia better than tangible interfaces for learning?
• Is speech as effective as a command-based interface?
• Is a multimodal interface more effective than a mono-modal
interface?
• Will wearable interfaces be better than mobile interfaces for
helping people to find information in foreign cities?
• Are virtual environments the ultimate interface for playing games?
• Are shareable interfaces better at supporting communication and
collaboration compared with using networked desktop PCs?
www.id-book.com 40
Summary
• Many innovative interfaces have emerged in last 30
years, including speech, wearable, mobile, brain, and
tangible
• This raises many design and research questions as to
decide which to use
▪ For example, how best to represent information to the
user so that they can carry out ongoing activity or task
• New smart interfaces that are context-aware and
monitor people
▪ Raising new ethical issues concerned with what data is
being collected and what it is used for
www.id-book.com 41
