How can we use AI to make

things better for humans?

ideo.to/ai
How can we use AI to make
things better for humans?
We believe that data science and machine learning can
help us design intelligent products, services, and systems
that improve people’s everyday lives.

But in order to have a truly positive impact, AI-powered

technologies must be grounded in human needs and work
to extend and enhance our capabilities, not replace them.

To call attention to that distinction, we use the term

“augmented intelligence” rather than “artificial intelligence.”
Data science is a tool that helps us build a smarter world,
but humans remain the architects.
How might we use
augmented intelligence
to solve dynamic, complex,
and evolving systemic
challenges in a human-
centered way?

ideo.to/ai
About These Cards
These cards aim to help interdisciplinary teams remain
human-centered while designing data-driven, intelligent
products, services, and systems. These new systems can
adapt to better achieve their intent over time, which presents
both familiar and unfamiliar challenges for design—familiar
because human-centered design has long sought to augment
humans’ capabilities by helping them solve problems;
unfamiliar because intelligent systems have the potential for
such massive scale that humans can no longer evaluate at
every phase.

IDEO joins this ongoing conversation from a place

of optimism. The cards reflect IDEO’s effort to illuminate
common challenges when designing with data. Awareness
of our own human tendencies is the first step in thinking
about ethical considerations. The following exercises help
ensure that the work is more ethically responsible, culturally
considerate, and humanistic. The activities do not offer
definitive direction, but rather initiate a set of actions
and prompts to stimulate conversations. They are tools
to be used and modified as you see fit.
Collaboration Is Key
When tackling the challenges that data can help
answer, it is often best to lean on your teammates.

All team members should be empowered to trust

their instincts and raise this Pause flag (found on the
other side of this card) at any point if a concept or
feature does not feel human-centered, even if they can’t
quite put their finger on why. It’s important to discuss
issues early, before project momentum leads the team
astray. There is a good chance someone else is having
similar thoughts and these conversations will help align
the team.
Design
Principles

ideo.to/ai
Design Principles

Don’t Presume the

Data is Not Truth Desirability of AI

Data is human-driven. Humans Just because AI can do something

create, generate, collect, capture, doesn’t mean that it should. When
and extend data. The results are AI is incorporated into a design,
often incomplete, and the process designers should continually
of analyzing them can be messy. pay attention to whether people’s
Data can be biased through what needs are changing, or an AI’s
is included or excluded, how it is behavior is changing.
interpreted, and how it is presented.
Unpacking the human influence on
data is essential to understanding
how it can best serve our needs.

Respect Privacy and the Unintended Consequences of AI

Collective Good are Opportunities for Design

While there are policies and laws Just as with any design endeavor,
that shape the governance, we know that we’re not going to
collection, and use of data, we get it right the first time. Use
must hold ourselves to a higher unanticipated consequences and
standard than “will we get sued?” new unknowns as starting points
Consider design, governance of for iteration.
data use for new purposes, and
your communication of how
people’s data will be used.
When to Use
These Cards

ideo.to/ai
When to Use These Cards
There are four primary use cases for when these cards will be of particular help.

Project Onset Early Prototyping

To help us think about what data To help teams think through

we are gathering (and deriving) how people may interact
and how it affects what we with a design and set them
design. What is missing? Is it up for successful user testing
representative? Is it biased? and concept evaluations.
How might users feel about
intelligent systems that learn
and change over time? How
much transparency should
be built into the design?

Decision and Planning Moments Anytime Exercises

To think about how proposed Don’t feel limited by these

approaches and solutions might moments. Many cards contain
affect the people they’re trying to exercises that should make the
serve. What are the potential future team feel confident that they
use cases for the output? What are are making good decisions.
the long term effects, and how will The cards offer prompts to
it change as system learns? help us remember the people
behind the numbers and
those who will encounter
the system we’re designing.
Blind Spots
Check
ideo.to/ai
 Blind Spots Check

How might we reduce

biases in our data?
Diversity is essential to ensuring that our learning and algorithms
are robust, and the product is culturally sensitive and inclusive.
By thinking about inclusion (and exclusion) in the data, we can
better uncover biases and begin anticipating consequences.

ACTIVITIES TO TRY

List what data you currently have and create another list with
examples of data needed to create a more complete picture.
What additions will make your data more representative of
the entire population or context?

List what information you have directly and what is a proxy.

Challenge yourself to explore whether the proxy has built-in
assumptions or biases that may impact what you’re capturing.

IN PRACTICE

An international hospitality brand wanted to know why potential

customers were abandoning their online travel booking before
completing their purchases. Company leaders believed that a better
interface would solve the issue. However, research revealed that
travelers preferred to speak to a representative by phone in order
to customize and plan their trip. This insight illuminated a blind
spot—the call center systems, which were antiquated, complex,
and didn’t “speak” to each other. By identifying and fixing the
blind spot, the company increased bookings.
The People
Behind
the Data
ideo.to/ai
 The People Behind the Data

Humanize the stories

behind the numbers
Sometimes qualitative research helps us understand what
additional data we need. Other times, reviewing data sets first
helps focus our ethnographic research. Most design challenges
benefit from a combination of both big and small data. Use this
to your advantage—talk to the people behind the numbers.
A human story alongside your data creates empathy.

ACTIVITIES TO TRY
Richly characterize some of the actual people whose stories
are being represented in the data. Discuss what could explain the
patterns, both typical and extreme, that you are seeing. What are
the underlying behaviors, events, or mitigating circumstances?
This probing can help you determine what questions to ask next.

“Translate” a few rows or columns of data into human stories —

especially when those stories highlight a range of experiences.
Give each story humanizing touches taken from actual stories
you heard in the field. What new considerations does this high-
light? Remember that you’re designing for a person, not a number.

IN PRACTICE

A large equipment company wanted to explore the tradeoffs between

usage-based repair and time-based. Rather than present graphs and
charts to show the difference between the two approaches, the team
created two contrasting stories demonstrating what happened in each
case. One told of an operator who overspent on parts that were replaced
before new ones were needed. In the other scenario, the operator had
to send his entire team home due to a part failure. Adding these simple,
relatable stories helped everyone in the room understand the real-world
implications of a data-driven solution.
Capturing
Minimum
Viable Data
ideo.to/ai
 Capturing Minimum Viable Data

Limit the amount of data

to gather and retain
Rather than collecting all the data we could, minimize privacy
concerns and maintain trust by capturing only information we
need and retaining it only as long as we need it. Ask yourself
how much detail you actually need to accomplish your project.
Do you need to sense who is in a room, or simply that someone
is there? Do you need to know what someone said, or just the
tone and volume of their voice?

ACTIVITIES TO TRY

List what data you are thinking about gathering and clarify why
you’re collecting each item and for how long it would be needed.
As a team, determine if there is less specific information you could
gather to get what you need and how you might be able to retain
it for as little time as possible. The European Union’s GDPR
regulations compel this kind of analysis.

Imagine that a close and skeptical relative is one of your users.

How would you explain what you are collecting, what you are
going to do with it, and why? Share these explanations with
potential users to get their feedback. Voicing concerns aloud
reveals weaknesses in your thinking that might otherwise go
unnoticed.

IN PRACTICE

A team designing an in-home security system was charged

with making it easier to use, while also offering a greater sense
of protection. Users wanted the system to monitor activity in
and around the home but also preserve privacy. To understand
the line between protection and privacy, the team mapped out
all human interactions that occur daily at home. This allowed
them to pinpoint times when visual recording was critical for
protection and when it wasn’t. Inspired by analog camera
shutters, the team designed a “privacy shutter” to open and
close automatically at specific points.
Mapping
the System
ideo.to/ai
 Mapping the System

Create and share a big-picture

view of how data flows
All data is shaped by humans; we decide what to collect, how to capture
it, and how to use it. Having a precise knowledge of any data’s journey—
its origin, path, and how it may have been transformed—can reveal
ambiguities, redundancies, inconsistencies, biases, gaps, and other
opportunities for design. Crucially, it also helps uncover key pieces
of the picture that can lead to unexpected (and often vital) insights.

ACTIVITIES TO TRY
On a whiteboard, draw a large-scale map of what you already
know about the part of the system you are interested in. Include
the big elements you’re aware of and leave plenty of whitespace.
Use Post-its to layer on details such as people, roles, tools, events,
and transformations. Then identify knowledge gaps and consider
how to fill them in. For example, you might ask more probing
questions of users, observe specifics more closely, role play how
the system works, or ask experts to fill in parts that are missing.

Recruit participants who have roles in the system. Ask each person
to walk you through their role and how they perform it. Pay close
attention to the tools and information they use and how that
information is captured. Map your findings to a timeline so you can
see the overlapping contributions of all stakeholders at a glance.

IN PRACTICE

A call center wanted to improve the customer service experience. Through

interviews, the design team heard that poor customer service was leading
callers to give up before fulfilling the purpose of their call. Design team
members observed calls and noticed that customers had to repeat them-
selves often and agents had trouble finding the information they should
have had easy access to. The team mapped all the people and data in the
system, charting the flow of information. Seeing pain points and bottle-
necks helped the team design a new experience in which agents had easy
and intuitive access to information and customers had less need to repeat
themselves, leading to higher satisfaction for everyone.
Monitoring
to Correct
Course
ideo.to/ai
 Monitoring to Correct Course

Mitigate potential
negative outcomes
Intelligent systems are dynamic, constantly learning from information
being generated and touched by humans. They do not exist in isolation;
they connect to and reflect the ever-changing contexts around them.
This means that over time, designs may veer off course. Good intentions
aren’t enough. Teams need to explore, identify, and properly communicate
the dependencies within a system so they can monitor its progress
as it learns.

ACTIVITIES TO TRY
Describe two future scenarios—one best-case and one worst-case—
that could impact your system through changes in input signals
and/or the humans participating. Don’t spend a lot of time working
on accuracy or likelihood of outcomes. The intention is to define
extreme contexts to explore different possible evolutionary paths.

What metrics could help you understand that your design is having
unintended consequences? Select a key set of metrics to help you
monitor progress early and often. As a team, explore how you can
build in safeguards, redundancies, or alerts to signal your design is
no longer acting as intended.

IN PRACTICE

A company in the service industry wanted to leverage millions

of customer data records to improve the purchasing experience.
Based on interviews with customers and sales agents, the design
team defined the different categories of information that these
agents used and wished to have at their fingertips. The team
conceptualized a new software interface that could surface multiple
customer data points at once, including past spending totals. But
a review of the new interface revealed that showing such financial
data put customers at a disadvantage if they were talking to an
aggressive salesperson. A sales agent could use a customer’s
average spending amount as a proxy for wealth, and be tempted
to sell a more expensive service. In the end, financial data was
removed from the new interface.
Respecting
Culture
ideo.to/ai
 Respecting Culture

How might we ensure

that our design is sensitive
to people’s contexts?
Machine learning and artificial intelligence systems are usually
constructed to provide efficiency and proficiency. While people find
these attributes valuable in certain contexts, humans may prioritize
privacy, routine, politeness, etc. in others. Such human priorities will
be compromised unless designers take care to introduce appropriate
cultural and contextual considerations into the system.

ACTIVITIES TO TRY
Think about people’s real lives. How will our design impact
their activities? Are there cultural rules, rituals, or taboos that
the system will force people to confront? When might they
prefer not to engage and why? Can we create solutions that
accommodate and support them?

List some culturally appropriate behaviors for the population

that will interact with the design. How rigid are they, and what
happens if they are broken? How close to violating these does
the design come, and what can be done to avoid this?

IN PRACTICE

In 2007, Facebook launched Beacon. It gathered data about

users’ transactions with external commerce websites and posted
those activities on their wall. The idea was to promote merchants,
but these posts were made without the consent of the users,
sharing information they may not want have wanted to be
made public. One user who received calls of “congratulations”
after Beacon revealed he’d bought his girlfriend an engagement
ring, but before he had a chance to even propose to her.
Designing
the Seams
ideo.to/ai
 Designing the Seams

Clarify each moment in a

person’s relationship with
an intelligent system
In an intelligent system, a “seamless experience” may not be appropriate.
People need to know when they are about to cross a critical threshold—
like when the system transitions from autonomous decision-making to
requiring human intervention. Identifying these moments and providing
visible seams help build trust with the user.

ACTIVITIES TO TRY

Create a journey map of the system, highlighting what it is doing

and how it “understands” why it should take an action based on
external context. Explore how this map intersects with a human-
centered map of the user experience, and prototype ways
to minimize ambiguity between human and machine agency.

For each moment, list all the ways in which the intelligent system
can take action on behalf of users (direct and indirect). Discuss
where each action falls on this spectrum:

System knows best Decide together Person knows best

IN PRACTICE

In semi-autonomous vehicles, there are moments when the car

can safely control itself as well as moments when a human driver
must take the wheel. How might design best convey the vehicle’s
capabilities and limitations to drivers? The design team set up
a driving simulation to explore a variety of channels including
voice, sound, and lights. Observing how and why drivers react
to these signals revealed critical moments in which they wanted
to take control, even though the vehicle was capable of driving
autonomously.
Balance Giving
and Getting
ideo.to/ai
 Balance Giving and Getting

Provide reciprocal value to

people who share their data
In many cases, human systems rely on clear, fair exchanges of value
between people and a product or service (e.g. money for pizza).
Other systems may rely upon contributions from participants and
highlight evident in-kind benefits of collaboration (e.g., a potluck
dinner). But exchanges of value can be distorted when a system
is tracking behavior, monitoring clicks, and stockpiling personal
information, because people may never learn the true value of the
information they have contributed, and may not benefit from
the exchange.

ACTIVITIES TO TRY
List the information you’re receiving from people in one column
and the benefits that you are providing in another (anything from
a smoother experience, to honoraria, to promoting positive social
change). Is the give-get loop balanced? Are the benefits obvious
to people when they share their information?

Interview potential users to learn how fair your system feels

to them. You might develop a “trading game” (see In Practice
below) to determine a mutually agreeable exchange.

IN PRACTICE

A company that hosts thousands of public events wanted to improve the

attendee experience. During research, the team noticed that event staffers
spent an enormous amount of energy capturing attendee data. Attendees
complained about having to give personal information, but also said
they didn’t mind giving up information if they saw a benefit. So the team
created a game, turning data such as email addresses and credit card
numbers into “currency.” Playing the game, attendees “bought” benefits
they felt were most relevant to their needs. For example, one person
bought a free ride to the event by giving his name, email, address and
zip code as currency. This exercise allowed customers to learn the value
of their data, and helped the company better solve customers’ needs
without asking for too much.
Anticipating
Future
Use Cases
ideo.to/ai
 Anticipating Future Use Cases

Guard against undesirable

uses of data
We gather and use information about people and communities,
intending to do so for their benefit. But others may have different
agendas. Teams change, strategies pivot, and data may be sold
to other companies. Always guard against potential future harm.

ACTIVITIES TO TRY
Imagine that your database gets hacked. Or your company is subject
to a hostile takeover. Or that your assets are put up for sale after a
bankruptcy. List a few companies, governments, or groups that might
try to use these assets—for good or bad. What might they use the
data for? How might you change what you collect, how you store it,
or how you structure it to avoid these potential consequences?

List what, if any, data should be excluded, guarded, limited, or

specially governed within your service. How could you accomplish
this without jeopardizing the beneficial intent of your design?
For example, would it be easy to anonymize names to make it
more difficult to cross-reference with other sources?

IN PRACTICE

In 2018, the City of Chicago launched CityKey, a single card that could
serve as an ID, library card, and public transit farecard. The card was
specifically designed for residents of Chicago who might have a hard
time acquiring a driver’s license or state ID—including undocumented
immigrants. In designing the CityKey, the City Clerk’s office wanted
to avoid replicating the experience New York City had faced with its
own IDNYC card. New York City had to go to federal court to protect
the personal information contained in its applicant database, which
the Trump administration sought to use for immigration enforcement.
Unlike New York’s program, Chicago’s CityKey database does not
retain images of any documents, nor any personal information about
CityKey holders.
Considering
the System
ideo.to/ai
 Considering the System

Ensure human-centered
benefits for individuals,
communities, and humanity
Interdependencies within the systems we design affect everyone.
Where we direct our focus will determine these choices and influence
outcomes across all levels of interaction. It’s important that we zoom
between micro to macro levels.

ACTIVITIES TO TRY
Create a matrix to examine the potential human consequences of design
outcomes at multiple levels, positive and negative. The prompts in the
sample matrix below are not intended to be comprehensive.

Potential Human Benefits Potential Problems

Reciprocity Invasion of privacy
Robust information Misuse of data
Individuals Trusted guidance or decision-making Distrust
& Families Loyalty Loss of control/agency
Reduced cognitive load Erroneous decisions

Diversity, inclusion Bias or exclusion

Fairness, justice Unfairness
Groups Access to services Exploitation
& Communities Participation Misunderstanding
Network benefits Sabotage

Society Effective sharing of resources Inequity

More opportunity; greater diversity Destabilization
& Civilization Simpler or more equitable systems Environmental harm
Systemic neglect
Cultural exclusion

Referencing this matrix, generate or evolve concepts to maximize

beneficial outcomes and mitigate problems at each level.

IN PRACTICE
An automotive company was interested in designing the future
of autonomous vehicles. The design team went beyond considering
how these vehicles would interact with their owners, imagining how
they might interact with other “smart cars” and “smart cities” as well
as pedestrians, infrastructure, communities, and neighborhoods.
These future considerations were incorporated into the final design
concepts, proposing new avenues of ownership and suggesting ways
for autonomous vehicle owners to communicate externally with
other cars and traffic systems.