Unit V

• Evaluating Prototype Models

 Syllabus
Design Testing with Users, Exploring Visual Design. Feedback Capture Grid,
Powerful questions in experience testing, Structured Usability Testing through
Solution Interviews, A/B Testing, Design Testing with Users, Choosing a Design
Testing, Usability Testing, Reflect- I like, I wish, I wonder"framework
,create a pitch, lean canvas lessons learned, Road map for implementation,
CAP, 4s.,Ethical Considerations in AI, Privacy and Security Concerns.
USER TESTING

Testing is the fifth stage in the Design Thinking process

Test: Put your prototype in front of real users and see how it fares! The testing
phase gives you the feedback and insight you need to iterate on your prototype
and make improvements.
What exactly is user testing and why
is it so important?
• Testing is, quite simply, the process of testing your prototype on real users.

• During the test phase, you’ll see how your target users interact with your
prototype, and gather valuable feedback.

• You’ll learn where your prototype succeeds and where it needs to be

improved.

• The insights gathered during the testing phase will enable you to iterate on
your prototype.
Why conduct user testing?
Together, prototyping and testing add huge value to the design process.

Not only does user testing help you to remain user-centric; it also makes good business
sense. By testing your ideas early and often, you are able to identify design flaws and
usability issues before you take the product to market. This has countless benefits for you,
the user, and the business!

So why conduct user testing?

• User testing saves time and money
• User testing reveals unexpected insights
• User testing improves user satisfaction:

• Example:Designing a Mobile App for Meal Planning

Feedback Capture Grid
• A Feedback Capture Grid is a useful tool in design thinking for
organizing and analyzing feedback from users during testing
phases.
• It helps teams synthesize insights in a structured way, allowing
for more effective iterations on designs.
• To test prototyped ideas quickly and simply and write down the
results for further development.
• The feedback capture grid supports the testing of ideas using
prototypes because it allows us to document test results in a
very simple form.
• It is primarily used when it comes to finding out how well an
idea solves a previously identified user problem.
• The feedback capture grid aims at acquiring profound
understanding as to whether the problem can be solved and
how it can be solved and whether the idea actually is the right
approach to a solution.
• Draw the grid and its four quadrants: “Things that worked,” “Needs to
change,” “New ideas to try,” and “Questions we still have”
• Evaluate ideas Fill in each quadrant with sticky notes.
• Use the Feedback Grid to capture ideas in real-time during a meeting or
workshop, or immediately following a Playback.
• Be specific and give constructive criticism.

• Cluster similar ideas and discuss. Search for patterns and themes. Take action
directly after the activity.
• Use the “Questions we still have” quadrant to inform an Assumptions and
Questions activity, “New ideas to try” to begin making Storyboards, or
“Needs to change” as a basis for action items.
Structure of the Feedback Capture Grid

Columns: Typically, the grid has several columns, which may include:
Feedback Type: Categorize feedback as positive, negative, or neutral.
User Quotes: Direct quotes from users that capture their thoughts.
Observations: Notes on user behavior during testing.
Implications: What the feedback means for the design (e.g., changes
needed).
Actions: Specific actions the team will take based on the feedback.
Powerful questions in experience testing
It evaluate ideas, prototypes, services, or a product by testing
them with real customers or users.
• “Powerful questions” is a great tool for gathering more findings in a test.
• As early as with low-resolution prototypes, we should pose the right
questions and develop them for high-resolution prototypes.
• The testing should always be done based on a specific prototype, however
During the test experience, the test person should interact as much as
possible with the prototype. The observers watch and listen attentively and
write down the results.

A simple test plan with goals, test environment, process, moderation,

and test participants is created.

More exact methods are used in later development, for example, unit
test, integration test, functional test, system test, stress test,
performance test, usability test, acceptance test, regression test, beta
test.
Powerful questions in experience testing

When conducting experience testing in design thinking, you can ask a variety of questions to gain
insights into users' experiences, including:
Overall impression
Ask users how they felt about their experience, what they liked or disliked, and if they would use the
product again.
Usability
Ask if the prototype does what it's supposed to, if users are confused, and if they can find what they're
looking for.
User needs
Ask what users' goals are, what motivates them, and what pain points the product needs to address.
User feedback
Ask users to verbalize their thoughts and actions as they use the prototype.
Comparison
Ask users to compare the product to a competitor's product.
Recommendations
Ask users how likely they are to recommend the product to a friend.
Income level
If the product includes payment tiers, ask users to confirm their income level.
Past experiences
Ask users about similar products they've used, what their needs were, and how well those products met
their expectations.
Open-ended questions that start with "what", "why", or "how" can help you get in-depth feedback from
users
 A simple test plan with goals, test environment,
process, moderation, and test participants is
created.

More exact methods are used in later development,

for example, unit test, integration test, functional
test, system test, stress test, performance test,
usability test, acceptance test, regression test, beta
test.
 Cont..
• What you can do with
the tool:

• Leave the fantasy world and conquer the real world by exploring the
prototype with real customers.

• Find out if it really works as intended for the

users/customers.

• Learn whether the idea will lead to

success or not.

• Get feedback: "love it", "change it" or "leave it".

• With the right questions, collect qualitative and quantitative data to identify whether
problems arise during experience or use.
 Solution interview
• To find out whether a solution is accepted by the user.
• As the name suggests, solution interviews are a tool used in
the test phase with advanced (high-resolution)prototypes.
• The goal is to test solutions that were developed in the
project and see whether they are accepted by the users
addressed.
• In order to make a clear distinction from the testing
tools already presented, the solution interview is
mainly used in the solution space.
• This is why we focus on the “acceptance” of a final
prototype or MVP.
• In this late phase, solution interviews supply insights on
the acceptance of a solution by the user, right up to the
pricing
Cont..
 Cont..
• Understand whether an intended solution is valued by
users, i.e. whether it is convincing in terms of
functionality, user-friendliness and user experience.

• To question the task on which the project is based,

i.e. to check whether the project concentrates on the
decisive questions.
• Understand the needs, behaviors and
motivations of users/customers more deeply.

• To measure the value of the solution for the user.

 Structured Usability Testing
• To test prototypes with potential users at defined and uniform conditions.
• Anything that is “operable” can be tested; this applies to both physical and digital
products.

• With usability testing, it can be checked whether something works effectively,

efficiently, and satisfactorily for the user.

• The testing should be as specific, frequent, and as early as possible.

• This requires real users who perform given specific tasks with the prototype at defined
and uniform
conditions. Observe everything and document it with video or tracking software,
whenever possible.
• The uniform structure allows for testing and comparing several ideas or variants on
the basis of the same criteria.
• Before testing is started, it is important to be aware of what is to be tested and how it is
to be measured.
• There are many different variants and versions of the usability testing tool: hallway,
guerrilla, laboratory test,
field test, and so on.
 Cont..
• Observe the interaction between user and system
(prototype) in defined test scenarios (tasks).
• Check and compare the correctness of the
assumptions, solutions, and concepts made with the
user.
• Get new inputs for improvement or completely new
ideas.
• Gain deeper insights into the problem by
testing existing products.
• Improve the suitability for use iteratively through
testing and subsequent optimizing.
 A/B Testing
• To review an assumption or compare two variants to find
out what the preferences of the users/customers are.
• The A/B test can be used as a stand-alone test or as an
expansion of a prototype test.
• The A/B test is a simple tool for testing two variants of a
prototype simultaneously. The test of the prototype usually
answers a question with different characteristics.
• This test is quite well suited to advance an existing
prototype/MVP or to test a new variant in comparison with a
basic prototype. It is important tomake it very clear before
the test what is to be tested and compared (e.g. by means of
key figures).
• Most users find it easier to give feedback when comparing
two prototypes than when they are asked to comment on one
prototype.
• Perform a true A/B test or several variants of a
prototype in the
• form of a multi-variants test or as split testing.
• Do a quantitative evaluation.
• Carry out a qualitative survey and evaluate the
number and content of feedbacks.
• Compare individual variants of a function or
a prototype (e.g. buttons, visuals,
arrangement).
Cont.
For example, the online streaming platform, Netflix, used A/B/n testing
to find which call to action button resulted in more sign-ups. A/B/n
testing extends A/B testing by incorporating more than one design
variant.
Design testing tool
When choosing a design testing tool for design thinking, you can consider the type
of testing you want to do and what you want to learn from it:

Usability testing: Helps you understand how users interact with your design, and
identify areas that might be frustrating or delightful.

A/B testing: Helps you make data-driven decisions between two design variations.

Prototype testing: Allows you to test your prototype with people who represent
your target audience. You can collect insights about user behavior, including facial
expressions.

Think aloud testing: Helps you identify gaps in your processes by asking users to
describe what they're thinking and feeling while performing a task.

User research data and heatmaps: Help you understand your customers' behavior
and preferences. You can use these tools to identify pain points, optimize user
experiences, and drive conversions.

Centralized repository: Helps you collect, analyze, and share user research data.
When choosing a design testing tool, you can consider things like:
• Ease of use: The tool should be easy to use and have an intuitive interface.
• Compatibility: The tool should support the devices and browsers you use, as well as
the operating systems and platforms you use.
• Integration: Consider how well the tool integrates with your existing development and
testing workflow.
• Scalability: The tool should be able to scale with your project needs.
• Features: Some tools offer features like heatmaps, scroll maps, and the ability to create
customized tests.
 Here are some design testing tools

Hotjar
A design thinking tool that helps businesses understand their customers' preferences and
behavior. It offers tools for user research data and heatmaps.

Lookback
A remote usability testing tool that allows you to observe how users interact with an
application in real-time.

UserZoom
A usability testing platform with advanced testing and research features.
Loop11
A code-free testing platform for creating customized usability tests. It
also provides metrics for data-driven decision making.

Crazy Egg
A user testing tool known for its heat maps and Scroll Map feature,
which shows when participants lose interest on a page.

UsabilityHub
A conversion rate optimization (CRO) tool that offers methodologies
to gauge user intent, feedback, preferences, and behaviors.

UXtweak
An all-in-one UX research tool that makes it easy to upload
prototypes and set up studies. It analyzes results and provides
insights.

TestRigor
An automated performance testing tool that helps create complex
tests with basic English language commands.
 Design Testing with Users
• Testing is, quite simply, the process of testing your
prototype on real users.
• During the test phase, you’ll see how your target
users interact with your prototype, and gather
valuable feedback.
• You’ll learn where your prototype succeeds and
where it needs to be improved.
• The insights gathered during the testing phase
will enable you to iterate on your prototype.
 Design Testing with Users
• Together, prototyping and testing add huge
value to the design process.
• Not only does user testing help you to remain
user-centric; it also makes good business sense.
• By testing your ideas early and often, you are able
to identify design flaws and usability issues before
you take the product to market.
• This has countless benefits for you, the user, and
the business!
 Design Testing with Users
So why conduct user testing?

• User testing saves time and money

• User testing reveals unexpected insights:
• User testing improves user satisfaction:
 Usability Testing
• Usability testing is the practice of testing how easy
a design is to use with a group of representative
users.
• It usually involves observing users as they attempt
to complete tasks and can be done for different
types of designs.
• It is often conducted repeatedly, from early
development until a product’s release.
 Cont
objectives are :
• Determine whether testers can complete tasks
successfully and independently.
• Assess their performance and mental state as they
try to complete tasks, to see how well your design
works.
• See how much users enjoy using it.
• Identify problems and their severity.
• Find solutions.
 Cont
• While usability tests can help you create the
right products, they shouldn’t be the only
tool in your UX research toolbox.
 Reflect- I like, I wish, I wonder,
• To provide constructive feedback and keep a
positive mood.
• We need feedback throughout the entire design thinking
process for the improvement of prototypes, stories, and
business models.
• “I like, I wish” is particularly suitable for sensitive projects.
By maintaining a positive mood, a relationship that is based
on partnership evolves between the feedback provider and
the feedback recipient.
• It can be used in the context of reflecting on the
collaboration as well as for a specific result. For example:
• “I like how you motivated us to conduct another customer
survey”; or “I wish that the prototype were tested in other
cultures as well.
• Practice a positive feedback culture. The use of “I like, I
wish, I wonder” helps in maintaining a positive mood.
• It is important to write down the feedback and
reflect on it later with the design thinking team.
• Feedback is a gift – always say thank you.
The "I like, I wish, I wonder" framework is a useful tool in design thinking for gathering
feedback and insights. Here's how it can be applied:
I Like
Purpose: Identify what is working well.
Example
"I like the user-friendly interface."
"I like the way the design incorporates feedback from users."
I Wish
Purpose: Highlight areas for improvement.
Example
"I wish there were more customization options."
"I wish the loading times were faster."
I Wonder
Purpose: Spark curiosity and explore possibilities.
Example
"I wonder how this could be integrated with other tools."
"I wonder what the impact would be if we targeted a different user group."
Cont
 Create a pitch
• To share the results and insights with the team
at the end of an iteration, and also at regular
intervals with the stakeholders.
 Create a pitch
• The term “pitch” actually comes from the advertising industry
and refers to a presentation in front of a potential customer.
• Agencies often compete in persuading the client and thus getting the
contract. The term “pitch” also became established in the start-up scene.
Here a pitch describes the presentation of one’s own business idea in a
short time in front of investors or a panel of judges.
• In design thinking, we use “pitch” for the brief presentation of the
results before the teams or the stakeholders.
• There are different types of pitches. They differ in terms of length, for
instance. The elevator pitch is the shortest type of presentation. The
point is to convey a summarizing and informative outline of the idea
within a very short period of time (often no more than approx. 30
seconds or 1 minute).
• Usually, only a few PowerPoint slides are used, or none at all. In
many pitch presentations, real prototypes are shown, making
the presentation more vivid.
• Depending on the project phase, the focus is on different questions.
 Create a pitch
• Show the team and stakeholders the current status of a
prototype, a project, or the ultimate solution.
• Structure ideas and highlight core information.
• Obtain feedback on the solution and important
functions, the customer needs, or the value
proposition.
• Convince the audience or decision-makers of the
project to get approval and resources for further steps
or the implementation.
Create a pitch
 lean canvas
• To translate a problem into a solution that takes into
consideration both the customer needs and my own
business context.
• The canvas supports us with structuring and visualizing our
innovation project. The completed lean canvas documents
the ultimate “problem/solution fit.”
• The blocks of the lean canvas lead through a logical
sequence, from customer problems to unfair
advantages.
• The canvas is better suited for designing the solution than
for exploring a problem.
• We use the lean canvas primarily to review the “problem/
solution fit” and
adjust it if necessary. This means that the collected data is
compared to the best solution that fits the behavior and
challenges of the customers.
• Summarize the results of the design
thinking iterations so everybody gets a
clear picture of the innovation project.
• Visualize and structure the hypotheses to review
them afterward and capture the findings in an
overview.
• Think and make observations about the
implementation or the business model to
identify risks entailed in the implementation.
• Compare different variants and business models
Cont
 Lessons learned
• To reflect on and record the insights gained
during and at the end of a design thinking project.
 Lessons learned
• The term “lessons learned” originates from project management.
• Lessons learned means the written recording and systematic collection
and appraisal of positive and negative experiences, developments,
findings, mistakes, and risks during a project.
• Different levels are considered, for example, technical, content,
emotional, social, and process-
related.
• The goal is to learn from the actions and decisions in order to design
future projects better.
• Thus lessons learned map the experience, knowledge, insights,
and the understanding the participants developed in the context
of project implementation.
• Design thinking projects usually produce a wide range of insights since
the methods are always applied in the context of the problem
statement, so each design challenge takes a different course.
• The tool helps to reflect on one’s own actions and learning. For more
extensive projects, its use is also recommended during the project, not
only at the end
• Collect and appraise experiences made
in the project in a structured manner.
• Learn from experience and make use of it in
the next project.
• Facilitate a positive attitude toward
mistakes and appreciate progress.
• Identify and document the findings; make
them applicable and
• usable.
 Road map for implementation
• To put the focus on the successful
implementation of potential
• market opportunities right from the onset.
 Road map for implementation
• The “road map for implementation” is, as it were, the road to success. With the open
approach of a complex problem statement, it is crucial to gain clarity about the
context as quickly as possible.
• The tool helps determine the critical factors that are relevant to the planning of the
• implementation project.
• The aim is to get a holistic view of the subsequent implementation as early in the
process as possible, to
• identify the critical path and possible risks in order to determine the necessary next
steps.
• The road map is created as early as possible, reviewed after each step, and adjusted if
• necessary.
• It is our compass, so to speak, for realigning the project again and again to an
initially vague and moving target.
• The execution of the implementation plan usually takes place later. Project
management
• methods, as they are used within the organization, help here.
Template and procedure:
Roadmap for
implementation
Cont
Innovation Tool using user needs,
CAP, 4s.
Innovation Tool using user needs,
CAP, 4s.
Innovation Tool using user needs,
CAP, 4s.
Innovation Tool using user needs,
CAP, 4s.
4S (Strategy, Structure, Style, Systems)
1.Strategy: Outlining the approach to solving the
problem, including goals, vision, and the methods
you’ll employ in the design process.
2.Structure: Referring to the organization of the design
project. This includes timelines, roles, and
responsibilities to ensure that the project is well-
coordinated.
3.Style: The visual and aesthetic aspects of the design.
This involves decisions about color, typography, and
overall visual identity that resonate with the audience.
4.Systems: Understanding how different elements of
the design interact with each other and fit into
broader systems. This could involve looking at
technology, processes, and user flows.
1. Strategy
Definition: The overall plan or direction a company or team takes
to achieve its goals. It’s the "why" behind actions.
Example: Apple's strategy involves focusing on innovation and
premium products to maintain a competitive edge. By investing in
research and development, Apple consistently releases products
that are ahead of market trends.

2. Structure
Definition: The way a company or organization is set up, including
its hierarchy, communication flows, and distribution of tasks.
Example: Amazon's structure is built around a decentralized
network of teams. Each team focuses on a specific product or
service, like AWS or Amazon Marketplace, allowing them to work
faster and more independently.
3. Style
Definition: The leadership and management style within an
organization, which affects company culture and decision-making.
Example: Google's management style is known for promoting
creativity and innovation. They encourage employees to spend 20%
of their time on passion projects, which fosters an open and
exploratory work environment.

4. Systems
Definition: The processes and tools that support the day-to-day
operations of the company. This can include IT systems,
communication tools, or workflows.
Example: Tesla's production system involves automated assembly
lines powered by advanced robotics. Their focus on automation
allows them to scale production while maintaining high-quality
standards.
Ethical Considerations in AI, Privacy and
Security Concerns.
1. Privacy Concerns
•What It Is: This is about keeping your personal information safe
and making sure it’s not used without your permission.
•Example: When you use a fitness app that tracks your steps, it
collects data about where you go. If the app shares that information
with advertisers without telling you, that’s a privacy issue.
2. Security Concerns
•What It Is: This is about protecting data from hackers and making
sure it doesn’t get stolen or misused.
•Example: Imagine a bank uses AI to process transactions. If hackers
break into the system and steal customer data, it puts everyone’s
money and information at risk.
 3. Bias and Fairness
•What It Is: Sometimes, AI can make unfair
decisions based on biased data, which means it
treats some people unfairly.
•Example: If an AI used for job hiring was trained
mostly on data from one gender or race, it might
favor that group over others, leading to unfair job
opportunities.
4. Accountability and Transparency
•What It Is: This is about making sure that when AI
makes a mistake, someone is responsible for fixing
it, and that people understand how the AI makes
decisions.
5. Regulatory Compliance
•What It Is: This means following laws that protect your data and privacy.
•Example: In Europe, there are laws (like GDPR) that say companies must ask
for your permission to collect your data and tell you how they will use it.

6. User Empowerment
•What It Is: This is about giving you control over your own data and helping
you understand what’s happening with it.
•Example: If a social media platform allows you to see what data it has about
you and lets you delete anything you don’t want it to keep, that’s empowering
for you as a user.
Summary
In simple terms, ethical considerations in AI focus on keeping your information
private and secure, ensuring fairness, being clear about how decisions are
made, following the law, and giving you control over your data. These factors
help build trust between people and technology.