Questionnaire Design

Dr A. F. Fagbamigbe
Epidemiology & Medical Statistics
College of Medicine
University of Ibadan
What is a
Questionnaire?

A set of Questions
designed to generate
the statistical
information from a
specific demographic
needed to accomplish
the research objectives
Surveys are more than a collection
of unambiguous questions
How questions are specified and
put together will influence the
respondents’ willingness to
participate & the responses they
provide
 Purposes of the Questionnaire
▪ Ensures standardization and comparability of the data
across interviews – everyone is asked the same questions
▪ Increases speed and accuracy of recording
▪ Facilitates data processing
▪ Allows the researcher to collect the relevant information
necessary to address the management decision problem
Improper design can lead to?
▪Incomplete information
▪Inaccurate data
▪Higher costs
 Criteria to consider

▪ Does it provide the necessary

information
▪ Does it consider the respondent
▪ Does it meet editing, coding and
data processing requirements
 Designing the Questionnaire
▪ Determine survey objectives: Plan what to measure.
▪ Decide on format. E.g. personal interview, telephone, self.
▪ Formulate questions to obtain the needed information
▪ Decide on the wording of questions
▪ Decide on the question sequence and layout of the
questionnaire
▪ Using a sample, test the questionnaire for omissions and
ambiguity
▪ Correct the problems (pretest again, if necessary)
 Questionnaire Design

• Ask Open – Ended Questions to generate key words

• Consider your mind – map
• Consider your analysis approach t-test, chi-squared
• Write your closed questions (types will follow).
 The Major Decisions in
Questionnaire Design
1. Content - What should be asked?
2. Wording - How should each question be phrased?
3. Sequence - In what order should the questions be
presented?

4. Layout - What layout will best serve the research

objectives?

The most difficult step is specifying exactly what

information is to be collected from each respondent
Content: Questions must meet 5 requirements

1. Are the questions relevant. Do they pertain to the

research problem
2. Are the questions accurate. Do they accurately depict
the attitudes, behaviors, etc. intended to investigate
3. Do respondents have the necessary information?
• Qualify respondents
4. Do respondents understand and interpret the question
correctly
5. Will respondents give the information?
 How should the questions be asked?
Format: How much freedom do we give respondents in
answering questions
Open-ended questions
– What do you look for most in a job?
– Is there anything else you would like to add about the product
Question Types – Open vs. Closed
What do you think are the reasons for football
hooliganism?

Do you think football hooliganism is

caused by: (tick if appropriate)
Lack of discipline at home
Players’ behaviour on pitch
Family breakdown
Youth unemployment
Poor schooling
Violence on T.V.
Other (please specify)
 Open Ended Questions: key advantages
▪ Wide range of responses and information can be obtained
▪Answers based on respondent’s not researcher’s frame of
reference – consumer’s terms
▪Lack of influence. Don't channel respondents thinking
▪ Can help interpret closed-ended questions - why
▪ Particularly useful as introduction to survey or topic
▪ When it’s important to measure the salience of an issue
▪ When too many possible responses to be listed or
unknown
 Open-ended questions: Key disadvantages
▪Ability and/or willingness of respondent to answer
▪ Interviewer’s ability to record answers quickly or summarize
accurately & probe effectively
▪ Interviewer’s attitude influences response
▪ Time consuming (interview sessions, tabulation, classification,
assignment, validation)
▪ Difficulty in coding
▪ Require respondents to be articulate
▪ Respondents may miss important points
▪ Non-response
Closed-ended questions (Fixed-alternative responses)
What do you look for most in a job?
____ Work that pays well
____ Work that gives a sense of accomplishment
____ Work where you make most decisions by yourself
____ Work that is steady with little chance of being laid off.
Advantages
▪ Ease of understanding
▪ Requires less effort on part of interviewer and respondent
▪ Ease of tabulation & analysis
▪ Less error prone
▪ Less interviewer bias
▪Less time consuming
▪ Answers directly comparable from respondent to respondent
Closed-ended questions (Fixed-alternative questions)

Disadvantages

▪ Middle/Neutral categories often selected inappropriately

(ignorance, safety)
▪ Less opportunity for self-expression or subtle qualifications
▪ Less involving for respondents
▪ Order of response categories can have major impact on results

Key tradeoff
Want to get respondent to address issues our research is
concerned with (Forced response) and at same time give
respondent opportunity to honestly opt out of question (i.e., Don’t
Know, No Answer, Neither Agree nor Disagree) so as not to dilute
data collected
 Dichotomous Questions
Should the Alberta Government give consumers an energy rebate?
1. Agree
2. Disagree

Advantages

▪ Easy to administer and tabulate

Disadvantages

▪ Prone to large amounts of error since polarized responses

prevent gaining information on the range of variation
▪ Fail to communicate any intensity of feeling
 Single vs. Multiple Response
Which of the following means do you use to
travel to college?

Bus

Car

Bike
What is your most usual means of travelling to
college? (Tick one box only)
Bus
Car
Bike
Ranked Response

Place in order of importance to you the following

features of a camping holiday (Indicate by numbering
from 1-5 in order where 1 is the most important)
Open air
Mobility
Cost
People
Atmosphere
Rated Response

(Circle the number under the initials that applies.

VI=Very important; I=Important; N=Neutral;
U=Unimportant; VU=Very Unimportant).
Indicate your view of the following aspects of a
camping holiday
VI I N U VU

Community Life 5 4 3 2 1

Low Cost 5 4 3 2 1

Outdoor Life 5 4 3 2 1
Precise Answers
Ask for precise answers if you think the
information is available and there are no other
constraints (e.g. too intrusive on privacy). For
example:

Give your age on lst September 2001 ……

years

is preferable to:

Are you…
Under 18 ….
18-65 …...
Over 65 ……
 Multiple Choice Questions
▪Are all possible alternatives included?
▪Too many alternatives
▪Position Bias

Scaled Response Questions

▪ Closed ended questions where the response choices are
designed to capture an intensity of feeling (Likert, Staple,
Semantic differential)
▪Easy to code and more powerful statistical tools
▪Main problem: Respondent misunderstanding
 Question Wording

Some general rules can be stated on question wording:

• Be concise and unambiguous
• Avoid double questions
• Avoid questions involving negatives
• Ask for precise answers
• Avoid leading questions
Question Wording
- Evokes the truth. Questions must be non-
threatening.
- Asks for an answer on only one dimension.
- Can accommodate all possible answers.

What brand of computer do you own?

A. IBM PC
B. Apple

- Has mutually exclusive options.

Where did you grow up?

A. country
B. farm
Question Wording
- Follows comfortably from the previous question.
- Does not presuppose a certain state of affairs
Are you satisfied with your current auto insurance?
(Yes or No)
- Does not imply a desired answer. (Leading
Question)
- Does not use emotionally loaded or vaguely
defined words. (e.g., most, least, majority)
- Does not use unfamiliar words or abbreviations.
(CPU, GPU)
Question Wording cont’n
- Is not dependent on responses to previous
questions. Avoid Branching

1. Do you currently have a life insurance policy ?

(Yes or No) If
no, go to question 3
2. How much is your annual life insurance
premium ?
- Does not ask the respondent to order or rank a
series of more than five items
 Question Wording
▪Can have major impact on how respondent interprets question
▪All respondents should interpret in the same way
Things to avoid
▪ Complexity: use simple, direct, conversational language
▪leading questions -- that suggest or imply certain answers
▪loaded questions -- suggest social desirability, or are emotionally
charged. Have you purchased a high quality Sony TV this year
▪Ambiguity and vagueness: Words such as “often”, “occasionally”,
“usually”, “regularly”, “frequently”, “many”, should be used with
caution. If these words have to be used, their meaning should be
explained properly.
▪Which province is bigger Manitoba or Alberta? – would your
answer be based on population or area?
 More things to avoid

▪long-worded questions

▪ double-barreled questions. Questions that refer to two

or more issues within the same question. Where
respondent may agree with only 1 part of multipart
statement.
▪ Do you think Nike offers better pricing and variety
than other brands

▪ Making implicit assumptions

▪Jargon
 More things to avoid
▪ burdensome questions - that may tax the respondent’s
memory
▪ How many tubes of toothpaste have you purchased in
the last 3 months
▪ Have you purchased toothpaste in the past week
▪Embarrassing, sensitive, or threatening questions:
▪ Have you charged more on your credit card than you
should
▪ Use third person do you think most people…..
 Question Sequence & Layout Decisions

Initial stages
▪ Screening or qualifying Questions: Have you been to the movie
theatre this month?
▪ Need to gain & maintain respondent’s cooperation
▪ Make questionnaire simple for interviewer to administer
▪opening questions should be interesting, simple, and easy to
answer.
▪ which theatre did you go to last?
Transition Questions
▪ What aspects of the theatre did you like best
▪Questions directly related to research objectives which
require more effort and get respondent thinking about topic

Difficult and Complicated Questions

▪ The following 10 questions relate to the characteristics of
theatres
▪Respondent now committed and can see an end in sight

Classifying and Demographic Questions

▪ What is your average annual household income
▪May not be answered (sensitive or threatening) but most
questions have been answered
 Other Considerations
▪Questions should flow logically from one to the next
▪General questions should be asked before more specific ones
▪Earlier questions should not influence response to later ones
▪Questions should flow from factual and behavioural questions to
attitudinal and opinion questions
▪Questions should flow from the least sensitive to the most sensitive.
▪Questions should flow from unaided (which brand do you prefer) to
aided questions (which brand do you prefer Tide, Cheer, Gain)

▪ Demographic questions should come at the begining.

1. initial questions - screening and rapport questions
2. Then demographic questions.
3. Middle and end - product specific questions
 Other Considerations
▪Use multiple questions instead of one
▪ Similar questions together
▪ consistent mindset for respondents
▪ Develop a logical flow
▪ Use transitions between sections (E.g. In this section we ask
questions about X)
▪Distinguish between instructions, questions and responses
Distinguishing question and responses:
CAPS or BOLD or Underline versus lower case or unformatted

INSTRUCTIONS LAYOUT A:
Do you agree, disagree or have no opinion that this company has:
• A good vacation policy - disagree/not sure/agree.
• Good management feedback - disagree/not sure/agree.
• Good medical insurance - disagree/not sure/agree.

INSTRUCTIONS LAYOUT B:
Does this company have:
___ Disagree Not Sure Agree
• A good vacation policy 1 2 3
• Good management feedback 1 2 3
• Good medical insurance 1 2 3

Matrix question
Filter and pivot questions should be used as necessary.

A FILTER question is one that screens out

respondents who are not qualified to answer a
second question.

A PIVOT question is a type of filter question that is

used to determine what version of a second
question to ask.
 Developing a logical flow

▪ If the questionnaire deals with several topics,

complete questions on a single topic before moving
on to a new topic
▪ If topics are related, ask questions on related topics
before asking questions about unrelated topics
▪ If you ask questions about behaviors over some time
period, follow chronological order backward in time
▪ When changing topics, use some transitional phrase
 QUESTIONNAIRE LAYOUT
The layout and physical attractiveness of a
questionnaire are important aspects
▪ Questionnaires should be designed to appear as
short as possible
▪ Questionnaires should not appear overcrowded
▪ Leave lots of space for open ended questions
▪ Questionnaires in booklet form are often
recommended
OPENING
▪Provide first name (at least)
▪Provide name of company doing research
▪Provide reason for survey and topic
▪State that no selling will be involved and no personal data
other than for statistical purposes
▪Tell respondent approximate time to complete
▪Reinforce that respondent’s time is appreciated
▪Invite to participate
AND CLOSING
▪Thank for time
▪Ask if they had a positive experience and remind them that
their opinions count
 Pretesting and Correcting Problems
Purpose of pretest: To ensure that the questionnaire meets the
expectations in terms of the information that will be obtained
▪Is question necessary – does it serve a purpose, will info be used
▪Missing important variables – does it provide the info needed
▪Match questions to objectives
▪Pretest Specific Questions For
• Variation
• Meaning
• Task difficulty
• Respondent interest and attention
• Ambiguous, ill-defined, loaded, double-barreled questions
▪Pretest the Questionnaire
• Flow of the questionnaire
• Skip patterns
•Length
▪Respondent Interest and Attention
 Questionnaire Design Flow Chart
Step 1: Specify what information will be sought
Step 2: Determine type of questionnaire and method of
administration
Step 3: Determine the content of individual questions
Step 4: Determine form of response for each question
Step 5: Determine wording for each question
Step 6: Determine sequence of questions
Step 7: Determine physical characteristics of questionnaire
Step 8: Revise steps 1-7, revise if necessary
Step 9: Pretest questionnaire, revise if necessary
Step 10: implement
A QUESTIONNAIRE
IS ONLY AS GOOD
AS THE QUESTIONS
IT ASKS