ASA Series

What Is a Survey?

What Is a Survey?
How to Plan a Survey ASA Series
How to Collect Survey Data What Is a Survey?
Judging the Quality of a Survey
How to Conduct Pretesting

What is a
What Are Focus Groups?
More About Mail Surveys
What Is a Margin of Error? Margin of Error?
Designing a Questionnaire
More About Telephone Surveys

Produced by
Section on Survey Research Methods
American Statistical Association
1429 Duke Street
Alexandria, VA 22314-3415 USA Section on Survey Research Methods
American Statistical Association

8/00
© 1998 American Statistical Association
 What is a
Margin of Error?

W hen results of surveys are reported

in the media, they often include a
statement like
“55 percent of respondents favor
Ms. Smith in the upcoming
mayoral election. There is a margin
of error of 3 percentage points.”
What does a statement like this mean? This
pamphlet attempts to answer this question
This pamphlet, What Is a Margin of Error? is the and to provide some cautions on the use of
eighth in the ASA series What Is a Survey? Here we the “margin of error” as
will attempt to define the often-used journalistic the sole measure of a
phrase “Margin of Error.” Our treatment is sometimes survey’s uncertainty.
more mathematical than in the other pamphlets in An estimate from a
the series; still it should be accessible to most read- Surveys are typical- survey is unlikely to
ers. Among the topics covered are how the number of ly designed to provide
observations in the sample, the type of sampling, and an estimate of the true exactly equal the
population size affect the margin of error. Some cau- value of one or more true population
tions are also provided on employing the margin of characteristics of a
error as the sole measure of a survey’s uncertainty. population at a given quanitity of interest.
time. The target of a
The What is a Survey? series is written primarily for
survey might be
the general public. Its overall goal is to improve sur-
vey literacy among individuals who participate in ■ the average value of a measurable quantity,
surveys or use survey results. The series is designed such as annual 1998 income or SAT
to promote a better understanding of what is scores for a particular group.
involved in carrying out sample surveys—especially ■ a proportion, such as the proportion of like-
those aspects that have to be taken into account in ly voters having a certain viewpoint in a
evaluating the results of surveys.
mayoral election

1
■ the percentage of children under three years Does sampling
of age immunized for polio in 1997 error render surveys
An estimate from a survey is unlikely to useless? Fortunately, The “margin of
exactly equal the true population quantity of the answer to this error” is a common
interest for a variety of reasons. For one question is “No.” But
how should we sum- summary of
thing, the questions maybe badly worded.
For another, some people who are supposed marize the strength sampling error
to be in the sample may not be at home, or of the information in a
that quantifies
even if they are, they may refuse to partici- survey? That is a role
pate or may not tell the truth. These are for the margin of error. uncertainty about
sources of “nonsampling error.” a survey result.
But the estimate will probably still differ Margin of Error
from the true value, even if all nonsampling Defined
errors could be eliminated. This is because The “margin of error” is a common summary
data in a survey are collected from only of sampling error, referred to regularly in the
some—but not all—members of the popula- media, which quantifies uncertainty about a
tion to make data collection cheaper or survey result. The margin of error can be inter-
faster, usually both. preted by making use of ideas from the laws of
Suppose, in the mayoral election poll probability or the “laws of chance,” as they are
mentioned earlier, we sample 100 people sometimes called.
who intend to vote Surveys are often conducted by starting out
and that 55 support with a list (known as the “sampling frame”) of
“Sampling error” Ms. Smith while 45 all units in the population and choosing a
means that results support Mr. Jones. sample. In opinion polls, this list often consists
This would seem to of all possible phone numbers in a certain
in the sample differ suggest that a
geographic area (both listed and unlisted
from a target pop- majority of the numbers).
ulation quantity, town’s voters, In a scientific survey every unit in the popu-
including people lation has some known positive probability of
simply due to the not sampled but being selected for the sample, and the proba-
“luck of the draw.” who will vote in the bility of any particular sample being chosen
election, would can be calculated. The beauty of a probability
support Ms. Smith.
Of course, just by chance, a majority in a
particular sample might support Ms. Smith
In a scientific survey every person
even if the majority in the population supports
Mr. Jones. Such an occurrence might arise due in the population has some
to “sampling error,” meaning that results in the known positive probability of being
sample differ from a target population quantity,
simply due to the “luck of the draw”—i.e., by selected into a sample.
which set of 100 people were chosen to be in the sample.

2 3
sample is twofold. Not only does it avoid bias-
es that might arise if samples were selected
based on the whims of the interviewer, but it
also provides a basis for estimating the extent
of sampling error. This latter property is what
enables investigators to calculate a “margin of
error.” To be precise, the laws of probability
make it possible for us to calculate intervals of
the form
estimate +/- margin of error.
Such intervals are sometimes called 95 per-
cent confidence intervals and would be
expected to contain
the true value of the
target quantity (in
Such intervals are
the absence of non-
sometimes called sampling errors) at
95% confidence least 95 percent of
the time.
intervals and would
An important
be expected to factor in determining
1988, The New Yorker Collection. Reprinted with permission.
contain the true the margin of error is
the size of the sam-
value at least ple. Larger samples
95% of the time. are more likely to Smith could plausibly range from 45 percent
yield results close to to 65 percent. This implies—because of the
the target population small sample size—considerable uncertainty
quantity and thus about whether a majority of townspeople
have smaller margins of error than more actually support Ms. Smith.
modest-sized samples. Instead, if there had been a survey of 1,000
In the case of the mayoral poll in which 55 of people, 550 of whom support Ms. Smith, the
100 sampled individuals support Ms. Smith, sample estimate would again be 55 percent,
the sample estimate would be that 55 percent but now the margin of error for Ms. Smith’s
support Ms. Smith—however, there is a margin support would only be about 3 percent. A 95
of error of 10 percent. Therefore, a 95 percent percent confidence interval for the proportion
confidence interval for the percentage sup- supporting Ms. Smith would thus be
porting Ms. Smith would be (55%-3%) to (55%+3%)
(55%-10%) to (55%+10%) or (52 percent, 58 percent), which provides
or (45 percent, 65 percent), suggesting that in much greater assurance that a majority of the
the broader community the support for Ms. town’s voters support Ms. Smith.

4 5
1988, Washington Post Writer’s Group. Reprinted with permission.
Probability Sampling Designs—The survey
researcher also has control over the design of
What Affects the Margin of Error? the sample, which can affect the margin of
Three things that seem to affect the margin error. Three common types are simple random
of error are sample size, the type of sampling sampling, random digit dialing, and stratified
done, and the size of the population. sampling.
Sample Size—As noted earlier, the size of a ■ A simple random sampling design is one
sample is a crucial actor affecting the margin of in which every sample of a given size is
error. In sampling, to try an estimate a popula- equally likely to be chosen. In this case,
tion proportion—such as in telephone polls— individuals might be selected into such a
a sample of 100 will produce a margin of error sample based on a randomizing device
of no more than about 10 percent, a sample of that gives each individual a chance of
error of 500 will produce a margin of error of no selection. Computers are often used to
more than about 4.5 simulate a random stream of numbers to
percent, and a sample support his effort.
of size 1,000 will pro- Telephone surveys ■ Telephone surveys that attempt to reach not
duce a margin of error only people with listed phone numbers but
often rely on the
of no more than about also people with unlisted numbers often rely
3 percent. This illus- technique of on the technique of random digit dialing.
trates that there are “random digit ■ Stratified sampling designs involve defining
diminishing returns groups, or strata, based on characteristics
when trying to reduce dialing.” known for everyone in the population, and
the margin of error by then taking independent samples within
increasing the sample each stratum. Such a design offers flexibility,
size. For example, to reduce the margin of error and, depending on the nature of the strata,
to 1.5% would require a sample size of well they can also improve the precision of esti-
over 4,000. mates of target quantities (or equivalently,
reduce their margins of error).
6 7
 Of the three types Interpreting the Margin of Error
of probability sam- In practice, nonsampling errors occur that
Stratified samples pling, stratified sam- can make the margin of error reported for a
are especially ples are especially poll smaller than it should be if it reflected all
advantageous when sources of uncertainty. For example, some
advantageous ... the target of the sur- respondents to the mayoral survey may not
when the target is vey is not necessarily have been eligible to vote but may have
to estimate differ- to estimate the answered anyway, while others may have mis-
proportion of an led the interviewer about their preferences.
ences in viewpoints entire population
Why isn’t the margin of error adjusted to
between different with a particular reflect both sampling and nonsampling uncer-
viewpoint but tainties? The answer is that, unlike sampling
groups. instead is to esti- error, the extent of nonsampling error cannot
mate differences in usually be assessed from the sample itself,
viewpoints between even if the sample is a probability sample.
different groups. For example, if there was a
Some things
desire to compare attitudes between individ-
that help assess
uals of Inuit (Alaskan native) origin versus Nonsampling errors
nonsampling
other Americans on their opinion about
uncertainties, occur that can make
drilling for oil on federal land, it would not
when available,
make sense to take a simple random sample
include the per- the margin of error
of all Americans to answer this question
centage of reported for a poll
because very few Inuit would likely fall into
respondents who
such a sample. Instead, one might prefer to smaller than it should
answer “don’t
take a stratified sample in which Alaskan
know” or “unde- be to reflect all sources
Native Americans compose one half of the
cided.” Be wary of uncertainty.
sample and non-Inuit compose the other
when these
one half.
quantities are
Sometimes samples are drawn in clusters in not given. Almost
which only a few counties or cities are sampled always there are people who have not made up
or only the interviewer visits a few blocks. This their mind. How these cases are handled can
tends to increase the margin of error and make a big difference. Simply splitting them in
should be taken into account by whoever proportion to the views of those who gave an
calculates sampling error. opinion can be misleading in some settings.
Size of Population—Perhaps surprising to It is important to learn if the survey results are
some, one factor that generally has little influ- actually from a probability sample at all. Many
ence on the margin of error is the size of the media surveys are based on what are called
population. That is, a sample size of 100 in a quota samples, and, although margins of error are
population of 10,000 will have almost the reported from them, they do not strictly apply.
same margin of error as a sample size of 100
Overall, nonresponse in surveys has been
in a population of 10 million.
growing in recent years and is increasingly a

8 9
consideration in the support, then it would not be terribly surpris-
interpretation of ing for another poll to report anything from a
Overall nonresponse
reported results. 10-point lead for Mr. Jones (such as 48 percent
Media stories typi- in surveys has been to 38 percent) to a 2-point lead for Ms. Smith
cally do not provide growing in recent (such as 44 percent to 42 percent).
the response rate, In more technical terms, a law of probability
even though these years and is dictates that the difference between two
can be well under 50 increasingly a uncertain proportions (e.g., the lead of one candi-
percent. When the date over another in a political poll in which both are
results are important
consideration in the
estimated) has more uncertainty associated
to you, always try to uncertainty of with it than either proportion alone.
learn what the non- reported results. Accordingly, the margin of error associated
response rate is and with the lead of one candidate over another
what has been done should be larger than the margin of error
about it. associated with a single proportion, which is
For more information, see “Judging the what media reports typically mention (thus
Quality of a Survey”—an earlier brochure in the need to keep your eye on what’s being
the What Is a Survey? series. estimated!).
Until media organizations get their reporting
Keep Your Eye on What is Being Estimated practices in line with actual variation in results
It is common for political polls to quote a across political polls, a rule of thumb is to mul-
margin of error of plus or minus 3 percent. It tiply the currently reported margin of error by
might happen, however, that in two separate 1.7 to obtain a more accurate estimate of the
polls between Jones and Smith in the same margin of error for the lead of one candidate
week one might have Jones ahead by 2 percent over another. Thus, a reported 3 percent mar-
in one poll while the other poll might have gin of error becomes about 5 percent and a
Jones ahead by 10 percent. How can this be? reported 4 percent margin of error becomes
A misleading feature of most current media about 7 percent when the size of the lead is
stories on political polls is that they report the being considered.
margin of error associated with the proportion There is a lot more to
It is common for be said about the use
favoring one candidate, not the margin of error
of the lead of one candidate over another. To political polls to of the term “margin of
illustrate the problem, suppose one poll finds error.” Surprisingly,
quote a margin
that Mr. Jones has 45 percent support, Ms. there is even some con-
Smith has 41 percent support, 14 percent are of error of plus troversy about its
undecided, and there is a 3 percent margin of or minus 3%. meaning. For those
error for each category. interested in reading
If we note that Mr. Jones might have any- more about this contro-
where from 42 percent to 48 percent support versy, a Sunday, June
in the voting population and Ms. Smith might 14, 1998, “Unconventional Wisdom” column by
have anywhere from 38 percent to 44 percent Richard Morin in The Washington Post may be a
good start.
10 11
Where Can I Get More Information?
In addition to the pamphlets in this series,
ASA also makes other brochures available
upon request:
■ Ethical Guidelines for Statistical Practice
■ Surveys and Privacy, produced by the ASA
Committee on Privacy and Confidentiality.
For the above brochures or other pamphlets
in the What Is a Survey? series, contact:
Section on Survey Research Methods
American Statistical Association
1429 Duke Street
Alexandria, VA 22314-3415 USA
(703) 684-1221/fax: (703) 684-2037
Email: asainfo@amstat.org
Web site: http://www.amstat.org/
sections/srms/
Besides the ASA, there are many other
associations that are concerned with the
proper collection and use of survey data:
■ The American Association for Public
Opinion Research (AAPOR) offers a number
of publications—perhaps the most relevant
of these is the one entitled Best Practices for
Survey and Public Opinion Research Survey
Practices AAPOR Condemns. To obtain
copies, call (313) 764-1555 or visit their Web
site at http://www.aapor.org.
■ The National Council on Public Polls
publishes another useful pamphlet, Twenty
Questions a Journalist Should Ask About Poll
Results. To obtain a copy, call (800) 239-0909.
■ The Research Industry Coalition, Inc.,
publishes a brochure, Integrity and Good
Practice in Marketing and Opinion Research.
To obtain a copy, call (516) 928-6803.
■ The Council of American Survey Research
Organizations publishes a pamphlet,
Surveys and You. To obtain a copy, call
(516) 928-6954, or visit their Web site at
http://www.casro.org.

12