IMPACT

EVALUATION
Impact evaluations aim to assess whether the predicted changes brought about through a project or
programme have happened or not. Many cover unexpected or negative change as well. Most impact
evaluations also try to assess the contribution of a development intervention to any changes identified.
They usually follow a rigorous and accepted research methodology.

There is no single, agreed definition of impact evaluation. contribute to wider learning. However, impact evaluations
The term means different things to different people. At one have the potential to provide more certainty about any
end of the scale, some people define an impact evaluation findings, because they are more rigorous in their approach.
as any evaluation that seeks to assess change (outcomes or On the other hand, they also tend to be more costly than
impact), rather than focusing only on activities and process. other kinds of evaluation, and may take longer to plan and
At the other end of the scale, some people only include implement. Therefore the costs and benefits of carrying out
evaluations that use complex, statistical methods, such as an impact evaluation always need to be carefully weighed
randomised control trials (RCTs). up.

The meaning of impact evaluation differs depending on CSOs are most likely to use impact evaluations in the
how impact is defined. For example, some define impact following situations.
narrowly, only including long-term changes in the lives of
targeted beneficiaries. Others use wider definitions that  Where there is significant pressure on a project or
also include changes in areas such as policies or capacities. programme to demonstrate that it is effective and/or
However, there is generally broad agreement that an represents good value for money. This is especially the
impact evaluation involves a few features that distinguishes case if there is a possibility of funding or support being
it from other kinds of evaluation. Although other kinds of reduced or withdrawn in the absence of robust
evaluation may cover one or more of the features in the list evidence of change.
below, an impact evaluation would normally be expected  Where rigorous evidence of change is required in order
to cover all of them (Stern 2015, Perrin 2012, Rogers 2012, to make decisions about whether or not to enter a new
ActionAid 2016). phase. Some projects or programmes are divided into
different phases, and an impact evaluation may be
• Impact evaluations seek to assess the changes
required at the end of one phase before another
brought about by a project or programme, rather
begins. This is both to decide whether or not to
than focusing on activities and outputs only;
proceed with the new phase, and/or how to revise the
• They aim to assess predicted change, although project or programme in the new phase.
many cover unexpected or negative change as
well;  Where decisions need to be made about whether
• They attempt to assess the contribution of a and/or how to adapt, scale-up, mainstream or replicate
development intervention (or set of interventions) pilot or innovation projects or programmes. Impact
to any identified change; and evaluations are used in these circumstances because
• They usually follow a rigorous and accepted more certainty is required about the changes brought
research methodology to identify change and about through these kinds of intervention.
contribution.  Where an organisation supports many similar projects
and programmes, and wishes to invest significant
Impact evaluations can be carried out on any kind of work,
resources in finding out whether and why these are
including service delivery, capacity development,
effective. For example, if an organisation supports a
mobilisation and policy influencing, and any sector of work
similar model of mobilising target populations in many
from health through to governance. An impact evaluation
different communities it may be worth designing and
can be applied at any level of work from small projects
implementing an impact evaluation in one or two
through to complex programmes covering multiple
communities in order to generate lessons. These
organisations, sectors and geographic locations.
lessons could then benefit all of the projects or
programmes following similar models.
When to use impact evaluation Ultimately, an impact evaluation should be undertaken
An impact evaluation can be carried out for the same when there is a clear need for the findings and a clear plan
reasons as any other kind of evaluation – e.g. to improve for how to use those findings, and when it is judged that
projects and programmes, demonstrate accountability, and the extra costs are matched by the potential benefits. An
© INTRAC 2017
impact evaluation should not be undertaken when the  Many theory-based methods rely on developing a
purposes could be served by quicker or cheaper theory of change for an intervention, and then looking
alternatives. Impact evaluations should therefore be for evidence at each stage of the theory to try and
reserved for interventions where they are most likely to be develop a plausible (believable) case that explains how
useful (Rogers 2012). effects have come about. Some theory-based methods,
such as process tracing and contribution analysis, also
In all of the situations described above, timing is critical. If involve the development and testing of alternative
an impact evaluation is done too soon enough time may theories of change. Theory-based methods may be
not have elapsed for impact to emerge. If too late the applied to an individual case (such as a policy change)
evaluation may not actually serve the purpose for which it or multiples cases (such as changes in organisations
was intended. receiving capacity development support).

Sometimes the need to undertake an impact evaluation is  Some case-based methods, such as Qualitative
clear from the beginning of a project or programme, in Comparative Analysis (QCA), rely on examining
which case the timing can be clarified from the start. But in combinations of causes that lead to a change or set of
other cases the need may emerge over the course of a changes across multiple cases. Case-based methods
project or programme. It is important to note, however, tend to rely on an assumption that there are many
that deciding to carry out an impact evaluation part way different potential pathways to change.
through a project or programme may restrict the options
available. This is because some of the methodologies Under certain conditions, there are other types of
commonly used for impact evaluations require specific methodology that could meet the criteria for an impact
types of baseline data to be collected or targeted evaluation.
beneficiaries to be chosen at random. This is not usually
 Many CSOs collect large amounts of quantitative and
possible to do midway through a project or programme.
qualitative information from targeted beneficiaries
during baselines, and then repeat the exercise later on.
The aim is to establish change by comparing the
Different approaches information collected at the two different points in
There is no single method or approach that can be used for time. If there is no control or comparison group there
all impact evaluations. The most common methodologies is always a risk that any estimated change is due to
can be roughly categorised into four types (see Stern 2015). other factors instead of the project or programme
It is important to note, however, that there are often large being evaluated, which leaves findings open to
areas of overlap between these four types. In addition, criticism. However, in some cases it is possible to argue
many impact evaluations include more than one of the that any change was almost certainly brought about
types. through the intervention concerned. For example, if
testing changes in long-held views on social issues such
 Some methods are based on statistical analysis. These as people living with HIV&AIDS, following a series of
typically collect information on a key outcome or awareness raising campaigns, it might be possible to
impact (e.g. crop yields, household assets, girls’ argue that any change was almost certainly the result
attendance at school) and a range of possible of the campaigns.
influencing factors. Estimates of impact can be
 Many CSOs facilitate evaluations using participatory
generated through comparing the association between
methodologies designed to assess change from the
the intervention and the outcome/impact. An example
point of view of communities, or different groups
would be comparing the association between the
within those communities. The critical thing here
number of training courses attended by farmers with
would be to assess the approaches and methodologies
overall changes in crop yields. If it could be shown that
used, and to come to a conclusion about how robust
farmers who had attended the most training courses
those approaches were. A properly planned and
had the highest increase in crop yields (on average) it
designed participatory evaluation, meeting high
would be possible the estimate the impact of the
research standards, would almost certainly count as an
project or programme, provided all other factors were
impact evaluation. A rushed or rapid evaluation with
equal.
minimal involvement of different community members
 Experimental methods compare performance in a would not.
group of individuals or organisations receiving support
 Likewise, some CSO evaluations are based around
with performance in a group not receiving support, or
participatory methodologies such as the Most
receiving a different kind of support. In experimental
Significant Change (MSC) technique or outcome
methods, these are known as comparison or control
harvesting. Again, if the methodologies are applied
groups. Impact is measured by calculating the
fully and properly the evaluations could be counted as
difference in performance between those receiving
impact evaluations. For example, if MSC was applied
support and the control or comparison groups.
during an evaluation as a proper methodology,
Experimental methods usually rely on large surveys,
including in-depth work with communities to generate,
and can be very expensive.
validate, select and communicate stories of change in a
transparent way, this could be called an impact

© INTRAC 2017
 evaluation. But if only one or two meetings were held to carry out evaluation work. This is why it is so important
with random communities to generate a few ‘change’ to carefully assess when it is appropriate to carry out an
stories this would not be classed as an impact impact evaluation.
evaluation.
Another key challenge, as stated previously, is that many
It is important to note that in many cases – depending on methodologies used in impact evaluations need to be
the definition – impact evaluations may not seek to assess decided from the start of a project or programme,
impact at all. Rather, they may seek to assess changes especially if they rely on baseline surveys or randomisation.
(outcomes) that may eventually lead to impact at some This can be difficult in more complex interventions where
time in the future. This is why many impact evaluations are goals and objectives evolve over time. In such cases it may
based around Theories of Change that predict both how be more appropriate to use theory- or case-based
intermediary changes (outcomes) result from a project or methodologies that do not require extensive baselines.
programme activities, and how they are intended to
contribute to long-term or sustainable change (impacts) in Finally, even if a formal impact evaluation methodology is
the future. adopted, there is still be a need to ensure that regular
monitoring information is collected over the course of a
project or programme. This helps to supplement evaluation
Challenges findings as well as shedding light on unexpected or negative
changes that were not predicted at the start of a project or
As already stated, a properly implemented impact programme. The adoption of a complex impact evaluation
evaluation will often be costly – both in terms of money does not remove the need for effective monitoring
and time spent – and may require specialist expertise. This throughout a project or programme. If anything it makes it
is often a challenge for CSOs that do not have large budgets more important.

Further reading and resources

Other relevant papers in this section of the M&E Universe can be found by clicking on the links below.

Evaluation Types of evaluation

Impact assessment Case-based evaluation

Participatory evaluation Theory-based evaluation

The publication by Stern (2015) below was written especially for BOND and is a simplified version of a longer report
commissioned by DFID. It provides a useful, up-to-date overview of impact evaluation, and was specifically written for CSOs.

The Better Evaluation website (www.betterevaluation.org) contains the largest set of resources in the world covering evaluation
in the social development sector. The site offers step-by-step guidance for those managing or implementing evaluations.
Experienced evaluators or those with an interest in evaluation are recommended to go to that site and search through the
different materials.

References
 ActionAid (2016). How to… Select a Methodological Approach for the Evaluation. Evaluation Technical Briefing Note, #7.
ActionAid, UK.
 Rogers, P (2012). Introduction to Impact Evaluation. Impact Evaluation Notes no. 1. InterAction, March 2012.
 Perrin, B (2012). Linking Monitoring and Evaluation to Impact Evaluation. Impact Evaluation Notes no. 2. InterAction, April
2012.
 Stern, E (2015). Impact Evaluation: A guide for commissioners and managers. BOND, May 2015.

© INTRAC 2017
 Author(s): INTRAC is a not-for-profit organisation that builds the skills and knowledge of civil
Vera Scholz and Nigel Simister society organisations to be more effective in addressing poverty and inequality. Since
1992 INTRAC has provided specialist support in monitoring and evaluation, working
Contributor(s): with people to develop their own M&E approaches and tools, based on their needs.
Anne Garbutt and Dan James We encourage appropriate and practical M&E, based on understanding what works in
different contexts.

INTRAC Training M&E Universe

M&E Training & Consultancy M&E Universe
We support skills development and learning on a range of For more papers in
INTRAC’s team ofhigh
themes through M&E specialists
quality offer consultancy
and engaging and
face-to-face, For
themore
M&Epapers in
Universe
training in all aspects of M&E, from core
online and tailor-made training and coaching.skills development the series
M&E Universe
click the
through to the design of complex M&E systems. series
homeclick the
button
Email: training@intrac.org Tel: +44 (0)1865 201851 home button
Email: info@intrac.org Tel: +44 (0)1865 201851

© INTRAC 2017