Research Excellence Framework

(REF) impact toolkit

Helen Tilley, Louise Ball, Caroline Cassidy

March 2018

CONTENTS

Module 1 Understanding impact

Module 2 Identifying your impact objective

Module 3 Stakeholder mapping

Module 4 Developing a theory of change

Module 5 Communicating your research

Module 6 Monitoring and learning

AIMS, AUDIENCE AND USES
This guide provides a simple step-by-step approach to planning for impact in six modules. It is based on the ODI’s Research
and Policy in Development (RAPID) Outcome Mapping Approach (ROMA) and draws heavily on the DFID-ESRC Growth Research
Programme (DEGRP) guidance on achieving policy impact (Young et al., 2014; DEGRP, 2017; Shaxson, 2016). These resources are
grounded in Outcome Mapping, see Box 1.

The step-by-step approach presented here aims to help researchers

think, in a structured way, about how to monitor and improve Box 1  Outcome mapping
the impact research will have on policy and practice. It will help
Outcome Mapping (OM) was first introduced by
researchers understand their impact pathways and develop a suite
Sarah Earl, Fred Carden and Terry Smutylo from the
of indicators they can use to monitor impact over the lifetime of their
International Development Research Centre (Earl
research project and for some years afterwards.
et al., 2001). It focuses on outcomes as defined as
changes in behaviour, relationships, activities and
The modules provide guidance in (1) formulating aspects of
actions, and in doing so recognises early changes
the Pathways to Impact statement, (2) developing a monitoring
and progress towards higher-level goals. OM differs
plan and collecting evidence to inform the reporting of impact
from conventional logic models by recognising that
in Research Excellence Framework (REF) case studies, and (3)
different actors affected by activities exist in different
writing a REF case study. A clear Pathways to Impact statement
environments, and it is explicitly designed for non-
will help define a good research communications strategy, to
linear, systemic change processes (Young et al., 2001;
ensure messages from research reach those who can make a
Jones and Hearn, 2009). It does not try to attribute
difference throughout the decision-making processes. During the
outcomes but rather explores plausible contributory
research and implementation of the communications strategy,
links between interventions and behaviour changes,
monitoring impact and collecting evidence will inform the writing
therefore allowing for complex and non-linear
of the impact case study for REF2021.
relationships between activities and results.
The modules are intended to be used in a flexible manner
depending on the purpose and characteristics of the research
project. If the research project structure (i.e. who the partners are or how the work is divided between them) or the context within
which it operates changes, plans should be revisited and, if necessary, adapted. Researchers wanting to dive into this in more detail
can ask for support from the RAPID team, who can review and comment on impact plans or have one-on-one discussions.

2
 MODULE 1

UNDERSTANDING IMPACT
Introduction
There are increasing demands upon researchers to define and demonstrate impact in order to secure funding for research. We
recognise the importance of conducting research to produce long-term contributions to the knowledge base and to intellectual
conversations, and our aim is not to convert you to the impact agenda. However, if you have picked up this guide, then it is likely
you are interested in influencing policy and practice with your research.

The impact of research on funding takes place through two

mechanisms. First, the REF exercise in 2014, and the forthcoming
2021 exercise, directly link the assessment of impact case
Take-away messages
studies to annual Quality-Related (QR) research funding,
with funding bodies using it to allocate the university block •• Start thinking about impact and how you will
grant between universities.1 In 2016/17, the block grant was measure it at the start of your research. This will
£1.58 billion (HEFCE, 2016). Second, research councils award help make it easier to understand and capture
competitive grants on the basis of funding applications that impact when it arises. Also, revisit your plan
require a Pathways to Impact statement outlining interventions regularly to adjust it as necessary.
that will enable your research to realise potential impact. •• Your role in achieving impact may not be
straightforward. Your influence may be direct,
In this introductory module, we discuss different definitions of but it may also be indirect. It may be that you only
impact and the challenges of measuring it. Measuring change is contribute to broad changes.
a challenging area for researchers. As well as the complexities •• Assessing the context is important when
involved in understanding contribution and causal chains, considering impact. Consider complexities such as
there are no standard metrics, and funders often set their own unexpected impact, impact over the longer term,
measures of success. For instance, publication and citation data contribution and co-creation and contextual factors.
are often used as a measure of research output but they do not
consider the impact research may have on policy and practice.

Impact and the challenges of measuring it

What does research impact mean, and how can you achieve it? Just as the term ‘impact’ is being more widely used, there
are increasing numbers of definitions of how it can be applied.2 In statistics, impact is defined as the measured difference in a
predefined indicator before the intervention and after the intervention.3 The Development Assistance Committee of the Organisation
for Economic Co-operation and Development (OECD-DAC) uses a broader definition of impact, as any long-term effect, whether
intended, unintended, positive, negative, direct or indirect.4

In the 2014 REF, impact is defined as an effect on, or change or benefit to the economy, society, culture, public policy or services,
health, the environment or quality of life, beyond academia (REF2014, 2014). Research Councils UK (RCUK) defines impact as
academic, economic and societal, and highlights the importance of balancing these, rather than just focusing on academic impact.5
This recognises both the contribution research can make to academic advances and the more diverse ways in which research can
contribute to economic performance within and outside of the UK, increase the effectiveness of public policy and improve quality of
life. RCUK also includes public engagement as an impact pathway; see Module 5.

Here, we apply the Economic and Social Research Council (ESRC)-preferred definition, summarised in Table 1, as this is research-
focused and therefore considers the ways in which research can have an impact. ESRC defines impact as conceptual, instrumental and

1 For more information see: www.gov.uk/government/news/government-launches- review-to-improve-university-research-funding, accessed 31 May 2017.

2 For a summary of definitions of impact used in international development, see Hearn and Buffardi (2009): https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-
files/10302.pdf, accessed 12 June 2017.

3 For example, the World Bank’s definition: ‘the indicator of interest with and without the intervention: Y1 - Y0’: White (2009): www.3ieimpact.org/media/filer_public/2012/05/07/
Working_Paper_3.pdf, accessed 20 May 2017.

4 For examples of intended, positive, negative and unintended, see Hearn and Buffardi (2009: p.9): www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10302.pdf;
also, see Hearn (2016): http://www.researchtoaction.org/2016/02/what-do-we-mean-by-impact/, accessed 30 May 2017.

5 For a discussion on pathways to impact see: http://www.rcuk.ac.uk/innovation/impacts/, accessed 31 May 2017.

4
capacity- building.6 We include enduring connectivity, to recognise that ‘research does not happen in isolation’ – as achieving impact
includes ‘building networks of people and organisations able to understand the research and make use of it’ during the research and
after the research has ended.

Table 1  Definitions of impact7

Impact type Examples

Conceptual: impacts on •• Until recently many people in East Africa thought that the local production of medicines could be detrimental but
knowledge, understanding and the project provided evidence and engaged with policymakers to help them understand that the local production of
attitudes medicines could actually improve health care quality and access and stimulate the local economy (McSherry 2017)
•• The Youth Forward initiative helped to strengthen the case for the Mastercard Foundation to change its
definition of youth from the standard UN definition (15 to 24 years) to one that is more in keeping with the
generally accepted definitions in many African countries (12 to 35 years)
Instrumental: impacts on •• As a result of research recommendations weather index insurance was incorporated into the Ugandan
changes in policy and practice Agricultural Strategic Plan and the Agriculture Sector Development Plan (McSherry 2017)
•• See DEGRP’s website section on impact
Capacity-building: impacts •• By providing a data set and developing an analytical framework the project jumpstarted relationships between
on the ability of researchers to UNECA and the National Statistical Agency to develop national accounts and employment data. (Cassidy 2017)
conduct similar work in future •• Interaction between consortia working on the Youth Forward initiative led to a sharing of best practice around
sexual and reproductive health (SRH) training for young participants. As a result, one consortium began also
training its young farmers in SRH

Enduring connectivity: impacts •• The project forged new and deepened existing connections with policy makers in Ghana’s Ministry of
on the existence and strength Environment, Science, Technology and Innovation (MESTI) and the Ministry of Trade and Industry (Fu 2017)
of networks of people and •• Youth Forward Committees that engage policymakers and share the learning from the initiative were set up in
organisations who understand and both Ghana and Uganda, strengthening the national networks of those interested in youth employment issues
can make use of the research

Identifying different types of impact

Research projects may be able to demonstrate all types of impact during their lifetime, but not in any predetermined proportion or
order. It is up to researchers to develop their argument on how to prioritise the different types of impact and how much effort to put
into achieving each one. This toolkit offers guidance on doing this.

Identifying the type of impact you are aiming for can help in recognising when a contribution has been achieved. However, this early
thinking, perhaps as part of developing your Pathways to Impact, should not be at the expense of being open to new and unexpected types
of impact that may arise during the course of your research. In both policy and practice, it is usually impossible to predict which types
of impact will be achieved and on what scale recognised in OECD-DAC’s broad definition, which includes unintended and indirect impact.

Policy change impact (within RCUK’s economic and societal definition of impact) is often overemphasised, although individual
research projects are not expected to directly influence decision-making. Instead, research is expected to make a plausible
contribution to decisions through the ‘web of influence’ through which research impact can be felt (British Academy, 2016).

In understanding impact, it is important to consider complexities such as unexpected impact; impact over the longer term;
contribution and co-creation; and contextual factors. These make measuring impact particularly challenging. Indeed, the 2014
REF case studies were assessed in terms of their (1) reach – ‘the spread or breadth of influence or effect on the relevant

6 For the ESRC-preferred definitions of impact see: www.esrc.ac.uk/research/impact-toolkit/what-is-impact/, accessed 30 May 2017.

7 These come from ESRC-preferred definitions of impact as set out in Meagher, L. (2012) ‘RELU societal and economic impact evaluation’. Evaluation prepared by Technology
Development Group (mimeo). (www.relu.ac.uk/research/Innovation%20in%20Prog%20Management/Relu_Impact_Evaluation_Final_Report_tcm8-22271[1].pdf). The fifth category
used to evaluate RELU, fostering interdisciplinarity, is not used here as it is not a focus of impact in REF.

5
constituencies’, and (2) significance – ‘the intensity or the influence or effect’. Therefore, these are key elements of impact that, as
far as possible, should be planned for and monitored during the research.

Contribution
What role do you, as the researcher, play in achieving impact? It may be that your influence is direct, or it may be indirect. In some
cases, we may be able to establish a direct link between a piece of research and a particular change, as the impact may result
directly from one research paper, for example.

However, in other cases the impact may result from a body of research involving many different researchers. It may be the case that
the impact is not realised until sometime after the research is finalised. This longer-term impact is assessed by the REF exercise as
‘historic evidence of impact’ (RCUK, 2014).

Demonstrating change and the significance of the change is therefore important, with proxies such as co-production being used to
illustrate contribution. In assessing the nature of your contribution, it is helpful to systematically consider how direct or indirect the
influence of your research on the impact has been, and whether there are other social or cultural factors that may have played an
important role. It may be possible to discern the following types of contribution or influence:

•• ‘Direct, attributable impacts’, such that without the research there would have been no impact;
•• ‘Plausible and distinct contributions to sustainable change’, such that there has been a plausible contribution to lasting change and your
contribution can be distinguished from other contributions (see Pasanen and Shaxson, 2016);
•• ‘Plausible and indistinct contributions’, where it is not possible to separate the contribution of your research from other influences, yet
its influence is likely to be merged with others and so therefore contributes to the combined impact.

Contextual factors
Impact through the production of evidence does not take place through a neutral bureaucratic process of identifying and implementing
a ‘best practice’ option. Decision-making processes are affected by the context in which they take place, how different stakeholders
use evidence to inform their positions, how they relate to each other and how evidence is brokered between them (Jones et al., 2012).

Du Toit (2012) points out that it is not simply about providing evidence: it is about ensuring these rigorous analyses inform goals,
strategy, policy design and implementation by adding value to what is already being done. ‘Pro-poor policy interventions take place in
complex, dynamic, open social systems’ (ibid., p.3) and they are generally implemented via messy partnerships (Guijt, 2008). Hearn
and Buffardi (2016) present dimensions of impact to arrive at a contextually appropriate understanding.

Some factors to consider when assessing the context, specific to each research project, include the following (Table 2):

Table 2  Assessing the context

1 Who are the key policy actors and decision-makers in the field?
2 Is there a demand for research and new ideas among these actors?
3 What are the sources of resistance to evidence-based policy-making?
4 What is the policy environment?
1. What are the policy-making structures?
2. What are the policy-making processes?
3. What is the relevant legal/policy framework?
4. What are the opportunities and timing for input into formal processes?
5 How do global, national and local political, social and economic structures and interests affect the room for manoeuvre of policy-makers?
6 Who shapes the aims and outputs of policies?
7 How do assumptions influence policy-making? To what extent are decisions routine, incremental, fundamental or emergent, and who supports or
resists change?

6
Planning for impact
It is important to understand the likely impact of research at an early stage by developing a Pathways to Impact statement, as it
helps in understanding ‘what you will do and how you will act to enable your research to connect with others and make a difference
conceptually and instrumentally’ (ESRC, 2017). It is also an important requirement of funding, as ‘Research Councils require
academics to consider the future impact of research at the point of applying for funding’ (RCUK, 2014b). Describing your Pathways
to Impact is a requirement for the vast majority of funding proposals and, even in the exceptional cases where it is not required, the
‘Pathways to Impact statement should be used to fully justify the reasons why this is not possible’ (ESRC, 2017).

This is not to suggest that the expected impact will remain constant throughout the research; rather, it should be updated and
adapted as the context changes and the research trajectory unfolds. Indeed, research councils (RCs) ‘recognise the emergent
and long term nature of research and of research outcomes’ and seek to support flexibility and autonomy in ‘project definition,
management, collaboration, participation, promotion and the communication of research outputs’ (RCUK, 2014a).

TASK: understanding impact in your research project

Time 10–20 minutes

Resources 1 x flip chart paper and pen per group
Purpose To start thinking about what impact means for your research project
Activity In your research team, consider the following questions:
1. What do you understand by impact?
2. What does impact mean for your research?
3. How does your organisation define or understand impact?
4. What are the differences between each of these?
5. Are these definitions written down or discussed?
Notes

7
 MODULE 2

IDENTIFYING YOUR
IMPACT OBJECTIVE
‘Impact objectives revolve around getting your research known and used amongst those who can benefit most from it’ (ESRC, 2017).

To have impact with your research, you need to be clear about what sort of change you want to see and what ‘impact’ means for your
research. An impact objective is a statement of how you intend to do this, and it is required by the Pathways to Impact statement and
the impact summary, where you need to outline who will benefit from the research and how. This will also help define a good research
communications strategy, to ensure messages from research reach
those who can make a difference throughout the decision-making
processes. It will determine the types of evidence you will collect Take-away messages
during the research and implementation of the communications •• Developing an impact objective helps you be
strategy, and in turn the monitoring of impact will inform the writing clear about the change you are aiming for and
of the impact case study for REF 2021. what ‘impact’ means for you. This will guide your
communications strategy and the evidence you
To identify your impact objective, it is important to situate your need to collect.
research within a broader policy context. A good impact-oriented •• Most research is done in collaboration in with
objective should be clear about why the changes you are proposing others. Building the capacity of others along the
are important, who they affect, what needs to be done about it and way may involve working in partnership, as part of
where you stand in relation to others who are also trying to bring a coalition.
about change (adapted from ROMA, Young et al., 2014). •• Having a good understanding of the wider policy
context is necessary before you can formulate
In this module, we provide guidance on determining your impact your impact objective.
objective, considering what you can reasonably expect to be able
to influence and how you may achieve such influence.

Box 2  The importance of collaboration

Most research is done in collaboration with others and, particularly in international development, there is a desire to build
your partners’ capacity along the way. This may involve working in partnership, or as part of a coalition – particularly if you
are working on a complex problem that involves a large number of actors.

The process of working together will involve discussions about objectives, aims and terminology. These may highlight
different values, language and assumptions partners have.

This process brings those involved together by developing a commitment to common values, rather than tightly defined objectives.
This helps provide the flexibility to navigate the implementation path and observe impact as it arises, particularly where it may
differ from the type of impact originally expected. The initial objective for the coalition, therefore, should be quite broad, leaving
room for each coalition partner to formulate sub- objectives that better reflect their particular goal and anticipated impact.

Source: Young et al. (2014).

Figure 1  Sphere of influence

Outputs: Outcomes: Impact

inputs, changes in and
activities behaviour context

Sphere of Sphere of Sphere of

CONTROL INFLUENCE CONCERN

9
 Box 3  The results chain

Research councils recognise that it is important to capture the outputs, outcomes and impact of the research they fund to
demonstrate the wider value of the research to the UK society and economy (RCUK, 2014).

The ‘results chain’ refers to the order in which results arise, starting with activities and inputs, that produce outputs, which
lead to outcomes. Outcomes can be described as ‘intermediate’ and either ‘final’ or ‘longer term’. These interact with
contextual features to produce impacts; the figure below illustrates the order.

Figure 2  The results chain

Activities/inputs Outputs Outcomes Impacts

Source: Young et al. (2014).

Box 4  Grounding your impact objective in the context

A relevant impact objective should be appropriate to the wider policy context. Therefore, having a good understanding of the
wider policy context is necessary before you can formulate your impact objective. Understanding the context includes not just the
nature of the problem but also the direction that current policy is taking (which may be no direction at all), or it may be that there
are competing policy directions (e.g. from different political parties) that need to be considered.

What you can reasonably expect to influence

The sphere of influence framework is useful to determine what is and what is not within our power to influence. It originates from
Outcome Mapping (OM), developed by Sarah Earl, Fred Carden and Terry Smutylo from the International Development Research
Centre (Earl et al., 2001). OM is a way of planning international development work and measuring its results. It is concerned
with results – or ‘outcomes’ – that fall strictly within the programme’s sphere of influence. The focus is on changes in behaviour,
relationships, actions and activities of the people, groups and organisations you work with directly (Young et al., 2014).8

Inputs, activities or outputs within the sphere of control are those where we have complete control over our actions. The sphere of influence
is where events are not completely in our control but where we do have the power to change the outcomes (Tsui et al., 2014). On the other
hand, the sphere of concern focuses on events occurring over which we have no control. They often represent actions taken by others
or the results of a cascading set of external factors. Although this is where the results that really matter lie (such as better education, quality
health care, secure livelihoods), to have an influence in this sphere you have to work with others. Results within the sphere of control
and sphere of influence are those that you can measure and can use to guide your strategy and engagement plan (Young et. al., 2014).

Table 3  FINER criteria for a good research question

F Feasible •• Adequate number of subjects •• Adequate •• Affordable in •• Manageable

technical time in scope
expertise and money
I Interesting •• Getting the answer intrigues investigator, peers and community
N Novel •• Confirms, refutes or extends previous findings
E Ethical •• Amenable to a study that institutional review board will approve
R Relevant •• To progressing knowledge in your field •• To policy •• To future
development research
Source: Adapted from Hulley et al. (2007), cited in Farrugia et al. (2010).

8 For more information, visit the OM Learning Community: www.outcomemapping.ca

10
TASK: identify your impact objective

Time 20–40 minutes

Resources 1 x flip chart paper and pen per group
Purpose To develop a clear and concise impact objective
Activity In your research team, answer these seven questions:

1. What is your research question and/or hypothesis?

See Table 3 for guidance on developing a strong research question.

2. What wider policy problem or issue does this feed into?

For example, if your research is on youth perceptions on agriculture in eastern Uganda, the wider policy problem may be
youth unemployment in the region.

3. How can your research contribute to understanding or solving this problem or issue?

What information or perspective will your research bring to light? What influence do you expect this to have? Will you engage
or encourage participation from local stakeholders?

4. Who will benefit from this research? How will they benefit?

Rather than just listing stakeholders or making broad statements about how your research will ‘provide new insights’ or
‘increase understanding’, the purpose of this exercise is to dig deeper. How will new insights or greater understanding make
a difference to people’s lives? Will the research achieve this by engaging with local stakeholders?
For example, a greater understanding of how young people perceive agriculture in eastern Uganda may help policy-makers
create policies that incentivise young people to access available jobs in agriculture – this will help young people build a
sustainable livelihood and feed their families.

5. What is your high-level impact objective?

This is the ultimate impact objective you would like to see the research achieve. This may take place beyond the end of the
research and it may be outside your sphere of influence. Critically, think about this and how you may need to adjust it to
reflect your responses to questions 1–4. It could be that you have more than one high-level impact objective.

6. What are your impact objectives within your sphere of influence?

Examples may be to build awareness of the research among a target audience, secure the commitment of a target group
of stakeholders to the research aims, influence specific policies or policy-makers on key aspect or encourage participation
among researchers or partner bodies (ESRC, 2016).

7. What would you point to, to indicate that change has happened and that your research has made a difference?

What observable changes among these groups might you see? Is it plausible that your research has contributed to these?
Think about this for your impact objectives both within your sphere of influence and for high-level impacts.
Notes

11
 MODULE 3

STAKEHOLDER MAPPING
Stakeholder mapping has various uses in developing effective Pathways to Impact and for putting in place an effective strategy for
achieving impact. It can be used to identify:

•• research users to consult during the research planning stage, to discuss the role they may play in planning the research, such as
co-investigators;
•• opportunities for building strategic partnerships and strengthening relationships;
•• and actively engage relevant users of research and stakeholders at appropriate stages.

Developing maps of how well the various stakeholders are aligned with the research, and the degree of interest they have in the
issue, is a valuable exercise. Building relationships and partnerships can greatly enhance the quality of research and can enhance
impact as these stakeholders may influence change.

An Alignment, Interest and Influence Matrix (AIIM) can help ensure you consider the full range of stakeholders who may be
influenced by your research in some way. It is a simple stakeholder mapping tool with three dimensions: the degree of interest
in the research topic; the degree of alignment with the thrust of the research; and the degree of influence they have within the
policy-making process.

Stakeholder mapping is best done in a group, so you can draw out discussions and perspectives around who may be interested in
your work and who you need to reach to achieve your impact objective.

13
Box 5  Alignment, interest, influence

Definitions

Alignment: Is the research likely to lead to policy the stakeholders broadly agrees with? Do they share the same sense of
potential importance and agree with your approach?

Interest: Are they interested in the issue your research is addressing? Are they committing resources to the issue? Do they
want change? Are they speaking about the issue?

Influence: How important are they to making change happen? Are they in a position of authority? Can they put pressure on
a decision-maker?

Groupings

Top right quadrant: Stakeholders in this quadrant are both interested in your issue and aligned with your approach. These
people are your collaborators and allies. You may want to ‘work in partnership’ with this group. Perhaps you could form a
community of practice, or set up meetings with key organisations or individuals to share ideas or coordinate activities.

Top left quadrant: Stakeholders in this quadrant agree with your approach, but they are not interested or prioritising
the issue. This could be for several reasons – they may not know much about the issue, have limited capacity or have
competing priorities. If there are influential stakeholders in this group, start to think about how you can develop their interest
or capacity to get involved.

Bottom right quadrant: Stakeholders in this quadrant are interested and perhaps active in your issue area, but they don’t
agree with your approach or position. If you need to get stakeholders from this group on board, you’ll need to challenge
their current thinking and persuade them of your position. Evidence usually works best for this group. Working through
champions who are aligned with your position can also be effective.

Bottom left quadrant: Stakeholders in this quadrant do not know much about the project, or if they do, they are not very
interested and probably do not agree with your views. If they are not

Influential in your project’s success, you may prefer to ignore them as a group. Ultimately, you can not do everything and
may need to focus your energies on the actors in the other three quadrants. It may also be hard to know who these actors
are, because they might not even be known to you. If they are potentially influential, you may need to just monitor them to
ensure that they don’t move into another quadrant and start to cause your project harm.

High alignment with approach

DEVELOP INTEREST WORK IN

OR CAPACITY PARTNERSHIP

Low interest or High interest or

engagement in issues engagement in issues

IGNORE CHALLENGE
OR MONITOR OR PERSUADE

Low alignment with approach

Source: Young et al. (2014).

14
TASK: map your stakeholders

To get the most out of this activity, it is best to do it as a group exercise, either with your immediate team or with those working
closely with you. If possible, have a good variety of people in the group, including research, communications and monitoring and
evaluation staff or the implementation team. The greater the range of people the more detailed your analysis will be. If your group is
larger than eight people, it may be helpful to allocate someone the role of facilitator.

Time 60 minutes
Resources 1 x flip chart paper, marker pen, up to 20 sticky notes per group
Purpose To identify:
•• research users to consult during the research planning stage;
•• opportunities for building strategic partnerships and strengthening relationships;
•• relevant users of research and stakeholders'
Activity 1. Place the flip chart paper on the wall or in the middle of the table and draw the matrix. Write your policy objective at the
top of the page.
2. Regarding your policy objective, list all the stakeholders you can think of. Write them down on the sticky notes (one per
note). Be specific and refer to different individuals or teams, rather than just saying the Ministry of Agriculture. This is
important as different teams may have different positioning.
3. Select those stakeholders who have the highest or the lowest interest or alignment.
4. Take turns to place the sticky notes on the matrix, placing the notes for the four stakeholders that are at the most
extreme points: so most interested/most aligned, least interested/least aligned, most interested/least aligned, etc. This
allows an analysis of their relative rather than absolute levels and prevents all stakeholders being concentrated in the
top right hand corner of the matrix.
5. As you place the sticky notes, explain to the group why you are placing the stakeholder in that position. Discuss whether
people agree or not. Remember you can have more than one sticky note per organisation; if an individual or team has a
different degree of interest, alignment or influence, separate them out.
6. Add dots to the sticky notes to show how influential the stakeholder is to the success of your research. Add three dots if
they have a lot of power or influence; two dots for reasonable influence; one dot for minimal influence; and no dots for
no influence. This will help you prioritise your efforts and resources later on.
7. Once all stakeholders are on the matrix, start to look at the groupings. Refer to Box 5: Alignment, interest, influence
matrix – groupings. Look at the individuals and groups that are most influential to you. Are they where you want them to
be on the matrix? If not, use the pens to draw arrows to where you would like them to move to.
8. Now add a ‘tick’ to any sticky note you have already had contact with or have access to.
9. Think about what strategies you may want to implement to respond to your findings.
Notes

15
 Box 6  AIIM in action

An AIIM analysis was conducted to support the objective of passing legislation to support mothers to breastfeed for up to two
years in Indonesia. This highlighted which stakeholders could be most effectively worked with and those whose perspectives
were conflicting – for example stakeholders supporting the supply of powdered milk instead of encouraging mothers to
breastfeed. Other stakeholders changed their position on the matrix at different stages of policy implementation. For example,
parts of the government supported breastfeeding after the legislation was passed but were not active in supporting the legislation
to be enacted at the district level.

Figure 3  Example of an AIIM map in action

Source: The regional pastoralist livelihoods advocacy project, June 2008. London: ODI.

16
 MODULE 4

DEVELOPING A THEORY
OF CHANGE
Outlining a Pathway to Impact involves understanding how change can come about and this, in turn, will underpin the changes that
are captured in the impact case study.

Developing a theory of change (TOC) involves refining ideas of how the research project’s work with stakeholders may lead to
change. Such change usually happens as stakeholders engage with the research, subsequently take on board the key messages
and finally internalise the issues to such an extent that they can
effectively act on behalf of the researchers to spread the message
themselves. This means developing a TOC.9 There are many
different types of TOC. In this module we provide guidance on Take-away messages
one that reflects the underlying outcome mapping (OM)-based •• Theories of change are not static: it is worth
principles of our approach. revisiting them at least annually to assess whether
plans need to be revised.
There are three steps in developing a TOC: •• Change is not always positive: capturing instances of
no change or negative change often provides valuable
1. Analyse the current context: what are the current ideas, interest insights. If you succeed in anticipating a backlash or
groups and processes. How do these influence policy-making? criticism, then you will be armed with a response.
2. For the key stakeholders, examine the changes in behaviour
you would:
a. expect to see, to indicate initial engagement with the research.
b. like to see, to indicate that the messages are being taken on board.
c. love to see, to indicate that the messages have been deeply internalised.
3. Identify what the research project and others will do, and check assumptions about how these are related.

Box 7  What is a theory of change?

A TOC helps researchers conceptualise what impacts they may have, and how they may be sequenced and achieved given the
wider context within which the project operates. In Vogel (2012), Julian Barr from Itad defines a TOC as follows:

A TOC is analogous to ‘Google maps—this is the territory, this is how we see our bit of the territory, and this is the route that
we think is best to take through it (though, like Google Maps, we recognise there may be a couple of different routes across the
territory, but we have explicitly chosen one). Based on our understanding of how the territory along the road works, this is how we
shall approach the journey, and these are some of the landmarks we expect to see along the way.’

Theories of change are not static: it is worth revisiting them annually, or more frequently if things are changing rapidly, to assess
whether plans need to be revised. This is particularly important for long-term research projects, when there may be changes in
context or partners. The same three steps would be followed, asking: has the context changed? Have the roles of key stakeholders
changed and what does this mean for the changes you would expect, like and love to see? Does this affect what the research
project does, or what others will need to do to bring about change?

Change does not have to be positive. Interesting lessons are often learnt from recognising negative changes, or where there has not
been any change. Often, the most interesting observations and lessons result from exploring why no change or negative outcomes
may have arisen. If, in your Alignment, Interest and Influence Matrix (AIIM), you’ve identified that a stakeholder is very interested but
not aligned with your approach, you may expect a backlash from them or critical comments, as they disagree with your approach.
This may be a good thing: if you’ve anticipated it, and it happens, and you have a plan to deal with it, then it can be framed as
an expected reaction and logged as an achievement. The more political the issue you are working on, the more likely you are to
encounter this sort of reaction. Box 8 considers orders of change in terms of the link between ideas and impact in a policy context.

9 What is presented here is one version of what a theory of change may be. See Vogel (2012) for a comprehensive review of theories of change: www.gov.uk/government/news/dfid-
research-review-of-the-use-of-theory-of-change-in-international-development.

18
 Box 8  How policy change happens

Orders of change were originally described by Hall (1993) to discuss the process of social learning. He explores the link
between ideas and impact in terms of policy- making. This is most useful at the planning phase to understand how deep
and transformational is the impact you are anticipating. Hall separates policy change into three levels:
•• First order change focuses on incremental changes. An example would be to increase the minimum labour wage in
response to protests.
•• Second order change occurs when policy instruments are completely altered. Following on with the above example, the
government would use a living wage instead of a minimum wage as a form of a social safety net.
•• Third order change takes place when a new hierarchy of policy-making goals is created. Hall calls this a ‘paradigm shift’.
For example, the minimum wage and the living wage policy would no longer be a priority as other goals would be a higher
priority, such as taking account of non-economic social contributions in assigning benefits.

It is important to remember that first and second order policy changes occur in normal policy-making. First and second
order of change often may not lead to the third order of policy change.

However, it may be that people are so resistant to change that a ‘love to see’ change may only be what Hall refers to as a
first order change. Depending on the policy context, it may be sufficiently ambitious to aim for a first order change.

Sources: Adapted from Tsui et al. (2014) and Hall (1993).

Step 1 Analysing the current context

This first step is often overlooked, but it helps develop a good baseline and thus a more accurate analysis of how change may
happen and the impacts that may be possible. The depth of analysis done at this stage may range from a full-blown political
economy analysis to a more focused understanding of the relationship between research evidence and policy.10

The key issues to consider are:

1. Knowledge and information. What knowledge is currently used in debates around the policy issue? For example, is it knowledge
from formal research; from past experience, perhaps published in evaluations, or else taken as ‘common knowledge’; is it widely
shared opinions; or do ideas come largely from ideologies and beliefs? Which of these types of knowledge dominates?
2. Actors and stakeholders. Whose voices are the strongest in debates? Whose evidence is currently seen as credible and why?
What networks exist between different stakeholders? Are there stakeholders whose voices are marginalised? Who decides what
knowledge counts, who arbitrates between contrary facts and opinions?
3. Knowledge intermediaries. Are there actors who broker debates, bringing new ideas into the discussions, synthesising evolving
understanding and communicating with stakeholders? Are there other active disseminators, such as lobby groups, civil society
organisations, press and media, who actively participate in debates around the issue?

For all of these three issues, consider the processes through which impact can take place. For instance, how does knowledge
circulate? Is it written down, is it oral? Is it public, or do critical debates and decisions take place in private networks behind closed
doors? Does the political context affect how knowledge of your issue flows around debates?11

The point is to look for any major aspects of the current context that may affect how change happens, particularly the sorts of changes you
may want to see as a result of your research. This analysis will also help you identify and capture unanticipated changes, if they arise.

10 See DFID’s how-to note: www.gsdrc.org/docs/open/PO58.pdf.

11 This does not necessarily mean party political issues (though could include them): it means issues where the balance of power between stakeholders is an important consideration in
how they act and how they relate to each other.

19
Step 2 Developing theories of change
We are encouraging an approach to developing theories of change based on the principles of OM (Earl et. al., 2001). This looks at
changes in people’s actions and behaviours, not changes in the things that are produced. It distinguishes between three different
levels of change:

1. Changes we would expect to see: the early positive responses to your work (such as attending meetings convened by the
research programme, giving feedback on a publication).
2. Changes we would like to see: active engagement with what you are doing (such as inviting you to attend one of their
meetings, asking for information on research-related issues).
3. Changes we would love to see: deeper transformations in behaviour that indicate that your messages have been completely
internalised (such as co-opting a collaborator onto a standing committee, incorporating one or more messages from your
research into a strategy document or taking a decision about resource allocation).

The lines between the different changes are blurred, and it is a matter of judgement as to which change falls into which category.

Figure 4 represents an example of how a research project may have outlined the expect, like and love to see for one of their key
stakeholders, the UK Department for International Development (DFID).12

Figure 4  Using outcome mapping to develop a theory of change

Given our understanding of the current context, these are the behaviours we would…

…expect …like …love

to see to see to see

Early positive responses Active engagement Deep transformation

to the research with the research results in behaviour

Table 4  Progress markers

Expect to see We would expect to see that the Drivers of Change conceptual framework is published in a DFID how-to note as it responds to DFID’s
identified needs to identify ‘the long term structural and institutional factors which enable or constraint reform in different countries’
and ‘to improve policy making and programming by identifying short and medium term opportunities to support strategic change’.
Like to see We would like to the framework captured in DFID’s guidance notes such that DFID’s approach change from one of providing
‘assistance to promote particular agendas around governance, growth or service delivery’ to consider how to provide support, taking
political feasibility into account.
Love to see We would love to see the framework informing DFID’s white papers to allow DFID to influence others e.g. through membership of the
OECD Development Assistance Committee.

Theories of change can be presented in table form, as below in Table 5. It is helpful to separate out your general statement of what
the changes will look like (‘Department X begins to actively seek out emerging research results’) from more specific indicators
(‘local collaborator is invited onto the standing committee for issue X’), which can be tracked. Working into the specific indicators of
change and ensuring their sequence s logical, and can be a helpful way of checking the overarching logic of the TOC.

12 This comes from the REF case study on Thinking and Working Politically from the University of York, available at: http://impact.ref.ac.uk/CaseStudies/CaseStudy.aspx?Id=43466.

20
It is important not to assume that all behaviours in the same category must happen at the same time across the different TOCs.
Change in social processes is decidedly non-linear; people may block change for reasons that aren’t immediately clear, or they may
suddenly get the point as a result of several things happening simultaneously, most of which are outside your control. Developing
different TOCs for different stakeholders helps you unpick the assumptions about how change happens. Research Councils UK
(RCUK) note how impact can occur at any stage of the research and often as a result of unexpected actions (2014). Therefore,
continual monitoring and updating of your TOC can help capture such surprises when they arise.

Table 5  An initial theory of change

General statement of change Which stakeholders are involved? Specific indicators

Current context:
Expect to see: early positive response
Like to see: active engagement
Love to see: deep transformation
in behaviour

Step 3 Identifying the research project's role and checking assumptions

The final step looks at the contributions to change made by the research and by others. This adds on two columns. The first sets
out what the research will contribute; the second looks at what we are assuming about how others will contribute:

Table 6  A final theory of change

General Which Specific What will What will others

statement stakeholders indicators the research contribute?
of change are involved? contribute?
Current context:
Expect to see: early positive response
Like to see: active engagement
Love to see: deep transformation
in behaviour

Having gone through these three steps, it should now be possible to revise the research project’s impact pathway, refining your
understanding of the different types of impact your research can have, being clear about the limits of your research, what needs to
be achieved through others, developing some concrete indicators of impact, including for the longer-term impacts of your research.

The Climate Development Knowledge Network (CDKN) has used progress markers to develop its monitoring and learning framework,
based on changes in behaviour in local communities. CDKN is jointly run by Price Waterhouse Coopers (PwC) and ODI; its main goal is
to influence countries in their climate change negotiations. Table 7 shows the progress markers it developed as an example.

21
Table 7  Example progress markers

Outcome challenge: the programme intends to see local communities recognising the importance of, and engage in, the planning of resource
management activities in partnership with other resource users in their region. These communities have gained the trust of the other members of the
partnership and the recognition of government officials, so they can contribute constructively to debate and decision-making processes.

They are able to clearly plan and articulate a vision of their forest management activities and gaols that is relevant to their context and needs. They call
upon external technical support and expertise as appropriate. They act as champions for model forest concepts in their communities and motivate others in
the partnership to continue their collaborative work.
Expect to see local communities
1 Participating in regular model forest partnership meetings
3 Establishing a structure for cooperation in the partnership that ensures that all local interests are represented (mechanics of setting up the structure)
4 Acquiring new skills for involvement in the model forest
4 Contributing the minimum human and financial resources necessary to get the model forest operational
Like to see local communities
5 Articulating a vision for the model forest that is locally relevant
6 Promoting the model forest concept and their experiences with model forests
7 Expanding the partnership to include all the main forest users
8 Calling upon external experts when necessary to provide information or meet technical needs
9 Requesting new opportunities for collaboration with other institutions and actors
10 Identifying opportunities for collaboration with other institutions and actors
11 Identifying opportunities for, and successfully obtaining, funding from a range of sources
Love to see local communities
12 Playing a lead role in resource management with a view to medium- and long-term benefits
13 Sharing lessons and experiences with other communities nationally and internationally to encourage other model forests

Source: Hamza-Goodacre et al. (2013).

22
MODULE 5

COMMUNICATING
YOUR RESEARCH
Your research will only have real world impact if it reaches the right people. Therefore, when planning your communications,
it’s important to think beyond one-directional ‘dissemination’ of research findings. Communications is much broader; it’s about
engagement and knowledge exchange. Research Councils UK (RCUK) defines it as a ‘multi-way exchange of knowledge between
academia and research users in business, public and third sectors’ (2014).

When we talk about ‘communications’, we use the word in the broadest sense. Communication can take many forms – formal
and informal. This includes digital engagement, social media,
publications, field visits, public events, emails or private meetings.
This takes investment: ‘growing effective knowledge exchange
with non-academic communities takes time, skills, confidence Take-away messages
and money’ (ESRC, 2017). •• Think ‘engagement’ not dissemination. We talk a
lot about research ‘dissemination’, but this is one-
The key to any communications activity seeking to engage directional and communication is much broader;
someone with your work is to: know who you want to reach; it’s about engagement and knowledge exchange.
know what you want to do; and work out how you want to reach •• Don’t leave it to the end. Communications is often
them – this means thinking about the channels and tools you will left to the end of the research process, but it is plays
use and what messages they will relate to. a critical role throughout the project’s lifetime.

It is often assumed that communications are done at the end of a

research project. In fact, knowledge exchange and engagement
play an important role throughout the project and research
lifetime. Engaging audiences early in the research process can help ensure relationships have crystallised by the time you come to
actually communicate your findings. It also gives you more time to plan, adapt and above all understand your audiences.

This module covers:

1. Knowing your audience

2. Crafting effective messages
3. Planning your outputs and activities
4. Presenting your research

Step 1 Knowing your audience

The audience is (or should be) at the heart of any communication. Before you can begin communicating your research, you
need to be clear about your aims: who do you want to reach and what do you want them to do with the information? Effective
communication material needs to be carefully tailored and adapted for different audiences.

It is helpful at this point to look back at your impact objective and reflect on the following:

1. Is there a problem you are working to address?

2. Are you offering a solution (or several solutions)?
3. Are you providing new information to support others to find solutions?
4. What do you want your communications to achieve?
5. Who do you need to engage with to achieve your goal?

24
TASK: identify your audiences

You will have already identified your key stakeholders and courses of action for each as part of Modules 3 and 4 of this toolkit.
This exercise allows you to identify your specific audiences for communications and engagement activities. Remember that your
audiences and communications objectives may change throughout the lifetime of your research. For example, at the beginning,
your communications may be about building partnerships and later on it may be about influencing action. It is helpful to have a
facilitator for this exercise.

Time 60 minutes
Resources 1 x flip chart paper, marker pen, up to 20 sticky notes per group
Purpose To develop a clear and concise impact objective
Activity 1. Write your impact objective on a large piece of flip chart paper so everyone can see it.
2. Put four sheets of flip chart paper on the wall. Label the first ‘Audience’, the second ‘Objective’, the third ‘Message’ and
the fourth ‘When’, and distribute sticky notes to participants.
3. Invite participants to write a key audience on a sticky note, contributing up to three each. Be as specific as possible
(i.e. rather than writing ‘policy-makers’, write ‘the Minister of Education’).
4. Invite participants to place their sticky notes one by one on the first flip chart paper.
5. Invite one or two participants to cluster the notes into groups and explain why they have chosen that particular grouping.
(You may also want to refer to your stakeholder maps.)
6. Get into groups of three or four people and think about why you want to engage with that audience. Is it to bring them
on board as a partner? Is it because you want them to share your research with others? Or is it because you want to
influence their actions with your research?
7. Invite one participant from each group to add the objectives to the second flip chart paper, so that they are aligned with
the respective audience groups. This may lead to some discussion, try to reach a consensus.
8. In those same groups, think about what kind of information the different audiences will most likely be interested in, or
need. Write them down on sticky notes.
9. Invite each group to add their ideas to the third flip chart paper for each audience group. Again, this may lead to some
discussion and debate among the groups, try to reach a consensus.
10. In the same groups, think about when you will want to engage with that audience. This will link to the objective, for
example if your objective is to build a partnership you may want to do that early in the research process. You may find
that you need to engage with some audiences throughout the research.
Notes

25
Step 2 Crafting effective messages
Identify your key message

A strong message should be at the core of any good report, presentation or other piece of communication. Research frequently focuses
on facts and data, but it is important to explore ways to tell a story through your messaging as well. Audiences, from academics to
decision-makers, are far more likely to want to interact with new information if they can relate to it in some way.

Key messages are ‘just the tip of the iceberg’ (Young and Quinn, 2012). They are what your audience most needs to know, and are
supported by the main information (the rest of the iceberg) as needed. If you give readers all the information in one go they are
unlikely to absorb it.

When planning your key messages, think about your audience. Who are they? What sort of message do you need to communicate
to them, and why? Thinking through several questions can help you shape the message then the channel for best conveying it:

1. Who are the audience? What role do they play? Are they specialists in the topic or are they non-technical? Are they people who
you already know?
2. What do they need to know? What are they currently working on? How can they use your information? Did they ask for the
information? If so, are you delivering what they wanted? Do they need to know more?
3. How do they best receive information? Do you know how they tend to prefer to receive communications? What has worked
well in the past (and what has not)? Is it better to approach them for a meeting or to send a report?

Make your message stick

There are multiple ways to develop strong messages. One great example is from Heath and Heath (2007). They talk about six
dimensions to a good message, which can be applied to all types of messaging, including evidence into policy. They are:

Simple. The best messages are always simple. This does not mean dumbed down. Rather, you are pulling out the key message
your audience needs to know. Keep it concise and don’t over-elaborate.

Unexpected. If you can make people sit up and listen you’re more likely to have impact. Counter-intuitive messages, a surprising
fact or a different angle can be one way to do this.

Concrete. Don’t be too abstract. Make your message tangible, relating to real life examples where possible. For example, rather
than saying 25%, say one in four people.

Credible. Your message can be clear, concise, well-constructed, but if it isn’t credible the audience is unlikely to listen. Is your
message backed up by evidence? Is it believable?

Emotional. We know emotions provoke reaction. Is there a way to explore emotions with your audience, make them connect or care
about the human story behind your research?

Stories. We use storytelling all the time for effective communications. Can your message tell a story rather than just hard facts?
What’s the impact of the evidence?

Next time you read a blog you find interesting or inspiring, a good exercise can be to spend some time looking at how much the
author uses the above dimensions when they try to convey their key messages.

26
Step 3 Planning your outputs and activities
Every research project needs to think about outputs and activities. While research may be published in an academic journal, if you
want to have an impact beyond academia you will need to think how to achieve this and explore other communication channels.

Remember that effective engagement is more than one-way

communication. As the Engineering and Physical Sciences Take-away messages
Research Council (EPSRC) writes, ‘done well, public engagement
can build trust and understanding between the research community •• Create packages not publications. Your impact will
and a wide range of groups, from policy-makers through to young be limited if you just produce one product; you need to
people’ (EPSRC, n.d.). Engaging stakeholders with your research tailor outputs and activities for different audiences.
can improve the quality of your research outputs and the impact, •• Be strategically opportunistic. Identify policy
raise your profile and develop your skills. Communication activities, windows and plan your research timetable to
therefore, are some of the most important actions you can take to make the most of these opportunities. However,
increase the impact of your research. unexpected or unplanned opportunities will also
emerge and you need to be ready to respond.
The outputs, channels and activities you use will depend on •• Don’t forget the messenger. Sometimes you may
what impact you want to have, who your audience is and what not be the appropriate messenger. For example,
resources you have. If you have communications support, involve you may not have the right profile, access or
them in the research planning stage to help identify early on what skills. You may need to partner up with other
outputs are likely to have the most impact and to make sure you organisations or individuals.
have the resources to do it.

Box 9  A spectrum of knowledge communication activities

This is a helpful framework to reflect on the different types of knowledge roles you can play. This may change throughout
your research lifetime, depending on what you’re trying to achieve and your resources. If you are clear about what your role
is and where you can achieve, you can then see where your limits are and where others can help.

INFORMATION KNOWLEDGE KNOWLEDGE INNOVATION

INTERMEDIARY TRANSLATOR BROKER BROKER

Influencing the wider

context to reduce
transaction costs
Translating research Actively engaging in and enable innovation:
Making information evidence for non- policy and practice build local capacity,
available: putting specialist audiences: debates: taking part in change incentive
research results into giving seminars, meetings, matchmaking, structures to
the public domain writing policy briefs convening, networking use research

Linear dissemination Co-production of knowledge,

of knowledge from producer to user social learning and innovation

Source: Shaxon, L. (2016).

27
Traditionally, we have always identified four key channels for communicating research – publications, events, digital and media. However,
today the lines between these channels are very blurred as digital communications becomes increasingly central to strategies.

Table 8  Example outputs, channels and activities

Types of output Why use it

Written formats
Journal article It presents your full research methodology and analysis. It is likely to be read by other academics or students so the language can
be technical (although should always be as clear as possible).
Research report It presents your full research and evidence. It is likely to have a broader audience than just other academics, so it needs to provide
a comprehensive and accessible overview of your research.
Brief Short, sharp targeted briefs can help communicate key recommendations to policy audiences.
Toolkit It presents practical tools, insights or experiences in a way that can be easily used or applied by other people.
Blogs and op-eds They can be used to provide an accessible and engaging ‘way in’ to bigger pieces of research or learning, as well as reaching
broader or targeted audiences.
Media release Designed to communicate key information to regional, national and international media outlets, it may be picked up by online and
print media publications. Media is especially useful for reaching influential people and policy-makers, as well as the general public
(depending on what media outlet picks up your work).
Digital tools
Website It can provide a central platform to host resources and make your work publicly available. By hosting everything in one place you
can track downloads or page views and find out who is engaging with your work.
Data visualisation It is used to present data visually, making it easier to understand or compare, to tell compelling stories or to draw attention to key
messages and statistics.
Online networks Tapping into existing or established online networks of people interested in research you’re working on is a good way to reach
wider, but targeted, audience groups. Newsletters can target a mass audience. It may be far more effective to ensure your research
appears in other newsletters, for example a university newsletter.
Email Often overlooked, email plays a critical communications role. A personal email from a researcher to a contact can be one of the
most effective ways to share research.
Types of output Why use it
Events and meetings
Public events They are one of the most common ways to showcase research on a public platform. They can be part of a conference, a public
panel or keynote event. Events may be streamed online or available for users afterwards through a video or podcast.
Policy dialogues They convene a range of policy stakeholders from different interest groups and with different perspectives, to focus on a policy issue.
Workshops They can be a good method to engage directly with key stakeholders and encourage small-group discussions.
Bi-lateral meetings Meetings are one of the most common ways to engage people early on with your project and to showcase results to several key
stakeholders, particularly decision-makers and donors.
Webinars Webinars can be a way to present new research online to audiences in different countries, and, particularly, can encourage new
learning and/or guidance for users.
Multimedia
Social media It allows you to reach larger networks of people who may be interested in reading, or using your research. This could be on Twitter,
LinkedIn and Facebook, as well as WhatsApp and SnapChat.
Multimedia This media may be better for reaching some audiences, such as young people or policy-makers. It can include videos, podcasts
and animation, but also photography.
Radio, TV broadcast Radio and TV have enormous potential to communicate to a wide audience. For example, in some developing and emerging
countries, community radio can be a critical way to reach key audiences.

28
TASK: map your communications activities

This exercise builds on TASK 1 ‘Identify your audience’. In this task you will start to identify the outputs, channels and activities you
need to reach your priority audiences. You can then determine who will do what and whether you have sufficient resources to deliver
your strategy. It is useful to note any capacity shortfalls you may be facing and possible interim strategies to address these. Activities
often take longer than you think, particularly those requiring working with external actors, such as developing partnerships.

If you have communications teams or support, it is important to include them in mapping your activities. It is also important to include
leadership of senior staff to draw on their experience. RCUK notes, ‘Commitment to include principal and senior investigator time on
knowledge exchange and impact activities, recognising that collaboration with users often requires a depth of experience and level of
understanding of the kind held by established researchers and which complements the skills of research associates’ (RCUK, n.d.).

Time 30-40 minutes

Resources 2 x flip chart paper, marker pens and sticky notes
Purpose To develop communication strategies for each audience group
Activity 1. Look back at the flip chart papers from the previous task. Re-familiarise yourself with your audiences, why you want to
engage with them and why.
2. Add a fifth flip chart to the wall titled ‘Communications activities’.
3. In the same groups, identify what communications outputs and channels may be best to reach the different audience
groups (e.g. a report, a policy brief, the media, an event, a face-to-face meeting, email etc.) Write these on sticky notes.
4. Invite participants to add their sticky notes to the flip chart paper.
5. Next, think about your resources (time, expertise and budget). Which of these activities are a ‘must’. Which are a ‘nice
to have’. Do you have the time, expertise and budget to deliver these activities?
6. Take a new flip chart paper and turn it landscape. Draw out the example table below and s
Notes

29
Table 9  Mapping activities example

Output/activity List all planned communications 1. Workshop 2. Journal article 3. Policy brief
outputs and activities.
Audience Who do you want to engage with Ministry X Academics in X field Minister of X
this output or activity?
Objective Why are you producing this output To establish a relationship To inform other academics To inform the minister
or carrying out the activity? Relate with the ministry; to get working in this research of our research results
this to your audience. a better understanding area, to support their work ahead of its annual
of the policy context for and share learning budget setting
the research; and to
identify the ministry’s
policy needs to inform our
research question
Staff time required How much – and whose – staff 15 days for lead researcher 30 days for lead 5 days, lead researcher
time is realistically needed to to plan, facilitate and write researcher to write and to write
deliver this work? up workshop coordinate peer review
3 days for
5 days for internal review communications support
Additional budget required What direct costs are involved? Flights, accommodation, Copy-edit Copy edit
sustenance
Peer review Design
Venue, catering
Proof-reading
Deadline By when does this output or End of scoping phase End of research phase End of research phase
activity need to be delivered (to (day, month, year) one (day, month, year) two (day, month, year)
meet the objective and to meet
your funding requirements)?
Priority/resources Is this a 'must have' or a 'nice to
have'? (Again, think back to your
overall impact objective.

Step 4 Presenting your research

Presentations

At some point during your research you are likely to need to present it – either formally or informally. This section examines some
of the ways in which you can improve your presentation skills and effectively share your key messages.

Tips for presenting well

•• Be clear about the purpose and audience for the presentation. This will help you focus on the information you most need to
prioritise and your key messages.
•• Structure your presentation. Can you identify one overarching message and three key points to share? The beginning should
capture your audience – quotes or surprising facts can work well. Then introduce the outline of your presentation. To close,
highlight your key points and any recommendations or next steps.
•• Plan your presentation and practice. Even the most experienced presenters practise, practise and practise.
•• Make sure it is clear and accessible. ‘If you can’t explain it simply, you don’t understand’ – so goes the quote by Einstein.
Try practising what you want to say with someone who does not know very much about the topic (a relative, friend, spouse or
colleague) and get them to give you honest feedback.
•• Stick to your time limit. Find out how long you will have to speak and prepare for less time than this, as it is normal to overrun. Also,
try to keep presentations short, or break it up with time for questions. Don’t just speak non-stop at the audience – they will switch off.
•• Find your passion. Stakeholders are more likely to listen to you if you sound confident but, above all, interested in what you are saying.

30
 Box 10  Pecha Kucha

Pecha Kucha means ‘chit chat’ in Japanese. It’s a presentation technique developed by two Japanese architects who felt
that architects had a hard time ‘getting to the point’. Pecha Kucha presentations are 20 slides, 20 seconds per slide,
6 minutes 40 seconds in total.

The 20:20 format gives you a structure to plan your presentation. You have only 20 seconds per slide before it automatically
moves to the next one, so it forces you to really identify the most important thing you want to say. It does, however, take
some practice and discipline. Pecha Kucha presentations work best when the slides are just images, with limited or no text.

Watch Pecha Kucha presentations: http://www.pechakucha.org/watch.

What not to do (common mistakes)

•• Don’t use your slides as a crutch. Slides are there to enhance your presentation, not duplicate it. People can’t listen and read at
the same time. If you need to, make notes, don’t read your slides.
•• Don’t overload your slides. Avoid putting too much information on one slide. Break up text with images, videos or quotes. Lots of
studies show people respond better to visual stimuli than text (Gallo, 2014). Be careful with graphs and data; they can be helpful
but can be difficult to read (see tips on data visualisation). Short videos can also be a good way to break up the presentation.
•• Don’t use PowerPoint until you have a structure. It pays off to spend time identifying your key messages and structuring your
presentation. Don’t start making your PowerPoint presentation until you’ve got a proper structure in place.
•• Do not assume PowerPoint is the only format. There may be other formats that are more appropriate for the presenting your
context and getting your message across. For example, Prezi is a visual tool that allows a different structure.

Briefs

Briefs don’t replace reports as you still need to be able to provide the detailed evidence. Instead, briefs are a useful tool to reach
specific audiences, particularly when they may not have time to read the full report or just need key information. They help present
your findings and recommendations in a way that is digestible.

The policy brief is a common form of brief used to target policy-makers. It typically has four functions: to explain/convey the
urgency of an issue; to present policy recommendations or implications on the issue; to provide evidence to support the reasoning
behind the brief; and to point the reader to additional resources on the issue.

Here are four main steps to help plan a policy brief:

1. Identify the purpose and overarching message of the brief.

2. Determine three recommendations/ implications if relevant.
3. Construct a logical line of argument for making these recommendations (your results or conclusions). When you are thinking
about your line of argument you can ask: what is the problem and how does my research fit into this? Why is it urgent/timely/
important? Why should the target audience know more?
4. Based on an understanding of the context around the issue, identify one or two entry points or hooks for the message. These are
commonly used at the beginning of reports, summaries or briefs to entice the reader into continuing to read.

Data visualisation and infographics

Data visualisation, the graphic representation of data or knowledge, can add value to your research by (ODI internal infographic guidelines):

1. Making data or information easier to understand or compare;

2. Telling complex stories in a condensed and compelling way;
3. Drawing attention to key audiences;
4. Engaging a wider audience.

Data visualisations showcase data or information in visual form, so there are a wide variety of options that fall under this category.
At the simplest level, a data visualisation could be a graph or chart, a timeline, a map or an illustration.

31
 According the UK Office for National Statistics, an infographic is ‘a self-contained visual story presenting information, data or
knowledge, with clear meaning and context and without bias’ (Office for National Statistics, n.d.). Infographics use visuals to tell a
story or relay a key message.

The type of data visualisation you use depends on the content you’re working with and the desired impact and target audience.
T pes of data visualisation
Ty
Figure 5  Types of data visualisation
ation
an

d
Charts and graphs Illustration Timeline
Criteria: data set Criteria: concept, idea or process Criteria: series of events or data
Good for: visualising and simplifying Good for: visualising and simplifying at regular time periods
data; highlighting trends and qualitative information Good for: showing changes over time
making comparisons

e. A B

C
Map based Infographic Narrative infographic
Criteria: data/information linked Criteria: data set with a clear message Criteria: data/information that tells
to multiple locations Good for: highlighting a surprising piece a self-contained story
Good for: showing geographical of data, a key message or visualising Good for: summarising a piece of
trends/distribution a research finding research, a concept or process succinctly
Source: ODI infographics guidelines

2
Box 11  Quick tips for producing data visualisation
1. Know your audience, purpose and the message.
2. Allow adequate time and budget.
3. Double check the data, its interpretation and sources.
4. Keep it simple.

If you don’t have in-house expertise or budget to commission something, here are a few tools you could try:
•• Picktochart – a tool to develop charts and infographics (www.piktochart.com).
•• Google Fusion tables – a tool to turn data into charts or maps.
•• Tableau public – a tool to create dynamic data visualisations (www.tableau.com/public).
•• Datawrapper – helps you create maps and charts (www.datawrapper.de).
•• Tiki-toki – helps you develop timelines (www.tiki-toki.com).
•• Ttdatavis – resources and tools for data visualisation creation (www.ttdatavis.onthinktanks.org/data-visualisation-resources).

32
MODULE 6

MONITORING AND
LEARNING FROM
YOUR RESEARCH
There has been discussion ‘about the link between underpinning research and impact, and the need to articulate this link in the
impact case studies’ as the links between research and impact were sometimes ‘tenuous’ (Manville et al., 2015). Putting in place
simple monitoring mechanisms early on can help pin down these links, even over the longer term. Also, ensuring there is evidence
to underpin the narrative in case studies is critical, as ‘the risk of relying on narratives to assess impact is that they often lack the
evidence required to judge whether the research and impact are linked appropriately’ (Penfield et al., 2014).

Monitoring and learning (M&L) was therefore designed to

respond to these demands. M&L covers: Take-away messages

1. Monitoring: to provide the evidence to demonstrate impact. •• Thinking about what information is required and for
2. Learning: to improve what you are doing); and what purpose, who will use the information, where
3. Accountability: to prove to different stakeholders what you it can be found and how often will help determine
are doing is valuable. resource needs for collecting data and what gaps exist.
•• It may be necessary to gather and generate evidence.
Putting simple systems in place helps you prepare for
unanticipated opportunities: a plan allows you to emphasise
reflection and aims to identify and respond to unanticipated opportunities that arise during the research. You can include space and
funding for events if necessary, for reflection for the research team as a group or as individuals during the research, with research
users if helpful.

An M&L framework helps you outline the planning and

management of associated activities, including timing, Box 12  Six levels for monitoring strategy and
personnel, skills, budget, deliverables and feasibility; and it management
describes the system for doing this. It outlines (1) what is
to be monitored; (2) how the monitoring will be conducted; 1. Strategy and direction: the basic plan the research
(3) who is responsible for what; (4) when will the monitoring project/programme/organisation is following in order to
take place (ongoing and/or at key intervals); and (5) the reach its intended goals.
resources required. 2. Management: the systems and processes the project/
programme/organisation has in place to ensure the
This module outlines: overall strategy is carried out and high-quality policy
research is produced (e.g. systems of peer/user review,
1. Six levels of monitoring.
quality assurance, planning cycles, etc.).
2. Gathering and generating evidence.
3. A task to develop an M&L framework 3. Outputs: the tangible goods and services a research
4. Methods for capturing impact. project/programme/organisation produces (e.g.
working papers, journal articles, policy briefs, website,
meetings, events, networks, etc.).
Six levels of monitoring 4. Uptake: direct responses to the research project/
programme/organisation (e.g. its research is
Six levels of monitoring are: strategy and direction; mentioned in a government policy paper, on a range of
management; outputs; uptake; outcomes and impact; and websites, referred to in a newspaper article, etc.).
context; Box 12 (Pasanen and Shaxson, 2016). It may be that 5. Outcomes and impact: changes in behaviour,
not each of these needs equal emphasis; for example, it may knowledge, policies, capacities and/or practices that
be that management systems could be lighter but there may
the research has contributed to, directly or indirectly
be more emphasis on outputs such as journal articles. The six
(e.g. a change in government policy implementation,
levels offer guidance on developing indicators and specific
monitoring tools that can be considered at each level and improved businesses, etc.).
provide a useful check to ensure nothing has been overlooked. 6. Context: changes in the political, economic and
organisational climate that have an impact on plans.
These are exogenous and cannot be expected to be
Gathering and generating evidence influenced by the project or organisation; however, they
will have an important impact on how things work and
As well as gathering evidence as it arises (this is much what is possible
easier than trying to find it much later when, for example,
people may have changed positions), it may be necessary to Source: Pasanen and Shaxson (2016).
generate evidence. We talk about each of these here.

34
Gathering evidence

You have identified the areas where you need to collect data by developing your progress markers and associated indicators from
Module 4. Starting with these ensures the information gathered will be directly applicable to monitoring performance. A good
performance question for each level of the objective hierarchy outlines the questions you would need to answer to know the extent
to which you are achieving the objective and to explain the success or failure of actual results.

In response to these, decide what information is required, who will use the information, where it can be found and how often. You
can then identify data sources for your indicators. Some may be collected at regular intervals, such as at the end of a research
phase, through a six-monthly download or through citation figures; others may be collected on an ad hoc basis through a feedback
log (Box 13). Then determine who is responsible for collecting these data and how they will report (where, to whom and for impact,
learning or accountability purposes). Alongside this, note any areas where you need more resources or inputs.

Box 13  A feedback log

A feedback log is a simple way to systematically record reports of the impact you are having. You can then analyse the
records to understand the nature of the work you are doing and how others perceive it. The log is simple; it is easy for
others to contribute to, requires minimum effort to maintain and can be easily used to demonstrate the work you do.

The type of information you collect can cover a range of outreach activities or it can focus on particular areas. To record
impact you can simply forward emails to the log. The unstructured nature of this approach makes it very easy to submit
evidence of the uptake of research outputs and feedback from audiences but it does require a little effort to maintain, for
example to compile into an Excel spreadsheet or table and to systematically use the data in reports.

Generating evidence

It may be necessary to purposefully produce evidence, for example by conducting a survey or requesting testimonies. Survey
Monkey or SurveyGismo could be used to conduct an online survey, although be prepared for response rates as low as 10%.
Following up with contacts to request testimonies is likely to elicit a higher response, although the content will be specific to the
involvement of the respondent. It is worth investing time in carefully considering the questions you ask, to ensure the replies
provide useful evidence to support the demonstration of impact. Asking a question that will provide specific feedback on the
anticipated future impact could be particularly useful. When seeking testimonies, open questions are most useful, and it may be
that an initial contact by the researcher that is directly followed up by support staff would elicit a fuller response.

35
TASK: develop a M&L framework

Complete the table below by indicating the priority purposes of the monitoring and describing where and when information is needed
and who needs to be involved.

Short description of the policy-influencing goal and outline strategy:

Purpose of the monitoring Priority What information is needed? Who will use the information? When is the information needed?

Being financially accountable

Improving research/project
operation
Adjusting strategy

Strengthening capacity

Understanding the context

Deepening understanding

Building and sustaining trust/

partnerships
Lobbying and advocacy

Determine which, if any, data are necessary to provide a picture at the start of the research. Identifying these baseline data will
make it easier to demonstrate the impact of the research on any improvements.

It is useful to circulate and obtain comments on your framework from stakeholders and those who will be involved in data
collection. This will help identify any gaps or challenges that might arise during implementation.

Methods for capturing impact

A possible low-intensity approach may use a combination of the following: (1) monitoring planned impacts against indicators in
the logframe at the end of the monitoring and evaluation (M&E) phase; (2) if available, using a Social Network Map to review the
number of relationships that have been strengthened over the past M&E phase, and hence the current scope of your possible
impact; and (3) extracting examples of actual impact from your impact logs.

A possible high-intensity approach may involve collecting stories related to the research at the end of the M&E phase, and going
through a systematic selection process to arrive at the stories that most appropriately capture the kinds of impacts that you are
aiming for.

Writing impact case studies

Simple, clear and linear narratives scored highly in REF 2014, for example when they (1) identified a problem; (2) outlined the
research conducted; and (3) described how the problem was alleviated (University of Cambridge, n.d.). Impact stories are a great
way to tell clear narratives. This involves taking information you have gathered and generated and using it to tell a story about your
impact. See Box 14 for examples from outside the REF.13

13 These are often called stories of change and the guidance has been adapted here to respond to the REF impact requirements.

36
Your impact case study can be structured as follows:

1 A summary containing key data and outlining the impact.

2 The problem and the context before your research started.

3 The research conducted:

Show you meet the qualifying criteria, including publication timing and researcher location. For collaborative
a research, describe your specific contribution.

b Describe the nature of the research, methodology and key findings.

c Summarise any underpinning research.

d Highlight collaborative work.

4.
4 Impact:

a Structure by beneficiary or impact type, and explain how the beneficiaries were engaged in the research.

b Report systematically against your progress markers, also highlighting unintended impacts.

c Include evidence such as quotes from interviewees and references.

d Report both quantitative and qualitative evidence.

e Clarify the contribution of the research to the impact and describe other influences upon the impact observed.

37
REFERENCES
British Academy (2016) Lord Stern’s review of the Research Excellence Framework: A response from the British Academy. London: British
Academy. Available at: http://www.britac.ac.uk/sites/default/files/British%20Academy%20Full%20%20Response%20Lord%20
Stern%27s%20Review%20of%20REF.pdf.
DEGRP – DFID-ESRC Growth Research Programme (2017) ‘Project Guidelines’. London: DFID- ESRC (http://degrp.squarespace.com/impact/).
Du Toit, A. (2012) ‘Making Sense of "Evidence": Notes on the Discursive Politics of Research and Pro-Poor Policy Making’. Cape Town:
Institute for Poverty, Land and Agrarian Studies, University of the Western Cape. Available at: http://www.plaas.org.za/sites/default/
files/publications-pdf/PLAAS_WorkingPaper21dutoit_0.pdf
Earl, S., Carden, F. and Smutylo, T. (2001) Outcome Mapping. Ottawa: International Development Research Centre. Available at:
https://www.idrc.ca/en/book/outcome-mapping-building-learning-and-reflection-development-programs.
EPSRC (n.d.) ‘Pathways to Impact’. Engineering and Physical Sciences Research Council (https://www.epsrc.ac.uk/funding/
applicationprocess/preparing/writing/resourcesimpact/).
ESRC (2017) ‘Developing a communications and impact strategy’. Swindon: Economic and Social Research Council (http://www.esrc.ac.uk/
research/impact-toolkit/developing-a-communications-and-impact-strategy/).
ESRC (2017) ‘Developing Pathways to Impact’. Swindon: Economic and Social Research Council (http://www.esrc.ac.uk/research/
impact-toolkit/developing-pathways-to-impact/).
Farrugia, P., Petrisor, B., Farrokhyar, F. and Bhandari, M. (2010) ‘Research questions, hypotheses and objectives’, Canadian Journal of
Surgery 53(4): 278–281. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2912019/#b2-0530278.
Gallo, C. (2014). Talk like Ted. London: Macmillan.
Guijt, I. (2008) Seeking Surprise: Rethinking monitoring for collective learning in rural resource management. Wageningen: Wageningen
University. Available at: http://edepot.wur.nl/139860
Hall, P. (1993) ‘Policy paradigms, social learning and the state: The case of economic policymaking in Britain’, Comparative Politics 24(3): 275–96.
Hamza-Goodacre, D., Jefford S. and Simister, N. (2013). ‘Supporting International Climate Negotiators: a monitoring and evaluation
framework’. CDKN Working Paper. London: Climate and Development Knowledge Network. Available at: https://cdkn.org/wp-content/
uploads/2013/11/CDKN_Working_Paper-ME-_Final_WEB.pdf
Hearn, S. and Buffardi, A.L. (2016) ‘What is impact?’. ODI Working Paper. London: Overseas Development Institute. Available at:
https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10302.pdf
Heath, C. and Heath, D. (2007) Made to stick: why some ideas take hold and others come unstuck. London: Random House Books.
HEFCE (2016) ‘How we fund research’. London: Higher Education Funding Council for England (http://www.hefce.ac.uk/rsrch/funding/).
IFC, GTZ, DFID (2008) Monitoring and Evaluation for Business Environment Reform: A Handbook for Practitioners. Investment Climate
Department. Washington D.C.: The World Bank. Available at: http://www.publicprivatedialogue.org/monitoring_and_evaluation/
M&E%20Handbook%20July%2016%202008.pdf
Jones, H. and Hearn, S. (2009) ‘Outcome Mapping: a realistic alternative for planning, monitoring and evaluation’. ODI Background Note.
London: ODI. Available at: https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/5058.pdf.
Jones, H., Jones, N., Shaxson, L. and Walker, D. (2012). Knowledge, policy and power in international development: a practical guide.
Bristol: Policy Press.
Manville, C., Guthrie, S., Hanham, M-L., Garrod, B., Sousa, S., Kirtley, A., Castle-Clarke, S. and Ling, T. (2015) Assessing impact
submissions for REF 2014: An evaluation. RAND, SFC, HEFCW and DEL. Santa Monica: RAND Corporation. Available at:
http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/Independentresearch/2015/REF,impact,submissions/REF_assessing_
impact_submissions.pdf
Office for National Statistics (n.d.) ‘Infographic Guidelines Version 1.0’. London: Office for National Statistics (https://gss.civilservice.gov.uk/
wp-content/uploads/2012/12/infographics-guidelines.pdf)
Penfield, T., Baker, M.J., Scoble, R. and Wykes, M.C. (2014) Assessment, evaluations, and definitions of research impact: A review.
Research Evaluation; 23 (1): 21–32. Available at: https://academic.oup.com/rev/article/23/1/21/2889056
RCUK (n.d.) ‘ESRC – Specific Requirements’. Swindon: Economic and Social Research Council (https://je-s.rcuk.ac.uk/handbook/pages/
GuidanceonCompletingaStandardG/CaseforSupportandAttachments/ESRCSpecificRequirements.htm)
RCUK (2014a) ‘Impact through knowledge exchange: RCUK position and expectation’. Swindon: Research Councils UK. Available at:
http://www.rcuk.ac.uk/documents/innovation/keposition-pdf/
RCUK (2014b) Pathways to Impact’. Swindon: Research Councils UK (http://www.rcuk.ac.uk/innovation/impacts/).
REF2014 (2014) ‘Assessment framework and guidance on submissions’ (http://www.ref.ac.uk/pubs/2011-02/).
Shaxson, L. (2016) ‘Achieving Policy Impact’. Guidance Note. London: DFID-ESRC Growth Resarch Programme. Available at:
http://degrp.squarespace.com/impact/.
Shaxson, L. and Pasanen, T. (2016) ‘How to design a monitoring and evaluation framework for a policy research project’. ODI Working
Paper. London: Overseas Development Institute. Available at: http://www.odi.org/publications/10284-how-design-monitoring-and-
evaluation-framework-policy-research-project

38
Tsui, J., Hearn, S. and Young, J. (2014) ‘Monitoring and evaluation of policy influence and advocacy’. ODI Working Paper 395. London:
Overseas Development Institute. Available at: http://www.odi.org/publications/8265-gates-monitoring-evaluating-advocacy.
University of Cambridge. (n.d.) ‘Impact Case Studies: Lessons Learned from REF 2014’. Cambridge: University of Cambridge.
(http://www.research-strategy.admin.cam.ac.uk/files/impact_recommendations.v3.docx)
Young, E. and Quinn, L. (2012) Making research evidence matter: a guide to policy advocacy in transition countries. Budapest: Open
Society Foundations
Young, J., Shaxson, L., Jones, H., Hearn, S., Datta, A. and Cassidy, C. (2014) ROMA: a guide to policy engagement and influence. London:
Overseas Development Institute. Available at: https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/9011.pdf.

39
ODI is the UK’s leading independent
think tank on international development
and humanitarian issues.

Readers are encouraged to reproduce

material for their own publications,
as long as they are not being sold
commercially. As copyright holder, ODI
requests due acknowledgement and
a copy of the publication. For online
use, we ask readers to link to the
original resource on the ODI website.
The views presented in this paper
are those of the author(s) and do not
necessarily represent the views of ODI
or our partners.

© Overseas Development Institute 2018.

This work is licensed under a Creative
Commons Attribution-NonCommercial
Licence (CC BY-NC 4.0).

All ODI publications are available from

www.odi.org

Overseas Development Institute

203 Blackfriars Road
London SE1 8NJ
Tel: +44 (0) 20 7922 0300
Fax: +44 (0) 20 7922 0399

Shaping policy for development odi.org | @ODIdev