Thematic Case Study

Using Problem Driven Iterative

Development Adaptation to Accelerate
the Progress of Indonesia Students’
Learning Outcomes

June 2020

2 INOVASI | Thematic Case Study: Gender and Education – June 2020

Contents
List of Figures .................................................................................................................................ii
List of tables .................................................................................................................................. iii
List of Acronyms, Abbreviations and Bahasa Indonesian Terms .................................................... iv
Executive summary ....................................................................................................................... 5
1 Purpose of the study ............................................................................................................... 7
2 Mini literature review ............................................................................................................... 9
2.1 PDIA and other flexible programs .................................................................................... 9
2.2 The PDIA process.......................................................................................................... 10
2.3 Evidence about PDIA and other flexible programs ......................................................... 11
2.4 ‘PDIA-like’ initiatives in the Indonesian education sector ................................................ 12
2.5 Conclusion and emerging research questions ............................................................... 13
3 Methods and limitations ........................................................................................................ 15
4 INOVASI’s adaptation of problem-driven iterative adaptation ................................................ 17
4.1 The historical evolution of INOVASI’s approach............................................................. 17
4.2 Authority: supporting district regulations and engagement with local actors ................... 20
4.3 Ability: structured technical learning or short courses .................................................... 23
4.4 Acceptance: shifting teachers’ mindset and generating buy-in among government officials
26
4.5 Crawling the design space ............................................................................................. 28
4.6 Conclusion ..................................................................................................................... 30
5 Evidence for problem-driven iterative adaptation .................................................................. 32
5.1 Guru BAIK results and comparative evidence ................................................................ 33
5.2 Jalan Andrews and indications of behaviour change at district level .............................. 38
5.3 Conclusion ..................................................................................................................... 44
6 Reflection on INOVASI’s experience with problem-driven iterative adaptation in Indonesia’s
education sector .......................................................................................................................... 45
References .................................................................................................................................. 49

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 i
List of Figures

Figure 4.1: Components of INOVASI’s problem-driven iterative adaptation ...................................................17

Figure 4.2: The Guru BAIK Cycle ....................................................................................................................18

Figure 4.3: Types of regulations supported by INOVASI ................................................................................22

Figure 4.4: Short courses; the in-on-in sessions sequence ............................................................................24

Figure 5.1: Guru Baik’s Impact on Students’ Average Literacy Scores (Out of 100) ......................................34

Figure 5.2: Guru Baik’s Impact on Students’ Numeracy Scores .....................................................................34

ii INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
List of tables

Table 2.1: Sections and research questions ...................................................................................................14

Table 4.1: INOVASI’s short-course modules ...................................................................................................24

Table 4.2: Summary of pre-pilot lessons learned ............................................................................................25

Table 5.1: Number of schools scaling-out Guru BAIK .....................................................................................35

Table 5.2: Literacy test scores: schools that received Literacy 1 and schools that received Guru BAIK and

Literacy 1: Southwest Sumba ..........................................................................................................................36

Table 5.3: Comparative effects on students’ average literacy scores between the Guru BAIK and Literacy

Boost pilots, by gender ....................................................................................................................................37

Table 5.4: Comparative effects on students’ average literacy scores between the Guru BAIK and Literacy

Boost pilots, by grades ....................................................................................................................................38

Table 5.5: Considerations for participating districts in Jalan Andrews ............................................................40

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 iii
List of Acronyms, Abbreviations and Bahasa Indonesian
Terms

AusAid Australian Aid

BOS schools operational funds (bantuan operasional sekolah)

BOSDA regional schools’ operational funds (bantuan operasional sekolah daerah)

CID Center for International Development, Harvard University

CLCC Creating Learning Communities for Children program

DBE Decentralized Basic Education program

DFAT Australian Department of Foreign Affairs and Trade

EPG education policy and governance

GEMBIRA mother-tongue transition pilot

Guru BAIK INOVASI’s classroom action research pilot

INOVASI Innovation for Indonesia’s School Children (INOVASI)

Jalan Andrews The process of extending PDIA to the district level (Andrews’ way)

KKG teachers’ working groups (kelompok kerja guru)

MBE Managing Basic Education

MeE monitoring and experiential evaluation

Merdeka Belajar Independent/free learning

MERL monitoring, evaluation, research and learning

MoEC Indonesian Ministry of Education and Culture (MoEC)

OECD Organisation for Economic Co-operation and Development

PDIA problem-driven iterative adaptation (PDIA)

PERMATA numeracy pilot

PISA Programme for International Student Assessment

POM Performance Oversight and Monitoring

PRIORITAS Prioritizing Reform, Innovation, and Opportunities for

Reaching Indonesia’s Teachers, Administrators and Students program

RISE Research on Improving Systems to Education program

SETARA Inclusive education pilot

iv INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
Executive summary

The Innovation for Indonesia’s School Children (INOVASI) program is a partnership between the
Australian Department of Foreign Affairs and Trade (DFAT) and the Indonesian Ministry of
Education and Culture (MoEC). The first phase of the program, with a budget of AUD49 million,
began in 2016 and ends in mid-2020. INOVASI has worked across 17 districts in four provinces,
namely: West Nusa Tenggara, East Java, East Nusa Tenggara and North Kalimantan. From the
outset, INOVASI adopted problem-driven iterative adaptation (PDIA) as its primary approach.
The way that INOVASI applies and interprets PDIA has evolved since the program began. In 2016,
the program used PDIA primarily to conduct classroom action research and also to encourage
teachers to develop a growth mindset. At that time, INOVASI applied PDIA at the classroom level,
asking teachers to identify their students’ learning problems and devise solutions. However, at the
second strategy testing session in 2017 the team decided that many teachers still lacked
fundamental competencies in literacy and numeracy. Hence, expecting them to identify problems
and solutions was unrealistic. After that, INOVASI instituted a short-course approach that was itself
an iteration since the approach was built on knowledge from previous donor-supported education
programs. Local iterations of INOVASI’s own short courses began in 2019 when local stakeholders
started to adjust the content of the courses and decide how best to deliver them in their own
districts.
Since 2018, INOVASI’s PDIA approach has also included a ‘thinking and working politically’
component. The program works with the districts to develop more appropriate regulations to
support better learning outcomes. Other aspects of the program that reflect this thinking are the
team’s efforts to influence district budget allocations away from infrastructure and towards activities
to improve the quality of learning and teaching. The most recent iteration is the application of PDIA
at the district level. Using PDIA, INOVASI works together with district-level officials in identifying
the district’s most pressing challenges.
To assess whether the PDIA approach has been effective, this study examines the evidence from
the Guru BAIK pilot and the most recent Jalan Andrews approach that is extending the use of PDIA
at the district level. There are four main reasons to examine the evidence from Guru BAIK. First,
Guru BAIK is the first pilot and lessons learned from this pilot have been embedded in all the other
pilots. Second, by focusing on Guru BAIK we prevent any overlap with other thematic case studies
that examine the other pilots on literacy, numeracy and inclusion in greater detail. Third, most
importantly, Guru BAIK has problem solving and growth mindset components. Finally, it is also the
only pilot that has a counterfactual, namely the Literacy Boost and the Literacy 1 pilot that do not
include these PDIA components. The evidence shows that Guru BAIK has more impact on student
literacy scores than Literacy Boost. The data also shows that teachers that benefitted from Guru
BAIK before participating in the Literacy 1 pilot performed better than those who only participated
in the Literacy 1 short course.
Furthermore, teachers with Guru BAIK experience contributed to higher learning outcomes among
children with special needs and those from poor socio-economic backgrounds. Nevertheless, there
is need for more data. The current available evidence is based only on a small number of districts.
Concerning the application of PDIA at the district level, there are some known challenges but also
positive indications. One of the barriers is that PDIA requires initiative and active involvement from
its counterparts but in the district bureaucracy, officials tend to prefer to wait for instructions. On
some of the positive findings, officials became more data-driven in identifying problems and in
looking for solutions. The application of PDIA also apparently helped to break down silos among

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 5
the various actors. District stakeholders became aware that addressing an education problem is
not the sole responsibility of the district education office but requires collaboration with other
stakeholders.
Finally, while there is clarity about what PDIA means as a concept, its application is subject to
different interpretations. Indonesia already has experience with other ‘PDIA-like’ initiatives that are
context-specific and seek to strengthen local solutions and decentralised decision making. These
interventions have also faced cultural and political challenges. Any aid program trying to implement
PDIA needs to anticipate those challenges.

6 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
1 Purpose of the study

The Innovation for Indonesia’s School Children program (INOVASI) is a partnership between
Australia’s Department of Foreign Affairs and Trade (DFAT) and Indonesia’s Ministry of Education
and Culture (MoEC). Funded by DFAT, the first phase of the education program runs from 2016 to
2020 1 and has a budget of AUD49 million. The aim of the program is to help the Indonesian
government make ‘accelerated progress towards [improved] learning outcomes for Indonesian
students’. As stipulated in the partnership arrangement, INOVASI does this:
…‘by generating evidence of tailored solutions to locally prevailing challenges and
promoting the adaptation and replication of these tested strategies to influence both policy
and practice’
… ‘[and] to provide decision-makers in Indonesia with evidence to make informed choices
about education policies, regulations, and resource allocations that can increase learning
outcomes in Indonesia’ (GoA and GoI 2015, p.1).
The purpose of this thematic case study is to explain INOVASI’s experience with using problem-
driven iterative adaptation (PDIA) as a strategy to achieve its objectives. While how to define, use
and benefit from PDIA are explored in the literature this study examines specifically how INOVASI
perceives and implements PDIA and unpacks the processes and decisions leading to this
understanding. There are two main reasons why these lessons could contribute to future adaptive
programs.
First, it is often a long journey from concept to implementation. Although development
professionals and academics are familiar with the concept of PDIA, factors arise when a program
is running that influence why particular elements of PDIA may be implemented according to the
textbook and why some others may not. Understanding this dynamic between concept and actual
practice can help inform donors’ expectations about the potential and limits of PDIA in the future.
Second, PDIA, thinking and working politically, (TWP) and other forms of flexible programming
originate from governance programs. A pioneer of PDIA, Lant Pritchett, during his visit to INOVASI
in December 2017 stated that INOVASI was at the ‘forefront’ of applying PDIA in an education
program. At that time, there were few (if any) other education programs2 that INOVASI could learn
from about how to implement PDIA. The approach emphasises ‘local solutions’ as elaborated in
INOVASI’s design document (DFAT 2014, pp. 5-7) and offers an alternative for a country as
diverse as Indonesia that has invested in its education sector but not yet achieved the desired
results. 3 Indonesia needs new solutions but how or whether PDIA addresses the country’s
education challenge is yet to be elucidated.
This study also presents available evidence of how PDIA has made changes at the classroom and
district levels. This stems particularly from Guru BAIK, INOVASI’s first pilot that implemented PDIA
with teachers at the classroom level and from the program’s experience in applying the strategy4

1
Following the six-month no-cost extension
2
Although not specifically using PDIA, the DFAT-funded Pathways program implemented by Palladium in the Philippines also applies
the principles of ‘flexibility, adaptability and responsiveness’. Pathways is an education program in the conflict-areas of Mindanao that
started in 2017, a year after INOVASI began. For more information, see Australian Embassy, the Philippines (2016, p.6)
3
Indonesia tends to rank low in the Programme for International Student Assessment (PISA) results – from 79 countries participating in
2015, Indonesia ranks only above Brazil, Peru, Lebanon, Tunisia, Macedonia, Kosovo, Algeria and the Dominican Republic (OECD
2018, p.7). Apart from PISA rankings, key stakeholders in Indonesia (such as the Minister of Finance, Sri Mulyani) have expressed
dissatisfaction with the low-quality education outcomes (Reily 2019).
4
Within INOVASI, this process is known as Jalan Andrews (Andrew’s way). Andrews’ way means the same as PDIA but the term is used
within the program to mark the shift from using PDIA at the classroom level to using it to empower program counterparts at the district
level.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 7
with sub-national government officials, known as Jalan Andrews. While Guru BAIK and Jalan
Andrews represent a small part of INOVASI’s overall work and INOVASI uses PDIA in its other
pilots, the results of the other pilots are discussed in other thematic case studies. Apart from
presenting the available evidence, this study discusses its limitations and offers some
considerations for future data collection.
The study begins with a 'mini' literature review and discusses the key elements of PDIA, how other
development assistance programs apply the approach and Indonesia’s experience with its own
PDIA-like interventions.

8 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
2 Mini literature review

Iterative problem solving in public policy has been a topic of discussion for several decades.
Political scientist cum economist, Charles Lindblom (1958, p.301), argues that policy alternatives
are designed incrementally: ‘A policy is directed at a problem; it is tried, altered, tried in its altered
form, altered again, and so on.’ This approach was later coined as ‘the science of muddling
through’ (Lindblom 1959). The principles of iterative problem solving started to be broadly
recognised among the development community again a few years ago. This began with articles in
journals (Pritchett and Woolcock 2013; Andrews 2015) and working papers (Pritchett, Samji and
Hammer 2013) before PDIA was broadly introduced through the publication of Building state
capability: evidence, analysis, action (Andrews, Pritchett and Woolcock 2017). Also, to further
share knowledge on PDIA, the Harvard Kennedy School provides a free online course for
practitioners and academics.
This literature review addresses some relevant issues in PDIA. First, how do practitioners and
academics understand PDIA in relation to other flexible5 approaches? Clarifying this is important
since, in practice, development assistance programs tend to use various approaches
simultaneously. Second, what does PDIA consist of? Without unpacking the PDIA concept it would
be hard to provide substantial comments about INOVASI’s experience with it. Third, what does the
literature say about others’ experiences with PDIA and is it suitable for an aid-funded program?
Fourth, what are the challenges of implementing PDIA in the education sector in Indonesia?
Understanding barriers within the Indonesian context means we can set realistic expectations of
what the approach can deliver and be aware of the challenges of context-specific, locally-based
interventions in the education sector in the country.

2.1 PDIA and other flexible programs

The question of how PDIA differs from other flexible approaches is an ongoing issue among
academics and development practitioners. For instance, Teskey (2017a) attempts to highlight
differences between ‘doing development differently’, PDIA, and thinking and working politically. He
categorises these three approaches as ‘the second orthodoxy’ that is more adaptive and more
mindful of power dynamics in the local context. This new approach differs from the ‘first orthodoxy’
that is more ‘prescriptive’ and based on ‘blueprint’ design. Andrews (2016), a key figure behind
PDIA, has a slightly different take. He believes that instead of being a separate approach, doing
development differently is the umbrella term that entails PDIA as well as thinking and working
politically.
Others disregard the distinctions between these approaches. Marquette (2019) is puzzled at how
development practitioners gloss over key differences in the various flexible programs: ‘I am struck
by how often people say “TWP/PDIA/adaptive management/PEA…whatever”‘. She shares a
similar position to Akmeemana (2018) arguing that PDIA is not for donors but rather for country
governments that have an ‘endogenous feedback loop from experimentation to adaptation’.
Therefore, instead of doing PDIA themselves, donor-funded programs should support this process

5
The term ‘flexible approach’ reflects the current definition used by the Adaptive and Flexible Working Group in DFAT Canberra. The
working group defines adaptive programs as a subset of flexible programs. Flexible programs are defined as having ‘systems for
monitoring changes in the context and adjusting’. Adaptive programs have ‘systems for trialling approaches, learning, building upon
them or trying something else’. INOVASI not only has monitoring and evaluation systems to help it adjust to the local context but also
has pilots to trial what works. Therefore, although never explicitly stated, according to the definition, INOVASI is an adaptive program. It
should be noted that these are working definitions and are likely to change in the future. For the definition see the presentation by the
Working Group of Adaptive and Flexible Programs on 14 December 2018 in Canberra.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 9
taking place within partner systems. Marquette continues that, unlike PDIA, thinking and working
politically is not an approach but rather a way of thinking that acknowledges that politics is a central
part of the development process.
What does this discussion mean for INOVASI? While experts continue to refine their definitions of
these concepts, INOVASI’s design document mandates that PDIA ‘underpins’ the program (DFAT
2014, p. 24). Thus, once the program began, the team needed to establish what PDIA meant and
how to embed the approach in day-to-day operations. This process of developing a working
definition of PDIA and deciding how to implement it in the education sector is elaborated in section
4.1. However the inconclusive discussions about flexible approaches described give a context to
the complexity that INOVASI faced in 2016 when knowledge and practical experience were limited.
This was not only a challenge for INOVASI but also for other programs wanting to be flexible.
To add to the complexity, PDIA can be implemented together with other approaches. The World
Bank’s Melayani program, for instance, apparently combines PDIA with two other approaches. The
first is ‘deliverology’, an approach based on the experience of Tony Blair’s Prime Minister’s
Delivery Unit (Barber, Kihn and Moffit 2010). The authors define this as ‘the art of getting things
done in government’ (p.vii). The other approach Melayani used is the World Bank’s own ‘rapid
assessment and action plan’. This approach helps the government at sub-national levels to deliver
reform by identifying barriers to desired outcomes in public service delivery and by proposing an
action plan to achieve those outcomes.
Moreover, the novelty of PDIA and other emerging flexible approaches is debated. Experienced
development practitioners believe that the principles are nothing new. For instance, at a
conference on doing development differently in Jakarta in 2017, many participants reportedly
packaged their interventions as ‘doing development differently’ when in practice they were simply
doing proper development work ‘based on data, designed and managed with extensive citizen
participation, and real-time monitoring’ (Teskey 2017b). This further illustrates the difficulty in
pinning down what PDIA is or is not.

2.2 The PDIA process

Moving on to the second topic in this literature review, we need to briefly unpack what steps are
involved in PDIA. Summarising from the literature (Andrews, Pritchett and Woolcock 2013;
Andrews, Pritchett and Woolcock 2017, pp. 169-175) the first step in PDIA is to identify and
construct local problems. Instead of arriving with particular solutions in mind, development
assistance programs need to consult with local stakeholders on the problems they consider as
priorities to resolve. This step also involves deconstructing the problem – breaking it down into
several more manageable parts. A fishbone (Ishikawa) diagram or the ‘five whys’ technique,
among others, might be used in this process
The second step in PDIA is ‘creating space for change’. This is when the triple-A analysis
(authority, acceptance, ability) takes place. Authority means securing buy-in from people in
positions of authority (usually formal positions but may also be informal positions of power). In
INOVASI’s case, people with the authority include staff members at the district education office, the
regent or mayor and the governor, as well as Ministry of Education and Culture (MoEC) officials.
Acceptance means having the consent of those affected by the reform, for example, teachers,
school principals and district officials. Ability refers to improving the technical capacities of teachers
on subject matter and pedagogy, as well as improving the ability of district officials in planning
reform efforts. It also means making sure that sufficient resources are available to carry out the
planned reform.
The final step is to look for and experiment with multiple alternative solutions and is described as
‘crawling the design space’. INOVASI seeks solutions that fit with the local context and adapts the
pilot accordingly. Sections 4.2 to 4.5 discuss how each of these steps in PDIA take place within the
program.

10 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
2.3 Evidence about PDIA and other flexible programs
What does the evidence say about whether PDIA works? The Center for International
Development (CID) at Harvard has facilitated PDIA-based workshops with government officials in
several countries. In Sri Lanka, CID worked on the issue of diversifying the country’s economy with
a team of local facilitators and government officials over seven months. One result of the process
was a methodology that officials can use to identify sectors that are suitable for diversification and
a database covering 70 sectors. The project also developed the officials’ capability in analysing
data. Andrews et al. (2017) argue that the ‘historical counterfactual’ – in other words Sri Lanka’s
previous efforts for economic diversification – did not materialise into concrete steps so the PDIA
process secured better results.
Another paper elaborates on the PDIA process in reforming the justice sector in Mozambique
(Andrews 2014). The country’s justice system was described as being unable to handle the volume
of justice cases in a timely manner. It had around 10,000 unsolved cases annually and a backlog
of 200,000 cases accumulated over time. Coordination in the justice sector includes several
government agencies with various responsibilities, making it difficult to get accurate data, for
example, on the number of judges, the location of prosecutors and available budgets. In eight
months, the PDIA team facilitated cross-agency discussions that resulted in a comprehensive
database.
The paper asserts that the PDIA process brought more tangible results under a much shorter
timeframe compared to a standard development intervention with a similar objective and larger
funding. The United Nations Development Programme (UNDP) implemented the System for
Justice Sector Planning, Monitoring and Evaluation project from 2007 to 2012 with a budget of
USD5 –7 million. This project was used by Andrews as a ‘counterfactual’ to the PDIA process.
Although by the end the UNDP-funded program provided a system to manage the justice sector
data, it was not used because it did not have local buy-in and hence did not get sufficient funding.
Both papers conclude that in terms of results, PDIA works better than standard approaches.
Lessons from other types of flexible approaches indicate that programs that conduct iterations to
find appropriate solutions tend to have better outcomes (for example: Brown et al. 2013; Harris
2016; Denney 2016; Mercy Corps and IRC, 2016; Hadley and Tilley 2017). Honig (2015) presents
some robust evidence of flexible programs as the preferred approach in development. After
examining a dataset of 14,000 projects and conducting eight case studies, he argues that
‘navigation by judgement’ or the ability of field staff to make decisions and shape the program is
more effective in fragile states as well as in ‘tasks like improving the government’s ability to
manage’ (p. 307). Under these circumstances, he concludes that central control from the donor is
less likely to be effective.
What is striking is that so far the literature seems to agree that PDIA and other types of flexible
programs are the better option for development assistance programs. Few publications to date
openly criticise the approach as being ineffective. One publication that highlights the challenges of
a flexible program is ‘Doing development differently at scale’ (Pellini, Karetji and Soekadis 2018).
The authors argue that large-scale programs, like the DFAT-funded Knowledge Sector Initiative
(KSI), have to deal with the contradiction of being adaptive and at the same time coping with rigid
implementation guidelines. Program operations often become a challenge for flexible programs.
Some activities carried out in large-scale programs are designed to follow a plan with fixed outputs.
Changing these plans often requires approval from the donor as well as the counterpart
government, making them far from flexible.
Moreover, the authors argue, flexibility is hampered by the need to spend close to 100 per cent of
the allocated budget within a given timeframe. Failing to do so can be perceived as being unable to
meet the program’s annual goals. This situation is particularly pertinent in activities implemented
with the government since their planning and hence the amendment processes are more rigid. The

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 11
issue of spending is also reported to shift the focus from finding solutions that may work to
delivering outputs (Pellini, Karetji, and Soekadis 2018, p 138).
The authors (p.141) further assert that flexible programs need staff with ‘strong facilitation and
moderation skills’ as well as an understanding of the sector’s political economy. Large-scale
programs often need to recruit a large number of staff members. Hence, recruiting those with the
appropriate experience for PDIA in a short time frame is often a challenge. INOVASI is likely to
have faced some of these operational challenges.
Smaller programs,6 such as the World Bank’s Melayani (2017–2018) program that had funding of
around USD400,000, are likely to be more flexible. Also, the Melayani program did not have any
pre-planned activities as these were all determined together with district officials. Based on local
consultations, the program worked on different issues in their three participating districts.
Bojonegoro opted to address infant and maternal mortality rates while Kubu Raya focused on
stunting and Belu on low education outcomes at the primary level (Melayani final presentation, 25
February 2019).

2.4 ‘PDIA-like’ initiatives in the Indonesian education sector

Following the fall of Suharto’s New Order, Indonesia commenced an ambitious decentralisation
process. Education is one of many sectors that have become a primary responsibility of lower-level
government. The first law regulating this was Law No 20 of 2003 on the National Education System
and it shifted responsibilities in the education sector mainly to district and city levels and even to
the school level. Districts and cities can now select, appoint, deploy, promote and dismiss teachers
by following national guidelines. Previously, this function was under the authority of the Ministry of
Education and Culture delegated to provincial governments. Districts and cities are now also
tasked with monitoring the performance of teachers and principals (Jalal et al. 2009, pp. 13-15).
Even before decentralisation, Indonesia intended to devolve some responsibilities to the local level.
Bjork (2003) explains the tepid implementation of the ‘local content curriculum’ in 1994. Ministry
officials and expert teams in Jakarta were concerned about the low transition rate from primary to
junior secondary schools. One of the main reasons they identified for this was parents and
students’ perception that spending additional years of schooling had little correlation with securing
access to better jobs. The local content curriculum was to enable greater links between formal
education and local economic potential, and thus stimulate interest in additional years of schooling.
In Java the local content may include farming whereas in Bali it may entail training in tourism, for
example. Teachers and principals were required to tailor the curriculum to the local situation.
Therefore in its intent, this curriculum is like PDIA.
Although this curriculum gave teachers more authority to innovate, adjusting learning to local
needs is time-consuming and required greater engagement with the community. Bjork argues that
many teachers were not keen to ‘augment their degree of influence’ (p. 205). The main reason was
that although ministry officials were not against the local content curriculum, incentive structures
such as criteria for measuring teachers’ performance remained the same. Instead of rewarding
much-needed creativity and initiative, the system credited teachers for their loyalty to the state and
obedience to their superiors. Bjork concludes that the local content curriculum was a case where
the national government expected behaviour change from teachers without actually making the
requisite structural changes to the education system (p. 215).
A similar issue occurred with school-based management. Initiated in 2003 to support
decentralisation, the intention was to give schools a higher degree of autonomy in managing their

6
Another program that is known to use PDIA in Indonesia is SMERU’s Diagnose, Design Adapt and Evaluate Local Education System:
from Schooling to Learning, part of the Research on Improving Systems to Education (RISE). The design of this program was presented
to INOVASI on 28 January 2019 but as it was ongoing during the report writing period, no results can be cited.

12 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
affairs. Law No 20 of 2003 on the National Education system mandates that ‘early childhood
education, basic education and senior secondary education are implemented based on the
minimum service standard by applying school-based management.’ School-based management is
also supposed to stimulate a more participatory approach to decision making by involving different
stakeholders, such as the school principal, teachers, parents and the local community. It is
assumed that parents care about the quality of their children’s education and by involving them,
they would closely monitor teacher performance, ensure that schools have the right policies (and
that these policies are implemented), as well as warrant that school resources are spent effectively
and efficiently (Barrera-Osario et al. 2009, p. 17). Conceptually, school-based management allows
schools to make decisions that suit their needs.
Nevertheless, an evaluation of how schools implement school-based management in Indonesia
concludes that many principals do not use their increased authority to develop more relevant
programs or improve the teaching and learning process in the classroom. This conclusion was
based on consultations with the schools’ stakeholders. Apparently, many school principals avoid
independent decisions and still rely on advice from district officials. Contrary to the principles of
school-based management, school principals felt that district officials had greater influence on
schools than the teachers. In addition, school committees’ participation was limited and the
committee simply signed-off on school policies without inputs from parents and the wider
community (Vernez, Karam and Marshall 2012). Hence, school-based management’s objective to
make schools more rooted in their local context was also challenged by existing behaviour
patterns.
At the school level, since 2005, government has provided schools with operational funds (Bantuan
Operasional Sekolah – BOS) to give them more autonomy in deciding how to improve education
quality. This funding can be used, for example, to fund teachers’ professional development or to
buy instructional materials (World Bank 2015). Decentralisation has therefore shifted the authority
to manage education affairs to sub-national levels so that local actors can better respond to local
challenges and this reflects the spirit of PDIA.
Another example of a PDIA-like initiative in Indonesia is the school-based curriculum, popularly
known as KTSP (kurikulum tingkat satuan pendidikan), that was introduced in 2006 and
implemented nationwide in 2009. The idea behind this new curriculum was that students across
Indonesia have different cultures, languages and socio-economic backgrounds so they also need
different teaching practices (Sulfasyah 2013, p. 3). Children in less developed areas in Papua, for
example, were not expected to learn at the same pace as children in large cities like Jakarta. Thus
the government realised some years ago that a national blueprint for education is ineffective.
However, although the new curriculum suited the Indonesian context, implementing it was deemed
sub-optimal since principals and teachers lack the appropriate skills (Siswono 2008; Sutrisno and
Nuryanto 2008, cited in Sulfasyah 2013, p. 9). Given these structural challenges in the education
sector, this thematic case study also discusses the expectations of a PDIA program in Indonesia.

2.5 Conclusion and emerging research questions

The literature review above leads to several research questions. Problem-driven iterative
adaptation and other types of flexible strategies are still evolving. The literature on implementing
PDIA and sharing lessons learned is expanding but remains scarce in the education sector. As
INOVASI started in 2016, the team had to work out the meaning of PDIA in the context of the
program and this understanding has also evolved over the years leading to the following questions:
Question 1: How did INOVASI’s PDIA approach evolve?
Question 2: How does INOVASI’s experience with PDIA compare to the existing literature?
Moreover, current evidence on PDIA and other flexible programs suggests that the approach works
better than conventional ways of delivering aid. However, we need further evidence on its

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 13
effectiveness, especially in relation to the education sector. This leads to the third question this
study addresses:
Question 3: Based on INOVASI’s experience, what evidence is available to demonstrate whether
PDIA works?
Finally, INOVASI is not the first program to emphasise locally-driven solutions in Indonesia’s
education sector. The country has former experience with similar interventions, such as the
schools’ operational fund, the local content curriculum, the school-based curriculum (KTSP) and
school-based management. While these may not strictly follow the PDIA steps elaborated in this
literature review, they all focus on adapting to local circumstances. So far, these experiences
suggest that the approach has not worked well in Indonesia. Many stakeholders at the school level
lack the technical capacities to identify and resolve problems. In addition, they feel more
accountable to their superiors than to the students and their parents. Yet INOVASI’s design
document anticipates PDIA bringing about behaviour change. It mandates the program’s
monitoring and evaluation system ‘to provide robust 7 evidence that enables decision-makers to
continually adapt INOVASI and its activities to maximise the extent to which it facilitates changes in
behaviour of local actors.’ Hence the last question this report aims to answer is:
Question 4: Considering the constraints for PDIA-like programs and INOVASI’s experience, what
are the lessons for future flexible aid programs in the education sector?
Table 2.1 shows which sections in this report address each of the four research questions.
Table 2.1: Sections and research questions

Section Research question

Section 4: INOVASI’s adaptation of PDIA Question 1

Question 2

Section 5: Evidence for PDIA Question 3

Discussion Question 4

7
The term ‘robust’ is later changed to ‘credible’, see section 4.5

14 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
3 Methods and limitations

The spirit of the thematic case study is to work as much as possible with evidence already
available in the program. This method involved several parts.
The first was a desktop review of project documents, such as six-monthly progress reports, guiding
program strategies, minutes of steering committee meetings, powerpoint presentations, pilot-
monitoring reports, blog articles written by staff members, studies conducted and commissioned by
INOVASI, as well as the broader literature on adaptive programming. Collecting program
documents is the primary source of data and information in this study. The second part involved
interviews with INOVASI staff, government counterparts at the national and district levels, and the
program’s beneficiaries, namely teachers and principals.
From June to September 2018, I participated in several INOVASI activities, primarily undertaken by
the monitoring, evaluation, research and learning (MERL) team. I was involved in the study on
suitable books for children in North Kalimantan as well as a spot check of Guru BAIK scale-out in
West Sumbawa. I also helped the team with the design of various monitoring instruments. My
involvement was part of my PhD fieldwork in understanding flexible programs in Indonesia. Initially
I was involved as an intern and later as a researcher in conducting this thematic case study. This
period of engagement was extended for a week in 2020 to understand INOVASI’s expansion of
PDIA at the district level, known as Jalan Andrews, that started in 2019 (see section 5.2).
Being embedded at INOVASI, I was free to communicate with everyone, from the senior
management team in Jakarta to provincial and district level staff members. I also had access to
INOVASI’s government counterparts at MoEC head office and at the sub-national levels.
Furthermore, I participated in several internal workshops 8 and activities held with counterpart
governments, such as the Temu INOVASI (innovation meetings organised by INOVASI).
INOVASI is a complex and dynamic program. Sometimes the understanding in Jakarta is not
immediately shared at the district level9 so being in this position helped me understand how events
unfold at different levels.
Using available program reports and evaluations has its strengths. I was able to draw on the
available knowledge collected over four years of implementing the program and by building on
available evidence the program already has, the study was also cost-efficient.
However these documents also have limitations in that the program reports are written for specific
purposes and do not always answer the research questions. For example, while monitoring reports
at the school level provide numerous data about students’ learning outcomes, they offer little
evidence on behaviour or mindset changes at the district education office. INOVASI focused on
these issues after it decided to expand its PDIA approach. There are indications of how this
expansion, referred to in this report as Jalan Andrews, began to stimulate officials to be more
problem-oriented. However, unlike at the classroom or school level where the program has a
structured data collection process, we can only explore how Jalan Andrews changed behaviour
among district officials through anecdotal evidence.

8
Such as a workshop with the policy team on policy changes, a workshop with grantees and the workshop on Jalan Andrews
9
For instance, shortly after the short course was introduced, staff members at the district level thought that the pre-pilots were
abandoned. Instead, the intention of the team in Jakarta was to strengthen teachers’ technical skills and to expand PDIA beyond the
classroom.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 15
In addition, while this report can add evidence about a flexible program, it cannot indicate whether
this is more suitable for a donor-funded education program in Indonesia than previous conventional
approaches. We can draw some lessons from comparing Guru BAIK and the Literacy Boost
program and comparing participants who received Guru BAIK and the Literacy 1 pilot and those
who received just the Literacy 1 pilot (section 5.1). Nevertheless, the available evidence from two
districts (Sumbawa and Southwest Sumba) and a small number of samples will not satisfactorily
answer the question.

16 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
4 INOVASI’s adaptation of problem-driven iterative adaptation

INOVASI began in January 2016 when PDIA was not as well-known and the key reference work,
Building state capability, was not yet published. To become acquainted with PDIA, INOVASI staff
members participated in the online course offered by Harvard University that uses journal articles,
video presentations and working papers as material. One staff member noted: ‘We started the
PDIA machine but we had to figure out how to operate it.’ Considering staff turnover and incoming
personnel, having a sufficiently solid understanding of PDIA across the whole program required
some time. Even up until the second half of 2018, staff members were still doing the online
course.10
This section documents the process as well as the decisions that have shaped INOVASI’s
understanding of PDIA. The strategy-testing sessions (based on Ladner 2015) provided the most
opportunities for all the teams to examine ongoing issues in using PDIA and alter the approach in
the program going forward.11 Figure 4.1 summarises the components of INOVASI’s PDIA.
Figure 4.1: Components of INOVASI’s problem-driven iterative adaptation

This section also discusses how INOVASI approaches the triple-A elements of authority,
acceptance and ability, as well as how it seeks and trials multiple solutions or ‘crawls the design
space’ with regard to its pilots.

4.1 The historical evolution of INOVASI’s approach

When INOVASI commenced in 2016, the principles of PDIA were perceived as similar to those in
classroom action research. This is a known method in education that helps teachers identify real
problems students face in their classrooms and requires teachers to come up with solutions.
Classroom action research is similar to PDIA in that it is context-specific and it enables teachers to
identify the particular needs of the student. For example, some students might prefer ‘solitary

10
Heyward (2018) provides a reflection of the course relevance for staff members
11
During the time of report writing INOVASI conducted four strategy testing sessions

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 17
reading activities’ while others prefer to do group reading (Stringer 2007, p. 68). Tailoring the
teaching method to each student’s needs is the goal of classroom action research.
Another influence on INOVASI’s approach came from collaborating with consultants from the Pulse
Lab 12Jakarta. They shared their experience with human-centred design and this led to INOVASI
introducing the concept of ‘growth mindset’ (Dweck 2007). This concept asserts that no matter
what talent or interests people have as initial traits, they can change through learning. A fixed
mindset assumes that our levels of intelligence or aspects of character are unchangeable. The
concept is used to help teachers to move from teacher-centred to student-centred teaching.
Moreover, INOVASI believed that beneficiaries, such as teachers, principals and government
officials, needed a change in mindset if PDIA was to be successfully rolled out. At that time the
beneficiaries generally expected an aid-funded program to come up with the solutions. Heyward
(2018) asserts that the conventional way for aid programs in the education sector is through top–
down mechanisms based on best practice. Therefore, participating in a PDIA program is a different
experience for counterparts as well as beneficiaries.
The two concepts of classroom action research and the growth mindset became the pillar of
INOVASI’s first pilot, Guru BAIK.13 The terms of reference for Guru BAIK stipulate that it is a ‘cyclic
process (see figure 4.2) for teachers to gather information about teaching and learning problems in
the classroom and implement experimental activities to solve the problems and improve students’
learning outcomes’ (INOVASI 2017, pp. 1–2). INOVASI implemented the first Guru BAIK pilot in
2017 in West Nusa Tenggara. The pilot used INOVASI’s initial version of PDIA that focused on the
classroom level so it was narrower in scope than the current version.
Apart from refining the technical approach of implementing PDIA in an education program, this
period was marked by the excitement of implementing PDIA. This can be seen from the
enthusiasm among staff members and reflected in ‘promotional’ products such as t-shirts and
badges saying: In PDIA we trust.
Figure 4.2: The Guru BAIK Cycle

Source: INOVASI (2017, p. 9)

12
Pulse Lab Jakarta is an initiative between the United Nations and the Government of Indonesia and specialises in using Big Data and artificial
intelligence to solve development problems.
13
Baik means good in Indonesian but the word is also an abbreviation of four key principles. First, belajar (learning): teachers are expected to
improve their competence through self-assessments of the teaching and learning process and its impact on students’ learning outcomes. Second,
aspiratif (aspirational): refers to the teachers’ willingness to listen to students’ aspirations about the teaching and learning process. Third, inklusif
(inclusive): means student-centred learning with the teacher considering the needs of each individual student. Fourth, kontekstual (contextual):
means facilitating a learning process that is rich in local context making it easier for students to understand the lesson and its application in their
day-to-day lives.

18 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
INOVASI (2019a, p. 3) provides a summary of INOVASI’s strategy testing sessions. The second
strategy testing in October-December 2017 concluded that INOVASI needed to expand the PDIA
approach beyond the classroom. Firstly, the team observed that most teachers in INOVASI’s
districts did not have the fundamental skills to identify problems and find appropriate solutions.
Secondly, INOVASI also realised that to improve learning outcomes districts required appropriate
policies to reallocate resources. Thus focusing on the classroom alone was insufficient. The first
analysis resulted in INOVASI adopting a short-course14 approach based on the Prioritizing Reform,
Innovation, and Opportunities for Reaching Indonesia's Teachers, Administrators and Students
(PRIORITAS) model that it began to implement in 2018. The second analysis led INOVASI to
establish the education policy and governance (EPG) team. Its main purpose is to assist districts in
creating policies that are not only supportive of literacy and numeracy but also have better
implications for gender and social inclusion. In addition, the policy and governance team helps
schools and districts to reallocate funding towards improving learning outcomes.
In other words, the education policy and governance team strengthens INOVASI’s strategy of
thinking and working politically.
This time of change was a hard period for INOVASI. Some staff members believed that
implementing the short-course approach was entirely different from PDIA. There was a strong buy-
in to the former understanding that encouraged teachers to identify their own problems and
solutions with as little assistance from INOVASI as possible. Apart from the team in Jakarta, the
move to the short courses was particularly slow in West Nusa Tenggara (INOVASI’s first province)
where strong loyalty to the earlier interpretation of PDIA made it hard for people to adjust.
The third strategy testing in July-August 2018 assessed the new strategies implemented since the
second strategy-testing session and looked at how the program could support scale outs. It
recommended providing more support to the districts’ pilots to ensure quality and sustainability.
Finally, the fourth strategy testing in June 2019 identified that INOVASI needed to build the
capability of district and provincial officials. This is crucial because a number of INOVASI’s pilots
are being scaled out using the districts’ own funding. If district officials do not have the capability,
these scaled-out pilots are not likely to produce the expected results and district officials will then
revert to conventional practices that did not work. INOVASI modelled its capability-building efforts
on the PDIA experience in Sri Lanka (Andrews et al. 2017). DFAT’s strategic review lauded this
effort, noting that ‘the move from PDIA as a specific tool or activity to a way of thinking has been
appropriate and meaningful for the design and implementation’ (Nichols and Bodrogini 2018, p.5).
The term ‘capability development’ is used to distinguish it from technical capacity development
(see section 5.2). Capacity building is more about improving a particular technical skill, for instance
improving teachers’ ability to teach literacy or numeracy, training schools in whole-school
development or helping the district education office to better plan for teacher deployment.
Capability development refers to improving the ability of local governments to use resources at
their disposal to solve problems. Whereas conventionally donors would do most of the work for the
counterpart government, in capability building, the INOVASI team is playing a supporting role,
particularly through monitoring and facilitating the local government’s working group (known as
pokja) meetings. In these groups, INOVASI works together with its counterparts to reflect on
current challenges and discuss the next steps in planning.
Inevitably, capacity and capability are sometimes intertwined. People with proper technical skills
may be more able to find solutions and to understand how to work with the resources at their
disposal. They might also be better informed about politically feasible second-best alternatives.

14
The short course is essentially teachers’ working group-based training.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 19
The focus on capability therefore does not replace technical competencies. It becomes an
additional element that can empower actors on the ground.
In summary, when INOVASI began in 2016, PDIA had only recently been applied in the education
sector whereas now the RISE program (under SMERU) and the World Bank’s Melayani program
also implement PDIA in the education sector. However, INOVASI was the first education program
in Indonesia to use the approach. Therefore, as this section demonstrates, adapting PDIA to the
Indonesian education sector required some trial and error.

4.2 Authority: supporting district regulations and engagement with local

actors
From a PDIA perspective, policy reform or development relates to generating the necessary
authority for reform and thus broadening the space for reformers. It also includes engaging with
local authorities to secure their buy-in (Andrews, Pritchett and Woolcock 2017, p. 158).
INOVASI has supported the development of 49 regulations at the district level. This policy work
meets one of INOVASI’s intermediate outcomes, namely ‘district governments adopt policy to,
directly and indirectly, support learning outcomes’. From a program perspective, working with
counterparts on regulations is also a requirement from the DFAT performance assessment
framework that counts the ‘number of districts with improved service delivery practices and
policies’.

INOVASI’s policy work has several purposes. First, regulations provide the legal bases for a
particular intervention. An example of this is from Probolinggo district in East Java that has a large
number of small primary schools with fewer than 50 students. Allocating a dedicated teacher for
every class is inefficient15 and INOVASI recommended that the district should implement multi-
grade teaching. Another reason for encouraging this approach in Probolinggo was the number of
teachers whose qualifications and experience were below the national standards in many schools
in the district (INOVASI 2019b, pp. 27-30). Once the district decided to go ahead with this idea in
2019, INOVASI worked with Probolinggo to develop the head of district regulation on multi-grade
teaching.
Second, the policy work aims to ensure that districts have sufficient budget to continue the pilots
when the project ends. The education policy and governance team assesses the funding available
for scaling out the pilots and if it is limited, the team helps district governments to shift allocations
from non-quality related activities to quality-related interventions. For example, in the Dompu
district in West Nusa Tenggara, schools needed fences to stop animals like goats coming into the
schoolyard and the team recommended they put up simple fences made from wood and wire,
instead of a more expensive version made of concrete and bricks. Schools would then be able to
use the saving, for example, to improve the quality of teachers and education personnel.16
Even when districts allocate funding for teachers and education personnel, the funds may not be
used to improve education quality. In 2019, INOVASI facilitated working groups in 16 districts
(excluding Malinau) to analyse district budgets from 2016 to 2019. The analysis showed that less
than one per cent of budget was allocated for teaching and learning in the classroom. This is in line
with findings from a DFAT-commissioned study that analysed five-year strategic plans, annual
plans and budget documents from 160 districts across Indonesia (POM 2016, pp. 6-7). With the
evidence that INOVASI collected showing the discrepancy between spending on infrastructure and

15
According to MoEC Regulation No. 17 /2017 a classroom at a primary school should consist of 20 students. Therefore, a primary
school consisting of grades 1 to 6 having less than 50 students is inefficient.
16
In Indonesia education personnel includes school principals, supervisors, the school’s administrative staff, laboratory assistants and
library staff.

20 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
on improving teachers’ quality, authorities were convinced to reallocate funding from physical
reconstruction to teacher training and teachers’ working groups.
Third, INOVASI’s contribution to policy development can facilitate cross-agency collaboration that
did not exist before. For example, working on the issue of inclusion, INOVASI brought together
different actors to work on the roadmap of inclusive education in Central Lombok, West Nusa
Tenggara. This district already had a head of district regulation on inclusion but it gave the district
education office full responsibility for inclusion. The roadmap involved other district stakeholders,
such as the district health office,17 the social affairs office18 and libraries.19 INOVASI also engaged
with the community and village empowerment office, a non-conventional partner for an education
program, to enlist village funds to support libraries in West Sumba.
Fourth, INOVASI seeks to contribute particularly to regulations that strengthen the quality of
teaching. For example, in Sidoarjo, the team reviewed the district head regulation on the regional
schools’ operational funds (BOSDA) for primary and junior secondary schools, resulting in a more
comprehensive new regulation.20 The previous21 regulation did not specifically regulate the use of
BOSDA for teachers’ working group activities. The new regulation stipulates a budget allocation for
a resource person, training materials, stationery, photocopying and activity banners to support
teachers’ working group activities. Also, the former regulation only regulated BOSDA allocations to
pay tuition fees for poor students while the new regulation broadens this to ‘social inclusion’. The
new regulation includes, for example, the purchase of educational aids for children with disabilities.
It also stipulates honorariums for one special teacher to assist children with disabilities and one
teacher to coordinate inclusive education activities at the school.
As figure 4.3 shows, INOVASI has produced nine district head or mayor’s regulations
(Perbup/Perwali). The strongest regulations at district level are local regulations (Perda), followed
by district head or mayor’s regulations that do not require approval from the local parliament. This
means the process involved in issuing them is shorter. From the nine regulations, four support
literacy, two focus on continuing professional development for teachers, two on inclusion and one
on guidelines for BOSDA allocations.
The level of regulations required depends on their purpose. If more experiment is needed, a district
head or mayor’s decree might be better. Unlike district head or mayor’s regulations that have to be
implemented across the district or city, a decree can have a limited scope. For example, in 2018, in
the Malinau district, North Kalimantan, INOVASI was still experimenting with integrating village
libraries with surrounding schools. Therefore, they pursued a decree rather than a regulation. If the
decree is effective, the plan is to issue a district head or mayor’s regulation in this area in the
future.

17
To coordinate children suffering from stunting which also hampers their learning outcomes.
18
The district social affairs office has data on children with disabilities who are out of school.
19
Libraries are included because they are a facility of life-long learning that cater to the needs of people from various backgrounds and
age groups. Therefore, they need to be conscious about inclusion and are able to provide the books and space that address the needs
of their members.
20
The regulation supported by INOVASI is District regulation No 6 of 2019.
21
District regulation No 19 of 2018.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 21
 Figure 4.3: Types of regulations supported by INOVASI

Notes: The graph is based on the 49 regulations that INOVASI supported ‘Others’ include assignment letters, district head instructions
and decision letters
Source: INOVASI MERL 2020

One decree by MoEC’s curriculum and book centre (Puskurbuk) is an example of how INOVASI’s
sub-national work can feed into national-level policies. In this instance, INOVASI’s baseline survey
(Survei Inovasi Pendidikan dan Pembelajaran – SIPPI) identified that 85 per cent of students in the
districts of Bulungan and Malinau (North Kalimantan) like to read. The barrier in facilitating
children’s reading interest was the limited availability of appropriate books, especially for early
grade students. The North Kalimantan team then advocated that the centre revisit the list of books
deemed suitable for early grade students. This effort led to only 260 books being identified as
appropriate from thousands of titles (INOVASI TASS 2019, p. 3). Since then the centre has
provided a new list of suitable book titles.
One point for reflection is that contributing to numerous district regulations is in itself an
achievement for INOVASI. Being invited to provide drafts for the districts to consider demonstrates
the trust between local actors and the program. Working on district regulations also contributes to
indicator #14 in INOVASI’s results framework, namely: ‘number of districts that make
improvements in educational service delivery policy’. INOVASI’s technical guidance note on
indicators further explains that ‘improved educational service delivery policy includes policies made
or implemented at the district level that have implications for improved education service delivery’
(INOVASI 2018b). Using this definition all participating districts can show improvements in
educational service delivery policy.
However, by presenting the indicator in these terms, the program is unable to describe with more
accuracy what improvement is being made. Is an improvement made because INOVASI
contributed to a policy that did not exist before? Or is an improvement made because a regulation
that was drafted with INOVASI’s support is of higher quality compared to a previous regulation? If
this is the case, how is it better? 22 More importantly, do improved regulations contribute to
improved service delivery? Thus, to be more meaningful, ‘improvements in educational service
delivery policy’ may need to be defined more narrowly in the future. If the program claims that
improvement occurred, the question that must be satisfied is: an improvement to what condition?

22
A senior staff member at INOVASI noted that district head regulations supported by the program are always equipped with the
necessary implementing guidelines explaining which personnel are charged with the implementation and technical guidelines explaining
how to implement certain activities. For example, a regulation may stipulate that teachers’ performance is assessed through the
competency assessment and character assessment. The implementing guideline determines the parties responsible for implementing
the two assessments (for example, school supervisors) and the technical guideline stipulates what the assessments consist of. Not all
regulations have these guidelines. If this is the case, having an implementing guideline and technical guideline can indicate a better
regulation that might be monitored in the future.

22 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
Finally, what is the relationship between a regulation and a pilot? Does a successful pilot become
adopted in a regulation or does INOVASI help to develop a regulation first before its pilot starts?
The answer depends on the context. In several districts or cities, INOVASI began with a pilot
before institutionalising the intervention through a regulation. Such a situation occurred for instance
in Batu city and in the districts of Bima, West Sumba, Probolinggo and Sidoarjo. However, the
district of Central Lombok declared itself as an inclusive district in 2013.23 To support the district’s
goals, INOVASI conducted the SETARA pilot on inclusive education in 2017, followed by
developing a more comprehensive roadmap for inclusion in 2018 (Hadiwijaya 2019).24

4.3 Ability: structured technical learning or short courses

As mentioned in section 4.1, after the second strategy testing, INOVASI recognised that it had to
make changes to its PDIA approach. The INOVASI six-monthly report in July 2018 notes:

‘The first round of pilot and pre-pilot activities demonstrated that the purely bottom–up
approach to PDIA relied too much on teachers generating solutions when they needed more
structured input to improve learning outcomes in the local context’ (INOVASI 2018a, p. 3).

As a response to the situation, at the beginning of 2018 INOVASI began to implement short
courses 25 on literacy and numeracy. Implemented in teachers’ working groups (KKG), these
courses serve as in-service continuing professional development for teachers. INOVASI selected
the teachers’ working groups because they are part of the existing educational structure 26 in
Indonesia. By using them INOVASI is also strengthening these groups. For instance, an INOVASI-
led study in West Nusa Tenggara concludes that most teachers’ working groups do not focus on
improving teacher quality. Instead, their activities revolve around socialising the 2013 curriculum,
organising educational competitions or developing exam materials (Akrom 2017, p. 48). INOVASI’s
pilots, therefore, encourage the groups to include content on improving education quality by
strengthening the groups’ continuing professional development function, as well as providing a
structure to the group activities and changing from the usual non-learning-oriented activities.
The short courses consist of ‘in and on’ sessions similar to both DFAT’s continuing professional
development training for school principals (ProDEP) conducted under the Education Partnership
(2012–2016) and MoEC’s own training arrangements. The courses are implemented during the
school calendar year from July to June and teachers meet five times a semester with each meeting
lasting three to five hours.27 During the ‘in’ sessions, teachers receive new material, reflect on the
implementation of the previous learning material in their schools and design an action plan for
carrying out the training they receive in their classrooms. The sequence of the activity is to reflect
on the previous learning they implemented before introducing new material. The process and the
activity in each meeting are depicted in greater detail in figure 4.4.

23
Through district regulation No 39 of 2013
24
A new district regulation was developed as the legal bases to implement the roadmap, namely district regulation No 31 of
2019 on the implementation of inclusive education in Central Lombok.
25
INOVASI uses the term short course and pilot interchangeably but they refer to the same activity.
26
Teachers’ working groups exist everywhere in Indonesia but in many districts it is unclear which section within the district education
office is responsible for administering their affairs. INOVASI’s education policy and governance team argues that this situation has led to
the poor quality of many groups. Given their unclear status, they are often treated as a ‘stepchild’ within the district education office.
27
Each meeting session is designed to be completed within three hours. Nevertheless, teachers often require more time to understand
the material. INOVASI is flexible in adjusting the delivery time of its training to suit participants’ needs.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 23
 Figure 4.4: Short courses; the in-on-in sessions sequence

Facilitators and peers observe how participating teachers execute their learning in the classroom
during the ‘on’ session and provide feedback. The in-on-in sequence is repeated until the training
materials (modules) are completed. Table 4.1 provides an overview of INOVASI’s modules. The
main content of the short course revolves around literacy and numeracy but, depending on the
local context and demand, specific elements can be included, such as community engagement,
mother-tongue transition and inclusion.
Table 4.1: INOVASI’s short-course modules
No. Module
1. Literacy 1
2. Literacy 2
3. Numeracy 1
4. Numeracy 2
5. Inclusion
6. Community module
7. Language transition
8. Leadership
9. Multi-grade

The pilots are also subject to adaptation. The first source of adaptation is derived from participants’
feedback about implementing the courses. In some locations, the short courses (the in sessions)
take place in shorter sessions over a longer period, in other locations they are longer sessions over
a shorter period. Adaptation also applies to the on sessions where individual teachers and
facilitators select their own format for mentoring that can happen in several ways. The teacher and
the local facilitator conducting the mentoring may agree on the number of meetings. Some meet
immediately in the classroom, others prefer to meet prior to teaching to discuss the lesson plan.
Mentoring can also either be done one-on-one or a local facilitator can mentor more than one
teacher (if there are parallel classes) in the same session. In this instance, a teacher teaches while
the local facilitator and the other teacher observe and provide feedback. If a teacher is new and
lacks confidence, the local facilitator can give an example of teaching or they can do team
teaching. Once the new teacher becomes more confident, he or she can teach during other on

24 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
sessions. This variation occurs at the school level as well as the working group level and the
selected options depend on a range of factors, including the local facilitator’s capacity, the
teacher’s capacity, the number of classes and the school’s location.
This is the co-design element of the pilot. At the time of writing this report, the education and
religious affairs district offices were in the process of adapting the modules for their continuing
professional development program. These adaptations form what the program coins the ‘co-design
process’. INOVASI technical guidance on the indicators defines co-design as ‘… involving local
stakeholders who are expected to be involved in the initial plan, design and implementation of the
intervention’ (INOVASI 2018b, p.9).
This study is unable to address some gaps in knowledge about how the pilots evolve.28 In 2017
and 2018, INOVASI implemented pre-pilots29 in fewer schools (conducted over a period of five
months) before they implemented the pilot in more schools in the district. The overarching lessons
learned from the pre-pilots were documented in various reports and are summarised in table 4.2.
However, the information is sporadic in that there is no structured documentation about lessons
learned from each pre-pilot in each district. We also lack details on how the lessons learned
shaped or contributed to the design of a particular pilot in a particular area, who from the local
stakeholders contributed to the co-design process or how INOVASI selected which inputs to
include from the pre-pilots and which inputs to drop.
Table 4.2: Summary of pre-pilot lessons learned
Pre-pilot lessons Source Lessons’ impact
The SETARA (disability inclusion) INOVASI Guiding program To make the distinction,
pre-pilot found that teachers were strategy (INOVASI 2018c, p. 8) INOVASI developed a tool
unable to distinguish children with known as the student learning
disabilities from children with minor profile (profil belajar siswa) that
learning problems is filled out by an assessor
teacher nominated by the
school and the district
education office
The GEMBIRA pre-pilot (language INOVASI Guiding program The pilot incorporated methods
transition) found that teachers had strategy (INOVASI 2018c, p. 8) from the Summer Institute of
difficulties helping children Linguistics using teachers’
transition from their local language gestures, pictures and local
to Bahasa Indonesia examples to help language
transition
The PERMATA (numeracy) and INOVASI Guiding program INOVASI rolled out the short
PELITA (literacy) pre-pilots strategy (INOVASI 2018c, p. 8) courses
identified that teachers required
foundational skills in literacy and
numeracy and that INOVASI’s
initial PDIA (classroom action
research) approach was
insufficient
The Guru BAIK pre-pilot (and pilot) Six-monthly progress report, July The growth mindset element
demonstrated that the growth 2019 (INOVASI 2019c, p. 17) was embedded in other pilots
mindset approach has facilitated a
change from teacher-centred to
student- centred learning

28
Finding out the information would require an interview with local stakeholders involved in the pre-pilots and pilots but given the limited
resources for the thematic case study, this was not possible.
29
Pre-pilots were implemented in West Nusa Tenggara, INOVASI’s first province. Other provinces had pilots that were already
embedded within the teachers’ working groups.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 25
Where we have better information is the relationship between pilots and culture. INOVASI deals
with two types of culture. First, INOVASI influences the school culture by creating a more open
relationship between teachers, principals and supervisors. Also, teachers are able to exchange
ideas more freely. Such a situation translates into the classroom where children are able to interact
and learn more from each other instead only from the teacher (Cannon 2020, p. 115).
The second type is how INOVASI’s pilots engage with the local culture. In this area there are
knowledge gaps in how exactly local culture in each province or district shape the pilots. For
instance, we lack information of how the culture in NTT would make literacy pilots different than
those in North Kalimantan. Nevertheless, there are some prominent examples. Pilots on mother
tongue in Sumba (Arsendy 2019) and Bima (Fillmore and Handayani 2018), as well as pilots in
Dompu addressing child jockeys and children leaving schools to accompany parents during rice
planting and harvesting are the few showing a strong connection between pilots and the local
culture.
To summarise, INOVASI has strengthened teachers’ ability in literacy and numeracy in order to
improve the overall quality of their teaching. Some endline surveys were still underway during the
writing of this thematic study and hence complete results from the INOVASI pilots are discussed
elsewhere. There is room for improvement in collecting evidence on how pilots adapt.
Nevertheless, to a certain extent, the program demonstrated how iterative adaptations can be done
by using pre-pilots and pilots.

4.4 Acceptance: shifting teachers’ mindset and generating buy-in among

government officials
INOVASI seeks to change the teaching practice from teacher-centred to student-centred. This goal
is in line with the spirit of the government’s most recent curriculum, Curriculum 2013 (K-13). It is
also consistent with the principles of active learning that has improved student learning as the
evidence suggests (for example, Michael 2006). However, changing teachers’ teaching practice
has been a challenge. The government has been trying to make the shift towards student-centred
learning since 1984 but without much impact. Donors’ programs, such as the Active Learning and
Professional Support program (British Council), the Creating Learning Communities for Children
program (UNICEF-UNESCO), the Decentralized Basic Education program (USAID), PRIORITAS
(USAID) and the Learning Assistance Program for Islamic Schools (AusAID) all attempted to assist
with the transition to student-centred learning.
Despite years of effort, research finds that active learning in Indonesia has been hampered by
several factors. Ragatz’s video study (2015) on the implementation of active learning among
eighth- grade students studying mathematics finds that between 2007 and 2011 teachers actually
became more teacher-centred in their approaches. Teachers who were surveyed explained that
their approaches were driven by the national exam that forced them to ‘teach to the test’ using
closed and routine problems that they felt would improve students’ scores. Although effective in
answering national exam questions, education experts believe that such an approach ‘does not
promote true understanding of mathematics’ (Ragatz 2015, p.130).
Ragatz’s study also finds that the effectiveness of active learning is not only linked to teachers’ skill
in carrying out the method but also with their beliefs. Some teachers have what he coins as
‘transmissionist’ belief. They perceive themselves as the source of knowledge which contradicts
active learning that encourages children to gain knowledge from interaction with their environment
and classmates. Children taught by teachers with a transmissionist belief tend to have lower
learning outcomes compared to others. Other teachers have a ‘connectionist’ belief system and act
as facilitators. Their belief system fits active learning and consequently their students have better
learning outcomes compared to the previous group. Ragatz finds that the most effective teachers

26 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
are those with a ‘flexible’ belief system. Depending on the circumstances, they apply rote learning
or more progressive methods (p. 132).
In another study, Sopantini (2014) argues that implementing active learning has been hampered by
a combination of technical, political and cultural factors. Many teachers still lack the technical skills
to implement active learning properly. Organisations such as the education quality assurance
agencies that were tasked with training teachers in active learning often did not have the
appropriate capacity (pp. 279–280).
Politically there are also barriers to active learning. The education quality assurance agencies at
the provincial level tend to consider that they represent the central government.30 While from a
bureaucratic perspective this is correct, district education officials were uncomfortable about often
being bypassed by these agencies that conducted training through the teachers’ working groups in
their areas. This process often neglects officials in the district education office, school supervisors
and principals to whom teachers report (Sopantini 2014, p. 286).
Sopantini further explains that cultural incentives may not align with a school implementing active
learning. Despite commitments from the school principal, schools might shy away from introducing
active learning because other schools in the vicinity are not ready to and doing active learning
alone could disturb the harmony among the schools. Seeking acceptance from others is common
in a collectivist society such as in Indonesia (Sopantini 2014, p. 292).
The evidence shows that changing teachers’ mindset to suit the needs of active learning was a
difficult undertaking. Similarly, changing teachers’ mindset to be more problem-oriented will not be
easy. The lesson from efforts to implement active learning is that without the proper belief system,
teachers will find it difficult to try out new alternatives to improve students’ learning outcomes.
Apart from the classroom level, INOVASI also worked at the district, provincial and national levels
to create acceptance among government officials. District-level stakeholders are engaged from the
early stages through the district planning meetings. At these meetings INOVASI, among others,
presents its baseline findings on education issues encountered in the district and uses the
meetings to verify the findings. District stakeholders also have a voice in determining the issues in
basic education to be addressed and the schools that should participate, as well as developing
criteria for the facilitators recruited.
Given the limited role of the provincial level in basic education,31 INOVASI consults the provincial
education office prior to engaging with the districts (particularly on which districts should be
involved). After that, the provincial education offices are mainly involved in joint monitoring
missions and the Temu INOVASI events. Although INOVASI’s activities are at the district level, it
also engages with national-level stakeholders at MoEC through steering committee meetings. Also,
INOVASI’s management unit consists of INOVASI staff members and echelon-2 officials from the
relevant MoEC directorates. These national-level mechanisms are used to share sub-national level
findings with MoEC. In so doing INOVASI does not bypass the central ministry.
INOVASI also generates acceptance at different levels of government by disseminating information
to appropriate stakeholders. This is done, for example, through the Temu INOVASI events held at
MoEC’s district, provincial or central offices. Here, districts share their learning from pilots with
other districts. The event not only showcases good practices but also stimulates debate about
policy issues. Totok Suprayitno, head of MoEC’s research and development agency was quoted in
the local newspaper, Tribunnews as follows:

30
Although located at the provincial level, the quality assurance agencies are part of MoEC, not part of the provincial or district
education offices.
31
The provincial level is responsible for senior secondary education.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 27
 ‘Temu INOVASI is an occasion to share ideas and good practices in Indonesia to improve
students’ learning outcomes. I hope that these good practices from West Nusa Tenggara,
East Nusa Tenggara, North Kalimantan and East Java do not stop at the participating
schools, but can gain momentum to spread to other schools in other locations’ (Ismanto,
2018).

The program also connects one level of government to another. For instance, the 2017 North
Kalimantan baseline data showed that 85 per cent of children liked to read. Yet, schools had
limited age-appropriate reading books. 32 Through collaboration with non-governmental
organisations, teachers, libraries, community members and universities, INOVASI increased the
supply of more appropriate books for lower grade students. INOVASI also raised the issue with
MoEC and it responded by issuing appropriate policies for schools to purchase more suitable
books (Heyward 2019a).
Various stakeholders need to buy into the reform efforts or innovations that the program offers.
Furthermore, as a program that requires beneficiaries to actively participate in identifying and
solving problems, INOVASI needs to create acceptance for its PDIA approach.

4.5 Crawling the design space

In its earlier stages INOVASI tried to implement monitoring and experiential evaluation (known as
MeE). As a concept, the addition of experiential – the small ‘e’ in MeE – represents a process
whereby implementing organisations ‘articulate the design space…then dynamically crawl the
design space by simultaneously trying out design alternatives and then adapting the project
sequentially based on results’ (Pritchett, Samji and Hammer 2013, p. 2).
INOVASI carried out the MeE by asking teachers to reflect on what teaching practices work and do
not work in improving literacy. While this encouraged teachers to be more reflective and critical of
their own performance, it is unclear how INOVASI used the MeE process ‘to crawl the design
space’. The MeE, baseline and endline surveys, conventional pilot monitoring (classroom
observations, interviews, small surveys) and reflections (discussions with local facilitators) were all
designed so that the program could learn from the processes and design appropriate solutions for
each context (INOVASI 2018c, p. 14).
However, there are challenges associated with how a program learns. Firstly, in an adaptive
program, the monitoring, evaluation, research and learning (MERL) team has to deal with the
tension of providing reliable data that might require a long process and at the same time delivering
the data fast enough for the program to make adjustments. INOVASI’s design document (DFAT
2014, p. 7) stipulates that one of the end-of-program outcomes is for ‘decision-makers to access a
robust body of evidence of what practice and policy implementation changes work to improve
student learning outcomes in Indonesia’.
Initially, INOVASI planned to obtain this ‘robust body of evidence’ through randomised controlled
trials (RCT). However the team decided that this method would only make sense after workable
context-specific solutions are identified and hence these trials could only be conducted in the
second phase. Abandoning the randomised controlled trials in the first phase meant changing the
end-of-program outcome from providing ‘robust’ evidence to ‘credible’ evidence. Although not

32
Children in grades one to three require books with lots of pictures and limited text but these books were not available in school
libraries. During a visit to one school in North Kalimantan in 2018, the author examined the school library’s record of students who
borrowed books and the book title. The record showed a student in grade two borrowing The danger of drugs (Bahaya narkoba) that was
full of text and had few illustrations.

28 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
derived from a statistically-robust process, the evidence should nevertheless be credible for
decision-makers.33
The INOVASI experience shows that for many local government officials, credible evidence does
not necessarily mean baseline and endline comparisons or statistically representative surveys.
They often consider, for example, that observation at the classroom or school level or informal
discussions with teachers on how particular training affected them are credible enough to justify
allocating local budget for scale out. For the program, such premature action by government
counterparts can be perceived as a risk (Heyward 2019b, pp. 7-11). Scaling out pilots that do not
deliver satisfying results would waste resources. However, for the district education office, in the
absence of good options to improve learning outcomes, INOVASI’s pilots seemed to be a workable
alternative despite this lack of evidence.
Secondly, at least in theory, there is a consensus that an adaptive program can learn from failure.
Failure in a pilot is seen as learning to improve implementation through the next iteration.
Regardless, despite the current appetite for flexible and adaptive programming, conventional
relationships between the donor, contractor and the counterpart government do not always provide
space to admit failure (Heyward 2019c). This dilemma faced by a flexible program is not unique to
INOVASI and has been described elsewhere (for example, Teskey and Tyrell 2017). As a result,
INOVASI has a clear idea of what works in a particular context. Conversely, the program struggles
to answer the question of what does not work.
Given these challenges, how does INOVASI ‘crawl the design space’? As mentioned in section
4.1, INOVASI adapts by using the strategy-testing sessions. These strategic discussions have
made fundamental changes to the overall pilot design over time. Heyward (2019c) explains that the
first round of pilots (January–May 2017) focused on the classroom and applied classroom action
research so teachers could identify their problems and devise solutions. INOVASI began the short
courses in the second round of pilots (July 2018– July 2019) and these had a heavy technical
content. In the third round of pilots (from July 2019) INOVASI began to:
‘…provide technically-informed teacher training, while building successful elements
from the Guru BAIK pilots back into the process, including giving teachers greater
ownership of the process and developing growth mindsets’ (Heyward 2019c, p. 2).

INOVASI staff members reported that local facilitators also played a role in embedding lessons
from Guru BAIK into the other pilots.34 Facilitators for Guru BAIK who later became facilitators for
other pilots also adapted some lessons from Guru BAIK.35
In addition, during the third round, INOVASI negotiated with districts on the type of pilots they
needed. Most opted for a deepening of the literacy and numeracy pilots, implemented through the
Literacy 2 and Numeracy 2 short courses (INOVASI 2019a, p. 25). Moreover, as section 4.3
explains, the pilots went through an iterative process in which minor adaptations from pre-pilots
were embedded to make improvements.

33
An internal paper ‘INOVASI: a strategy for scale-out and beyond’ (Heyward, 2019b) elaborates on what types of evidence are
accepted by decision-makers. It also explains the risk coming from a lack of evidence that INOVASI experienced when a number of its
pilots were scaled out by districts before the endline surveys were completed.
34
The first local facilitators were lecturers, coming mainly from outside the district. INOVASI then recruited teachers, principals and
supervisors from the district as local facilitators. This allowed teachers to experience peer-to-peer learning from those working in the
same area and facing similar issues. Changing the local facilitators from lecturers to local teachers and education personnel is also an
example of INOVASI’s iteration.
35
This study is not able to identify how exactly the lessons from Guru BAIK were adapted to other pilots. It may occur through the
modification of modules or introduction of the Guru BAIK cycle as part of another pilot. Finding out how the process happened would
require fieldwork in West Nusa Tenggara.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 29
The literature explains that one way of ‘crawling the design space’ is by using positive deviance.
INOVASI tried to identify positive deviance in the province of East Java that benefitted for a long
time from donor support. The program conducted a stocktake study that identified 165 good
practices and six particularly prominent examples emerged from these (INOVASI undated). The
challenge was to differentiate between positive deviance and good practices. While the concept of
positive deviance is explained (for example, Pascale, Sternin and Sternin 2010), how it differs from
a good practice is not well explained.
Spreading a good practice was also more complicated than anticipated. In some areas, it was easy
to relate the stocktake result to the pilot, for example in the Sumenep district where the stocktake
identified good practices in literacy and the district also had a literacy pilot. In Probolinggo
however, the stocktake study identified good practices in numeracy, yet the district opted for a
literacy pilot. The lesson here is that spreading good practices may not be in line with the district’s
interest. Even if a good practice or positive deviance can be identified, securing actors’ buy-in is
another challenge.
Moreover, INOVASI’s practice of experimenting with solutions or ‘crawling the design space’ is
different from the idea of taking small bets to test alternatives, as discussed in numerous
publications (for example, Bain, Booth and Wild 2016, p. 20; Mercy Corps and IRC 2016, p. 9;
Green 2018, p. 17; Brinkerhoff, Frazer and McGregor-Mirgahni 2018, p. 4). Instead, what we have
seen so far are improvements of individual pilots.

4.6 Conclusion
The development community might currently have an idea of how an aid-funded program like
INOVASI should implement PDIA. However, when the program began in 2016, shaping a PDIA-
based program in education was an iterative process itself, involving cycles of trial-and-error. The
lesson here is that iteration is not only needed to find appropriate solutions but also to work out
how to implement a new approach like PDIA.
The elements of the triple-A analysis – authority, ability and acceptance – can be seen in the
various aspects of INOVASI’s work. The program’s contribution to over 40 district and city level
regulations gave it the authority to help reallocate district funding, bring different parties together to
collaborate and institutionalise the intervention. The experience, particularly from North
Kalimantan, shows that a development program that works with pilots might benefit by aiming for a
regulation with limited scope (such as a decree) to test interventions in a narrower space (for
example, in a few villages). Once the program is comfortable with how the intervention works, it
can aim for a regulation to be implemented across the whole district or city.
Moreover, INOVASI creates acceptance by building relationships at the district, provincial and
national levels. A program working largely at the district or city level needs to share information and
involve the provincial and national levels as much as possible. This approach is important to reach
the end-of-program outcomes by 2023, namely ‘the policy and practice changes identified are
reflected in Indonesian government (district, provincial, national) policies, regulations and plans.’
Apart from creating buy-in for reform efforts, for a PDIA-based program, generating acceptance
also means changing the mindset of counterparts and beneficiaries to accept the PDIA approach.
Government officials and teachers had experience of receiving solutions from aid programs
whereas in a PDIA program they are expected to contribute to the solutions.
From the perspective of the aid program, faced with a situation where the answer is uncertain,
most likely staff members will start with an alternative they believe most likely to succeed. This is
how INOVASI began the short courses. Previous experience from the PRIORITAS, Decentralized
Basic Education (DBE) and Managing Basic Education (MBE) projects showed that the short
course is the most feasible alternative. The question is how to take on these lessons by

30 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
encouraging diverse innovations. It is also important to learn from other government approaches,
other donor projects and emerging local innovations.
Another lesson from INOVASI is that not all stakeholders can contribute to developing a solution in
a meaningful way. Many teachers lacking foundational skills in literacy and numeracy struggled
with identifying their challenges in the classroom. Therefore, INOVASI needs to build their ability
first. Consistent with the idea of iterative adaptations, INOVASI’s pilots have built on lessons from
pre-pilots or previous pilot cycles. Where INOVASI differs from the literature is that INOVASI is not
using pilots as small bets to find the most appropriate solution. Instead, it seeks to improve
individual pilots based on learning.
What such practice means for flexible and adaptive programs merits a more in-depth discussion in
the future. On one hand, avoiding the process of eliminating less successful pilots appears to be
resource-efficient and does not lead to a waste of taxpayers’ money. On the other hand, treating all
pilots as equally successful could be a missed opportunity for identifying the optimum solution
within a context.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 31
5 Evidence for problem-driven iterative adaptation

The mini literature review (section 2.4) reports that Indonesia has its own PDIA-like initiatives that
have so far been less successful than anticipated. Despite the increasing authority that districts
and schools have to manage their own affairs, local solutions rarely emerge due to traditional
behaviour patterns that have not caught up with post-Reformasi regulatory changes. This section
explains what INOVASI has achieved with PDIA in the Indonesian context, despite these
constraints and section 6 then discusses what can be realistically expected from a flexible
approach in Indonesia.
The INOVASI design document states that the program should ‘focus on identifying
transformational changes which can be adopted by the wider system and facilitate change’ (DFAT
2014, p. 21). This aim to be transformational is shared by other flexible programs. For instance, the
Department for International Development’s Engaged Citizens Pillar (Derbyshire and Donovan
2016, p. 21), the Asia Foundation’s waste reform initiative in Phnom Penh (Denney 2016, p. 30)
and DFAT’s Pacific Leadership Program phase three (Mander-Jones 2017, p. 10) all aspired to be
transformational. But what does being transformational mean? Aid scholar Pablo Yanguas (2018)
argues that ‘transformational change is predicated at institutional change’. While formal institutional
changes, such as regulations or laws, can be achieved within the program cycle, informal
institutional changes that are ‘conventions and codes of behaviour’ (North 1990, p. 4) take longer
to reform. For a program to be transformative, it needs to deal with both the formal and informal
institutions in a particular sector.
In the education sector, persuading teachers and government officials to be problem-oriented
instead of simply compliant with regulations, for example, is an informal institutional change.
Another example is the shift in teachers’ mindset to accept that children can learn from each other
rather than have the teacher as the sole source of knowledge in the classroom. This section
presents data and anecdotal evidence of whether PDIA works and what may indicate informal
institutional change.
However, these are early signals. Whether teachers and government officials who were influenced
by the program are undergoing transformational changes can only be measured in the longer term,
some years after program completion. Nevertheless this section presents some evidence on what
the program has achieved in ensuring that real positive change reaches the classrooms and is not
merely reflected in formal regulations.
The first part of this section looks at the results from Guru BAIK, INOVASI’s main pilot for changing
teachers’ mindsets. We also briefly compare results from Guru BAIK and the Literacy Boost
program (a conventional pre-designed pilot known also as Gema Literasi in Indonesia) since these
were both implemented in North Lombok and Sumbawa districts in West Nusa Tenggara.
Furthermore, a small number of schools in Southwest Sumba received Guru BAIK and the Literacy
1 short course (known as Guru BAIK plus) and we compare results with the schools that received
just the Literacy 1 short course in the same area. Being implemented in the same districts means
we can hold many factors constant that could affect learning outcomes (for example, district
leadership, available funds in the education sector, teacher quality, students’ backgrounds). The
second part of this section looks at anecdotal evidence indicating how the program may have
changed the mindset of district officials.

32 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
5.1 Guru BAIK results and comparative evidence
While Guru BAIK represents a small part of INOVASI’s pilots, the results are important in
understanding the effect of PDIA at the classroom level since the program used this early pilot to
test how the PDIA cycle works. While this report only highlights the results of Guru BAIK, other
thematic case studies (Cannon 2020; Fearnley-Sander 2020; Gibson 2020; Sprunt 2020; van der
Heijden 2020) explain the results from other INOVASI pilots.
Guru BAIK changed during its implementation. The early version of the pilot targeted randomly-
selected teachers in partner schools and, at that time, participating teachers were not linked to a
particular teachers’ working group. After the second strategy testing, when INOVASI began to
implement its pilots in teachers’ working groups, local government budget funding (APBD) for Guru
BAIK was connected to these groups. The idea was that teachers learn best in communities of
practice, especially in communal societies like Indonesia. In addition to teachers, the budget
allocation included principals and supervisors since teachers could not implement the new skills or
ideas without the approval of their superiors.
INOVASI was in its early stages when Guru BAIK began and being new to the concept of PDIA, it
took a step-by-step approach. The focus was on the process rather than the outcomes of
increased learning and whether they could be achieved during the project cycle. INOVASI
expected teachers to identify their own problems and their own solutions.
Although the endline study had not been completed, INOVASI was concerned that the early pilot
approach was not achieving substantial gains in learning outcomes or a comprehensive systemic
change to make a difference. Therefore, as section 4 explains, INOVASI decided to focus on
technical short courses and work politically to influence the districts’ policies. This led to redefining
the meaning of PDIA within the team and a change in approach. A number of team members
contested this pivotal change creating a difficult time for the team.
INOVASI’s director explained the rationale behind the program’s new approach to PDIA:
‘When we talked to our specialists in the field (around 2017) there was a view that you
can’t tell the teachers what to do, we are not allowed to give them any clues or tips. That’s
antithetical to the whole exercise. They have to figure it out. They have to own it. In my
view, that was too far. In a way, you come in, you have all the knowledge and experience,
and it was sort of unethical not to share it a little bit with people in villages that had missed
(proper content) in their teacher education’ (Heyward 2020, comment at the Australasian
Aid conference).
The results of Guru BAIK were presented in the report, Guru BAIK: building teachers’ capacity in
West Nusa Tenggara, Indonesia (INOVASI 2019d). The study is based on a survey conducted in
49 schools in the Sumbawa district. While the baseline was implemented in North Lombok and
Sumbawa in 2016, when the endline was carried out in August 2018, the North Lombok district
was hit by a severe earthquake and the endline survey had to be cancelled. The evaluation was
based on treatment and control schools. Using a control group that did not receive the pilot gives a
more reliable picture of its impact than using just pre-test and post-test data. An evaluation can tell
whether progress was due to an intervention or due to factors external to the program (for
example, an improved curriculum or other government training).
The evaluation of Guru BAIK is mixed but we start with the positive results. Guru BAIK targets
teachers and such interventions are widely known to take some time to yield concrete results in
students’ learning outcomes. The Guru BAIK training took place from January to May 2017. The
baseline was conducted in 2016 and the endline was conducted in 2018. To account for measuring
the two years 36 duration between the baseline and endline, a psychometrician was involved in
36
This means that children that were in grade one during the baseline were in grade three dur-ing the endline survey

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 33
adjusting the tests. In the time period, Guru BAIK increased average literacy scores from 50.5 in
the baseline to 51.7 in the endline (INOVASI 2019c, p. 22). The evaluation figure 5.1 depicts Guru
BAIK’s impact on students’ literacy scores.
Figure 5.1: Guru Baik’s Impact on Students’ Average Literacy Scores (Out of 100)

Guru BAIK also had a positive impact on students’ numeracy scores (figure 5.2). However, Guru
BAIK’s impact on both literacy and numeracy tended to decline in the higher grades (INOVASI
2019d, p. 23).
Figure 5.2: Guru Baik’s Impact on Students’ Numeracy Scores

An issue that emerges is that although the impact of Guru BAIK on students’ literacy and
numeracy (figures 5.1 and 5.2) is modest, there was a strong buy-in from local governments.
Looking at the numbers alone, it is surprising that government officials were enthusiastic about
scaling out a program that had a net impact of 1.20 points in literacy and 2.21 in numeracy (from a
scale of 0 to 100).37
Nevertheless, up until early 2020, districts were scaling out Guru BAIK in numerous schools (table
5.1). Dompu, Bima, North Lombok, Central Lombok, Sumbawa and West Sumbawa districts in
West Nusa Tenggara province had all allocated funding for scale out as early as 2017, long before
the endline was completed in 2018. Securing these allocations usually takes time so this early
response was only possible due to the socialisation process that began in 2016, informing district-
level officials about the concept of Guru BAIK and convincing them of its usefulness. Although

37
This is the author’s assumption and is not based on any interviews with local government officials in districts that have allocated
funding for Guru BAIK.

34 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
funded by the district budget, INOVASI assisted with implementing these district-run Guru BAIK
pilots.
Teachers were also tested on their own skills in literacy and numeracy and their literacy scores
show particularly strong results. The scores in Guru BAIK treatment schools increased whereas
teachers’ scores in the control group declined by the endline. The pilot also encouraged teachers
to use data (such as test scores and community reports) to analyse problems and this activity
increased from 47.06 per cent in the baseline to 83.33 per cent in the endline. A similar trend was
recorded in the control group with a smaller improvement from 62.79 per cent to 83.78 per cent
although the starting point was higher. The changes in the control group indicate that external
factors influenced changes in all surveyed schools.
Principals were also satisfied with students’ learning outcomes. For instance, during the baseline,
28.57 per cent of school principals in the treatment schools reported that they were satisfied with
students’ learning outcomes. In the endline this increased to 42.86 per cent who made the same
comment. For the same category in the control group, 20 per cent of principals in the control
schools were satisfied in the baseline, increasing to 53.33 per cent (INOVASI 2019d, p. 29) in the
endline – a higher increase than in treatment schools. Such data means that external factors, other
than Guru BAIK, were driving the principals’ positive perceptions.
Guru BAIK results were less encouraging in other areas. For instance, the percentage of children
who were excited about going to school dropped from 100 per cent in the baseline to 93.83 per
cent in the endline. Conversely, in the control group, for the same category, the percentage
increased from 95.10 per cent to 96.59 per cent (INOVASI 2019d, p.24). The report suggests that
students might find the shift to active learning ‘more demanding’38 but qualitative findings from a
spot check of Guru BAIK scale outs funded by the local budget indicate that teachers actually find
children happier and more active in the classroom (INOVASI 2018d, p. 8).
Another less positive result was that the percentage of teachers in Guru BAIK schools using
workplans actually declined from 78.12 per cent in the baseline to 62.50 per cent in the endline. A
similar trend was found in the control group where the number dropped from 80.65 per cent to
54.55 per cent. Although no further explanation was provided, it can be assumed that teachers in
general used workplans less.
Table 5.1: Number of schools scaling-out Guru BAIK

District 2017 2018 2019 Total

Bima 16 25 21 62
Dompu 11 10 N/A 21
North Lombok N/A 20 N/A 20
Central Lombok 10 10 11 31
West Sumba N/A N/A 24 24
Southwest Sumba N/A N/A N/A N/A
Sumbawa N/A 10 N/A 10
West Sumbawa 7 50 25 82
N/A = data not available
Note: INOVASI collects data on district allocations for pilots’ scale-out as the amount of funding allocated is reported in six-monthly
reports. However, it was too difficult to unpack the bulk of funding to identify how much was specifically allocated for Guru BAIK.
Hence, this table only presents the number of Guru BAIK scale-out schools, not the amount of the funding.

38
A senior staff member from INOVASI finds this surprising since students are usually happier as learning becomes more engaging,
enjoyable and more student-centred.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 35
A district facilitator in one of the districts that allocated funding to scale out Guru BAIK explained
that despite not having seen any endline results, officials were convinced about the quality of the
program. The program’s association with Australian aid seems to give it a positive image.
Regardless of the absence of evidence, they believed that this program would be a better option
than other alternatives available to them.
When an official at the district education office was interviewed in 2018 and asked why West
Sumbawa district allocated funding for Guru BAIK despite not seeing any evidence that it worked,
the official stated: ‘We believe that this is a good program’. In other words, the positive perceptions
of officials were the driver of scale out for Guru BAIK in 2017 when the endline survey was not yet
available. After that, their observations and discussions with participating teachers probably
concluded that the pilots had been worthwhile and that led to further allocations in 2018 and 2019.
Furthermore, INOVASI’s experience with Guru BAIK in Southwest Sumba district presents
evidence of the pilot’s impact on mindsets. Southwest Sumba has seven schools that received
Guru BAIK plus the Literacy 1 short-course pilot (Guru BAIK plus) and there are seven other
schools that received the Literacy 1 short-course pilot without Guru BAIK. Purba and Sukoco
(2019) reveal that teachers that received the Guru BAIK pilot in addition to the Literacy 1 short
course had a stronger impact on basic literacy scores in general. Moreover, they also perform
better for children with special needs, those from poorer economic backgrounds and those who
use a different mother tongue (table 5.2).

Table 5.2: Literacy test scores: schools that received Literacy 1 and schools that received Guru BAIK
and Literacy 1: Southwest Sumba

% of students who passed the basic Guru BAIK plus

Literacy pilot 1
literacy test (letter, syllable and word (Guru BAIK + Literacy pilot 1)
(% increase)
recognition) (% increase)

All students 75% 113%

Gender
Male 81% 135%
Female 72% 97%
Special needs
Students with special needs 41% 193%
Students without special needs 84% 100%
Socio economic status
Top 88% 49%
Middle 53% 58%
Bottom 88% 121%
Students’ mother tongue
Indonesian 41% 59%
Local language 111% 131%
Note: Some figures are above 100 per cent because what is measured is the percentage of increase from baseline to endline. The
(𝑒𝑛𝑑𝑙𝑖𝑛𝑒−𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒)
equation used to calculate the figure presented in this table is 𝑋 100%
𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒

The data shows that Guru BAIK has increased awareness about challenges in the classroom in
general and those faced by disadvantaged children in particular (Purba and Sukoco 2019, p. 96).

36 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
Based on this limited sample of 14 schools, this finding supports INOVASI’s decision to embed
Guru BAIK into most of its pilots in the third round.
Turning to the Literacy Boost pilot that Save the Children implemented with INOVASI funding from
March 2017 to November 2018. At that time INOVASI was still finalising its main approach and the
Literacy Boost pilot was an ‘off the shelf’ solution to demonstrate immediate gains. This need
arose because MoEC was still unfamiliar with the nature of flexible programs and wanted to get a
clear idea of the concept underpinning INOVASI. Literacy Boost is an internationally-implemented
program by Save the Children and can also be used as a counterfactual to the context-specific
PDIA.
Literacy Boost has several components and focuses on training teachers in literacy skills and
working with communities to ensure that children learn outside school. First, it trains master
trainers – school supervisors, principals and officials from the district education office – who then in
turn train teachers in teaching literacy. Second, the pilot increases children’s access to books.
Literacy Boost distributed 15,800 books to 50 target schools in North Lombok and Sumbawa.
Third, it creates reading camps run by facilitators in the partner schools’ vicinity. The reading
camps encourage students to do reading activities in a less formal environment outside the
classroom. Fourth, the pilot also raises awareness among parents about the need to encourage
and assist their children to read at home. By the end of the pilot, 1,064 parents had participated in
the activities. Fifth, it also develops the reading buddy program where students from the fifth grade
become reading peers for students in grades one to three. Literacy Boost matched 504 students
from the lower grades to 348 students in higher grades (Save the Children and INOVASI 2019, pp.
10-13).
The main difference between Guru BAIK and Literacy Boost is that Guru BAIK specifically targets
teachers while Literacy Boost also involves the broader community, such as parents, students in
higher grades and local volunteers. In terms of content, Guru BAIK is about helping teachers to
identify and solve problems. The Literacy Boost focuses on delivering technical material to
strengthen literacy teaching.
North Lombok and Sumbawa were the two districts that had both pilots but the earthquake in 2018
meant that collecting data in North Lombok was not possible so any comparison had to be based
on the data from Sumbawa. The data shows that Guru BAIK had a stronger effect on all students’
literacy scores but particularly on boys’ scores (table 5.3). Even with its declining effect on student
literacy in grades two and three, Guru BAIK still had a stronger effect than Literacy Boost.
However, in grade four, Guru BAIK appears to have had a negative effect (table 5.4). Further
research is needed to investigate this anomaly.
Table 5.3: Comparative effects on students’ average literacy scores between the Guru BAIK and
Literacy Boost pilots, by gender

Guru BAIK Literacy Boost

*
Category Coefficient SE SD p.value Effect size Coefficient SE SD p.value Effect size
All 1.95 0.77 2.53 0.0056 0.2076 0.578 0.789 11.38 0.2318 0.0615
Boys 2.32 1.09 2.12 0.0170 0.2466 0.789 1.09 15.72 0.2355 0.0839
Girls 1.61 1.09 1.49 0.0689 0.1715 0.382 1.13 16.3 0.3679 0.0407
Key: SE = standard error; SD = standard deviation; p. value = probability
Source: INOVASI (2019d,p. 22)
Note: Students from grades one to five were surveyed for the baseline and students from grades three to six were surveyed in the
endline. The team opted for this method to track the same students (INOVASI 2019d, p.17).

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 37
 Table 5.4: Comparative effects on students’ average literacy scores between the Guru BAIK and
Literacy Boost pilots, by grades

Guru BAIK Literacy Boost

Grade Coefficient SE SD p.value Effect size Coefficient SE SD p.value Effect size
1 2.28 0.17 1.28 0 1.78 0.03 0.3 2.54 0.46 0.01
2 1.94 0.11 0.93 0 2.09 0.01 0.39 3.01 0.49 0.01
3 1.07 0.15 1.32 0 0.81 0.52 0.31 2.51 0.05 0.21
4 -0.63 0.25 1.9 0.01 -0.33 0.1 0.54 3.2 0.43 0.03
Key: SE = standard error; SD = standard deviation; p. value = probability
Source: INOVASI (2019d,p. 22).
Note:
[Mean of treatment group]−[Mean of control group]
𝑒𝑓𝑓𝑒𝑐𝑡 𝑠𝑖𝑧𝑒 = . There are many ways to interpret how large an effect size is. Cohen, for instance,
Standard Deviation
has the following criteria for effect size: small (0.2), medium (0.5), large (0.8). Therefore, with the exception of the Guru BAIK effect size
among students in grades 1-3 in Table 5.4 all other effect sizes are small. The effect size is calculated in order to compare Guru BAIK
and Literacy Boost.

Apart from the effects on learning outcomes, it appears that Guru BAIK had better buy-in from local
government counterparts. Literacy Boost was only scaled out in 10 schools in 2018 in the Moyo
and Medang islands. After implementing the pilot in urban areas, the government of Sumbawa
wanted to test its effect in remote islands. However, the scale out was short-lived and was
discontinued in 2019. Instead, Guru BAIK in the Sumbawa district was scaled out in 2018 and
2019 in a total of 25 schools. There were plans for further scaling out in 2020 but the funding was
reallocated to fund the scale out of INOVASI’s Numeracy 2 pilot. The North Lombok district that
also had experience with Guru BAIK and Literacy Boost, opted to scale out only Guru BAIK in 20
schools in 2018.
District facilitators in Sumbawa and North Lombok believed that the district’s preference towards
Guru BAIK was influenced by the district education office relationship with INOVASI. INOVASI had
been working in these districts for longer than Save the Children and this may have led to the
stronger buy-in for Guru BAIK.
Based on the experience of Guru BAIK and Literacy Boost, the second strategy testing
recommended a more technically and politically informed approach (section 4.1). This shows that
INOVASI iterates within the pilots but also uses learning across pilots.

5.2 Jalan Andrews and indications of behaviour change at district level

Beginning in mid-2019, INOVASI expanded PDIA from the classroom to the district level and this
approach is known among program staff members as Jalan Andrews (literally, ‘Andrews way’, after
the principal PDIA author, Matt Andrews). The goal of this expansion is to assist district
governments to work in a more problem-driven manner instead of just continuing as usual. In so
doing INOVASI is expanding its approach from classroom to district level.
Government officials tend to strictly follow guidelines and regulations rather than focus on
problems and ways to address them. These attitudes among officials are understandable since
they do not want to be seen to deviate from the rules. At the same time, by focusing too much on
regulations, they can lose focus on the real development problems at hand. During a steering
committee meeting in 2018, Totok Suprayitno, INOVASI’s main counterpart at MoEC, noted:

38 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
 ‘We observe that many teachers are not innovative because they are too focused on
regulations, there is too much compliance. Teachers are more afraid that they are not
implementing regulations from the national level, enforced by school supervisors, than not
teaching children properly…These are facts and are exacerbated by our regulations that
are artificial and administrative…School supervisors are visiting schools, providing
feedback, making corrections and conducting quality assurance not based on students’
learning outcomes, but whether or not teachers are compliant with existing regulations.
[Moreover], the National Profession Certification Body (BSNP) has the eight national
education standards. [We] assume that if schools are applying these standards, they will
automatically produce students with good learning outcomes’ (INOVASI 2018e, p. 6).
Shifting PDIA from the classroom to the district level is intended to deal with this overemphasis on
the superficial, bureaucratic aspects of teachers’ work at the expense of the substance – ensuring
children learn – what Lant Pritchett would call a form of isomorphic mimicry (Pritchett 2013).
Officials at the district education office and other government organisations need to agree that the
ultimate purpose of regulations is to improve learning outcomes instead of simply securing
teachers’ compliance. This does not mean that the program encourages teachers and principals to
bypass the regulations or laws. The idea is to identify reform space within existing regulations and
focus on achieving development outcomes.
However, unlike at the school level, the evidence to capture behaviour change among government
officials is limited. This is inevitable since INOVASI focuses on changes at the classroom and
school level. Furthermore, at the time of writing this report, PDIA at the district level had only been
tried out for six months and was subject to continuous improvements so it was too early to expect
transformative change in how government operates. Based on his experience of applying PDIA in
Mozambique, Andrews (2015, p. 299) speaks of the Hawthorne effect whereby authorisers might
act differently in the presence of external parties. Whether the approach created lasting change
remains to be seen in the years to come. Therefore, this section only aims to explain how the
approach was implemented and describe any indications of behaviour change after the six-month
period.
The required steps for extending the use of PDIA – the Jalan Andrews process – were adopted
from Andrews et al. (2017) and were socialised among provincial teams in June 2019. They are
listed in box 5.1. Moreover, Table 5.5 explains the considerations in selecting each participating
district/city in each INOVASI province.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 39
 Table 5.5: Considerations for selecting participating districts in Jalan Andrews
Province District/city Consideration
West Nusa Tenggara Central Lombok  Closer proximity to Mataram where
the INOVASI provincial office is
located, allowing easier access for
providing support
 Most conducive working
relationship with the district
education office
North Kalimantan Bulungan  INOVASI has a strong working
relationship with Bulungan
 The district education office
personnel has a strong appetite for
change
East Java Probolinggo  No district election in the near
future
 Education is the main priority for the
district government
East Nusa Tenggara Kupang  Initially the district of East Sumba
was selected but was eventually
dropped due to change in
leadership.
 The provincial government
requested DFAT’s assistance
during the steering committee
meeting

40 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
Having strong working relationships was the dominant criteria for provincial offices in selecting the
participating districts. PDIA requires close collaboration and a degree of openness from district
officials meaning that trust is crucial. For instance, data played an important role in identifying
problems yet many districts struggled to provide reliable and valid data. Admitting this situation to a
donor-funded program required an element of trust.
Each district approached a different problem in the education sector. Bulungan focused on literacy
and particularly scaling out the literacy pilot in grades four, five and six through the teachers’
working groups and using the district budget allocation. The main consideration was that many
students in the higher grades are still unable to read. Scaling out to other grade teachers would
also ensure greater sustainability of the program since the learning outcomes of children who were
already taught by trained teachers in lower grades could be threatened once they graduate to
higher grades and have untrained teachers.
The district of Central Lombok opted to strengthen literacy by addressing the issue of weak
teacher-cluster working groups. The difference with Bulungan was that the Central Lombok scale
out focused on the overall strengthening of the teachers’ working group, but remained focused on
teachers in lower grades. Their goal was to ensure that lessons from INOVASI pilots were applied
by a larger number of schools in the district.
The Probolinggo district chose to tackle a broader challenge, namely strengthening human
resource capability at the district education office. In September 2019, the head of the district
education office sent a formal letter to INOVASI and requested assistance to address this matter.
The need to improve the capability of staff members arose in response to the district head’s focus
on improving education (particularly through multi-grade teaching and the use of village funds for
education).
The East Nusa Tenggara province implemented Jalan Andrews in Kupang, the province’s capital.
Initially, the district of East Sumba was selected due to the distinctive role of the Sumba Education
Forum in improving education. In addition, East Sumba scaled out INOVASI’s program in all
schools in the district (other participating districts in the province allocated funding for scale out but
only for a limited number of schools). However, implementing Jalan Andrews in East Sumba had to
be abandoned due to an unconducive political situation following the competition between the
district head and his deputy39 that drove a split between their supporters in the bureaucracy.
The main problem to be addressed in East Nusa Tenggara was the high level of out-of-school-
children. Out of 1.35 million school-aged children in East Nusa Tenggara, 111 thousand children
were not in school. As a solution, the province requested INOVASI’s assistance in developing a
‘grand design’ for education.
The facilitating teams’ experiences reveal that selecting problems might be more complicated than
assumed. North Kalimantan and West Nusa Tenggara opted for manageable problems that could
be addressed within the six-month trial period40 but Kupang and Probolinggo selected problems
that are likely to take years to solve. Improving the capability of staff members of the district
education office (Probolinggo) and developing a grand design that will require buy-in from all
districts in East Nusa Tenggara41 (Kupang) are unlikely to show an impact in such a short period.
At the same time, these demands were real district demands – they were real problems for local
stakeholders and authorities requested assistance through official channels. For the team on the
39
Both competed for the chairman of Golkar’s regional executive board at the district level, see http://www.nttonlinenow.com/ new-
2016/2019/02/15/golkar-resmi-berhentikan-gideon-mbilijora-dari-jabatan-ketua-dpd-ii-sumba-timur/
40
The six-month trial period was also adopted from Andrews et al. (2017)
41
Securing buy-in from districts is necessary since basic education is under the authority of district governments. The provincial
government instead is responsible for senior secondary education. The relationship between the province and district in the education
sector is not hierarchical but rather reflects the division of labour.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 41
ground, avoiding assisting with such demands could lead to a deteriorating relationship with
officials and hamper future work.
Moreover, to start off the Jalan Andrews process, the teams in Central Lombok, Probolinggo and
Kupang conducted an analysis workshop to identify and break down the root problems. The team
in Bulungan decided to reflect on the pilots’ implementation in the district and thus build on the
knowledge the team had from working in the district for two years. The North Kalimantan team also
felt that this reflection method would be more effective.
However in North Kalimantan the reflection assessed not only the effectiveness of the pilots but
also the quality of pilot implementation and the budgeting process. The reflection session was
used to make some changes (iteration). First, instead of conducting training for facilitators at the
district level, given the geographical challenges, the training was conducted in seven different
clusters across Bulungan district.42 Second, schools in remote areas can appoint facilitators from
within the schools, allowing effective mentoring to take place. Third, the module was adjusted to fit
the capacities of teachers in Bulungan. The process was conducted by sending modules to
facilitators two weeks prior to the local facilitator training. Local facilitators would discuss the
module in small groups and identify the challenges faced in delivery, and whether the content was
clear enough for teachers. Based on their input, the North Kalimantan team revised the modules.
Fourth, the budget from the teachers’ working group was funded through the schools’ operational
funds (BOS and BOSDA) instead of from the district budget (APBD). This diversified the funding
sources so that district funds were only used for the facilitators’ training, and monitoring and
evaluation. Fifth, the INOVASI team involved officials from the district education office that have
the specific mandate to conduct training through the teachers’ working group instead of those that
simply had the capacity (but not the authority) to do so (see box 5.2).
There were several challenges in using the PDIA approach. First, all teams secured authorisations
from the top, through the head of district (Bulungan, Probolinggo, Central Lombok) and the
governor (Kupang). They then formed taskforces 43 to plan concrete steps in addressing the
selected development problem. Despite having the authority from the highest level of government
and reporting achievements every two weeks (known as ‘push periods’), participation in some
taskforces waned over time. Nevertheless, the remaining taskforce members made some
progress.

Also, the PDIA approach needs active participants and culturally this is not always possible. There
are active champions who would like to see change but some lower-level officials tend to wait for
instructions from the top. This reflects the conventional bureaucratic culture in Indonesia.
Furthermore a participant from the district education office in Central Lombok reported that the
district’s focus on education quality is not always in line with the support they obtain from Jakarta.
MoEC provided more support for infrastructure than for improving the quality of learning and
teaching.
Box 5.2 describes some of the specific challenges encountered in implementing PDIA in one of the
districts.

42
Districts in Kalimantan encompass a vast area. The Bulungan district alone is almost 14 thousand square kms (the North Kalimantan
province with five districts/cities encompasses an area of around 72 thousand square kms). As comparison, the East Java province,
consisting of 38 districts/cities has an area just below 48 thousand square kms.
43
The team members making up the taskforces slightly varied in each district. Generally they include the head of the education office,
various heads of section within the education office, the district planning office, the regional religious affairs office, the local public library
and representatives from teachers’ working groups.

42 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
INOVASI also succeeded in involving different stakeholders in the taskforce (see footnote 43)
showing that the district education office did not need to take sole responsibility for education.
Thus the program helped break down silos among district bureaucracies and showed how
cross-agency collaboration can be done.

44
The name of the district is not mentioned following the advice of the provincial team who considered that some of the information
provided could be sensitive for publication.

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 43
Participating districts could also be becoming more problem-based in their approach. One official
involved in the PDIA process reported: ‘Our programs (at the district education office) were based
on routine work (for example, the national exam, school exam, infrastructure development) not
based on the problems they wanted to address.’ In other words, some officials were beginning to
see the merits of identifying problems and working towards appropriate solutions instead of
continuing business as usual.

5.3 Conclusion
Overall, this section shows that Guru BAIK has improved learning outcomes in literacy and
numeracy, albeit only making minor improvements at this stage. In the district of Sumbawa, Guru
BAIK had more impact in improving literacy than the Literacy Boost program. However the districts’
decisions to scale out Guru BAIK were not driven by statistical improvements in learning outcomes.
They had positive perceptions of the program through the socialisation process and from their
officials’ own impressions of its impact through school observation and discussions with
participating teachers.
The evidence in Southwest Sumba shows that combining Guru BAIK with the Literacy 1 pilot led to
better results than having just the Literacy 1 short course pilot. This supports the decision to
combine Guru BAIK with other pilots in round three to optimise learning outcomes. However the
experience of Southwest Sumba represents only one out of 17 districts participating in INOVASI.
The comparison between Guru BAIK and Literacy Boost as well as between Guru BAIK plus and
the Literacy 1 short-course were possible due to staff members’ efforts in finding reliable evidence.
These comparisons were not part of the design from the outset and as a result some questions
remain. For example, although Guru BAIK plus was more effective than just Literacy 1 in
Southwest Sumba, would the evidence hold in the other 16 INOVASI participating districts?
Current data cannot answer this question. This experience shows the need for a more structured
evidence base, particularly for a program that deals with multiple pilots.
Moreover, other adaptive programs can learn from Guru BAIK’s experience in Sumbawa. All
programs tend to see scale outs as an indication of success or as a sign of the counterpart’s buy-
in. However Guru BAIK’s scale-out in 2019 was cancelled because the district funding was
allocated to another INOVASI pilot, Numeracy 2, showing that a program’s own pilots can end up
competing for districts’ limited funding. Districts have their own considerations in scaling out pilots
from a donor-funded program. As development practitioners we need a better understanding of the
drivers behind the districts’ decision making. More importantly, we should be able to answer the
question of whether the district’s decision to scale out Numeracy 2 will lead to better learning
outcomes than if they continued to fund Guru BAIK.
Moving on to behaviour change at the district education office, INOVASI’s experience with PDIA at
the district level (and provincial level in the case of East Nusa Tenggara) shows promising
indications of change. Sub-national education offices became more data driven in their decision
making and collaboration involved different actors. Whether these indications will lead to lasting
institutional changes in the district (or provincial) education offices’ way of working remain to be
seen. To draw stronger conclusions, we need to collect data more systematically in this area in the
future.
Finally, INOVASI has shown that there are champions for change at the district level. These
officials realise that business-as-usual is not sufficient to reach the desired development outcomes.
However, to sustain the impact of PDIA, we need a critical mass of champions. If this happens in
the future we could see a problem-based culture replacing routine planning in participating districts.

44 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
6 Reflection on INOVASI’s experience with problem-driven
iterative adaptation in Indonesia’s education sector

INOVASI’s experience shows that the concept of PDIA and how it works are clear but development
assistance programs require time to operationalise the strategy. The evolution of PDIA from 2016
when it was interpreted as classroom action research to its application at the district level through
Jalan Andrews in 2019, reveals the trial-and-error process within the program.
The first use of PDIA in classroom action research in the Guru BAIK pilot was supported by a
strong commitment within the INOVASI team to this specific interpretation of PDIA. The approach
was conceptualised as a cycle of activity and team members were trained to strictly apply each
step in the cycle. This commitment was reinforced with a particular branding evident in t-shirts and
badges for example, declaring ‘In PDIA we trust’. This excitement was understandable since
implementing such a new approach might bring fundamental changes to Indonesian education and
staff members were enthusiastic about its potential.
The irony is that this formulaic approach restricted the flexible nature of PDIA. This experience
offers three main lessons: (1) approaches like PDIA are best interpreted and applied in flexible
ways; (2) PDIA in the education development context needs to encompass a political or policy
dimension alongside the technical classroom-based dimension; and (3) solutions to local problems
(including classroom practice problems) are best informed by established professional knowledge
(for example about how to teach reading) and good practices (such as have been sustained from
earlier projects), as well as local initiative and innovation. In other words, ‘crawling the design
space’ should include considering established good practice and ‘positive deviance’.
Learning these lessons and reinterpreting PDIA within INOVASI was a difficult process. In
essence, the program had successfully built loyalty to this early interpretation of PDIA, making it
difficult for everyone to accept changes. Ultimately, discussions within the team and with the donor
about the nature of PDIA resulted in deeper understanding and iteration in the program’s approach
– but it required adjustments to the team make up. This in turn resulted in some loss of momentum
and re-learning.
The broader lesson is that adaptive programming requires an open mind on the part of donors and
implementers – and a flexible approach to personnel as programs adapt and evolve. INOVASI’s
second round of pilots built on the early Guru BAIK approach and a lengthy ‘pre-pilot’ process of
exploring problems relating to literacy, numeracy and inclusion in the local context. The piloted
solutions took the form of short courses – continuing professional development – delivered in local
cluster-based teachers’ working groups. The short-course approach itself was an iteration from
approaches in earlier projects (CLCC, MBE, DBE, PRIORITAS) that have evidently produced
sustainable improvements (Cannon 2020) and INOVASI adapted this approach. The program was
spread over a six to ten month period and was more affordable and sustainable as it uses routine
(fortnightly or monthly) teachers’ working group sessions in local schools and adapts generic
course materials to local context in response to the outcomes of the pre-pilot problem exploration.
The development community needs a consensus on whether we need to reinvent the wheel every
time we use PDIA or how an adaptive program can build on existing practices without missing out
on any innovations that might work better. Furthermore, the earlier implementation of Guru BAIK
shows that expecting local actors to come up with local solutions may be harder than anticipated.
Teachers with inadequate training struggled to identify problems and more so to find proper
solutions. This needs more time and they should be able to find their own solutions in the near
future. Aid programs often take place within limited timeframes and hence are under pressure to
show changes over the program cycle. Despite the initial challenges, Guru BAIK demonstrates

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 45
positive effects. INOVASI’s comparisons between Guru BAIK and Literacy Boost as well as
between the Guru BAIK plus and the Literacy 1 pilot alone (section 5.1) provides additional
evidence that a flexible approach works better than a conventional approach.
As PDIA builds on identifying local problems, aid programs using this approach might need to be
mindful of the types of problems it addresses. The experience of Jalan Andrews shows that without
clear boundaries, local actors may opt to address problems that are priorities for them but are not
realistic to solve within the short timeframe (for example, strengthening the human resources of the
district education office or significantly reducing the number of out-of-school children). Moreover,
the question of whether or not INOVASI is using PDIA has been raised on several formal and
informal occasions. Sections 4.1 to 4.5 show how the program implements elements of PDIA and
raises some questions about the limited evidence available, for instance, on how iterations occur.
A program with the mandate to use PDIA is required to show how the approach works. At the
same time, to reflect the spirit of PDIA it should not be reduced to templates that demonstrate a
commitment to the approach but not its substance. A program under pressure to find local
solutions might disregard available solutions from past experiences (for example, the short
courses). A PDIA program must find the balance between being able to show evidence and learn
lessons about the approach but at the same time avoid PDIA becoming a straitjacket.
A separate case study is needed to understand the different applications of PDIA in INOVASI’s
four provinces. Informal discussions among staff in Jakarta often stressed that it worked
particularly well in North Kalimantan. Yet the team sometimes deviates from instructions and
adapts guidelines from Jakarta to its situation. For instance, while other provinces started Jalan
Andrews with problem exploration, the North Kalimantan team decided to start by reflecting on the
implementation of their pilots.
What is not evident within INOVASI is how pilots led to teachers taking ‘small bets’ – experimenting
with different solutions or 'crawling the design space'. This merits discussion not merely to tick one
of the PDIA boxes but because it relates to a core principle in INOVASI and in most flexible
programs, namely identifying solutions that fit with the local context. Small bets require failure and
despite all the support for PDIA and other flexible programs, admitting failure is still hard to accept
for all stakeholders involved in any aid program. Instead of abandoning pilots that do not work,
INOVASI uses iterations to improve (almost) all pilots. While this approach might avoid wasting
resources, improving most of the pilots instead of eliminating the less effective ones hampers the
process of identifying solutions that will work in a particular context.
Pilots might work well when managed by dedicated staff from a development assistance program.
However, there is no guarantee that they will have the same quality once transferred to and
managed within counterpart government systems. Hence, working within partner systems in phase
two might be a more appropriate approach. What an aid program should prevent is reverting to
conventional relationships with the counterpart government, taking over the government’s job
instead of empowering them to manage their own affairs. Therefore, it is important that the
capability approach developed within INOVASI is transferred into the second phase.
During the first phase, INOVASI has built strong relationships with counterparts in central and sub-
national governments. It has also developed a deeper understanding of PDIA over the last four
years. Having this modality will enable a faster engagement with government officials in phase two
where INOVASI will work within partner systems. The Government of Indonesia will fund most of
the pilots with INOVASI providing technical support and guidance on the PDIA approach. The
merger between INOVASI and the Technical Assistance for System Strengthening (TASS)
program will also enable a stronger link-up between central and sub-national governments.
On implementing PDIA within government systems, a lesson from phase one is that patience is
crucial. Even practitioners with years of experience in international development who participated
in the Harvard course struggled to operationalise the approach. Government officials juggling many
urgent issues and possibly less confident studying in English will take even longer to grasp the

46 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
concept. Furthermore, INOVASI is considering not using the term PDIA in phase two although it
will emphasise the core elements of PDIA, namely: identifying problems; basing policy on evidence
and seeking stronger solutions through iterations. Based on INOVASI’s experience, those new to
PDIA might focus on complying with PDIA demands rather than determining what PDIA is or is not.
During my PhD fieldwork (apart from this thematic case study), I interviewed government officials
about their experience with flexible programs. Some support the principles but stressed the
challenges that would come from implementing them within government planning systems. A few
disagreed with the flexible principles stating that making too many changes would complicate their
logframe they needed to develop. This forward planning tool started with USAID in 1969 (Natsios,
2010, p.16) when donors spread the use of logframes to non-governmental organisations and
partner governments. Hence, partner governments are not to blame if their systems still embrace
fixed-planning and are less open to short–term iterations.
Education experts trying to support schools implementing active learning in the last decades know
that mentoring is a central aspect of success (for example, Cannon 2010, p. 91). It is not sufficient
to teach theories in workshops. Teachers and principals need mentoring in applying what they
learned in training. Similarly, helping local governments embrace the principles of PDIA would
require mentoring. A friend compares learning PDIA to learning to ride a bike. ‘You will fall many
times in the beginning but eventually you will get it. But those teaching you to ride a bike cannot
expect you to do well in the first instance.’
At the time of report writing, it was too soon for the Jalan Andrews process to provide solid
evidence of behaviour change at the district and provincial education offices. Institutional change
takes time. As a result of the six-month Jalan Andrews process, district actors became more data-
driven and problem-based. These are indications of officials moving away from routine planning
practices. The findings cannot answer Andrews’ (2015) concern about people acting differently due
to the Hawthorne effect caused by the presence of external parties. While this study can share
some lessons, other open-ended questions can only be answered in the future. Whether
INOVASI’s application of PDIA led to more sustained practices due to its context-specific
adaptations and whether PDIA led to better learning outcomes compared to conventional aid-
funded interventions can only be examined through evaluations in the years to come (possibly
during phase 2).
Finally, the literature review section explains that Indonesia had experience with PDIA-like
interventions in the education sector. The local content curriculum, school-based management and
the school-based curriculum all intended to strengthen locally-based solutions and sought to
empower schools to make more independent decisions. Reviews of these programs show limited
success because teachers and principals did not have the appropriate skills, incentives structures
were not aligned to support creativity and stakeholders were hesitant to step up their efforts. Local
solutions demand more effort from local stakeholders and, for many, such an initiative can be
perceived as more work and more burdensome compared to just following instructions.
In a similar spirit, MoEC’s emerging policies embrace the principles of Merdeka Belajar (freedom
for learning) that give teachers more freedom. Part of Merdeka Belajar is the national standard
based school exam (Ujian Sekolah Berstandar Nasional – USBN). This allows teachers to develop
a standard written exam or other means such as group work or essays to determine students’
competency (MoEC 2020). Merdeka Belajar also aspires to create a fun learning environment that
enables students to choose from a variety of sources to learn. The teachers’ task is, therefore not
only delivering the curriculum but also facilitating this process (Apandi 2020). These initiatives by
the government should look back and assess why comparable efforts in the past did not work well.
Despite the good intentions, the evidence shows that if stakeholders are not ready, such initiatives
can lead to ‘premature load-bearing’ (Andrews, Pritchett and Woolcock 2017, pp. 53-76).

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 47
Likewise, an aid program seeking to implement PDIA in the education sector must be mindful of
these historical country experiences. Although not using the label of PDIA, others have tried
locally-based solutions, decentralised decision making and context-specific interventions before.
Instead of enforcing a purist PDIA approach where local actors are encouraged to find their own
solutions, a flexible development assistance program might start by understanding the cultural and
structural constraints that similar approaches encountered within the sector in the past.

48 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
References

Akmeemana, S 2018, comment during the Australasian Aid Conference 2018, viewed 8 July
2019,
<https://www.youtube.com/watch?v=6LmOJlVCLWY>

Akrom, MA 2017, A case study of a teachers’ cluster working group in Moyo Hulu,
Sumbawa, INOVASI, Jakarta, viewed 10 July 2019,
<https://www.inovasi.or.id/en/publication/report-a-case-study-of-teachers-cluster-working-
group-in-moyo-hulu-sumbawa/>

Andrews, M 2015, ‘Doing complex reform through PDIA: judicial sector change in
Mozambique’, Public Administration and Development, 35(5):288–300.

Andrews, M, Ariyasinghe, D, Batuwanthudawa, T, Darmasiri, S, de Silva, N, Harrington, et

al. 2017, Learning target for economic diversification: PDIA in Sri Lanka, CID Faculty
working paper no 332, Center for International Development at Harvard University, Boston,
viewed 18 May 2019,
<https://bsc.cid.harvard.edu/publications/learning-target-economic-diversification>

Andrews, M 2016, ‘PDIA: it does not matter what you call it, it matters that you do it’,
Building State Capability Blog, viewed 19 October 2019,
<https://buildingstatecapability.com/2016/10/03/pdia-it-doesnt-matter-what-you-call-it-it-
matters-that-you-do-it/>

Andrews, M, Pritchett, L, Woolcock, M 2013, ‘Escaping capability traps through problem-

driven iterative adaptation (PDIA)’, World Development 51:234–244.

Andrews, M, Pritchett, L, Woolcock, M 2017, Building state capability: evidence, analysis,

action, Oxford University Press, Oxford.

Australian Embassy, the Philippines 2016, The Pathways program, Powerpoint presentation,
viewed 10 December 2019,
<https://philippines.embassy.gov.au/files/mnla/Austender%20-
%20PATHWAYS%20Industry%20Briefing%20Presentation.pdf>

Apandi, I 2020, ‘Guru penggerak dan merdeka belajar’ (The progressive teacher and the
independent learning principle), Kompasiana, viewed 1 March 2020,
<https://www.kompasiana.com/idrisapandi/5e2fe194d541df18ac2fea02/guru-penggerak-
dan-merdeka-belajar>

Arsendy, S 2019, ‘Mother-tongue language education: improving education quality while

preserving culture’, Indonesia at Melbourne, the University of Melbourne, viewed 17 May
2020
<https://indonesiaatmelbourne.unimelb.edu.au/mother-tongue-language-education-
improving-education-quality-while-preserving-culture/>

Bain, KA, Booth, D, Wild, L 2016, Doing development differently at the World Bank: updating
the plumbing to fit the architecture, Overseas Development Institute, London.
<https://www.odi.org/publications/10555-doing-development-differently-world-bank-updating-
plumbing-fit-architecture>

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 49
Barber, M, Kihn, P, Moffit, A 2011, ‘Deliverology: from idea to implementation’, McKinsey &
Company, viewed 17 April 2019,
<https://www.mckinsey.com/industries/public-sector/our-insights/deliverology-from-idea-to-
implementation>

Barrera-Osorio, F, Fasih, T, Patrinos, HA, Santibanez, L, Decentralized decision-making in

schools: the theory and evidence on school-based management, World Bank, Washington
DC.

Bjork, C 2003, ‘Local responses to decentralization policy in Indonesia’, Comparative

Education Review, 47(2): 184–216.

Brown, T, Richardson, L, Sigdel, S, 2013, Hitting a moving target: delivering accountability,

transparency and inclusion in Nepal, The IDL Group, Triple Line consulting, Foundation for
Development Management, GRM International, viewed 7 March 2019,
<http://www.tripleline.com/wp-content/uploads/2014/01/Hitting-a-moving-
target_27Jan14final.pdf>

Brinkerhoff, DW, Frazer, S, McGregor-Mirghani, L 2018, Adapting to learn and learning to

adapt: practical insights from international development projects, Policy Brief no. 2378-7937,
RTI International, Durham.

Cannon, R 2010, Impact evaluation: a study on the impact of the MGP-BE project on
participating districts, communities and schools in Indonesia, UNICEF, EU, MoEC, Jakarta,
viewed 30 January 2020,
<https://www.researchgate.net/publication/313990914_Impact_Evaluation_Study_A_Study_
of_the_Impact_of_the_UNICEF_MGP-
BE_Project_onParticipating_Districts_Communities_and_Schools_in_Indonesi>

Cannon, R 2020, Continuing professional development and sustainability: studies of CPD

and the sustainability of benefits in Indonesia from INOVASI’s pilots, Jakarta, INOVASI.

Derbyshire, H, Donovan, E 2016, ‘Adaptive programming in practice: shared lessons from

the DFID-funded LASER and SAVI programs’, SAVI and LASER, viewed 15 May 2019,
<http://savi-nigeria.org/wp-content/uploads/2016/08/Laser_Savi_Report-online-version-final-
120816pdf.pdf>

Denney, L 2016, ‘Reforming solid waste management in Phnom Penh’, Working politically in
practice series, case study no 8, The Asia Foundation and Overseas Development Institute,
Viewed 25 May 2019,
<https://asiafoundation.org/publication/reforming-solid-waste-management-phnom-penh/>

Dweck, CS 2006, Mindset: the new psychology of success, Random House, New York.

.Department of Foreign Affairs and Trade (DFAT) 2014, Innovation for Indonesia’s school
children: final design, DFAT, Jakarta, viewed 10 April 2019,
<http://dfat.gov.au/about-us/business-opportunities/tenders/Documents/inovasi-design-
document.pdf>

Department of Foreign Affairs and Trade (DFAT) 2018, ‘The Working Group on Adaptive
and Flexible Programs’, email received from Sarah Boddington on behalf of the working
group on 14 December 2018.

Fearnley-Sander, M 2018, INOVASI guiding program strategy: the program development

approach and strategic implementation choices, INOVASI, Jakarta.

Fearnley-Sander, M 2019, INOVASI fourth strategy testing report, INOVASI, Jakarta.

50 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
Fearnley-Sander, M 2020, Terms of Reference: Reflections on Jalan Andrews: INOVASI’s
implementation of PDIA with provincial and district policy makers, INOVASI, Jakarta.

Fillmore, N, Handayani, W 2018, ‘Language matters: language and learning in Bima,

Indonesia’, the Devpolicy Blog, the Australian National University, viewed 17 May 2020,
<5https://devpolicy.org/language-matters-language-and-learning-in-bima-indonesia-
20180328/>

Government of Australia & the Government of Indonesia (GoA & GoI) 2015, Subsidiary
arrangement between the Government of Australia and the Government of Indonesia
relating to the Innovation Facility for Indonesia’s School Children, the Government of
Australia and the Government of Indonesia, Jakarta, Indonesia.

Green, D 2018, How change happens, Oxford University Press, Oxford.

Hadiwijaya, A 2019, ‘Regulasi yang menumbuhkan peluang, bukan yang kaku’ (Regulations
that create opportunities, not those that are rigid), presented at INOVASI Learning Day in
2019.

Hadley, S, Tilley, H 2017, Governance for growth in Vanuatu: review of a decade of thinking
and working politically, Overseas Development Institute, viewed 19 May 2019,
<http://dfat.gov.au/about-us/publications/Documents/governance-for-growth-phase-2-
program-design-document.pdf>

Harris, D 2016, ‘Leather sector reform in Bangladesh’, Working politically in practice series,
case study no. 7, The Asia Foundation, San Francisco, viewed 8 September 2019,
<https://asiafoundation.org/wp-content/uploads/2016/06/Leather-Sector-Reform-in-
Bangladesh.pdf>

Heyward, M 2018, ‘PDIA course journey: INOVASI’s experience with PDIA to solve the
wicked hard problem of education in Indonesia’, Building State Capability Blog, viewed 3
January 2019,
<https://buildingstatecapability.com/2018/08/06/pdia-course-journey-inovasis-experience-
with-pdia-to-solve-the-wicked-hard-problem-of-basic-education-in-indonesia/#more-2579>

Heyward, M 2019a, ‘The problem with “best practice”: using PDIA to find solutions for
Indonesian education’, Devpolicy Blog, viewed 25 October 2019,
<https://devpolicy.org/the-problem-with-best-practice-using-pdia-to-find-solutions-for-
indonesian-education-20190917/>

Heyward, M 2019b, INOVASI scale out: a strategy for scale out and beyond, INOVASI,
Jakarta.

Heyward, M 2019c, draft blog for innovationXchange, received via email from Mark Heyward
on 10 December 2019.

Heyward, M 2020, discussion at the Australasian Aid Conference, viewed 18 February 2020,
<https://www.youtube.com/watch?v=lFXCobTjB-4>

Honig, D 2015, ‘Navigating by judgement: organizational structure, autonomy, and country

context in delivering foreign aid’, doctoral dissertation, Harvard University, Graduate School
of Arts and Sciences, viewed 17 August 2019,
<https://dash.harvard.edu/handle/1/17467366>

INOVASI 2017, Terms of reference for Guru BAIK, INOVASI, Jakarta

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 51
INOVASI 2017, Guru BAIK process: evaluation report, INOVASI, Jakarta, viewed 28
February 2020,
<https://www.inovasi.or.id/wp-content/uploads/2018/01/GB-Midterm-evaluation-report-
05012018_final.pdf>

INOVASI 2018a, Six-monthly progress report, July 2018, INOVASI, Jakarta.

INOVASI 2018b, INOVASI indicator technical guidance, INOVASI, Jakarta.

INOVASI 2018c, ‘Minutes of fourth steering committee (SC) INOVASI & TASS’, INOVASI,
Jakarta.

INOVASI 2019a, Laporan studi: analisis konteks lokal dan relevansinya terhadap
pembelajaran kelas rangkap (Report: local context analysis and its relevance for multi-grade
teaching), INOVASI, Jakarta.

INOVASI 2019b, Six-monthly progress report, July 2019, INOVASI, Jakarta.

INOVASI 2019c, Guru BAIK: building teachers’ capacity in West Nusa Tenggara, Indonesia,
INOVASI, Jakarta, viewed 25 November 2019,
<https://www.inovasi.or.id/en/publication/report-guru-baik-building-teachers-capacity-in-west-
nusa-tenggara-indonesia/>

INOVASI, TASS 2019, ‘Minutes 6th steering committee meeting INOVASI-TASS, 14 March
2019’, INOVASI, Jakarta.

Ismanto, F 2018, ‘Forum Temu INOVASI diharap jadi solusi peningkatan mutu pendidikan’
(Innovation meeting forum is expected to provide a solution to improve the quality of
education), Tribunnews (online edition), 29 November 2018, viewed 10 December 2019,
<https://www.tribunnews.com/nasional/2018/11/29/forum-temu-inovasi-diharap-jadi-solusi-
peningkatan-mutu-pendidikan>

Jalal, F, Samani, M, Chang, MC, Stevenson, R, Ragatz, AB, Negara, SD 2009, Teacher
certification in Indonesia: a strategy for teacher quality improvement, World Bank, Jakarta.

Ladner, D 2015, ‘Strategy testing: an innovative approach to monitoring highly flexible aid
programs’, Working politically in practice series, case study no. 3, The Asia Foundation, San
Francisco, viewed 8 November 2019,
<https://asiafoundation.org/publication/strategy-testing-an-innovative-approach-to-
monitoring-highly-flexible-aid-programs/>

Lindblom, CE 1958, ‘Policy analysis’, American Economic Review 48(3): 298-312.

Lindblom, CE 1959, ‘The science of muddling through’, Public Administration Review 19(2)
(Spring, 1959): 79-88
Pritchett, L, Samji, S, Hammer, J 2013, ‘It’s all about MeE: using structured experiential
learning (“e”) to crawl the design space’, CGD working paper 322, viewed 17 August 2019,
<https://www.cgdev.org/sites/default/files/its-all-about-mee_1.pdf>

Mander-Zones, Z, Heather-Latu, B, Collins, M 2017, Independent evaluation of the Pacific

Leadership Program (PLP) phase 3, Department of Foreign Affairs and Trade, Canberra,
viewed 17 May 2019,
<https://dfat.gov.au/about-us/publications/Pages/pacific-leadership-program-phase-3-
independent-evaluation-management-response.aspx>

52 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
Marquette, H 2019, ‘What we are missing by not getting our TWP alphabet straight’, From
Poverty to Power, viewed 8 July 2019,
<https://oxfamblogs.org/fp2p/what-were-missing-by-not-getting-our-twp-alphabet-straight/>
Mercy Corps and IRC 2016, Adapting aid: lessons from six case studies, Mercy Corps, IRC,
viewed 12 August 2019,
<https://www.mercycorps.org/sites/default/files/Mercy_Corps_ADAPT_Adapting_aid_report_
with_case_studies.7.21.16.pdf>

Michael, J 2006, ‘Where is the evidence that active learning works?’, Advances in
Physiology Education 30(4): 159–167.

Ministry of Education and Culture (MoEC) 2020, ‘Mendikbud tetapkan empat pokok
kebijakan pendidikan Merdeka Belajar’ (The Minister of Education and Culture regulates four
key components of independent learning), viewed 1 March 2020,
<https://www.kemdikbud.go.id/main/blog/2019/12/mendikbud-tetapkan-empat-pokok-
kebijakan-pendidikan-merdeka-belajar>

Nichols, P, Bodrogini, PW 2018, Aide memoire: strategic review – education sector

investments, INOVASI-TASS (draft), Australian Government, Department of Foreign Affairs
and Trade, Jakarta.

Natsios, A 2010, ‘The clash of the counter-bureaucracy and development’, Center for Global
Development Essay, viewed 2 February 2020,
<www.cgdev.org/content/publications/detail/1424271>

North, DC 1990, Institutions, institutional change and economic performance, Cambridge

University Press, Cambridge.

Organisation for Economic Co-operation and Development (OECD) 2018, 2015 PISA:
Results in focus, OECD, Paris, https://www.oecd.org/pisa/pisa-2015-results-in-focus.pdf
Pellini, A, Karetji, P, Soekadis, A 2018, ‘Doing development differently at scale’, in A Pellini,
B, Prasetiamartati, KP Nugroho (eds), Knowledge, politics, and policymaking in Indonesia,
Springer, Singapore, pp. 131-146.

Performance Oversight and Monitoring (POM), Kajian Renstra nasional dan daerah terkait
guru dan tenaga kependidikan (Study on national and districts’ five-year strategic plan on
teacher and education personnel), POM, Jakarta.

Pritchett, L 2013, The rebirth of education: schooling ain’t learning, Center for Global
Development, Washington, D.C.

Pritchett, L, Samji, S, Hammer, J 2013, It’s all about MeE: using structured experiential
learning (“e”) to crawl the design space, HKS Faculty Research working paper series
RWP13-012, viewed 10 October 2019,
<https://bsc.cid.harvard.edu/publications/its-all-about-mee-using-structured-experiential-
learning-e-crawl-design-space>

Purba, R, Sukoco, G 2019, ‘Rethinking classroom action research: a worthy effort for quality
learning?’, paper presented to the Second International Conference on Educational
Assessment and Policy, Jakarta, 24 September 2019, viewed 13 January 2020,
<https://iceap.kemdikbud.go.id/index.php/iceap/article/view/208/63>

Ragatz, A 2015, Indonesia: a video study of teaching practices in TIMSS eight grade
mathematics classrooms, understanding what teaching practices are used, why they are
used, and how they translate to student learning, the World Bank, Jakarta, viewed 12
November 2019,

INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 53
<http://documents.worldbank.org/curated/en/886911472471847117/Indonesia-A-video-
study-of-teaching-practices-in-TIMSS-eighth-grade-mathematics-classrooms-understanding-
what-teaching-practices-are-used-why-they-are-used-and-how-they-relate-to-student-
learning>

Reily, M 2019, Menteri Muhadjir lawab kritik Sri Mulyani soal sistem pendidikan’ (Minister Muhadjir
answers Sri Mulyani's criticism about the education system), Katadata, Jakarta,
https://katadata.co.id/berita/2019/03/14/menteri-pendikan-jawab-kritik-sri-mulyani-soal-sistem-
pendidikan

Sopantini 2014, ‘Reforming teaching practice in Indonesia: a case study of the

implementation of active learning in primary schools in North Maluku’, PhD thesis, University
of Tasmania, Hobart, viewed 17 August 2019,
<https://eprints.utas.edu.au/22399/1/Whole-Heyward-thesis.pdf>

Stringer, ET 2007, Action research, third edition, Sage Publications, Thousand Oaks.

Save the Children and INOVASI 2019, Literacy Boost: improving literacy for early grade
students in West Nusa Tenggara, Indonesia, INOVASI, Jakarta, viewed 13 January 2020,
<https://www.inovasi.or.id/en/publication/final-report-gema-literasi-improving-literacy-for-
early-grade-students-in-west-nusa-tenggara-indonesia/>

Sulfasyah 2013, ‘Investigating the implementation of the Indonesian KTSP (school-based

curriculum) in the teaching of writing in year two’, PhD thesis, Edith Cowan University, Perth.

Teskey, G 2017a, Thinking and working politically: are we seeing the emergence of a
second orthodoxy, Governance working paper series issue 1, viewed 27 August 2019,
<http://www.abtassociates.com/AbtAssociates/files/99/9972961e-e010-4adc-b50b-
efde2b211083.pdf>

Teskey, G 2017b, ‘So is “Doing Development Differently” a movement now? And if so,
where is it going?’, From Poverty to Power, viewed 1 May 2019,
<https://oxfamblogs.org/fp2p/so-is-doing-development-differently-a-movement-now-and-if-
so-wheres-it-going/>

Vernez, G, Karam, R, Marshall, JH 2012, Implementation of school-based management in

Indonesia, RAND Corporation, Santa Monica.

Widagdo, H 2019, ‘Tantangan-tantangan implementasi Jalan Andrew di Kaltara’ (Challenges

from implementing Jalan Andrews in North Kalimantan), internal working paper, INOVASI,
Jakarta.

World Bank 2015, Mengkaji sepuluh tahun Bantuan Operasional Sekolah (Examining ten
years of the schools operational fund), World Bank, Jakarta, viewed 11 November 2019,
<https://www.worldbank.org/in/news/feature/2015/06/15/reviewing-ten-years-of-indonesia-
school-grants-program>

World Bank 2019, Melayani final presentation, 25 February, Mercure Hotel, Jakarta.
Yanguas, P 2018, Why we lie about aid: development and the messy politics of change, Zed
Books Ltd, London.

54 INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020
INOVASI | Using PDIA to Accelerate the Progress of Indonesian Students’ Learning Outcomes – June 2020 55