MID TERM REPORT FOR THE COURSE OF PROGRAM EVALUATION

Submitted to: Dr. Aneel Salman

Submitted by: Muhammad Faizan

Mphil-GPP-S16-2nd semester

National Defence University, Islamabad

1. Identify and explain the most significant features which must considered while

formulating an evaluation plan?

Introduction:
Program evaluation is a detailed process lingering through systematic ways of defining

and projecting social problems which needed to be addressed.

Evaluation plan:
Evaluation plan or simply evaluation is the systematic technique to address the issues and

problems embedded in a program or project which is either going to be projected or

already in phase or completed. This process leads to investigate the effectiveness of

social programs. Evaluation is a steady process of hectic routines in which one has to

keep in mind the various attributes of the program. When starting an evaluation, certain

criteria are selected so that the evaluator does not transcend the limits of selective

evaluation plan. This criterion depends upon the nature of program and also in which

phase the program already is.

Features of evaluation plan:
Some of the most important features which must be under consideration while

formulating an evaluation plan are listed below:

i. Purpose of evaluation
ii. Logic models
iii. Identifying issues and Framing of relevant questions
iv. Engaging stake holders
v. Data analysis and its interpretation
vi. Setting up a timeline for evaluation activities
vii. Budget

Purpose of evaluation:

The foremost feature of an evaluation plan is the purpose of evaluation. Why we are

going to evaluate a certain program? What actually we are looking for improvement in a

certain program? How this evaluation will benefit or lead way for impact on society as a
whole? Most importantly, purpose of evaluation revolves around program improvement,

program accountability and, knowledge generation. Program improvement and program

accountability falls under the category of primary purpose while knowledge generation is

mostly termed as secondary purpose of evaluation. Formative evaluation (qualitative) is

conducted if our objective is improvement of a program while summative evaluation

(quantitative) is conducted for program accountability.

Logic models:

Logic models are also an important aspect of evaluation. Logic models usually help the

evaluator to construct a guide map embedded with some logic showing a pathway from

policy interventions to policy outcomes. It can also give an easy access to the

stakeholders and clients to notice the goals which are to be projected and delimiting the

risks which underlines the program evaluation.

Identifying issues and Framing of relevant questions:

Third purpose of any evaluation must be to identify those specific issues which are to be

raised or kept in mind while evaluation of any program improvement or program

accountability. The result will be the tight scrutiny of the whole process and the conflict

of interests arising in that particular climate. Parallel, formulating of relevant questions

pictures how strong the evaluator has drawn the attention of the target population to those

specific concerns and issues. By doing so, the overall phenomenon will generate

participatory research (evaluation).

Engaging stakeholders:
Thoughtfully arguing, I think that engaging stakeholders should be the pre-requisite for

any policy program or policy analysis. The reason is that the term stakeholders abhors

to both parties, i.e. Client (individual) or target population and the donor who is actually

financing either policy program or policy analysis. An evaluator is always fulfilling the

interests of the stakeholders because either they are the victims to whom help is given or

implementing a project or program for a donor funded agency or corporation.

Data analysis and its interpretation:

Data analysis is another feature of evaluation plan, that from where an evaluator is

getting the data and whether the data is qualitative or quantitative. The type of data to be

selected and analyzed also depends upon the nature of evaluation, like formative or

summative where we use qualitative and quantitative data respectively. Triangulation is

also a type of data analysis where we use both qualitative and quantitative data.

Setting up a timeline for evaluation activities:

Creating a timeline for proposed activities and daily routines is a useful activity in all

ways. While doing evaluation, timeline limits the lengthy process into small sub units and

its proposed impacts. It also helps the evaluator not to move out of the circles once the

timeline of proposed activities is drawn. Stakeholders can also have an insight about

whole program and can give meaningful results.

Budget:

Budget is also an important part for all types of project proposals and evaluations. In

budgeting, one should remember that it is all about number games. The numbers
 depicting and perceiving shall draw the attention of the benefactor. There is no free lunch

in todays world so while giving proposals or conducting evaluation, the budget of the

program shall be in a range of acceptability and to be selected among various

competitors.

Conclusion:

By keep intact to aforementioned features, an evaluator will have the edge of delivering

some extra ordinary work. The reason is that it follows a pathway from identifying a gap

and the filling of it through sub units of a whole process.

2. Why it is necessary to consult stakeholders before conducting an evaluation?

Discuss the importance of context and stakeholders in program evaluation; provide

at least three practical and contrasting examples each.

Now a days evaluation is not an objective research rather a subjective or participatory

research where we actively engage the stakeholders as the primary benefactor in most

cases. The question is to what extent the nature of participation shall be? How the nature

of stakeholders and their interests do influence? And, at what stage of the evaluation, the

participation should be? These questions are technical in nature as they abhor the

practicality of clash of interests between individuals.

Significance of Stakeholder in conducting an evaluation:
A stakeholder is usually a person who directly sponsors or finance, a program or an

evaluation. Become an important key player by having the monetary strings, stakeholders

can influence certain program processes, manipulate or forged impact results, and can

exploit the whole program if there is a clash of interest. At the other end stakeholders can

give insights of various social dimensions by engaging them in some meaningful

participation. It is explicitly clear that stakeholders do possess some important space in

program evaluation but their nature and type must be examined very deeply. Doing so,

one can easily face the bold gestures of stakeholders and can gain political support if the

stakeholder belongs to political arena.

Engaging stakeholders should be the pre-requisite for any policy program or policy

analysis. The reason is that the term stakeholders abhors to both parties, i.e. Client

(individual) or target population and the donor who is actually financing either policy

program or policy analysis. An evaluator is always fulfilling the interests of the

stakeholders because either they are the victims to whom help is given or implementing a

project or program for a donor funded agency or corporation.

Participation of stakeholders gives a clear understanding on how to associate with

important and sensitive stakeholders. By combating to these techniques an evaluator can

hold the stakeholders in an active participation and can get a clear consensus on the

whole program.
Importance of context and stakeholders in program evaluation:
Consulting the stakeholders is very necessary, as it will help the project to develop

properly in an area. It will tell us about the demographic value of the area so that the

person conducting the project or an evaluation can reserve himself to the concerned area.

If the program evaluation belongs to political sphere, respected political persons shall be

taken into confidence for political support and implementation. If that person is taken in

confidence then the gross root level of the project will get strengthened. It will also

increase the interest of the political people, and the local community as well. Thus

community, local government, provincial government, federal government, and security

agencies are the basic stakeholders for a project or an evaluation to initiate properly and

work efficiently.
Demographically the area coming under the project must be illustrated so that the loss of

funds could not occur and it shall be utilized in most efficient way. The demographic
 classification of the area will tell us about the union councils, villages, tehsil, and district

etc. If the program or evaluation is related to government agency, then for the survey or

the project; government statistics division should also be consulted as they have the best

information regarding an area.

However it is up to the evaluator that to what extent he is drawing the participation of

concerned stakeholders. In some cases stakeholders shall be consulted while formulating

a program plan. It will help in clearing misconceptions about the project and will unite

the evaluator-stakeholder partnership in some concrete form. Other case of engaging

stakeholders in program evaluation would be at mid point to share the process and

progress of program assessment, or at last while doing actual evaluation of the

implementation to see whether the concerns of the stakeholders are resolved or not.
The perfect example of consulting the stakeholders should be the Pakistan bureau of

statistics surveys" who consults the local community, local police, local government, and

local people before starting or evaluating the survey. Another example could be of World

Bank who recently conducted a survey on biomass resource mapping in Pakistan. For this

purpose, they consulted government of Pakistan at first stage, and then linger through

each province. Agriculture department of each province and in each district were

consulted in order to fulfill the survey. Then they contacted different universities for

hiring students.
3. How you will measure the effectiveness of progress in terms of targeting the most

vulnerable in a project intervention being implemented in earthquake affected areas

of the country through a donor funded program.

While measuring the effectiveness of progress in terms of engaging the most vulnerable

in a project intervention being implemented in earthquake affected areas of the country,

we usually adopt case studies. In these case studies we examine the affected areas upon

two criterion; livelihood effect and health effect.

The effectiveness of the progress in a project couldnt be measured only by targeting the

most vulnerable. As it is not a broad term, it will help the most vulnerable to rehabilitate,

but the less vulnerable component would still be at effect because of less or not being

targeted. In case of earthquake affected areas, the vulnerable component; if it is human,

then effectiveness could be measured through the assessment of basic needs of the

individual, the health conditions, and basic facilities provided to the individual through

the project. If the project aims to target individuals as a community, then it should target

all the components of the basic livelihood of the human that are: health, education,

infrastructure development, and food. If all the components are provided by the project

then automatically the vulnerable component, i.e. human will survive and live a better

life. Thus measurement should be done on the basis of components involved to improve

the livelihood of vulnerable component.

In earthquake areas, individuals left are mostly in desperate form as they have lost their

body parts and are paralyzed for their entire life. Positive policy interventions by mostly

donor funded agency in this context would be rehabilitating them with artificial limbs and

transplant so that they can be engaged in local community building. Hence strengthening

the progress, by helping the marginalized people in most effective way.

Another policy intervention would be participatory learning approach (PLA). In this

approach the donor funding agency trains the local community for specific time period,

and when the program has been implemented, the local people trained can carry the

policy interventions with the help of local community, aligning them to build a stronger

community.
 All these policy interventions primarily depend upon the project type and the source of

funding. Measurement of the effectiveness could be done through observation or by

survey by actually engaging the local community and realization of their needs and

complexities. Every service delivery and rehabilitation provided to the vulnerable

component must be measured, and then the results should be illustrated and guidelines

shall be provided.
Source of funding and the donor agency in itself can affect the delivery services; the

vulnerable component of livelihood can ascend or descend the ladder of progress of

policy interventions. In case of Pakistan, national agencies of funding cant get the right

measurements due to unavailability of resources, and cant reach each and every

individual. Foreign funding agencies do have access to most of the tools and resources

due to availability of excess of funds which they can use to engage the local community

for helping the affected areas of the earthquake. UN Programs under the title of UNHCR

are best known for such types of natural calamities and rehabilitation.
4. Program theory helps in distinguishing between theory failure and

implementation failure. View the framework for assessing program theory,

particularly assessment of logic and plausibility.

Program theory is a detailed process in which, inputs are taken into account for achieving

certain goals or outcomes through necessary intervention of causal relations. It defines

the how and why of a program which are explicitly given in a program theory. But

here, it shall be taken into notice that while evaluating a program theory, we evaluate

program not a theory. The reason is that program theory depicts the importance of the

necessary interventions which will affect certain indicators to produce desire results.
Program theory has two main components; impact theory and process theory. In impact

theory we evaluate; proximal (immediate results) and, distal (long term results)
outcomes, while process theory deals with the service utilization, and organizational plan.

Assessing program theory is done relatively in conjunction with the aspects of both,

program process or impact. Program theory does help in distinguishing between theory

failure and implementation failure. This distinction portrays that whether program theory

is not formulated in logical sense or implementation of the said program is somewhat

difficult. By theory failure one should assume that, the theory upon which plausible

conditions are met have some ambiguities or are not constructed logically. Might be, it is

too much idealized with high ambitions. A theory contrasting to the realities is already

renounced failed. Implementation failure is somewhat linked to the final process in

which, the program implementation fails either due to lack of political support or socially

contradicting programs.
The most effective framework for assessing the program theory is by using the logic

model that presents a sensible and plausible model of how the program will function

under certain conditions. It gives a pictorial chart representing the main components of

program theory and their connectivity to one another in a logical form. The logic model

provides us basis for the expected performance of the program. Its elements comprise of

objectives, inputs, activities, resources, outputs, outcomes, impact, and any internal or

external influences. Logic models help us to present our intended goals and outcomes

which are to be achieved by providing necessary resources and activities to proposed

inputs. These logic models can also be of help in improving program designs to achieve

critical goals of a program theory. It is also clearly indicated through the importance of

program theory with respect to stakeholders. Program theory with the help of logical

models can give an overall view of the proposed plan and policy design, and can address

the concerns of stakeholders in a plausible way by redesigning or affirming to a certain

 program theory. Logic models give a complete and explicit view of the hierarchy of the

problem and proposed interventions which will produce specific outcomes.

While making a logical model, we gather all the necessary information and evidences

which supports our problem. In second stage, proposed activities are related to those

evidences which must be analyzed or addressed by providing specific logics through the

context of the problem. The logical models through causality, predicts the outputs, which

will lead to certain outputs. While program theory is a detailed process, explaining

program in a theoretical manner which is usually lengthy and have some bounded

uncertainties, logic models shows the plausibility of concerns that by doing so what can

happened accordingly.
5. Explain the difference/s between policy analysis; program evaluation and research

with examples of each from South Asia.

Although, policy analysis and program evaluation comes under the term of policy

process, yet they exhibit or possess certain differences which are implicit in nature.

Policy analysis is that part of policy process where we identify certain problems and give

guideline for successful interventions by policy makers or policy analyst. Program

evaluation simply deals with the content of program, its past and future. By doing so,

program design or policy process can be measured in terms of its effectiveness, and how

strong its impact could be.

The difference between analysis and evaluation is directly linked to the policy-making

context, within which social science operates or delivers. The use of analysis is most in

demand when underutilized resources are available, when there is a great deal of

confidence in the efficacy of public action, and when policy-makers want to forge new

program initiatives.
Retrospectively, policy analysis and program evaluation belongs to objectivity in which

we tend to look for the gaps which are not filled yet with the past activities. Contrary
research is that field in which we test the hypothesis proposed for our re-searching the

context. Research is mainly derived by the subjectivity phenomena, through which

speculations granted can be tested.

It is also vital to note here that evaluation is an assessment criteria, an evaluator must

know how to collect information, what information can be converted into necessary

evidence, and what recommendation can be handed to strategic partners. On the other

hand, researcher is randomly approaching the hypothesis due to weak judgment of some

core issues within.

The domain of evaluation and research also differs as research is all about individual

curiosity for solving scientific queries based on knowledge and theory. Whereas

evaluation judges the merit or worth of a program, in process (policy analysis) or

implemented (evaluation).
Example:
Role of NGOS in promoting women entrepreneurship; a case study of KP, is an example

of typical research where we will be testing the above hypothesis through a specific

methodology and look for the results; whether NGOS are really promoting women

empowerment or not.
Bhasha dam and its attributes come under the arena of policy analysis. In this context our

problem is well defined but the political feasibility and non-provision of funds had

delayed the process.