JNKVV

College of Agriculture Jabalpur

Department of Extension education

Subject -:Evaluation and impact assessment(Ext507)

Presentation on :-Evaluating the evaluation

Submitted To Submitted By
Dr .Kamini Bisht Neha Jat
Assistant professor 220110006
Dept. of Extension education Msc previous year
 Evaluating the evaluation
Evaluation
Latin word valeue means “ to be strong” be of value

Richard et al.
“Evaluation is the systematic gathering of information for purpose of making
decisions.”

“A systematic methods for collecting ,analyzing and using data to answer

questions about projects ,policies and programs, particularly about their
effectiveness and efficiency .”
Evaluation process
 10 steps in programme evaluation
1. Identify and describe programme
2. Identify the phase of the programme
3. Assess the feasibility of implementing an evaluation
4. Identify and consult key stakeholders
5. Identify approaches to data collection
6. Select the data collection techniques
7. Identify population and select sample
8. Collect analyse and interpret data
9. Communicate findings
10. Apply and use findings
Identify and describe the programme
A program description is a summary of the intervention being evaluated. Before you
can evaluate the program you must be able to clearly describes its purpose, activities
and components as well as the outcomes it is intended to achieve in the short ,
medium ,and long term .the description will also illustrate the program's core
components and elements, its ability to make changes, its stage of development, and
how the program fits into the larger organizational and community environment.

THERE ARE SEVERAL SPECIFIC ASPECTS THAT SHOULD BE INCLUDED WHEN

DESCRIBING A PROGRAM.
 Statement of need
A statement of need describes the problem, goal, or opportunity that the program
addresses . Important features to note regarding a program's need are: the nature of
the problem or goal, who is affected, how big it is, and whether (and how) it is
changing.
Expectations
Expectations are the program's intended results. They describe what the program has to
accomplish to be considered successful. For most programs, the accomplishments exist on a
continuum (first, we want to accomplish X... then, we want to do Y...). Therefore, they should be
organized by time ranging from specific (and immediate) to broad (and longer-term)
consequences.
Activities
Activities are everything the program does to bring about changes. This also shows how different
program activities, such as education and enforcement, relate to one another.
Resources
Resources include the time, talent, equipment, information, money, and other assets available to
conduct program activities. It may also point out situations where there is a mismatch between
what the group wants to do and the resources available to carry out these activities.
Stage of development
A program's stage of development reflects its maturity. At least three phases of development are
commonly recognized: planning, implementation, and effects or outcomes.
Context
A description of the program's context considers the important features of the environment in
which the program operates. This includes understanding the area's history, geography,
politics, and social and economic conditions, and also what other organizations have done.
Logic model
A logic model synthesizes the main program elements into a picture of how the program is
supposed to work. It makes explicit the sequence of events that are presumed to bring about
change. Often this logic is displayed in a flow-chart, map, or table to portray the sequence of
steps leading to program results.
 Creating a logic model allows stakeholders to improve and focus program direction.
There are three phases of evaluation
Design;-
By focusing the evaluation design, we mean doing advance planning about where the
evaluation is headed, and what steps it will take to get there. A well-focused plan is a safeguard
against using time and resources inefficiently.
Implementation
In the implementation phase, program activities are being field tested and modified . Evaluation
during a program’s implementation may examine whether the program is successfully
recruiting and retaining its intended participants, using training materials that meet
standards for accuracy and clarity, maintaining its projected timelines, coordinating
efficiently with other ongoing programs and activities, and meeting applicable legal
standards.

Completion
Following completion of the program, evaluation may examine its immediate outcomes or long-
term impact or summarize its overall performance, including, for example, its efficiency and
sustainability.
eg. Number of people who received the empowerment education or any program service would
not be considered a program outcome unless participation in and of itself represented a change
in behavior or attitude (e.g., participating in a program to treat substance abuse ).
Types of evaluation

Needs Assessment
 A Needs Assessment can be useful for determining whether a problem or
need exists within a community, organization or target group and then
describing that problem. The Needs Assessment is an ongoing process
and is helpful for assessing whether a new policy/program/intervention
may be necessary.

 Baseline evaluation-: Need assessment is a form of baseline to find out

the target group and their perceived needs or expectations from the
programme .
Implementation/Process/Formative Evaluation
A process / formative evaluation is undertaken during the program implementation stage to
determine whether the program is going as per plan and changes if any required to met the
objectives. Formative evaluation are undertaken in long term program and project for cross checking
and corrective measures.
Impact/Outcome/Summative Evaluation
An outcome / summative evaluation is useful for focusing on the results of an initiative, whether they
are short-term, intermediate or long-term .it undertaken once the program active a stable state of
operation or towards the end of the program to find out its results effectiveness, impact and further
course of action .
Follow up
it is undertaken long after completion of the program to see whether there are any long term
changes among beneficiaries .when follow up evaluation are repeated at set time intervals to
study long term benefits , sustainability of results and outcomes they are also called longitudinal
evaluations.
Assess the feasibility of implementing an evaluation
There are several basic questions decide when to evaluate
 Is the programme important or significant enough to call evaluation?
 Is there a legal requirement to carryout an evaluation?
 Will the results of the evaluation influence decision making about the
programme?
 Is there enough time to complete the evaluation?
 Are sufficient funds available to carry out the evaluation?
Identify and consult key stakeholders
 Stakeholders are those who are invested in the program or the outcomes associated
with the program. As such, they are interested in the evaluation or what will be done
with the results. Stakeholder can be individuals, groups and/or entire organization.
Three principle groups of stakeholder
 Persons involved in program operations
Staff and partners
 Persons affected by the program
Clients ,their families ,social networks ,community groups
 Users of the evaluation findings
Policy makers ,managers ,administrators ,funders
Identify approaches to data collection (quantitative,
qualitative ,mixed)
Quantitative Data Is numerical (age or income) and can be quantified and statistically
analyzed
 It can draw connections between study factors
 it proves the answers such question as “who was involved’’ “how many”, or “what were the
outcomes” for an evaluation.
 Most common examples of quantitative data collection are assessments/tests and surveys.

Qualitative Data Is narrative and non-numerical data (Ethnic identify or relationship status ) .
 It describes qualities, opinions or feelings of a study group.
 It answers “how” and “why” for an evaluation.
 Common examples of qualitative data collection are interviews and focus groups
Mixed method
The evaluation of community engagement may need both qualitative and quantitative methods
because diversity of issues addressed eg. population ,Type of project and goals .
Using this approach to gather and evaluate data may assist to increase the validity and reliability
 The main differences between quantitative and qualitative data
methods.
In terms of scope,
 Quantitative data is broader and provides less detailed information across a larger group
of study cases.
 While qualitative data affords more detailed, in-depth information on fewer study cases.
Data Collection
 Quantitative data collection employs structured instruments with questions with pre-
defined response options, such as close-ended questions or rating scales.
 Qualitative data collects information through semi-structured instruments with open-ended
questions where responses are not pre-defined.
Data Format
 As mentioned earlier, quantitative data is numeric and can be statistically analyzed.
 Qualitative data is narrative information that provides insight and context to a program
study.
Data Analysis
 Quantitative data relies on statistical approaches to study the data. This can range from
simple frequencies to more complex statistical analyses such as regression models.
 Qualitative data depends on content analysis in which themes or patterns are identified,
coded, and summarized.
Results
 The results from quantitative data are usually descriptive and, in many cases, can be
generalized to a larger population.
 Qualitative results tend to be exploratory and provide meaning, explanation, and
Qualitative data results cannot be generalized to a larger population .
 Although quantitative and qualitative data are often presented as mutually exclusive, using
a mixed method approach (collecting both quantitative and qualitative data) can ultimately
provide the most complete understanding of your program.In fact, many program
evaluations will incorporate both quantitative and qualitative data in their evaluation.
 Select data collection techniques survey
interviews,and questionnaires with different types
Survey
Surveys are used when you want information from a defined group to get a overall idea of the
study situation or to get a count of a particular characteristic among study cases. • They are
standardized instruments that collect data from a specific group and are usually comprised of
well-defined, close-ended questions. • Surveys are typically administered through mail, email,
online, in-person, or via phone. Email or online surveys are generally less costly than the other
administration modes.

Surveys have a long list of advantages

• They can be a quick and efficient way to obtain information.
• They allow you to obtain information from a large group of people and, depending on how it is
administered, can allow respondents to be anonymous.
• Survey instruments are standardized and allow for an easier way to analyze and compare
results using a range of statistical methods.
 surveys have some disadvantages.
 Surveys may be administered to a group that is not representative of a larger study population.
There may also be some self-reporting issues that may influence results.
 High response rates are ideal, but may be difficult to achieve with a survey. High response rates
are necessary to ensure quality data and sound data analysis.
 Because surveys usually consist of close-ended questions with pre-defined responses, they may
result in a lack of in-depth information on a specific topic or concept.
Questionnaire
A questionnaire is a set of questions for gathering information from individuals. You can
administer questionnaires by mail, telephone, using face-to-face interviews, as handouts, or
electronically (i.e., by e- mail or through Web-based questionnaires).

 When should you use questionnaires for evaluation?

 When resources are limited and you need data from many people. You can disseminate
questionnaires relatively inexpensively. Your costs will increase if you need to do a lot of
follow-up to get a sufficient response rate.
 To gather data about knowledge, beliefs, attitudes, and behaviors. Questionnaires are
helpful in gathering information that is unique to individuals, such as attitudes or
knowledge.
 When it is important to protect the privacy of participants. Questionnaires are helpful in
maintaining participants’ privacy because participants’ responses can be anonymous or
confidential. This is especially important if you are gathering sensitive information
 Types of Questionnaire
Open ended-used in case where new facts are to be searched out. Mainly used for
intensive studies of a limited number of cases .Respondents is free to express his view and
ideas .
Close ended –used when categorized data are required. Inform and choose answer from
asset of provided responses no liberty to express his own judgement .
Advantages of questionnaires include –
 Inexpensive way of collecting data.
 Generates a large amount of data.
 Usually has an easy-to-understand and easy-to-respond design which makes it easy to
understand and respond to.
 The responses can be easily quantified. It can also be used to compare and contrast other
research.
 It’s easy to analyse the results of a questionnaire.
 The qualitative and quantitative data collected from a questionnaire helps the
surveyor to create new strategies and learn about the trends in the audience.
 Questionnaires often let its respondents maintain their anonymity.

Questionnaires have many disadvantages too, like –

 Respondents are sometimes dishonest while answering the questions.
 When it comes to open-end questions, it becomes difficult to understand and
interpret the responses.
 A questionnaire isn’t a perfect tool to know about the feelings and emotions of the
respondents.
 Lack of personalization demotivate the respondents and many don’t even
respond to it.
interviews
 An interview is accurately defined as a formal meeting between two individuals in which the
interviewer asks the interviewee questions in order to gather information. An interview not only
collects personal information from the interviewees, but it is also a way to acquire insights into
people’s other skills.

 Structured interviews have predetermined questions in a set order. They are often closed-
ended, featuring dichotomous (yes/no) or multiple-choice questions. While open-ended
structured interviews exist, they are much less common.

 Semi-structured interviews are a blend of structured and unstructured interviews. While the
interviewer has a general plan for what they want to ask, the questions do not have to follow a
particular phrasing or order.
 An unstructured interview is the most flexible type of interview. The questions and the
order in which they are asked are not set. Instead, the interview can proceed more
spontaneously, based on the participant’s previous answers.

Advantages
 Interviews generally have better response rates than surveys .
 Provide rich, in-depth data with the ability to explore a range of topics with respondents .
 They allow for flexibility in questions/probes to clarify questions that some respondents may
not understand.
Disadvantages
 interviews can be very time-consuming to complete and result in less anonymity for the
respondents because they are oftentimes administered in person.
 Depending on how structured the interview questions are they can be difficult to analyze
and compare. They can also be costly, because they require a skilled interviewer to obtain
relevant information from the respondent.