Chapter - 5 Measuring the Impact of Agricultural Extension: Evaluation and Improvement Jagadeeswari Boppana*, Jagan Mohan Reddy’, M, Ravinder Naik’, Uday Kumar! ™Ph, D. Scholar, Department of Agricultural Extension Education, PJTSAU, Rajendranagar *Director, FEI, Rajendranagas, Hyderabad “Professor, Dept. of Agril Extension Education, Rajendranagar “Ph.D. Scholar, Department of Agricultural Extension Education, PITSAU, Rajendranagar Email: jagadceswari317@gmail.com Abstract ‘Assessment of the intervention on the beneficiaries is very significant for assessing the intervention utility. Impact assessment helps in understanding the results, outcomes or changes occurred due to the new technology/intervention on the target group. It is important to assess the impact of the technology to determine its success, how it has impacted the beneficiaries and the local community, and also to use the findings of the assessment for recommending changes in the policies. Impact assessment methodologies have evolved over the year and the present chapter focuses on the need for impact measurement in extension and the approaches for impact assessment. Keywords: Impact assessment, Difference-in-differences method, Non-experimental design, Experimental design, Quasi experimental design, Social impact, Environmental impact54 Introduction Agricultural extension services play a pivotal role in fostering rural development, enhancing food security, and improving the livelihoods of millions of farmers around the world. These services, provided by governments, non-governmental organizations, and other stakeholders, aim to equip farmers with knowledge, skills, and resources to adopt modern agricultural practices and technologies. However, to ensure the effectiveness and efficiency of these programs, there is an imperative need for impact measurement. Measuring the impact of agricultural extension programs is not merely a matter of administrative necessity; it isa fundamental requirement for the successful execution of agricultural development strategies. This need stems from the increasing demand for transparency, accountability, and evidence-based decision-making in the realm of agricultural development. It is also driven by the urgency of addressing complex and interrelated challenges such as climate change, population growth, and resource constraints. An effective Impact measurement should assess the mechanisms by which beneficiaries are responding to the intervention (Khandker ef a/., 2009). In this context, this chapter will explore the pressing reasons for measuring the impact of agricultural extension services and various approaches and tools for the measurement of impact of Agricultural extension. Impact assessment Impact assessment is intended to determine more broadly whether the program had the desired effects (both positive and negative) on individuals, households, and institutions, and whether those effects are attributable to the program intervention (Baker, 2000). Impact analysis measures the significant changes that had occurred due to a program or any kind of intervention. These changes consider what had changed, to whom, how important the change was, how long the change lasts and how that intervention had contributed to the change. Differences between Monitoring, Evaluation and Impact S.No [Monitoring _| Evaluation Impact 1 [Measures on-| It measures performance of| Measures the long-term going activities | project against objective _| changes; positive or negative, intended or not55 2 |Itisimportant _ | Itisrequired in the middle or | Can be included at all stages during the at the end of project and or afier the end of project implementation of the project 3. | Tefocuses on Focus is on the intervention | It studies the affected intervention population 4 |Focusison the | Here the focusis on the _| Focus is on impact outputs outcomes 5 |Whatis being | Whathas happened? Did we | What has changed? For whom? done achieve what we intended to | How significant it is for them? achieve? Impact measurement framework COOOOO Inputs include all the resources that are used in the project or intervention. ‘This includes financial capital, human capital, infrastructure. Processes are the activities done in order to achieve the outputs and outcome of the intervention Outputs are the final products developed for delivery. These are tangible results of an activity. Outcomes are the medium-term results or the benefits attained by the beneficiaries of the intervention Impact is the long-term sustainable consequence of the intervention. Need for Impact measurement of Agricultural Extension Impact assessment is needed to be carried out as it helps ‘To promote accountability in the allocation of resources across public programs Policy makers decide whether programs are generating intended effects. To fill gaps in understanding what works, what does not, and how measured56 changes in well-being are attributable to a particular project or policy intervention Based on effect, the impact of any intervention can be » Positive or negative Direct or indirect Primary or secondary Intended or unintended Short/mediun/long term Steps for Impact Assessment 1. v Rw ¥ ¥ wean an Technology/Project or intervention whose impact is to be measured should be selected . Target domain should be identified . Define the level of impact assessment . Select the comparison method for impact assessment Before and after method ‘With and without method Comparison ‘with and without’ combined with “before and after” method. . Select the sample size for evaluating impact Design the baseline survey Define indicators for impact assessment . Select appropriate methodology . Data collection and analysis Quantitative and Qualitative tools for impact assessment Quantitative tools Qualitative tools Structured interview schedule Key informant and semi-structured interviews Surveys Participatory rural appraisal Questionnaires Rapid rural appraisalST Checklists Focus group discussion Telephone surveys Participant observation Case study Different approaches for impact assessment Impact assessment approaches can be classified into various types like quantitative, qualitative participatory methods or mixed approaches. Qualitative approaches focus on the potential impact of the intervention, mechanism of the impact through in depth and group-based interviews. Quantitative approaches focus on assessing the degree and extent of impact quantitively. These approaches generate a robust measure of the impact of a programme on their target beneficiaries, by comparing a treatment group (those receiving the intervention) to a control group (those not receiving the intervention). Impact assessments can be carried out at different levels of aggregation—individual research projects, specific research programs, or the research and technology system as a whole. ‘Types of Impact Assessments Based on the time period of assessment, impact can be measured at any stage L.e., ex-ante (before the intervention), ex-post (after the intervention) or at any stage of the intervention. Ex-ante assessment: Ex ante assessments are undertaken before the project or program is initiated as an aid in priority setting. It forecasts the potential impacts as part of the planning, design and approval of an intervention. The output of the ex-ante impact assessment consists of “project documents,” which include the “log frame ” and the “Ex-ante Evaluation Table.” It should be performed based on the concept and evaluation viewpoints such as necessity, priority, adequacy, effectiveness, impact,and sustainability and the results should be incorporated into the implementation plan. Ex-post assessment: Ex post assessments are undertaken after diffusion of an intervention has been initiated, to assess actual impacts on the ground. Ex-post assessments have immediate benefits and reflect reality. Ex- post assessments can be much more costly than ex ante assessments because they require collecting data on actual outcomes for participant58 and nonparticipant groups, as well as on other accompanying social and economic factors that may have determined the course of the intervention. An added cost in the ex- post setting is the failure of the intervention, which might have been predicted through ex -ante analysis. Based on the research design, impact assessment can be of various types i.c., non-experimental, quasi~experimental and experimental research designs. Non- Experimental Design (Sivakumar ef a/., 2017) In the non-experimental research design, researcher cannot control, manipulate or alter the predictor variable or subjects, but instead, relies on interpretation, observation or interactions to come to a conclusion. Hence, one must rely on correlations, surveys or case studies, and cannot demonstrate a true cause-and-effect relationship. Different types of assessments in non-experimental design are > Pre-test/post-test design: In this design, comparison group is not present and the pre and post changes in any specific phenomenon occurred due to the intervention is measure. As there is no comparison group is present here, all the changes in phenomenon cannot be attributed solely to the intervention. » Time-series designs: The changes in outcome indicator over time is estimated to determine trends. It gathers data prior to, during, and after the implementation of an intervention or program to determine the impact over a period of time. » Cross-section Surveys: Cross-section surveys query respondents within a target population at one point in time. > Post-test only design: Here the impact of the intervention in terms of change in any specific phenomenon on the group is studied without considering the aspects before the intervention. Quasi-experimental approaches Researchers cannot randomly assign participants to groups and thus lack control over extraneous variables. Different approaches are Pre-test/Post-test with non-random assignment to intervention or control groups In this design, data are collected before and after the intervention. Assigning subjects to the intervention and comparison groups is non-random.59 ‘Two group post-tests only with non-random assignment In this, after the program has concluded, data from non-individuals and participants who got the intervention are collected. In order to isolate effects of the intervention, matching participants and non-participants is carried out. Double difference or difference-in-differences (DID) method It estimates the effect ofa specific intervention by comparing the changes in outcomes. over time between a population that is beneficiary of intervention (the intervention group) and a population that is not (the control group). It can be applied in both experimental and quasi-experimental designs and requires baseline and follow-up data from the same treatment and control group. Regression discontinuity design AA pre-test — post-test comparison method design that elicits the causal effects of interventions by assigning a cut-off or threshold above or below which an intervention is assigned; ‘Two types of regression discontinuity designs: Sharp method which assigns a discrete cut-off point for both beneficiaries and non- beneficiaries and compares means of treatment effects to assess the impact; Fuzzy method was used in instances where a few eligible people are excluded from the intervention or became ineligible due to other reasons. The fuzzy version is the widely used method. Experimental design Observations are made about an intervention. One or more control groups are used, but groups are created using random assignment. Different types of experimental designs for impact assessment are Randomized evaluations In this, programme benefits are extended to a randomly selected treatment group (beneficiaries), while keeping an identical group as control. The progress of the treatment and control groups on selected impact indicators is tracked over time (Khandker ct al., 2010)60 Pre-test/Post-test with random assignment to intervention or comparison groups Here subjects are randomly assigned to a group that receives the technological intervention (study or treatment group) or a comparison group that does not receive the intervention (control or non-treatment group). Data for each group are collected before and after the intervention. At the end of the experiment, differences between the intervention and comparison groups can be attributed directly to the effect of the intervention. Post-test only randomized experiment Two groups are randomly assigned the subjects and treatment conditions. Data collected only after the intervention. Participatory impact assessment It is an extension of Participatory Rural Appraisal (PRA). It involves the usage of participatory tools combined with more conventional statistical approaches specifically to measure the impact of technology/intervention on the beneficiaries. Mostly extension research employs a qualitative participatory approach to assess the impacts using PRA, focus groups, case studies, participant observation etc. Emphasis of participatory/ collaborative forms of assessment is engaging stakeholders in the assessment process, so they may better understand assessment process, and the program being evaluated and ultimately use the evaluation findings for decision- making purposes. Stakeholders can be involved in any stage of assessment. Mixed method approach of impact assessment An integrated approach that draws on tools and techniques from at least two different social science disciplines for defining hypotheses, sample selection, evaluation design, data collection and analysis. Combines quantitative and qualitative approaches. Quantitative impact assessment techniques work well for a limited set of pre- established variables (preferably determined and measured ex ante) but less well for capturing unintended, less expected (indirect) effects of interventions. Qualitative methods or descriptive (secondary) data analysis can be helpful in better understanding the unintended or less expected effects of interventions. Economic impact assessment techniques Economic impact assessment is a formalized analysis of the overall economic impact of An intervention/ project or technology.61 Methods of Economic Analysis Partial budgeting technique: Partial budget is useful in analyzing the effects of a change from an existing plan. This budget only considers revenue and expense items that will change with a defined change in the plan. Partial budgeting= (added income/reduced costs)-increased costs/reduced income) Net present value: This is simply the present worth of the cash flow stream. Sometimes, it is referred to as Net Present Value (NPV). The selection criterion of the project depends on positive value of NPW when discounted at the opportunity cost of the capital. This could be satisfactorily done, provided there is a correct estimate of opportunity cost of capital. NPW is an absolute measure, but not relative. NPV= Rv(1+i)t Benefit cost ratio: Benefit-cost ratio (BCR) is an indicator showing the relationship between the relative costs and benefits of a proposed project, expressed in monetary or qualitative terms. If a project has a BCR greater than 1.0, the project is expected to deliver a positive ‘net present value to a firm and its investors. B-C Ratio= PV [Net benefits| PV[Costs] Internal rate of return: > The internal rate of return (IRR) is a metric used in financial analysis to estimate the profitability of potential investments. » The internal rate of return (IRR) is a rate of return on an investment. The IRR of an investment is the interest rate that gives it a net present value of 0, or where the sum of discounted cash flow is equal to the investment. The IRR is calculated by trial and error. Adoption Quotient: Adoption is conceived as a 2-part process: » (1) cognitive adoption, involving obtaining knowledge and critically evaluating the practices; and (2) behavioral adoption, involving actual uses of the practices.62 > The variables involved in the 2-part process are identified and used to construct the adoption quotient (AQ), a ratio scale designed to quantify adoption behavior of an individual. » The validity of the AQ scale was tested on 10 high-AQ and 10 low-AQ_ farmers.’ The correlations obtained suggested that the scale is valid. Other types of impact assessments are Social Impact Assessment Social Impact Assessmentis a process of research, planning and the management of social change or consequences (positive and negative, intended and unintended) arising from policies, plans, developments and projects. It is the process of identifying and managing the social impacts of industrial projects. It can also be applied to policies, plans and programmes. Steps of social impact assessment includes: understand the issues; Predict, analyse and assess the likely impact pathways; Develop and implement strategies at the last design and implement monitoring programmes. Environmental Impact Assessment Environmental Impact Assessment (EIA) is a process of evaluating the likely environmental impacts of a proposed project or development, taking into account inter-related socio-economic, cultural and human-health impacts, both beneficial and adverse. It is the assessment of the impact of planned activities on the environment, including impacts on biodiversity, vegetation and ecology, water, and air, Conclusion Impact assessment is required to measure the change occurred due to the intervention. There was a notable transformation in the impact assessment methodologies over the period. Impact assessment is required to assess the impact in term of economic viability, environmental sustainability and socially equitable. Hence, appropriate assessment method has to be chosen according to the specified objectives, given context and expected results. Impact assessment definitely paves a way for smooth implementation of the intervention/technology. References Baker, J. L. (2000.) Evaluating the impact of development projects on poverty: A handbook for practitioners. Washington D.C.: World Bank.63 Maredia, M. K., Byetlee, D., & Anderson, J. R. (2000). Ex post evaluation of economic impacts of agricultural research programs: a tour of good practice. Paper presented to the Workshop on “The Future of Impact Assessment in CGIAR: Needs, Constraints, and Options”, Standing Panel on Impact Assessment (SPIA) of the Technical Advisory Committee, Rome, May 3-5, Rome. Karemulla, K. Introduction to Tools and Techniques For Impact Assessment Of Agricultural Research And Development Projects. 105 FoCARS, Digital repository of course materials. Retrieved from https://naarm.org.in/ focarsrepository/files/9.%20 Techniques %20for% 20Impact%20Assessment.pdf. Khandker, $. R., Koolwal, G. B., & Samad, H. A. (2009). Handbook on impact evaluation: quantitative methods and practices. World Bank Publications. Sivakumar, P.S., Sontakki, B. S., Saravanan, R., & Mittal, N. (Eds.). (2017). Manual on good practices in extension research & evaluation. Agricultural Extension in South Asia (AESA). 191-202. Retrieved from https://Awww-g-fras.org/en/ knowledge/gfras-publications. html?download=669:manual-on-good-practices- in-extension-research-and-evaluation. White, H., & Raitzer, D. A. (2017). Impact evaluation of development interventions: A practical guide. Asian Development Bank.