Program Development and Evaluation
Course Objectives:
1. Recognize how evaluation fits within the program development process
2. List three reasons why public health program evaluations are important
3. Explain the difference between formative and summative program
evaluations
4. Describe at least five aspects of a public health program that can be
measured
5. Identify at least two factors external to a public health program that may
affect its evaluation
IDENTIFY THE PROBLEM
Public health problems can be identified in multiple ways, all involving the
collection of information.
Health officials can obtain information from:
Health care provider records
Hospital intake forms or records
Laboratory results
THE PROCESS
All successful program planning models have at least three components: Plan,
Implement, and Evaluate
Program evaluation – the systematic collection of information about the
activities, characteristics, and outcomes of programs to make judgements
about the program, improve program effectiveness, and/or inform decisions
about future program development
Evaluation should be part of a public health program from the
beginning.
Two broad categories of evaluation:
1. Summative
2. Formative
Summative – occurs after implementation and examins the outcomes and
impact of a program as compared to the intended goals
Summative evaluation includes:
Outcome evaluation. Did the program achieve its objectives?
Were the program's goals met?
Impact evaluation. What long term effects has the program had?
Cost-effectiveness. Did the program produce sufficient benefits
related to its cost? Can other programs produce these benefits at a
lower cost?
We evaluate outcomes to:
Demonstrate accountability to funders and stakeholders.
Help make informed decisions about the program.
Replicate or model the program.
Formative – asks if a program is being planned, developed, and implemented
as inteded and if the program is reaching the target audience
Formative evaluation includes:
Process evaluation. Is the program operating as designed?
Coverage evaluation. Is the program reaching its intended
audience?
We evaluate the process to:
Monitor activities and identify areas for improvement.
Monitor progress towards meeting the program's goals and
objectives.
Find out if the program is efficient and acceptable.
Demonstrate the program's worth and promote it.
Ideally, both formative and summative evaluations should be used.
Primary users – specific persons who are in a position to do or decide
something regarding the program.
Stakeholders – persons, groups, organizations, or systems that affect, or
may be affected by, the program.
Engaging stakeholder should be the first step in focusing an evaluation. The
program evaluation should be conducted by a team with different
perspectives. If an evaluation is conducted primarily by one person, they
should obtain input from different parties involved.
THE PLAN
In determining the program goal, describe what the program will change and
who will be affected.
Then once the goal statement is agreed upon, it is necessary to develop
program objectives that meet the SMART criteria.
SMART goals are:
Specific
Measurable
Attainable
Realistic
Timely
Once objectives are developed, the next step is to develop a logic model.
This serves as a road map that links the planned work to the expected results.
Logic models include:
Inputs and resources
Activities
Outputs
Short-term outcomes
Intermediate outcomes
Long-term outcomes
Once the logic model is developed, you can use it as a tool for evaluating the
process and desired outcomes of the program
IMPLEMENT
Implementation – refers to conducting or carrying out the activities of the
program; could include one or multiple activities
The activities should be:
Linked to the goal and objectives of a program
Based on a theory, model, or prior research
Appropriate for the target audience – the people who receive or benefit
from the program’s activities
The activities in a public health program can be organized in various
categories:
Administrative systems changes
Behavioral or environmental changes
Communication or outreach
Community mobilization or advocacy efforts
Direct service interventions
Education
Infrastructure-building
Lab diagnostics
Research initiatives
Surveillance or outbreak investigations
FORMATIVE
Formative evaluation produces information that helps form and refine the
program. It includes examining the process and coverage.
Process Evaluation – assesses the fidelity of a program; how the process
affects the outcome; better results are achieved by asking process
questions first
A formative evaluation can help:
Identify strengths and weaknesses of a program.
Monitor activities and identify areas for improvement.
Monitor progress towards meeting the program's goals and
objectives and carrying out its activities.
Find out if the activities are effective, efficient, and acceptable.
Formative evaluation helps you to assess two main
things: process and coverage.
- Process measures help in determining if a program is being
implemented as planned
Data and information about the number of activities
Staff abilities
Available resources
- Coverage measures the extent to which your program reaches those in
need
Continuously evaluating throughout program planning and implementation
aids in reaching the target outcomes, goals, and objectives.
SUMMATIVE
Summative evaluation is used to determine if the program worked. This
focuses on what needs to be changed as a result of implementing the program.
Summative evaluation can help measure the program outcomes and if the
program has achieved its goals and objectives.
Measuring the impact of a program can help decide the program’s future
direction, compare it to similar programs, and/or gain support for additional
resources and funding.
Summative evaluations measure the effect of the program short-term,
intermediate, and long-term.
Program logic modeling helps clarify the summative questions we ask to
measure outcomes, including:
What did we accomplish? Did we achieve our outcomes? Why or
why not?
What is different as a result of our actions?
How expensive was the program?
Did we meet our goals and objectives?
Were there any unintended effects of the program?
A summative evaluation can focus on individuals, groups, organizations or
populations.
A summative evaluation may measure changes in:
Awareness or knowledge
Attitudes, beliefs or motivation
Behaviors or skills
Policies and procedures
Health status or quality of life
Environmental or social conditions
Organizational capacity, service utilization or access
Is the program cost-beneficial?
Summative evaluations should occur after a program has been in place long
enough to produce measurable results.
Summative evaluations strive to measure the program effect.
- Program effect - the outcomes that are attributable to or caused by the
program
Many external factors can influence the outcomes of your program such as:
Other programs
Outside interventions or policies that may be targeted at the same
audience
Economy
Changes in funding can influence program implementation
Political climate
Competing priorities for local, state, and federal resources,
including time, talent, and funding
Social and environmental climate
Popular trends or cultural beliefs can have an impact on the target
audience's beliefs and behaviors
There can be multiple program outcomes:
No change in expected outcome
Opposite direction of expected outcome
There may be more disease in a population after a public health
program is implemented due to increased reporting.
Unintended outcomes are identified
You may identify outcomes that you did not expect to occur as a
result of the program.
Expected outcomes
The outcome is what you anticipated.
Results are neither negative nor positive.
The information provided from results can be evaluated and used to make
judgements, improve effectiveness, and make informed decisions about future
programs.
