This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License.
Your use of this
material constitutes acceptance of that license and the conditions of use of materials on this site.
Copyright 2007, The Johns Hopkins University and Jonathan Weiner. All rights reserved. Use of these materials
permitted only in accordance with license rights granted. Materials provided AS IS; no representations or
warranties provided. User assumes all responsibility for use, and all liability related thereto, and must independently
review all materials for accuracy and efficacy. May contain materials owned by others. User is responsible for
obtaining permissions for use from third parties as needed.
Introduction to Health Program Evaluation
Jonathan Weiner, DrPH
Professor of HPM
Johns Hopkins University
Goals of Session
To provide a framework for understanding health program
evaluation
To provide insights and perspectives useful in the
undertaking of applied evaluative studies
To provide methodologic guidance useful in undertaking
health program evaluation
Section A
Introduction to Health Services Evaluation
Health Services ResearchEvaluation Paradigm
Science
base
New Knowledge for Society
Research
Methods and basic
scientific knowledge
Provider/Program Focus
Evaluation
Improve quality
and efficiency
Program
management
Non-policy
Policy
Policy research
Assess program effectiveness
Policy makers (public and private)
Health Program Evaluation Paradigm: Goals and Objectives
Goals and
Objectives
Program
Management
Outside
Funding
Source or
PolicyMaking
Body
Service Delivery
Goals and
Objectives
Program
Management
Program
Control
Treatment
Units
Delivery
of
Service
Patients
in
Original
State
Outside
Funding
Source or
PolicyMaking
Body
Evaluation: Source of Data
Goals and
Objectives
Program
Management
Program
Control
Treatment
Units
Delivery
of
Service
Structure
Process
Outside
Funding
Source or
PolicyMaking
Body
Patients
in
Original
State
Comparison Change
Evaluation
Outcome
Patients
in Final
State
Evaluation: Source of Data
Goals and
Objectives
Program
Management
Program
Control
Treatment
Units
Delivery
of
Service
Structure
Process
Outside
Funding
Source or
PolicyMaking
Body
Patients
in
Original
State
Comparison Change
Evaluation
Outcome
Patients
in Final
State
External Factors Outside Environment
Goals and
Objectives
Program
Management
Program
Control
Treatment
Units
Delivery
of
Service
Structure
Process
Outside
Funding
Source or
PolicyMaking
Body
Patients
in
Original
State
Comparison Change
Evaluation
Outside
Environment
Outcome
Patients
in Final
State
10
Health Services Evaluation (HSE) and Health Services
Research (HSR) Differences
HSE is always used for decision making
The research questions of HSE are derived (at least in part)
from the program goals
There is a judgement as to ideal criteria against which the
program is judged
HSE always takes place in a real world setting
11
HSE and HSR Similarities
Both apply the same scientific techniques, though they are
more difficult to apply in HSE
12
Formative and Summative Evaluation
There are Two Main Types of Evaluation:
Formative (during formation of program)
Emphasizes internal operational objectives
Can be viewed as a type of ongoing quality improvement
Focuses on efficiency improvement
Summative (helps sum a program once fully implemented)
Allows policymakers to assess whether goals have been
met
Focuses on effectiveness (including cost vs. benefits)
13
Formative Evaluation
Science
base
New knowledge for society
Research
Methods and basic
scientific knowledge
Provider/program focus
Evaluation
Improve quality
and efficiency
Formative
evaluation
Program
management
Non-policy
Policy
Policy research
Assess program effectiveness
Policy makers (public and private)
14
Summative Evaluation
Science
base
New knowledge for society
Research
Methods and basic
scientific knowledge
Provider/program focus
Evaluation
Improve quality
and efficiency
Program
management
Non-policy
Policy
Policy research
Assess program effectiveness
Policy makers (public and private)
Summative
evaluation
15
HSE from Three Perspectives
1. From the organizations perspective
Demonstrate program effectiveness
Justify past expenditures
Required by the funders
Gain support for new programs
Provides mechanism for improving program by
identifying weaknesses
16
HSE from Three Perspectives
2. From the funding agency and/or publics perspective:
How much bang for the buck?
Should the funding be continued?
Can the program be improved?
To document the success of the funders program
17
HSE from Three Perspectives
3. From the evaluators perspective
The desire to contribute to the field
The desire to enhance professional advancement and/or
earn a living
A sympathy with the programs goals
18
Section B
Health Program Evaluation Continued
Factors that Have Increased Importance of HSE over the
Last Several Decades
Budget constraints
New attitudes about government programs
New technologies
Increased desire for public and private organization
accountability
20
Factors that Have Increased Importance of HSE over the
Last Several Decades
Increasing complexity and interdependence of programs in
the system
Increases in the role of evidence-based public health and
health care management
Increased concern over quality of care
Increased concerns with equity and disparities
21
The Basic Theory Behind HSE
1. Find out the programs goals
2. Translate the goals into measurable indicators
3. Collect data on indicators for those receiving care from the
program and those not receiving care
4. Compare indicators between patients in the program and
outside the program
22
A Special Note about The Role of Economic Evaluations
This lecture (and the course overall) focuses on an empirical
impact evaluation approach
Later in the course we will introduce you to the economic
evaluation frameworks of cost-benefit and cost-effectiveness
Both perspectives have their place
At times these two approaches can be combined into one
evaluation
23
Dimensions of Health Program Goals or Objectives
1. The nature of the changes to be effected
2. Relative importance of objectives
3. Target client/patient population
24
Dimensions of Health Program Goals or Objectives
4.
5.
6.
7.
Short-term vs. long-term effects
Magnitude of effect
Stability and duration of change
Complementary nature or conflicting nature of multiple
goals
25
When Goals Are Not Already Defined, Evaluator Must . . .
Wait for personnel to reach consensus
Develop goals independently after consulting with program
staff
Develop goals with staff collaboratively
Do evaluations without specific goals, focusing on
exploratory analyses
26
Reasons Why Goal-Oriented HSE Rarely Works Smoothly
Goals are usually unclear, non-existent or not realistic
Programs often accomplish positive effects besides
documented goals
Health services programs are usually very complex
How do you disaggregate the effects of each component
of total program to provide useful feedback?
27
Reasons Why Goal-Oriented HSE Rarely Works Smoothly
Health outcomes (usually ultimate program goals) are
extremely difficult to show a cause-effect relationship
because of multi-factorial nature of human health
The four-step approach will answer whether or not the goals
have been reached (but the question of why the goals have
or have not been reached is also a key issue)
28
The Dosof a Good Evaluator
Be trusted by all key players
Remain objective
Be knowledgeable about program
Remain autonomous
Become involved in implementation of results
29
A Few Words about other Course Learning Opportunities
The lab exercise
The individual learning exercise
Required and extra readings
30
