Archived Information

CHAPTER 7

CONCLUSIONS AND LESSONS FOR THE

EISENHOWER PROGRAM

This is the second report of the National Evaluation of the Eisenhower Professional
Development Program, Part B (Title II of the Elementary and Secondary Education Act). The first
report was based on six exploratory case studies of school districts conducted during the first months
of the evaluation (Birman, Reeve, & Sattler, 1998). The purpose of that report was to obtain an
initial description of the program and the issues that it faced in different local contexts. This second
report describes the current status of the program, based on data from national probability samples of
districts, State Agency for Higher Education (SAHE) grantees, and teachers, as well as on data from
10 in-depth case studies in five states. The third report of the evaluation, to be issued in early 2000,
will augment the cross-sectional results reported here with a longitudinal account of teachers'
experiences in Eisenhower and other professional development activities and changes in their
teaching practices.

The Eisenhower Professional Development Program is the U.S. Department of Education's

largest investment that is solely focused on improving the quality of teaching practice. Part B of the
Eisenhower Professional Development program, with a 1999 appropriation of approximately $335
million, provides funds through states to school districts and institutions of higher education and
nonprofit organizations. These funds primarily support professional development in mathematics
and science.

In its 1994 reauthorization of the program, Congress made explicit that Eisenhower-assisted
professional development activities should be designed to improve teacher practice and, ultimately,
student performance. The law also places particular emphasis on serving teachers in schools with
high poverty rates. Furthermore, the reauthorized Eisenhower program embodies policymakers'
intention that the program support systemic education reform and deeper learning among teachers.

In the analyses conducted for this report, we examine the extent to which these intentions
have been achieved. Our conclusions about the effectiveness of the Eisenhower program are divided
into four main parts. First, we ask, what is the nature of current classroom teaching practices in
mathematics and science, and what directions does current classroom practice suggest for
professional development (Chapter 2)? Second, what evidence do we find that teachers experience
high-quality professional development in Eisenhower-assisted activities (Chapter 3)? Third, what
evidence do we find that districts develop their portfolios of professional development activities and
manage and operate their projects in ways consistent with the intent of the legislation (Chapters 4 and
5)? Fourth, how well do SAHE grantee projects perform with respect to the requirements and intent
of the legislation (Chapter 6)?

These major areas correspond to the main elements in our conceptual framework for the
evaluation (see Exhibit 7.0).

7-1
 EXHIBIT 7.0
Conceptual Framework for This Evaluation

Context for Eisenhower-assisted Activities

(District Size and Poverty; SAHE-grantee Features)

District and SAHE- District and SAHE-

grantee Management of grantee “Portfolios” of
Eisenhower-assisted Eisenhower-assisted
Activities Professional
Development Activities
Building a Vision: Teacher Experiences in
Alignment and Portfolio Features Eisenhower-assisted
Teaching
Coordination Professional Development TeachingPractice
Practice
Teacher Recruitment
and Selection **** * ** ** ** *
Implementation:
Continuous Improvement Source:
**** Source: National Profile Source:Longitudinal
Longitudinal
and Planning Study
(Teacher Activity Survey) StudyofofTeacher
Teacher
Source: National Change
**** and Case Studies ChangeandandCase
CaseStudies
Studies
Profile (District
Source: National Profile Coordinator and SAHE-
(District Coordinator and grantee Interviews) and
SAHE-grantee Case Studies
Interviews) and Case
Studies

Chapter 5 (District) Chapter 4 (District) Chapter 3 Chapter 2

Chapter 6 (SAHE Grantee) Chapter 6 (SAHE Grantee)

In the following section, we briefly review the data on which our evaluation is based. Then,
we turn to the main conclusions and implications of the study.

STRENGTHS OF THE D ATA

Before considering the findings of the National Evaluation of the Eisenhower Program, it is
important to take stock of the quality of the data on which the findings rest. The data set has a
number of important strengths. First, our descriptions of the nature and quality of professional
development provided through the Eisenhower program are based on national probability samples,
each with an excellent response rate. The national probability sample of district programs and SAHE
grantees has an 88 percent response rate for district program coordinators and 87 percent for SAHE
grantees. The national probability sample of teachers who participated in Eisenhower professional

7-2
development activities has a response rate of 72 percent. The 72 percent response rate is especially
high when considering the multistage process necessary to complete the sample. 1

Second, the two probability samples are complementary. Data from our telephone interviews
with district program coordinators and SAHE-grantee project directors are backed up by teacher
participant accounts of what they experienced and its quality.

Finally, we have taken a number of steps to maximize the validity and reliability of the
national survey data. For example, although the telephone interview and teacher survey data are
based on self-reports, most of the data represent an accounting of behaviors, not direct judgments of
quality that might be more likely biased in a positive direction. In addition, the survey results are
cross-validated through case study data that are rich in potential to explain the descriptive statistics
and path analyses from the surveys.

The three strands of the evaluation are designed to produce an integrated portrait of the
Eisenhower program from many perspectives. Because the evaluation involves a variety of research
methods and has collected data from groups of individuals who view Eisenhower-assisted activities
from different vantage points, it is able to provide an accurate description of program-funded
activities and analyses of the features of these activities and their effects on teacher practice.

The data we analyze in this report do not provide direct estimates of changes in teaching
practice over time. Our conclusions about the effectiveness of Eisenhower-assisted professional
development are based on teachers’ reports of the extent to which participation enhanced their
knowledge and skills and improved their teaching. Data from the second and third waves of our
longitudinal study, to be examined in our third report, will provide additional information on teacher
change.2

None of our analyses address directly how teacher participation in Eisenhower professional
development leads to gains in student achievement. The work reported here, however, is not
completely divorced from student achievement. The characteristics of professional development on
which project directors and teacher participants reported in the surveys are carefully grounded in the
available literature on professional development and student achievement. To the extent that the
Eisenhower program provides professional development with characteristics identified as effective in
the literature, we conclude that they are of high quality, and we tentatively infer that they should lead
to benefits for students.

1
District coordinators and SAHE-grantee project directors had to submit the complete list of professional
development activities provided during the prior year and the number of participants. Two activities were selected
from each district with probability in proportion to size, and from those, complete rosters of teachers were collected
from which two teachers were randomly selected and surveyed.
2
We also took a number of steps to maximize the validity and reliability of the evaluation's national survey data.
For example, most of the survey questions ask teachers and administrators to provide an accounting of behaviors,
not direct judgments of quality that might be more likely to be biased. The substantial variation in the responses
teachers and district administrators provided to these behavioral items, as well as the consistency in teacher and
district administrator responses, tends to bolster our confidence in the validity of the data.

7-3
 CURRENT T EACHING P RACTICES IN M ATHEMATICS AND SCIENCE

The first analyses presented in this report focus on what we have learned from the baseline
wave of our longitudinal survey about teachers’ classroom practices in mathematics and science at
the elementary, middle, and high school levels. The data for these baseline analyses, which are
presented in Chapter 2, are from teachers in 30 schools, across 10 districts in five states. Teachers
reported on the content they taught and their pedagogical practices for the 1996-97 school year. In
addition, as part of our case studies, we conducted classroom observations of two teachers in each of
the 30 schools.

These data allow us to identify weaknesses in classroom practices that might appropriately be
addressed by Eisenhower-assisted professional development activities. One standard we used to
judge the quality of teacher classroom practices is the degree of alignment with the National
Assessment of Education Progress (NAEP). The content tested on the NAEP is one appropriate
standard against which to judge the quality of content offered in our nation's classrooms because the
NAEP is considered to be the nation's report card. The National Education Goals Panel uses the
NAEP to monitor progress against our national education goals. The NAEP is, of course, not without
its critics. Some would like the NAEP to be more aligned with the National Council of Teachers of
Mathematics (NCTM) mathematics standards and National Research Council (NRC) science content
standards. Others would like the NAEP to place a heavier emphasis upon what they see as more
basic skills.

To allow comparisons to the NAEP, we obtained all mathematics and science items for
NAEP assessments in grades 4, 8, and 12. We then conducted content analyses of these items using
the same descriptors of content that were used in the questionnaires completed by teachers. A useful
byproduct of this work is the description of the content tested on the NAEP mathematics and science
assessments. The two-dimensional content maps we prepared, focusing on the topics and student
performance goals tapped by the NAEP items, provide an easily-interpreted picture of the content
emphasized on the NAEP, as well as the content not tested.

Our findings on current teaching practice are consistent with other research, especially the
Third International Mathematics and Sciences Study (TIMSS). We find that teachers in our 30 case
study schools cover a broader range of content, in less depth than is reflected in the NAEP. Teachers
might be characterized as teaching for exposure rather than mastery. We find a disturbingly slow-
moving curriculum. Much of the content taught at one grade is repeated in the next. Like TIMSS,
we find teachers not covering key content areas on which student achievement has been shown to be
weak. For example, we find little teacher emphasis on geometry and measurement in mathematics.
In contrast, we find teachers emphasizing basic topics, such as number sense and calculation, to a
greater extent than is found on NAEP. The emphasis on basic topics is especially true in high-
poverty schools.

One positive finding is that teaching practices appear to be moving toward a better balance
across performance goals for students. In earlier surveys of teachers' practices, a greater emphasis
was found to be given to student memorization of facts and mastery of low-level algorithmic skills,
while less emphasis was given to students' ability to apply their knowledge and to communicate with
others (Porter et al., 1993). Our longitudinal study baseline data show a much more balanced
emphasis in instruction across these varying performance goals.

7-4
 While our survey findings concerning content are supported in part by our observations of
teachers, they also occasionally are challenged. When our observations reveal differences between
what we saw and what teachers report, the differences tend to be in the direction of teachers’ over-
reporting solving novel problems and applying mathematics and science to real-world situations.
Still, our observations are largely supportive of the descriptions of classroom practices provided by
teachers in response to the survey questions. As reported by Cohen (1990), teachers appear to be
getting the message from national professional content standards that better balance needs to occur
across performance goals, and they are trying to move in that direction. In at least some cases,
however, their practice is not yet there.

Our findings about teacher classroom practices are important in two respects. First, the
inadequacies identified in current instruction suggest important targets for Eisenhower and other
professional development. Second, the weaknesses in instruction we have identified represent areas
in which we hope to track improvement over time, using our longitudinal study, and to relate
whatever improvement is observed to characteristics of professional development experienced by the
teachers.

TEACHERS' EXPERIENCES IN EISENHOWER-ASSISTED A CTIVITIES

We examined teachers' experiences in Eisenhower-assisted activities in three ways. First, we

assessed the effectiveness of Eisenhower-assisted activities by asking teachers to report on the extent
to which participation enhanced their knowledge and skills. Next, we examined the quality of
Eisenhower-assisted activities, be assessing the extent to which supported activities share features of
quality identified in the literature on professional development. Finally, we used our data to assess
the strength of the relationship between features of the activities in which teachers participated and
teachers' self-reported outcomes.

Effectiveness of Eisenhower-Assisted Professional Development Activities

To measure the effectiveness of Eisenhower-assisted professional development activities, we
asked teachers to report on the extent to which participation enhanced their knowledge and skills in
six domains: (1) in-depth knowledge of mathematics/science; (2) curriculum; (3) instructional
methods; (4) approaches to assessment; (5) use of technology; and (6) approaches to diversity. 3

When asked directly, many teachers in SAHE-grantee activities and somewhat fewer teachers
in district activities report that participation in Eisenhower-supported professional development has
led to enhanced knowledge and skills and changes in their classroom teaching practice. For
enhanced knowledge and skills, we asked teachers questions about six domains, ranging from
enhanced knowledge of mathematics and science, to instructional methods, to approaches to
diversity. For example, 48 percent of teachers in district activities and 68 percent of teachers in
SAHE-grantee activities report enhanced in-depth knowledge of mathematics or science; and 63
percent of teachers in district activities and 79 percent of teachers in SAHE-grantee activities report
enhanced knowledge of instructional methods. Percentages are lower for use of technology and

3
The term "Eisenhower-assisted activities" reflects the fact that district Eisenhower funds can support some or all of
the cost of professional development activities.

7-5
approaches to diversity. Similarly, we asked about improvement in classroom practice, and the
percentages were parallel. Once again, there is considerable variability among activities in reported
change in practice, and teachers in SAHE-grantee activities report somewhat more change in practice
than do teachers in district activities. Overall, our data on knowledge and skills and change in
teaching practice suggest that teachers believe they are benefiting from their participation in
Eisenhower-assisted professional development.

A comparison of our data for SAHE-grantee activities with the results obtained for 34
exemplary summer institutes in mathematics and science, supported by the National Science
Foundation, the Department of Education, and other agencies, indicates that teachers participating in
SAHE-grantee activities report enhancement of knowledge and skills in mathematics and science
content roughly comparable to the results for the 34 exemplary activities.4 SAHE-grantee activities
meet benchmarks for reported teacher outcomes set by other exemplary professional development
programs. Teachers participating in district Eisenhower activities, however, show weaker results
than do teachers in the 34 exemplary activities.

The Quality of Eisenhower-assisted Activities

Our analysis of the quality of Eisenhower-assisted professional development activities, which
appears in Chapter 3, is grounded in the recent literature on the characteristics of professional
development. Over the past decade, a considerable literature has emerged on professional
development, teacher learning, and teacher change (Corcoran, 1995; Darling-Hammond, 1995;
Hargreaves & Fullan, 1992; Hiebert, 1999; Lieberman, 1996; Little, 1993; Loucks-Horsley et al.,
1998; Richardson, 1994; Sparks & Loucks-Horsley 1989; Stiles, Loucks-Horsley, & Hewson, 1996).
The research literature contains a mix of large- and small-scale studies, including intensive case
studies of classroom teaching, evaluations of programs designed to improve teaching and learning,
and surveys of teachers about their pre-service and in-service preparation and in-service professional
development experiences. In addition, there is a large literature describing "best practices" in
professional development, drawing on expert experiences.

Despite the size of the literature, however, relatively little systematic research has been
conducted on the effects of professional development on improvements in teaching or on student
outcomes, and very little has been conducted comparing the effects of alternative forms of
professional development. The research that has been conducted, however, along with the
experience of expert practitioners, does provide some preliminary guidance about the characteristics
of high-quality professional development (See Loucks-Horsley, 1998.) In particular, several recent
studies suggest that professional development that focuses on specific mathematics and science
content and the ways students learn such content is especially helpful (Cohen and Hill, 1998;
Kennedy, 1998). To measure the quality of Eisenhower-assisted activities, we integrated and
operationalized the ideas in the literature on "best practices" in professional development.

4
See Carey, N., & Frechtling, J. (1997, March). Best practice in action: Follow-up survey on teacher enhancement
programs. Arlington, VA: National Science Foundation. Carey and Frechtling indicate that 44 percent of
participants in outstanding teacher development programs reported that the programs enhanced their knowledge and
understanding of science content to “a great extent” (value of 5 on their 5-point scale). If we isolate the percentage
of participants in SAHE-grantee activities who reported that the activity enhanced their mathematics or science
knowledge “to a great extent” (value of 5 on the 5-point scale), the percentage is 41 percent. The comparable
percent for district activities is 24 percent.

7-6
 As we reported in Chapter 3, we focus on three “structural features," or characteristics of the
structure of a professional development activity. These structural features include:

♦ the form or organization of the activity––that is, whether the activity is organized as a
reform type, such as a study group, teacher network, mentoring relationship, committee
or task force, internship, individual research project, or teacher research center, in
contrast to a traditional workshop or conference;

♦ the duration of the activity, including the total number of contact hours that participants
are expected to spend in the activity, as well as the span of time over which the activity
takes place; and

♦ the degree to which the activity emphasizes the collective participation of groups of
teachers from the same school, department, or grade level, as opposed to the participation
of individual teachers from many schools.

In addition to these structural features, we focus on three “core features” or characteristics of

the substance of the professional development experience itself:

♦ the degree to which the activity has a content focus––that is, the degree to which the
activity is focused on improving and deepening teachers’ content knowledge in
mathematics and science;

♦ the extent to which the activity offers opportunities for active learning -- that is,
opportunities for teachers to become actively engaged in the meaningful analysis of
teaching and learning, for example, by reviewing student work or obtaining feedback on
their teaching; and

♦ the degree to which the activity promotes coherence in teachers’ professional

development, by incorporating experiences that are consistent with teachers’ goals,
aligned with state standards and assessments, and encouraging of continuing professional
communication among teachers.

We used these structural and core features to judge the extent to which the Eisenhower
program supports high-quality professional development. We found that most Eisenhower-assisted
activities are traditional in form, such as workshops, courses, or conferences; and relatively few
Eisenhower-assisted activities are reform types of activities, such as study groups, networks, or
mentoring relationships. About 79 percent of teachers in district activities are in traditional types of
activities— primarily workshops and conferences. About 76 percent of teachers in SAHE-grantee
activities are also in traditional activities—primarily workshops and college courses. In contrast,
approximately 22 percent of teachers in district activities and 26 percent in SAHE-grantee activities
are in reform activities, including teacher networks, study groups, mentoring, committees and task
forces, internships, and resource centers.

On average, SAHE-grantee activities are of longer duration than district activities. The
average amount of time teachers in district activities report spending in Eisenhower-assisted
activities is 25 hours, compared to 51 hours for teachers in SAHE-grantee activities. However, while
district activities are shorter than SAHE-grantee activities, the average length of district activities, in

7-7
hours, has approximately doubled since the last evaluation was conducted in 1988-89. 5 In addition, a
higher percentage of SAHE-grantee activities than district activities span an extended period of time.
For example, 46 percent of teachers in SAHE-grantee activities are in activities lasting at least six
months, including 2 percent lasting more than one year, 20 percent lasting 10-12 months, and 24
percent lasting 6-9 months. Twenty percent of teachers in district activities are in activities lasting at
least six months.

On the third structural feature, collective participation, districts outperform SAHE grantees.
Although relatively few district or SAHE-grantee activities emphasize the collective participation of
teachers from the same department, grade level, or school, district activities give more emphasis to
collective participation than do SAHE-grantee activities. Twenty percent of teachers in district
Eisenhower-assisted activities report participating with other teachers in their department or grade
level, compared to 7 percent of teachers in SAHE-grantee activities; and 19 percent of teachers in
district activities report participating with all teachers in their school, compared to 11 percent of
teachers in SAHE-grantee activities.

On the last three dimensions of quality—content focus, active learning, and coherence—we
find that SAHE-grantee professional development is more likely to place an emphasis on these
features than district professional development.

Two thirds of teachers participating in SAHE-supported Eisenhower activities participate in

activities that place a major emphasis on content, compared to fifty-one percent of teachers in district
Eisenhower-assisted activities. Similarly, the percentage of teachers reporting a major emphasis on
each of 18 separate indicators of active learning is higher for SAHE grantees than for districts, but
relatively few teachers in either district or SAHE-grantee activities report some elements of active
learning. For example, only 5 percent of teachers in district activities and 16 percent of teachers in
SAHE-grantee activities report that their teaching was observed by the activity leader.

Teachers in both district and SAHE-grantee activities report that the activities have elements
that promote coherence with other aspects of their professional experiences. For example, more than
three quarters of teachers in both district and SAHE-grantee activities report that their activities are
aligned with state and district standards. Somewhat more teachers in SAHE-grantee than district
activities, however, report that their Eisenhower activities built on prior professional development
(39 percent compared to 31 percent) or were followed up with later activities (70 percent compared
to 53 percent).

Our main finding is that there is enormous variability in quality among Eisenhower-assisted
activities. This is not surprising, given that the Eisenhower program operates as a funding stream
within broad guidelines. A great deal of discretion is left to districts and SAHE grantees to decide on
the nature of the professional development that they will provide, though SAHE grantees have to
design programs within the constraints of the SAHE's competition. The variability in quality
indicates that some districts are providing the same kinds of high-quality professional development
activities that SAHE grantees provide. For example, although many district activities are short, 2
percent of teachers in district activities are in activities spanning more than one year, 8 percent are in
activities lasting 10-12 months, and 10 percent are in activities lasting 6-9 months. That some

5
The 1988-99 evaluation collected data on duration from districts rather than teachers, so a comparison of results
from the 1988-89 and the current evaluation should be interpreted as providing an indication of the general
magnitude of the change rather than a precise numerical estimate (see Knapp et al., 1991).

7-8
districts provide activities of extended duration, with collective participation, a major focus on
content knowledge, a major emphasis on active learning, and coherence with teachers' other
experiences, represents an “existence proof” that it is possible for districts to provide such activities.

We do not know, however, whether all districts possess the organizational capacity and
staffing to provide quality professional development. Districts lacking capacity may well be able to
purchase high-quality professional development (e.g., from institutions of higher education) if they
are aware of and insist upon the dimensions of quality we have identified.

Finally, the average differences in quality we observe between district and SAHE-grantee
activities are associated with parallel differences in cost. Thus, we conclude that high-quality
professional development is more expensive than lower-quality professional development. SAHE
grantees spend over twice as much per teacher participant as do districts. We estimated that SAHE
grantees spend about $512 per participation, in comparison to $185 per participation for districts. 6
High-quality professional development is more expensive.

Why the SAHE-grantee activities are of higher quality, on average, than the district activities
is not completely clear. One possible explanation is that districts receive a formula allotment while
the SAHE grantees must compete for their award. Both the competition and the SAHE guidelines for
the competition may push in directions of higher quality. 7 In addition, IHE/NPOs may, on average,
possess more state-of-the-art knowledge about the attributes of high-quality professional
development than districts do, and they may have greater capacity to deliver professional
development with these attributes. Finally, as noted, high-quality professional development is
considerably more expensive per participation. Undoubtedly, districts feel a responsibility to provide
professional development to all of their teachers. This may push them in the direction of professional
development with lower costs per participation.

Features of Professional Development that Promote Improvements

in Teaching
We used our national probability sample of teachers who participated in Eisenhower-assisted
professional development to estimate the strength of the relationships among features of the
professional development and self-reported teacher outcomes—enhanced knowledge and skills and
changed teaching practice. Our analyses, reported in Chapter 3 (Exhibit 3.21), estimate standardized
path coefficients while controlling for the subject area of the professional development experience
(mathematics or science); school poverty, percent minority, and grade level; and teacher gender,
certification, and years of experience.

We find that activities with more positive structural features tend to provide professional
development experiences with more positive core features, which in turn tend to produce more
positive teacher outcomes. In particular, activities of longer duration, both in time span and in
contact hours, tend to place more emphasis on content than shorter activities, provide more
opportunities for active learning, and provide more coherent professional development. Activities
with more collective participation also tend to provide more opportunities for active learning and

6
A “participation” is a teacher participant in an Eisenhower-assisted activity. Teachers who participate in more than
one activity are counted separately for each activity in which they participate. The dollar per participation figure for
districts includes federal Eisenhower dollars only, and does not count the 33 percent matching requirement.
7
We were not able to conduct a systematic analysis of SAHE competitions.

7-9
offer more coherent professional development. The three core features, in turn, all have independent
effects on teachers’ reports of enhanced knowledge and skills. Teachers who participate in activities
that place a major emphasis on mathematics and science content, for example, are substantially more
likely to report enhanced knowledge and skills than teachers in activities that do not emphasize
content. (For a more complete description, see Chapter 3.)

Thus, our results expand the existing literature by providing empirical support for the
contention that professional development that emphasizes content knowledge, active learning, and
coherence leads to enhanced teacher knowledge and skill and change in teacher practice. In addition,
our results highlight the importance of professional development that engages teachers for an
extended number of contact hours, and that spans several months, as opposed to being concentrated
in a few days or weeks.

Our results also support the literature in indicating that reform types of professional
development tend to produce more positive outcomes than traditional types, but the effect is smaller
than we had anticipated. In our analyses, we find effects of the distinction between traditional and
reform activities, and they are in the predicted direction, but they are generally not direct effects on
teacher outcomes. Rather, the effect of reform versus traditional professional development activities
operates indirectly through the structural and core features identified above. That is, reform activities
tend to produce better outcomes primarily because they tend to be of longer duration. Traditional
and reform activities of the same duration tend to have the same effects on reported outcomes.
Further, we find a large amount of variability within both traditional and reform-type activities.
Traditional activities sometimes have positive characteristics, such as opportunities for active
learning, and reform activities sometimes have negative characteristics, such as limited duration.

Further, our analyses show that the difference between districts and SAHE grantees, in terms
of enhanced teacher knowledge and skill and greater change in teacher practice, is explained almost
entirely by the SAHE grantees' placing a greater emphasis upon duration, subject-matter content,
active learning, and coherence. In short, if districts placed a greater emphasis upon these
characteristics, we would hypothesize that teachers would report their knowledge and skill enhanced
to the same extent as teachers in SAHE-grantee activities, and that they would be just as likely to
report changing their teaching practice.

DISTRICT M ANAGEMENT AND OPERATION OF

EISENHOWER-ASSISTED ACTIVITIES

We now turn from describing teachers’ experiences in Eisenhower-assisted activities to a

description of the ways that districts manage and operate their Eisenhower programs. Each district
receiving Eisenhower funds generally uses the funds to support a collection of professional
development activities. The collection of professional development activities that a district supports
with Eisenhower funds can be viewed in its entirety as its “portfolio” of Eisenhower-assisted
professional development activities. In our analysis, we examine the factors that influence the
quality of the portfolios of professional development activities that districts offer and the extent to
which districts engage in efforts to target teachers of high-need students.

7-10
 The measures we use to characterize the quality of a district’s portfolio of Eisenhower-
assisted activities are: (1) the percent of participations in reform types of activities, (2) the average
span of time of activities (i.e., number of days, weeks, or months), (3) the number of opportunities
for active learning in in-district workshops and institutes, and (4) the degree of collective
participation in in-district workshops and institutes.8

In assessing district management and operations, we focus on the role of several provisions
emphasized in the legislation: (1) the coordination (co-funding) of Eisenhower-assisted activities
with other sources of funding for professional development; (2) the alignment of Eisenhower-assisted
activities with state and district standards and assessments; (3) the participation of teachers and
school-level staff in planning Eisenhower-assisted activities; and (4) the use of a process of
continuous improvement, including monitoring progress against measurable objectives and
performance indicators.

The Relationship of District Management and Features of Professional

Development
In Chapter 5, we report the results of a path analysis of the relationship between these
provisions of the legislation and the characteristics of quality professional development that districts
provide. The path analysis is based on a national probability sample. Our path analysis of district
professional development portfolios (Exhibit 5.11) controlled for district size, poverty level,
consortium, and cluster status. We began our analysis of districts by examining co-funding (one
aspect of coordination) and alignment. Consistent with the literature on systemic reform (e.g.,
Fuhrman, 1993; Smith & O'Day, 1991), we find that co-funding Eisenhower professional
development activities with other professional development initiatives has important direct and
indirect effects on the quality of the Eisenhower-assisted professional development offered by
districts. For example, districts that engage in more co-funding of Eisenhower activities with other
programs tend to support a greater proportion of reform activities than districts that engage in less co-
funding, and they tend to provide more opportunities for collective participation. In addition,
districts that engage in more co-funding tend to engage in more extensive continuous improvement
efforts, and they tend to involve teachers more widely in planning, both of which are related to
increased opportunities for active learning. Alignment of professional development with state and
district standards and assessments also has positive effects. Districts that align professional
development with standards and assessments are more likely to offer reform types of activities. In
addition, these districts are more likely than others to engage in continuous improvement, which is
related to increased opportunities for active learning. These results (along with others reported in
Chapter 5) lead us to conclude that the intentions of the program concerning strategies such as co-
funding, alignment, continuous improvement, and teacher involvement in planning are appropriate,
in the sense that they appear to lead to higher-quality professional development.

Thus, through analyses of our data, we find support for the appropriateness of the key
provisions of the Eisenhower legislation. Generally, these provisions tend to encourage districts to

8
The data from our national sample of teachers show that each of these dimensions is related, either indirectly or
directly, to improvements in teachers’ knowledge and skills and changes in teaching practice; thus, we consider each
of these dimensions as an indicator of high-quality professional development, whether it has a direct effect on
teacher outcomes, or operates indirectly (e.g., a reform approach affects teacher outcomes indirectly through its
effect on duration).

7-11
offer professional development with effective structural and core characteristics, as identified in the
literature and supported by our evaluation.

Variation Among Districts in Management and Operations

Given that co-funding, alignment, teacher participation in planning, continuous improvement,
and targeting are intended features of the program, and given that our evidence indicates that these
strategies generally lead toward higher-quality professional development, we ask to what extent each
is being implemented (Chapter 4).

Co-funding and working closely with other federally supported professional development
programs (where the programs are available) occur most often with professional development
programs that have a mathematics and/or science focus similar to that of the Eisenhower program.
Thus, working closely with other programs and co-funding is more likely between the Eisenhower
program and National Science Foundation programs than it is with other Department of Education
programs. For example, among teachers in districts with an NSF-funded Urban Systemic Initiative
(USI) that supports professional development, over 80 percent are in districts where the Eisenhower
project co-funds with the USI. Among teachers in districts with Title I, Part A funds, 50 percent are
in districts where the Eisenhower project co-funds with Title I.

Most districts report being engaged in efforts to align Eisenhower-assisted professional

development with standards and assessments. Alignment is more likely for state standards and
assessments than it is for district standards and assessments and more likely for mathematics than
science. In addition, Alignment is, however, more likely for standards than for assessments. This
may reflect the fact that the ESEA requirement of aligned assessments had not yet gone into effect;
alternatively, these results may reflect that districts first establish standards, and then seek to revise
their assessments to be aligned with the newly developed standards.

The program intends that teachers participate in planning Eisenhower-assisted activities. Our
results indicate that 99 percent of teachers are in districts that report involving teachers in planning
professional development. Thus, we conclude that this provision of the law is relatively well-
implemented. Still, our case-study data reveal that the Eisenhower legislation's 80/20 rule, calling
for at least 80 percent of funds to be used "in a manner that is determined by such teachers and staff"
and "to the extent practicable, takes place at the individual school site," is not well-understood and is
sometimes not even known to exist. We conclude that this provision of the law could be clarified
and given more emphasis in the information provided to projects from the Department of Education
and from the states.

Our data suggest that the use of performance indicators to guide the continuous improvement
process is not yet widespread in districts that receive Eisenhower funds. Fewer than one in five
teachers are in districts that collect data on performance indicators established to guide district
professional development efforts. One-third of the teachers are in districts that have developed such
indicators. The majority of teachers are in districts whose Eisenhower coordinators are not aware
that their state has performance indicators. Clearly, the development of state and district
performance indicators is a goal of the program that is not generally being implemented, at least not
yet. There are other ways in which districts show some commitment to continuous improvement.
Needs assessments through teacher surveys and informal conversations are common. Nearly every
district reports evaluating their professional development activities. These evaluations, however, are
frequently based on teacher satisfaction surveys or participation counts. It is less common to use

7-12
observations of teachers’ subsequent classroom practice to evaluate the effectiveness of Eisenhower
professional development.

Districts report a strong emphasis on recruiting teachers from high-poverty, low-achievement

schools, and slightly less emphasis on recruiting teachers from Title I schools. Despite these reported
efforts at targeting, and even though districts with larger numbers of high-poverty students receive
greater funding, teachers from high-poverty schools are only slightly more likely than others to
participate in Eisenhower-funded activities. Apparently, current targeting strategies have limited
effectiveness. Perhaps these results are explained, in part, by the fact that most participants in
Eisenhower-assisted activities are volunteers and teachers from high-poverty schools are no more or
less likely than other teachers to volunteer for professional development. It is possible that districts
do not have adequate strategies to shape the incentives and constraints that determine which teachers
volunteer.

Differences in Management and Operation of Eisenhower-assisted Activities

by District Poverty and Size
Throughout our analyses of district data, we tested to see where the management and
operation of Eisenhower-assisted professional development differ significantly according to the
district poverty level or the size of the district. All of our analyses simultaneously controlled for size
and poverty, so any effects are independent of one another. We also tested for the interaction
between these two variables. Of the two variables, district size is much more frequently related to
the nature of the Eisenhower professional development provided than is poverty. Generally, larger
districts are more likely to provide high-quality professional development than are smaller districts.
This is true for features of the activities provided—such as active learning, collective participation,
and the span of time over which the activities extend—as well as district management strategies,
including alignment with standards and assessments, frequency of co-funded projects, and a
commitment to continuous improvement. Also, when several small districts form consortia to deliver
Eisenhower professional development, the quality is higher in ways that parallel that for large
districts. We speculate that larger districts and consortia have a better infrastructure and greater
capacity to provide high-quality professional development. Higher-poverty districts also are more
likely to co-fund Eisenhower professional development with other professional development, and
they are more likely to be committed to continuous improvement. Large districts and high-poverty
districts have a greater variety of funding sources in addition to Eisenhower, increasing opportunities
for co-funding, and perhaps creating a complexity that demands a commitment to continuous
improvement.

SAHE-GRANTEE MANAGEMENT AND O PERATION OF

EISENHOWER-ASSITED A CTIVITIES

SAHE grantees are subject to the same provisions for alignment, continuous improvement,
and targeting that districts must follow. However, SAHE grantees also are subject to some additional
requirements or guidelines. In particular, the 1994 reauthorization emphasizes the importance of
coordination between SAHE grantees and districts in planning and providing professional
development. Each SAHE grantee is required to enter into an agreement with one or more districts

7-13
for the provision of professional development. Further, SAHE-grantee projects are shaped by the
priorities and guidelines that the SAHE sets in structuring its competition for the Eisenhower
awards.9

The Relationship of SAHE-grantee Management to Features of Professional

Development
In Chapter 6, we report the results of our analyses of the relationship between intended
program strategies, such as alignment, coordination, and continuous improvement, and the quality of
the professional development that SAHE grantees provide. The path analysis is based on a national
probability sample.

As reported in Chapter 6, Exhibit 6.26, coordination with districts (e.g., use of feedback
mechanisms, support and extension of district activities, co-funding and district involvement in
planning, implementing and monitoring) is positively associated with the quality of the professional
development provided. For example, SAHE grantees that coordinate more extensively with districts
provide professional development with a larger number of contact hours. Coordination also has a
positive effect on the use of strategies for continuous improvement which, in turn, increases the
active learning. In addition, coordination has an indirect effect upon targeting, through its effect
upon continuous improvement. These results for coordination parallel those found for the district
sample.

Variation Among SAHE Grantees in Management and Operations

We have discussed the legislative requirements for alignment, coordination, co-funding,
continuous improvement, and targeting, and their role in shaping effective professional development
in SAHE-grantee projects; we now discuss the extent to which SAHE-grantee projects are
implementing these management strategies (Chapter 6).

Nearly all SAHE-grantee projects are aligned with state standards, and most are aligned with
state assessments and district standards, but less than half are aligned with district assessments.
SAHE grantees work closely with districts in several ways, such as communicating with district staff
and using district needs assessments, but report low levels of other key components of coordination,
such as co-funding and working with the Eisenhower coordinator. Also, SAHE-grantee projects tend
to be much more affected by state than district indicators. However, SAHE grantees report
moderately high levels of engagement in most other continuous improvement efforts, such as
conducting needs assessments and evaluations. These findings suggest that, although the average
quality of SAHE-grantee activities is relatively high, it could be further improved by strengthening
the coordination between SAHE grantees and districts and giving more emphasis to the use of district
indicators.

In addition, although most SAHE-grantee project directors report targeting teachers of low
achieving, high-poverty students and teachers in Title I schools, these teachers are much less likely
than others to participate in SAHE-grantee activities.

9
Relative to SEAs, SAHEs have a smaller number of grantees, and thus may be able to monitor their grantees’
projects to help ensure the implementation of quality activities. However, we did not examine the SAHE’s
monitoring role.

7-14
Differences in SAHE-grantee Management and Operation by Institution Type
and Departmental Affiliation
In our analysis of SAHE grantees, we compared activities housed in education,
mathematics/science, and other departments, and research/doctoral and other types of institutions of
higher education. Our results indicate that, on average, projects housed in education departments are
stronger in important ways than are projects in mathematics/science departments. Education
departments sponsor professional development activities that last more than twice the number of
contact hours and span a longer time period. They engage in more types of coordination with the
districts from which they draw their teachers, and they place more emphasis on continuous
improvement than do mathematics/science departments. Education department projects are
especially strong when housed in research/doctoral universities, where they are more likely to put a
high focus on content and provide numerous opportunities for active learning.

These findings of the superiority of professional development provided through education

departments, in contrast with mathematics/science departments, may be due to the fact that education
departments have greater expertise in student and teacher learning of subject matter, as well as
experience in coordinating with practitioners. Faculty in these departments are the main contributors
to the literature on effective professional development in mathematics and science. Projects in
mathematics/science departments, on the other hand, have a strong content focus. Others have noted
that, in the training of teachers, quality might be enhanced by better collaboration between educators
and scientists. Apparently, the same can be said for continuing teacher education in the form of
Eisenhower professional development. Each type of department brings unique strengths to designing
and delivering professional development that could be complementary in meaningful collaboration.

LESSONS FOR P ROFESSIONAL D EVELOPMENT

These findings of the National Evaluation of the Eisenhower Professional Development

Program have a number of lessons for professional development generally and for future Eisenhower
legislation and program operations specifically.

First, the program should seek ways to encourage the use of all features of professional
development that are related to teacher knowledge and skills, and change in practice. The program
should continue emphasizing "sustained and intensive" professional development activities. More
specifically, this evaluation suggests that professional development should focus on deepening
teachers' content knowledge and knowledge of how students learn particular content, on providing
opportunities for active learning, and on encouraging coherence in teachers' professional
development experiences. Eisenhower professional development should pursue these goals using
activities of greater duration and collective participation. Clearly, there is considerable room to
increase the frequency of these attributes of effective professional development in both district and
SAHE-grantee activities. While reform professional development is more effective than traditional
professional development, the advantage of reform activities is explained primarily by greater
duration.

7-15
 Second, the program should develop additional approaches to targeting teachers in high-
poverty schools. While increasing the participation of teachers in high-poverty schools is an
important goal of the legislation, current approaches to targeting these teachers are insufficient to
achieve this goal.

Third, federal indicators might be improved by specifying more clearly the dimensions of
high-quality professional development, focusing on annual improvements rather than absolute levels,
and setting higher standards. The federal indicators are not as directive as they might be in
specifying the dimensions of high-quality professional development. Our analyses suggest that the
federal indicators should guide local practice toward greater emphasis upon content knowledge,
active learning, and coherence, and that districts should pursue these attributes of professional
development through offering activities of greater duration and with more emphasis upon collective
participation. In addition, several of the federal indicators are stated in terms of absolute levels to be
achieved by a specified date (e.g., “at least 50 percent of teachers”). It would be better to have
indicators stated in terms of the amount of improvement needed annually, until an acceptably high
level of performance is reached. Finally, the federal indicators set some standards that are too low.
For example, in targeting teachers from high-poverty schools, the standard is simply to exceed the
national average. Furthermore, in the targeting indicator, participation in Eisenhower professional
development is stated as a dichotomous variable (participate or not), while clearly teachers
participate in Eisenhower-supported professional development in varying amounts and types.

Fourth, the program should pay attention to building district capacity to foster continuous
improvement efforts. If using data to make decisions is to be a serious endeavor, then districts may
need assistance in determining the types of data that would be useful and in interpreting them. We
believe local evaluation can be done in ways much stronger than current practice, which relies
heavily upon participation counts and teacher satisfaction surveys. We recommend that local
evaluation of professional development focus on instruction, by assessing the degree to which the
professional development is characterized by well-implemented attributes of high-quality
professional development: a focus on content, active learning, and coherence, delivered with
sufficient duration and collective participation. Such evaluations might be based on a variety of
sources of data, including well-designed surveys of participating teachers, and the observation of
teachers to assess the extent to which they have made appropriate changes in their instruction.
Although the assessment of student achievement might be used as part of a balanced evaluation of
professional development, we suggest that local evaluation efforts should not focus on assessing the
effects of professional development on student achievement. There are simply too many intervening
variables between professional development experiences and subsequent gains in student
achievement to make such studies feasible at the local level. Further, collecting and analyzing high-
quality data on gains in student achievement is expensive and requires technical skills that may not
be present at the local level. Clearly, there needs to be more research that looks at the relationships
between features of professional development and gains in student achievement, but this is better
done as a part of well-designed major research studies.

Fifth, more information is needed on the characteristics and conditions that give some
districts the capacity to provide high-quality professional development. We have speculated that
districts could provide the same types of professional development as do SAHE grantees. There are
existence proofs of districts doing exactly that. What we do not know from our analyses is the
percentage of districts that have the capacity to provide such professional development. Our analyses

7-16
indicate that larger districts have greater capacity and, to some extent, so do high-poverty districts.
This larger capacity may be explained, in part, by their larger district staff and, in part, by their
greater Eisenhower funds. Consortia that tie together several small districts into one unit for
providing Eisenhower professional development also seem, on average, more effective than small
districts. The issue of district capacity is one that should be closely monitored in future research.

Sixth, districts could increase the quality of the professional development they provide by
focusing their Eisenhower money on a small number of teachers, rather than spreading it across a
large number of teachers. Not surprisingly, high-quality professional development costs more per
participant than does lower quality professional development. Districts may feel a greater
responsibility to reach a large number of teachers than do SAHE grantees, and this is reflected in the
cost per participant. The question is, should districts continue to spread the money from the
Eisenhower program across as many teachers as possible? Or, should they focus the money on a
small number of teachers, so that they can provide higher quality, more influential professional
development? Our results suggest the money should be focused. This recommendation also interacts
with the finding about targeting. More effective targeting might, at the same time, provide a
rationale for more focused expenditures.

Seventh, one reason that SAHE-grantee professional development is, on average, of higher
quality than district professional development may be that SAHE grantees have to compete for funds.
We do not know as much about these SAHE-sponsored competitions as we would like. We find that
the majority of SAHE grantees have been receiving Eisenhower support for several years. Still, in all
cases, IHE/NPOs interested in Eisenhower support must develop a proposal and have it judged
worthy of funding. In contrast, districts receive money from the state educational agency on a
formula basis, with no competition. Perhaps having districts compete for funds would push them
toward higher quality professional development.

Eighth, there is considerable evidence that, on average, education departments in institutions

of higher education provide higher quality professional development than do mathematics and
science departments. The one exception is that mathematics/science departments in non-
research/doctoral institutions focus more on content knowledge than do education departments. We
do not have empirical data on inter-departmental collaboration, but, in our case studies, we did not
see much evidence of these two types of departments collaborating and combining their expertise to
provide high-quality professional development. Lack of collaboration between education and
mathematics/science departments is a well-recognized problem in pre-service teacher education and
may be an issue for in-service education, as well.

Ninth, SAHE-grantee projects should be engaging in higher levels of coordination with

districts, as called for in the legislation. The evaluation shows that such coordination is related to the
provision of high-quality professional development. Therefore, more attention might be paid to
supporting and developing opportunities for SAHE grantees to coordinate and work with districts in
mutually beneficial ways—ways that allow grantees to exercise their expertise in developing
professional development projects while benefiting from district expertise in serving the needs of
their teachers and students.

7-17
 Finally, the evaluation supports the importance of programs that fund professional
development activities within specified subject areas. Over the past 15 years, the Eisenhower
Professional Development Program has provided continuous support for professional development
activities for mathematics and science. This evaluation highlights the importance of the content
focus of professional development activities and the role that the Eisenhower program has played in
building capacity in these two subjects in school districts. Generic professional development that
focuses on teaching techniques without a content focus does not appear to be effective. If the
Congress is considering expanding the program, it should consider creating analogous programs in
other academic subject areas, rather than eliminating the content focus on mathematics and science.

7-18