Testimony of

The Hon. Kathy Patterson

Before the

Council of the District of Columbia

Hearing on Measuring What Matters:

March 19, 2021

Virtual Platform
The John A. Wilson Building
1350 Pennsylvania Avenue N.W.
Washington, DC 20004

1
Good afternoon, Mr. Chairman and members of the Committee of the Whole. I am Kathy Patterson,
D.C. Auditor, and I am joined by Erin Roth, director of education research for the Office of the D.C.
Auditor. Our consultants on this project, by Dr. Brandan Keaveny and Dorothyjean Cratty, of
DataEthics, are also available for questions.

We are pleased to share the results of the recently published education data audit mandated by the
Council two years ago in the District of Columbia Education Research Practice Partnership
Establishment and Audit Act of 2018. We based the work not only on the specific requirement in the
act, but the more expansive provisions of the legislation as introduced, and testimony before the
Council on the kind of information parents, teachers, principals and other stakeholders wish to see to
better guide their efforts to improve education achievement.

The recently finished audit, Measuring What Matters: More and Better Data Needed to Improve D.C.
Public Schools, finds that the District's state education agency, the Office of the State Superintendent
of Education (OSSE) has failed to build necessary data systems and collect the data needed to
adequately support students and schools in the District. More specifically, schools, teachers, and
families are left without important information needed to best serve our students’ needs, growth,
mobility, and supports over time—information like accurate attendance and absenteeism data,
growth on standardized tests over time, and tracking of individual students through the years.
Importantly, the burden of this failure is disproportionately borne by our most vulnerable students
and schools.

Our topline findings:

• The District of Columbia does not have a Statewide Longitudinal Data System (SLDS) despite
$10 million in federal funding and $25 million in District funding. The U.S. Department of
Education defines such as system as one that “collects and maintains detailed, high quality,
student and staff-level data that are linked across entities and over time, providing a
complete academic and performance history for each student.”

• Without such a system the District has limited ability to effectively assess and address
learning loss post-COVID.

• Similarly, the District has limited ability to accurately assess and address racial equity issues
across our public schools.

• We documented issues with the validity of data in OSSE publications including school report
cards.

• We documented the risk of compliance challenges with federal data submissions.

• The District’s State Education Agency has not fulfilled its mandates as a state agency by
choosing not to collect critical data from all schools to fulfill the requirements of an SLDS.

2
Our state education agency does not serve students directly. Rather, the focus of their responsibility
is to monitor the educational progress of all students and schools using a proxy and that proxy is
data. To the extent that needed data are non-existent, partial, flawed, not reported, or not used, our
state education agency is not meeting its fundamental responsibility to know and understand our
students and schools. This failure not only puts at risk basic monitoring of educational services but
also compliance reporting like student discipline and safety, District-wide educational improvement
work, and the potential for future federal grants.

We realize that the average reader or listener and even public officials may be challenged to fully
understand concepts like “longitudinal data” – it means information that combines multiple pieces of
information on individual students over time to see patterns and trends. The testimony that follows
separates the issues into these buckets: the data we do not collect, data we collect only in part, data
we collect but do not share, how we could better use the data we do collect today, what our
education consumers could be receiving, and what an aspirational data system might look like. We
end with a section on fact checking to correct the record on the findings based on erroneous
statements in the public record over the last week.

What we don’t collect

Right now, in the District, we do not collect three categories of critical information:

• student courses, credits, and grades

The first of these three are fundamental for understanding and tracking educational progress in the
District. We cannot assess student progress, equity in access, or develop evidence-based
improvement strategies without student course data. Student courses, credits, and grades are
needed to develop robust early warning systems that would help us identify students at risk of
academic failure and disengagement. Importantly, the courses a student takes and the grades they
receive have been shown to be better predictors of high school graduation and postsecondary
success than standardized test scores. But we do not collect them.

Teacher-student links show who is teaching whom. In D.C., we do not know who is teaching whom.
By not tracking these links we cannot assess how teachers are distributed among students and across
schools, Wards, or courses. The Council mandated that OSSE create a plan to implement a school
climate survey in every school serving 6th grade to 12th grade students by this school year. Instead
OSSE delivered a plan recommending that each LEA choose its own climate survey, again leaving the
District without comparable information on student engagement and supports across all students
and all schools. 1 And what metrics are states relying on heavily right now during the pandemic?

1
https://lims.dccouncil.us/downloads/LIMS/44147/Introduction/RC23-0153-Introduction.pdf

3
Precisely these:

• Student courses, credits, or grades for all 6th through 12th graders
• Student supports and other "whole child" data, like school climate survey data for students,
parents, or teachers, or other school-based interventions and supports like response to
intervention (RTI), positive behavioral interventions and supports (PBIS), and more.

The absence of this data was underscored in testimony before this Committee last week.
Councilmember Janeese Lewis George asked Acting State Superintendent Shana Young how many of
the District’s 9th graders are on track to graduate and Ms. Young said she could not provide the
answer. She cited the lack of standardized test scores as a main reason. Had the District been
collecting course, credit, grades, climate surveys or other student support data, this critical
information on 9th grader progress would have been available.

What we collect only in part

Most of what OSSE does collect is collected only in part, which means that the District is severely
limited when it comes to meaningful data use. Like driving a car with faulty parts in key places like
the steering column, wheels, and brakes, and not being able to reliably turn or stop, to actually use
District-wide education data brings problems and risks due to data collected only in part. Some of
these data are not collected District-wide, meaning we are missing certain students, schools, or
teachers, while others are missing large swaths of data for most students, and other key data
elements are unusable due to inconsistent definitions both over time and within the same year but
with differences by sector, i.e. DCPS versus public charter schools.

These elements include:

• Student enrollment (entries and exits from specific schools).

• Student enrollment by program type.

• Student attendance.

• Student discipline.

• Teacher role, demographics, qualifications, certifications, retention, and salary.

• Robust early childhood, career and technical, and adult education data.

• Postsecondary, workforce, and other agency data linked to education.

4
Figure1 summarizes the status in the District of each of the standard data element groups.

Figure 1: Current District SLDS Standard Data Element Collections and System Capabilities

What we collect but don’t share

In addition, there are data that we do collect but don't share publicly, including data collected on
contracts or grants paid with District tax dollars. For example, as noted the teacher data we collect is
incomplete, and we nonetheless report these data to the federal government while not reporting it
locally. We report multiple data points across many areas to the Office of Civil Rights for the Civil

5
Rights Data Collection (CRDC) and also fail to report this information locally. The Every Student
Succeeds Act specifically asks that states share locally the CRDC data reported to the federal
government.2 Finally, although we pay for a contract with the National Student Clearinghouse which
reliably tracks the post-secondary enrollment, persistence, and graduation rates for more than 90%
of all postsecondary students across the country, we routinely fail to report these data locally. After
ODCA began this audit, OSSE began last year to report on postsecondary enrollment but is still not
reporting on persistence and graduation.

What we could do with what we have but don’t

Although our data is limited and problematic, the data collected represent students that deserve and
require focus and monitoring at the statewide District level. We can immediately begin increased
data use and reporting, ideally driven by repeated stakeholder demand for analysis in key areas like
teacher turnover, demographics, student learning, and mobility. Actual use of data is the best known
and most effective strategy for increasing quality and sustainability. As data are used and made
accessible, more education stakeholders have opportunities to engage and communicate needed
adjustments to increase reliability and validity. Importantly, as data are used, more District residents,
practitioners, and policymakers become invested in building more data capacity.

For example, even though we do not have needed course, credit, and grade information, we can
build a limited, initial early warning system right now that would return critical attendance, behavior,
and achievement data to educators. We could create high school feedback reports, the most in-
demand state level reporting by LEAs across the country. These reports show levels of incoming
achievement, attendance, and behavior for each high school as well as include key metrics on access
and outcomes within each high school. We could do this tomorrow while we are also investing in a
long-term pathway to better data and systems.

What D.C. education consumers don’t have

Right now, District education stakeholders, practitioners, and policymakers do not have access to the
same level of reporting and analysis that is available to their counterparts in other states. And we are
not speaking of elite, high income school districts—this data use is supported with federal grant
money and collaborative state work across the country. Accessing data and using interactive tools in
Washington, Kentucky, Arkansas, Montana, Texas, Rhode Island, Wisconsin, Illinois, Hawaii, Alaska,
Tennessee, and Maryland, to name just a few, yields analysis not available here. These include
longitudinal data on teachers, schools, and students interwoven to show educational opportunities
and contexts. Many of these states also link to early childhood, postsecondary, and workforce
outcomes in addition to longitudinal use of k-12 data.

Figure 2 on the following page shows a chart from Arkansas’s teacher data collection showing the
breakdown of teacher experience by race-ethnicity, and this is just one of the many slices of data
available on Arkansas’s teachers.

2
https://www2.ed.gov/policy/elsec/leg/essa/rptcardpubliccomment3282019.pdf

6
Figure 2: Experience by Teacher Subpopulation, 2020

7
Figure 3 is an example of one of Washington’s data dashboards showing college enrollment,
persistence beyond the first year, what were the highest degrees the students attained, and their
median earnings up to 12 years after graduation.

Figure 3: Washington data dashboard

Finally, Figure 4 on the following page shows the kind of information a Montana teacher will have
this fall as she plans to address the learning loss of a new classroom of students including
coursework, grades, attendance, and behavior over time for each of his students. District teachers do
not have this information available to them.

8
Figure 4: Montana Student Level Report

9
Moving forward with P-20W+

The audit doesn’t simply show what we do not collect. It also shows that we can immediately start
using existing data better—to assess equity, meet basic stakeholder demands, and better support
students in need. District students are graduating and entering a competitive post-secondary and
workforce world and members of the Council do not have data that demonstrates how District
students are doing. Do they have equitable access to postsecondary enrollment, persistence, and
degree attainment? Are they prepared for the workforce? What are their earnings? We do know that
red flags aren’t being heeded. In 2017, the University of the District of Columbia shared with the
State Board of Education that 126 out of 128 newly enrolled students from DCPS high schools needed
remedial coursework. Your performance oversight records show that students have complained to
the Ombudsman that they cannot access the transcripts they need for work or college. We
recommend using the data we do collect to give you answers that are available now.

We have been talking about our need to complete the build-out of a Statewide Longitudinal Data
System, referring to p-12 education data and recommend that that be a clear priority for the District.
But it isn’t an end point. The best practices review undertaken as part of the audit yielded a wealth of
information on what innovative state governments are doing beyond linking what happens in the
elementary grades with middle school success or preparation for high school. Forward looking states
have moved ahead with P-20W+ data systems that bring specific data securely from across systems,
linking early childhood, K–12, postsecondary, workforce, and other sectors to identify how well our
education system functions and where the gaps are.

Such systems in Kentucky and Ohio, for example, provide the in-state and out-of-state employment
outcomes for postsecondary graduates from both states and enable you to see the employment
outcomes associated with specific institutions including majors declared and credentials earned.

The Kentucky Center for Statistics (KYSTATS) was created by legislation in 2012 and collects and links
data to evaluate education and workforce efforts in the Commonwealth. This includes developing
reports, responding to research requests, and providing statistical data about these efforts so
policymakers, practitioners, and the general public can make better informed decisions.

Two years ago the California legislature passed the Cradle-to-Career Data System Act to create a
statewide data infrastructure to help state agencies improve education and workforce policies and
programs, and support statewide research efforts. The law required a working group to get feedback
from researchers, policy experts, local education and community college leaders, advocates, and civil
rights groups. According to the Data Quality Campaign, this type of engagement is necessary to
secure stakeholder buy-in and ensure the recommendations represent a collaborative effort. The
working group has also prioritized transparency—all meetings have been open to the public, with
additional resources and materials available online.

10
They are attempting to launch multiple statewide tools, including something as detailed as an
electronic transcript exchange app to help high schools, colleges, and universities efficiently share
information and permit families to access the resources they need from anywhere in the state.

This is just a quick glance at strides other states are making to better utilize data capacity and
linkages to help drive policy and individual decision-making. We obviously can’t run until we’re
walking securely. We will not have a Statewide Longitudinal Data System until we have reached some
kind of consensus on what exists in the District today—what we do have and what we do not have.
Which brings me to the last section of our testimony.

Fact-checking

Mr. Chairman, the late Senator Daniel Patrick Moynihan had a frequently quoted phrase that while
we are all entitled to our opinions, we are not entitled to our own facts. In that spirit I would like to
conclude by correcting the record on several statements that have been made in recent weeks
related to the education data audit.

• “This report resurrected and relitigated old debates on the governance of our schools.”

This is wrong and had an audience of one, the Mayor. It sought to tie the audit’s well-documented
findings about OSSE not living up to its mandate to a quite separate discussion on whether mayoral
control of schools should be taken up again. Although the report recommends legislation it is the
plain fact that the clearest path to data collection and a true statewide longitudinal system is by
Executive directive and not through the more time-consuming enactment of legislation and certainly
not through any change in government structure. Columnist Jonetta Rose Barras quoted me as saying
that, “This audit is a perfect opportunity for the Mayor to use her authority over public education in
the District to direct her new state superintendent to finish the job of building a statewide data
system.”

The only way in which governance more generally is touched on in the report is by comparing what is
statutorily required of our state education agency with what has actually been accomplished and in
this we do fault OSSE for not yet completing the task of building a comprehensive data system.

• “OSSE lacks the authority to direct charter schools to submit data.”

This also is inaccurate and is somehow linked to a misunderstanding of the autonomy granted to the
District’s public charter schools. As one of the authors of the Council’s charter school authorizing
legislation, I can attest that the purpose of autonomy was to have “freedom to” not “freedom from.”
Freedom to experiment with new approaches to education like a longer school day or a residential
campus—not freedom from providing information on race and ethnicity of teachers or what math
course a school’s 8th graders are taking. The D.C. Code is explicit on OSSE’s authority to require data
submissions as and when OSSE chooses.

11
Specifically, D.C. Code §38-2609 states that,

“Upon the request of the State Superintendent, necessary data pertaining to students,
teachers, and school levels shall be submitted to the OSSE for the purpose of constructing,
updating, or maintaining the EDW system by:
(i) The University of the District of Columbia;
(ii) A public school;
(iii) A public charter school; or
(iv) An entity administering a publicly funded educational program.

(B) The requested data shall be submitted within a reasonable time, as determined by the
OSSE, following a request, and in a standardized format to be established by the OSSE.3

• “We have access to longitudinal, student-level data from across the education continuum,
from birth to adulthood, and can draw linkages across this continuum to analyze the
relationship between inputs and outcomes.”

This is a carefully worded statement that seemingly implies that we have a longitudinal system. And
yet our questions would be: if we have access, where is that access displayed? And for what
purpose? The fact is that ODCA and our contract partners audited every document published by the
Office of the State Superintendent of Education. We found no instance of longitudinal data use. Not
one.

I mentioned earlier the statement Ms. Young made last week when asked how many 9th graders are
on track to graduate. The District could act right now to use the data it does collect on attendance
and behavior and prior test scores and growth to make a rudimentary 9th grade early warning
system. Unfortunately, there is no evidence of this type of longitudinal data use in OSSE reporting or
planning. If the Executive persists in making this claim we recommend that the Council ask for the
evidence.

• The Office of the Deputy Mayor for Education and OSSE “spent two years working
collaboratively with the Auditor and her staff.”

This is true in part and wrong in part. We greatly appreciate the time and effort members of the
OSSE data team spent with ODCA staff and contractors to produce this audit. An extraordinary
amount of time, though, was required on my part and the part of my General Counsel and outside
counsel we had to hire to provide guidance on federal privacy law in order to continually renegotiate
our access to education data to which we are entitled by the Home Rule Charter. As we were

3
D.C. Code §38-2609.

12
wrapping up fieldwork for this audit and requesting the newly-available 2018-2019 school year data,
the Executive denied this request—again, for data to which the D.C. Auditor is statutorily entitled.
This request was elevated to the Deputy Mayor who denied us that access. As we wrote in our
performance responses to this Committee in February a year ago:

In recent weeks we have experienced a District government agency under audit using its own
regulatory status in a manner that has hindered the ongoing audit. The Office of the State
Superintendent of Education, with the tacit approval of the Deputy Mayor for Education, has
withheld information requested as part of the education data audit mandated by the D.C.
Council. In the interest of completing the audit this spring we have chosen to forego use of
ODCA’s subpoena authority to secure the information and will report the refusal to respond
to a legitimate data request as a finding in the audit.

Conclusion

The Office of the D.C. Auditor has presented this education data audit as a good faith effort to meet
the statutory requirement and to provide documented and credible evidence on what education
data the District collects and uses. Mr. Chairman, I continue to hope that the Mayor and her
education team will finally permit the ODCA team of experts who undertook this work to meet and
discuss the findings and recommendations with the data team at the Office of the State
Superintendent of Education, a request I have made continually since November. I believe progress is
possible based on consensus and communication.

This concludes our testimony and we are happy to respond to questions.

13