In an earlier post, I argued that school based solutions to the problem of the poor performance of students in central city schools were not likely to succeed because they ignored the impact of the concentration of disadvantaged students on student achievement. The data showed that 79% of the variation in performance in school performance in upstate New York metropolitan areas was related to the concentration of economically disadvantaged students within them.
Discussions about the benefits of charter schools tend to be heated – inflamed by ideological differences. But whatever one’s feelings are about the virtues of preserving public education, or of competition in improving educational opportunity, before making judgements, we should examine the available data about their effectiveness.
At the outset, it should be noted that evaluating the true impact of charter schools is difficult. Ideally, the performance of charter and public schools should be compared by selecting and assigning students at random and following their progress over a period of years. But, in reality, students in charter schools are not selected at random, and matched samples of public school students are not available for comparison. Published analyses on the subject have pointed out the need to adjust performance comparisons of students at public and charter schools for selection bias, because charter school students are to a large degree self-selected.
Where competent analyses comparing charter and public schools have been done, the findings have been mixed. One review of the available studies concluded:
“Taken in the aggregate, the empirical evidence to date leads one to conclude that we do not have definitive knowledge about the impacts of public charter schools on students and schools. But in reviewing the existing evidence, one is also struck by the fact that the impacts of charter schools appear to be very contextual. Some public charter schools are better than others. Some are very successful in meeting student needs, and others are not very successful…. Consequently, the impacts of public charter schools should not be painted with one broad brush stroke. Each should be judged on its own evidence and performance.”
Other studies have found significant advantages for charter schools in central cities. Atila Abdulkadiroglu, Joshua Angrist, Susan Dynarski, Thomas J. Kane and Parag Pathak, in “Accountability and Flexibility in Public Schools: Evidence from Boston’s Charters and Pilots” found:
“A consistent pattern has emerged from this research. In urban areas, where students are overwhelmingly low-achieving, poor and nonwhite, charter schools tend to do better than other public schools in improving student achievement. By contrast, outside of urban areas, where students tend to be white and middle class, charters do no better and sometimes do worse than public schools.”
My research is based on a reanalysis of state education data on the performance of students on the 2015 Statewide Student Assessment. It cannot provide a controlled analysis of the performance of charter school students, compared with those in public schools. For that reason, the data available to me cannot produce conclusive evidence about the effectiveness of charter schools.
Because publicly available data is cross-sectional, it provides information about the performance of students at a given point in time, but unlike longitudinal studies, it does not directly measure their gains over a year or years. For that reason, when a cross-sectional study finds out-performance, or under-performance, there is the danger of making an attribution error, because we don’t know whether the out-performance or under-performance was a characteristic of the student population that was unrelated to the effectiveness of the schools being evaluated. For example, the students at out-performing schools might have characteristics related to their selection that would predispose them to perform better than other students.
With those limitations in mind, it is worth looking at the New York State Education Department data on student performance from the 2015 Statewide Student Assessment, controlling for the concentration of poverty in schools, to see whether students at charter schools do significantly better than those at public schools with similar concentrations of disadvantaged students. The chart below shows the performance of students in public and charter schools in all counties in metropolitan areas, except for the City of New York:
Note that data was available for only 33 charter schools outside New York City, so conclusions from this group of schools must be regarded as tentative. Still, a few things stand out. First, the performance of charter schools was quite varied – several charter schools were among the worst performers compared to schools with similar concentrations of disadvantaged students, while a number of others, particularly those with high concentrations of disadvantaged students performed better. Second, for charter schools, unlike public schools, student performance was not related to the concentration of poverty.
As a group, students at charter schools did slightly better than at public schools with the same concentrations of disadvantaged students. However, the fact that 24% (8 of 33) schools exceeded the percent of students predicted to pass by 20% or more, based on the concentration of poor students, is significant. Only 1.9% of public schools outside New York City had student performance reaching that level. And, as Abdulkadiroglu, et. al. found, the benefit from charter schools was most significant for students in schools with high concentrations of poor students.
The performance of the better charter schools in urban counties outside New York City was significantly better than average schools with high concentrations of disadvantaged students, but not as good as at schools with few poor students. Most of the better performing charter schools had about 40% of students passing the Statewide Assessment, compared with as many as 60% in schools with few disadvantaged students.
School Performance in New York City
The concentration of disadvantaged students in New York City schools is associated with 52% of the variation in student performance between the schools. Compared to public schools in urban counties outside New York City economic disadvantage is a less powerful predictor of student performance in City schools – 52% vs. 79%. For charter schools, the relationship between the concentration of poverty and student performance was very weak – explaining only 8% of the difference in student performance. As with other counties, the performance of charter schools was quite heterogeneous. Students at charter schools in New York City as a group did better than those at public schools with similar concentrations of disadvantaged students. At the same time, a number of Charter schools performed less well than the average of public schools with the same concentration of poor students.
The weaker relationship between the concentration of poverty and student performance in New York City schools appears to be in part a consequence of the city’s policy of creating specialized schools with selective admission criteria. For example, the Medgar Evers College Preparatory School includes questions about student performance on the Statewide assessment in its application form. Another example is the TAG Young Scholars School, which describes its admission policy this way: “Prospective students must be tested by The New York City Department of Education to determine whether they qualify for a seat in one of the City’s Gifted and Talented programs.” Note that while charter schools often use lotteries to select students, they are not permitted to use test performance as a selection criterion.
These selective public schools raise the issue of causal attribution, since unlike schools that do not choose students based on test scores, it is likely that student bodies enter the selective public schools at higher levels of performance than students at other public and charter schools, and that their better performance may primarily be a result of selection criteria, rather than teaching at the schools.
Some charter schools and public schools in New York City did as well as schools with low percentages of disadvantaged students. Some of the best performing public schools with high concentrations of disadvantaged students use test performance as one criterion for admission. Since charter schools are not permitted to exclusively serve high performing populations, the performance of the best charter schools is more remarkable. At 34 of 148 (23%) of charter schools, 20% or more students than were expected to pass based on the concentration of disadvantaged students passed the statewide assessment. Among public schools in New York City, including those that have selective admissions, 8.9% of schools exceeded their predicted performance level by 20% or more.
While this data cannot prove that the excellent performance of some charter schools was the result of the schools themselves, rather than some other factor, it is consistent with studies that have shown charter schools to be advantageous for disadvantaged students in central cities.
Much of the discussion about the performance of schools, and how to improve outcomes, has focused on the common core and its testing requirements. The purpose of these requirements was to provide a universal set of assessment tools that would provide comparable data about student progress across systems.
The results of the testing have been disappointing to many, since, as the figures above show, large percentages of students did not achieve passing grades. For example, Governor Cuomo’s 2015 The State of New York’s Failing Schools report stated, “It is incongruous that 99% of teachers were rated effective, while only 35.8 percent of our students are proficient in math and 31.4 percent in English language arts. How can so many of our teachers be succeeding when so many of our students are struggling?”
Governor Cuomo’s proposal to improve student performance included the creation of a teacher evaluation system that relied more heavily (50%) on the performance of students in standardized tests, a process to make it easier to remove substandard teachers, and a process to place under-performing schools in receivership. Several of the proposals have problems. Teacher evaluation systems that rely heavily on the progress of students on standardized tests suffer from statistical defects that result in low reliability of results – a subject for a future blog post. The process for identifying under-performing schools does not effectively identify schools that are under-performing relative to the concentration of students in poverty within them.
Most significantly, by focusing almost exclusively on accountability for under performing teachers and schools, the proposal does not offer a strategy for overall improvement of New York’s schools. Accountability focused methods focus on remedying or removing the worst five or ten percent of schools and teachers in the system, but do nothing to help the great majority achieve better results.
If New York’s education system is to make strides in improving student outcomes, it must encourage schools and teachers to adopt known classroom teaching strategies and effective curriculum choices that have the potential to improve overall outcomes. Since a significant number of charter schools have achieved excellent student outcomes, it would be helpful if the strategies they use could be considered for adoption in schools that do not perform well. The state should focus on finding ways to encourage the use of effective strategies, by disseminating information and incentivizing their adoption.
Considerable research has been done on the strategies employed by effective charter schools in improving student performance. For example, “Getting Beneath the Veil of Effective Schools: Evidence from New York City,” by Will Dobbie and Roland G. Fryer of Harvard University found that: “traditionally collected input measures – class size, per pupil expenditure, the fraction of teachers with no certification, and the fraction of teachers with an advanced degree – are not correlated with school effectiveness. In stark contrast…an index of five policies…explains approximately 45% of the variation in school effectiveness.” They are consistent with the approaches used by “no excuses” model charter schools that emphasize selective teacher hiring, extensive teacher feedback, increased instructional time, and a focus on discipline and academic achievement.
For most schools in cities with high concentrations of disadvantaged students in central cities, academic performance remains poor. In some of these schools less than 10% of students received passing grades on the statewide assessment, and the overwhelming majority of schools with concentrations of disadvantaged students of 90% or more had less than 20% of students passing.
But almost one quarter of charter schools and a few public schools have broken the link between poverty and poor school performance. At these schools, more than 40% of students passed the statewide assessment, despite very high concentrations of poverty within them.
Accountability based approaches aimed at weeding out ineffective teachers, or taking control of schools from boards of education will benefit only a small minority of students statewide. Instead, we should focus on making use of what works in improving student performance at the best charter schools, encouraging poor performing schools to adopt effective techniques.