Parents in central cities seeking good educations for their children face the disconcerting reality that relatively few city school children pass standardized tests required by New York State, suggesting that the schools are failing. In Rochester, Buffalo, Syracuse and Schenectady, less than 20% of students passed state required English Language Arts and Mathematics exams in 2016 and 2018.
A considerable body of research shows that student performance is strongly related to socioeconomic status. Parents’ incomes and educational status predict almost three-quarters of the variation in performance between schools and school districts where parents are relatively affluent and well educated and those who are not. In Buffalo, Syracuse, Utica and Schenectady, more than 80% of students were disadvantaged. In Rochester, more than 90% were.
Socioeconomic status and student performance on tests are strongly related to economic status once students become young adults. Only 23% of students from low-income families whose math scores were below average had above average socioeconomic status as young adults, according to a study by Anthony P. Carnevale, Megan L Fasules, Michael C. Quinn and Kathryn Peltier Campbell (“Born to Win, Schooled to Lose”). Eighty percent of students from high income families whose performance was above average on math tests in 10th grade had above average SES as adults.
But socioeconomic status (SES) does not tell the whole story. There are variations in performance that are not accounted for by economic and educational status. Adjusted for SES, students in Albany, Rochester and Syracuse performed below expectations. In Rochester, only 9% of students in grades three through eight passed the state required exams in 2018. Based on the percentage of economically disadvantaged students and district educational levels, 17% were expected to pass. In Syracuse, only 13% passed. 18% were expected to pass. In Albany 18% passed, compared to 31% predicted by the two factors.
Although we might infer that unexplained performance differences are the result of differences in school quality, we cannot be certain that it is the cause. Because the available data has a limited number of variables, we cannot know why students in Albany, Rochester and Syracuse performed more poorly than expected. Poor performance could be the result of differences in classroom instruction, but it might also be the result of other factors, such as student differences that existed in pre-school years. We know that as early as third grade – the lowest grade level at which the tests are given – students in the poorly performing school districts did less well than would be predicted by their educational and economic statuses. This suggests that differences in early childhood might be important.
There are better measures of the effect of schools on student learning than comparing achievement on tests at a single point. One approach is to compare the educational growth of students in different schools and school districts. This approach reduces the effects of non-school related differences.
When student educational growth is compared different patterns appear. Differences between urban school districts are smaller using the educational growth measure than the SES/Education model. But differences within school districts are large in some cases. And charter schools are more successful in some cities than in others in providing better settings for student educational growth.
Charter Schools
Given the bleak academic performance of central city school students, it is not surprising that charter schools were established. The fundamental promise of these schools was to offer learning environments that would be more conducive to student success than existing district operated schools. Supporters of the movement believed that charter schools, freed of bureaucratic rules and limits on school districts ability to incentivize teachers to help students perform better, could produce better results for city children.
Charter schools have been controversial. Teachers unions have argued that they divert needed resources from existing public schools, and that they produce little real benefit for schools. Assessed in aggregate across the nation, evaluations of them have been mixed, some showing benefits, others showing that students at charter schools do less well than at comparable schools operated by school districts.
Defenders point to consistent differences in charter school performance by state, reflecting differences in the way charter schools are authorized and evaluated. For example, a recent study, “Charter School Performance in New York” issued in 2017 by the Center for Research on Educational Outcomes (CREDO) at Stanford University found that overall, students at charter schools performed significantly better than those at comparable school district operated schools. CREDO compares the educational growth of students at charter and district operated schools to evaluate differences in outcomes. The chart (reproduced above from the CREDO report) shows that overall, students at charter schools in New York State performed significantly better in mathematics than those in schools run by school districts, by about 1/10 standard deviation (one tenth of a standard deviation is equivalent to a move from the 50th percentile to the 54th percentile.) In English, the difference amounts to a change from the 50th percentile to the 51st percentile.
The difference in performance in New York state was largely driven by charter school performance in New York City. Upstate differences between charter school and district school performance were not statistically significant. However, CREDO data shows that Uncommon Schools in Rochester was associated with a positive difference of 0.11 standard deviation in English compared with district-operated schools – a difference of 4.37 percentiles. In math, the difference was larger – 0.26 standard deviation, or 10.26 percentiles.
Not all charter schools outperformed their traditional public school counterparts. There were large variations in charter school performance. Nearly half significantly outperformed district schools, but slightly more than half performed at levels that were not significantly different from schools operated by school districts or performed worse.. (p. 11).
New York State has recently begun publishing data on student growth at schools administering the state’s third through eighth-grade exams. The state defines student growth as follows: “[A] Student Growth Percentile (SGP)…measure[s] a student’s improvement or growth relative to other students, considering the students’ prior academic histories…The SGP indicates whether a student grew more than or less than students with similar test histories in the state. New York defines Elementary/Middle-Level (EM) Growth as “three years of student-level growth in ELA and mathematics combined.”
Analyzing student progress to assess the quality of schools and school districts offers significant advantages over simple comparisons of student test scores. Student progress comparisons exclude the effect of differences in student performance that result from differences that are not related to school quality. Even so, there are weaknesses in the student growth measure unless some additional control measures are used. In New York’s case, three additional controls are included in constructing the measure: percentage of English Language Learners in the classroom, percentage of students with disabilities, and percentage of students in poverty.
Educational Growth in School Districts
Although Albany, Rochester and Syracuse performed relatively poorly compared with what the SES/education model predicted, the State’s student educational growth data did not show the same performance deficit. Large upstate cities except Binghamton showed student educational growth that was within two percentiles of the average for all school districts in the counties within which they were located. Utica did best, with the average student in the district being in the 52nd percentile in the state, 2.95 percentiles higher than the county average (49%).
In the upstate metropolitan counties studied, the districts that performed the best were a mix of large and small, suburban and rural places. In the best performing district – Whitney Point — the average student’s performance growth placed her or him in the 55th percentile.
The weakest performing districts were also a mix of district types. In the worst-performing district, the average student’s performance growth placed him or her in the 41st percentile.
Although the gap between the best performing district and the worst is large – 15 percentiles – overall differences between districts in the upstate metropolitan counties studied were small – 64% of the districts were within plus or minus 2% of the average for all districts.
As a practical matter, although the data does not show that city school districts like Albany, Rochester and Syracuse are doing a particularly poor job of educating students, most in these districts perform poorly, and high percentages drop out before completing high school. And, as the following section shows, there are large variations in student educational growth among schools within the same school districts.
Educational Growth in Schools
The following section is based on student growth data published by New York state for 2017-2018. Charter schools from three counties – Albany (Albany), Monroe (Rochester) and Erie (Buffalo)– were included in this analysis because data was available for five or more charter schools in each of these counties.
This chart shows the average EM growth percentile at each school in the three cities compared with other schools. Three quarters of the schools had EM growth percentiles that were within a range of only six points – between the 46th and 52nd percentiles.
Charter school performance varied in the 2017-2018 New York State data. The best (Buffalo Academy of Science – 69th percentile) and worst performing school (Charter School of Inquiry in Buffalo – 39th percentile) were 30 percentiles apart in student growth. Similarly, there were large variations in district operated schools in each of the cities. For example, at the best performing district operated school (the Montessori School in Albany), average EM student growth was in the 60th percentile, while at the worst school (PS 82 in Buffalo), the average student was in the 38th growth percentile compared to students state-wide.
School sizes are relatively small. Random variations in performance could occur because of the small number of test subjects in each. To prevent misinterpretations of differences in school performance because of sample variability, I use the method applied in the CREDO study. For schools to be labeled better or worse performing than average, a statistical test was employed. For schools to be considered better or worse than average, the statistical test had to show with 95% confidence that the school’s student growth percentile was different from the average (49th percentile) of school districts in the areas examined.
NYSED data from 2017-2018 shows that in the three cities and counties studied, charter schools did perform better overall than district-operated schools. 37% of charter schools had average EM growth percentiles that were significantly above average, compared with 23% of district operated schools. Even so, 63% of charter schools’ average EM growth percentiles were average or below average. For district operated schools, 77% were average or significantly below average. But there were large variations in the performance of district operated and charter schools in each of the three cities studied.
Buffalo
Buffalo’s district-operated schools had the highest percentage of significantly above average district-operated schools and the lowest percentage of above average charter schools in the three cities.
For students in Buffalo, in many cases choosing to attend a charter school offers no real benefit. Thirteen district-operated schools had EM growth percentiles that were significantly above average, compared with three charter schools. For students at average or above average district operated schools, there are few charter school alternatives where students on average show significantly higher educational growth. Eight district schools had growth percentiles that were significantly below average. Students at these schools might benefit from seeking to enroll in a charter school that had above average or average student growth scores.
Albany
The overall performance of district operated schools in Albany was significantly weaker than charter schools in the city and was the weakest of the three cities studied. Two of five charter schools had EM growth percentiles that were significantly above average, while three were average. Almost half (47%) of district operated schools had average growth percentiles that were significantly below average, while only 13% were significantly above average. There were exceptions, however. At the city’s Montessori Magnet School, the average EM growth percentile was 60%, while at William S. Hackett middle school, the EM growth profile was 55% — also above average.
Many Albany students in district operated schools might benefit by seeking to enroll in charter schools, particularly those attending the seven schools with below average growth percentages. For students at district operated schools with average EM growth percentiles, relatively little might be gained. For example, students at the Delaware Community school would be moving from a school with an average EM growth percentile of 51.8 to a charter school that would at best have an average growth percentile that is 3% higher.
Rochester
Rochester’s district operated schools overall performed slightly better than Albany’s, with 29% having EM growth percentages that were significantly below average, compared with 47% in Albany. But, as in Albany, only 13% of district schools were significantly above average. As a group, charter schools in the Rochester area performed the best among those in the three cities, with 55% having significantly above average EM growth percentiles.
Six of nine schools with significantly above average EM growth percentiles were charter schools, compared to only three district operated schools. Among schools that had EM growth percentiles that were significantly below average, seven of nine were district operated schools.
Most schools operated by the Rochester school district had EM growth percentiles that were statistically average. For many of these students, seeking to transfer to a school ranking higher might not offer a significant advantage, given the amount of variation in school performance that could be associated with statistical sample “noise.” For students in below average schools operated by the Rochester School district, moving to a better performing school could provide greater educational opportunity.
Conclusions
Although differences in student growth between city school districts were not large, within school districts there were relatively large differences between the best and worst performing schools. In two of three cities – Albany and Rochester – charter schools as a group outperformed district operated schools. But in those cities some charter schools performed poorly, and some district operated schools performed well.
For students, avoiding schools whose performance is significantly below average could be beneficial to student growth. But, geographic accessibility and realities of competition for limited seats in better performing schools can make it difficult to get into them.
For policy makers there are several challenges. Attempts to turn around poorly performing schools do not have a promising track record. Brian Backstrom of the Rockefeller Institute at the State University of New York writes, “Over the past half-century, billions of dollars have been spent across the nation on efforts to transform persistently low-performing public schools — most of them urban, most of them low-income, and most of them disproportionately enrolled with students of color — into models of success. It hasn’t worked.”
School turnaround efforts are often frustrated by forces that contribute to organizational inertia, ranging from unions that fear that members will lose their jobs to uneven implementation because of differences in senior and middle level managers’ commitment and ability and insufficient long term financial commitments.
Backstrom points out that the most effective approach to improving schools that perform poorly has been to close them. But doing so faces practical impediments – most notably that better alternatives must be available to displaced students if efforts are to succeed. This can be difficult, because existing, better performing schools usually have enrollment constraints that limit the number of additional students that they can accommodate.
Where city school districts have seen improved results, charter schools often play an important role. The challenge here is two-fold. One issue is that scalability is a problem. Evidence suggests that some charter school operators with proven track records – KIPP and Uncommon Schools are two examples – are more likely to provide statistically superior results than independently operated charter schools, but the successful organizations are constrained by their organizational capacity to grow while maintaining quality. Charter schools take time to establish and, in many cases, need substantial private financial resources to support their efforts. Quality control is a concern as well. Student performance at some charter schools is significantly worse than at most district operated schools. These schools do not add meaningful choice to those seeking greater educational growth. Here, the State Education Department should be vigilant in monitoring charter school performance.
————————————————
An abridged version of this post appears in the Rochester Beacon.