A Closer Look at Student Performance in Upstate City Schools

Parents in central cities seeking good educations for their children face the disconcerting reality that relatively few city school children pass standardized tests required by New York State, suggesting that the schools are failing.  In Rochester, Buffalo, Syracuse and Schenectady, less than 20% of students passed state required English Language Arts and Mathematics exams in 2016 and 2018.

A considerable body of research shows that student performance is strongly related to socioeconomic status. Parents’ incomes and educational status predict almost three-quarters of the variation in performance between schools and school districts where parents are relatively affluent and well educated and those who are not.  In Buffalo, Syracuse, Utica and Schenectady, more than 80% of students were disadvantaged. In Rochester, more than 90% were.

Socioeconomic status and student performance on tests are strongly related to economic status once students become young adults. Only 23% of students from low-income families whose math scores were below average had above average socioeconomic status as young adults, according to a study by Anthony P. Carnevale, Megan L Fasules, Michael C. Quinn and Kathryn Peltier Campbell (“Born to Win, Schooled to Lose”). Eighty percent of students from high income families whose performance was above average on math tests in 10th grade had above average SES as adults.

But socioeconomic status (SES) does not tell the whole story. There are variations in performance that are not accounted for by economic and educational status. Adjusted for SES, students in Albany, Rochester and Syracuse performed below expectations.  In Rochester, only 9% of students in grades three through eight passed the state required exams in 2018. Based on the percentage of economically disadvantaged students and district educational levels, 17% were expected to pass.  In Syracuse, only 13% passed.  18% were expected to pass.  In Albany 18% passed, compared to 31% predicted by the two factors.

Although we might infer that unexplained performance differences are the result of differences in school quality, we cannot be certain that it is the cause. Because the available data has a limited number of variables, we cannot know why students in Albany, Rochester and Syracuse performed more poorly than expected. Poor performance could be the result of differences in classroom instruction, but it might also be the result of other factors, such as student differences that existed in pre-school years. We know that as early as third grade – the lowest grade level at which the tests are given – students in the poorly performing school districts did less well than would be predicted by their educational and economic statuses. This suggests that differences in early childhood might be important.

There are better measures of the effect of schools on student learning than comparing achievement on tests at a single point. One approach is to compare the educational growth of students in different schools and school districts. This approach reduces the effects of non-school related differences.

When student educational growth is compared different patterns appear. Differences between urban school districts are smaller using the educational growth measure than the SES/Education model. But differences within school districts are large in some cases. And charter schools are more successful in some cities than in others in providing better settings for student educational growth.

Charter Schools

Given the bleak academic performance of central city school students, it is not surprising that charter schools were established. The fundamental promise of these schools was to offer learning environments that would be more conducive to student success than existing district operated schools.  Supporters of the movement believed that charter schools, freed of bureaucratic rules and limits on school districts ability to incentivize teachers to help students perform better, could produce better results for city children.

Charter schools have been controversial.  Teachers unions have argued that they divert needed resources from existing public schools, and that they produce little real benefit for schools.  Assessed in aggregate across the nation, evaluations of them have been mixed, some showing benefits, others showing that students at charter schools do less well than at comparable schools operated by school districts.

Defenders point to consistent differences in charter school performance by state, reflecting differences in the way charter schools are authorized and evaluated.  For example, a recent study, “Charter School Performance in New York” issued in 2017 by the Center for Research on Educational Outcomes (CREDO) at Stanford University found that overall, students at charter schools performed significantly better than those at comparable school district operated schools.  CREDO compares the educational growth of students at charter and district operated schools to evaluate differences in outcomes.  The chart (reproduced above from the CREDO report) shows that overall, students at charter schools in New York State performed significantly better in mathematics than those in schools run by school districts, by about 1/10 standard deviation (one tenth of a standard deviation is equivalent to a move from the 50th percentile to the 54th percentile.)  In English, the difference amounts to a change from the 50th percentile to the 51st percentile.

The difference in performance in New York state was largely driven by charter school performance in New York City.  Upstate differences between charter school and district school performance were not statistically significant.  However, CREDO data shows that Uncommon Schools in Rochester was associated with a positive difference of 0.11 standard deviation in English compared with district operated schools – a difference of 4.37 percentiles.  In math, the difference was larger – 0.26 standard deviation, or 10.26 percentiles.

Not all charter schools outperformed their traditional public school counterparts. There were large variations in charter school performance.  Nearly half significantly outperformed district schools, but slightly more than half performed at levels that were not significantly different from schools operated by school districts or performed worse.. (p. 11).

New York State has recently begun publishing data on student growth at schools administering the state’s third through eighth grade exams. The state defines student growth as follows: “[A] Student Growth Percentile (SGP)…measure[s] a student’s improvement or growth relative to other students, considering the students’ prior academic histories…The SGP indicates whether a student grew more than or less than students with similar test histories in the state. New York defines Elementary/Middle-Level (EM) Growth as “three years of student-level growth in ELA and mathematics combined.”

Analyzing student progress to assess the quality of schools and school districts offers significant advantages over simple comparisons of student test scores. Student progress comparisons exclude the effect of differences in student performance that result from differences that are not related to school quality. Even so, there are weaknesses in the student growth measure unless some additional control measures are used. In New York’s case, three additional controls are included in constructing the measure: percentage of English Language Learners in the classroom, percentage of students with disabilities, and percentage of students in poverty.

Educational Growth in School Districts

Although Albany, Rochester and Syracuse performed relatively poorly compared with what the SES/education model predicted, the State’s student educational growth data did not show the same performance deficit. Large upstate cities except Binghamton showed student educational growth that was within two percentiles of the average for all school districts in the counties within which they were located. Utica did best, with the average student in the district being in the 52nd percentile in the state, 2.95 percentiles higher than the county average (49%).

In the upstate metropolitan counties studied, the districts that performed the best were a mix of large and small, suburban and rural places. In the best performing district – Whitney Point — the average student’s performance growth placed her or him in the 55th percentile.

The weakest performing districts were also a mix of district types. In the worst performing district, the average student’s performance growth placed him or her in the 41st percentile.

Although the gap between the best performing district and the worst is large – 15 percentiles – overall differences between districts in the upstate metropolitan counties studied were small – 64% of the districts were within plus or minus 2% of the average for all districts.

As a practical matter, although the data does not show that city school districts like Albany, Rochester and Syracuse are doing a particularly poor job of educating students, most in these districts perform poorly, and high percentages drop out before completing high school. And, as the following section shows, there are large variations in student educational growth among schools within the same school districts.

Educational Growth in Schools

The following section is based on student growth data published by New York state for 2017-2018. Charter schools from three counties – Albany (Albany), Monroe (Rochester) and Erie (Buffalo)– were included in this analysis because data was available for five or more charter schools in each of these counties.

This chart shows the average EM growth percentile at each school in the three cities compared with other schools. Three quarters of the schools had EM growth percentiles that were within a range of only six points – between the 46th and 52nd percentiles.

Charter school performance varied in the 2017-2018 New York State data. The best (Buffalo Academy of Science – 69th percentile) and worst performing school (Charter School of Inquiry in Buffalo – 39th percentile) were 30 percentiles apart in student growth. Similarly, there were large variations in district operated schools in each of the cities. For example, at the best performing district operated school (the Montessori School in Albany), average EM student growth was in the 60th percentile, while at the worst school (PS 82 in Buffalo), the average student was in the 38th growth percentile compared to students state-wide.

School sizes are relatively small. Random variations in performance could occur because of the small number of test subjects in each. To prevent misinterpretations of differences in school performance because of sample variability, I use the method applied in the CREDO study. For schools to be labeled better or worse performing than average, a statistical test was employed. For schools to be considered better or worse than average, the statistical test had to show with 95% confidence that the school’s student growth percentile was different from the average (49th percentile) of school districts in the areas examined.

NYSED data from 2017-2018 shows that in the three cities and counties studied, charter schools did perform better overall than district operated schools. 37% of charter schools had average EM growth percentiles that were significantly above average, compared with 23% of district operated schools. Even so, 63% of charter schools’ average EM growth percentiles were average or below average. For district operated schools, 77% were average or significantly below average. But there were large variations in the performance of district operated and charter schools in each of the three cities studied.

 Buffalo

Buffalo’s district operated schools had the highest percentage of significantly above average district operated schools and the lowest percentage of above average charter schools in the three cities.

For students in Buffalo, in many cases choosing to attend a charter school offers no real benefit. Thirteen district operated schools had EM growth percentiles that were significantly above average, compared with three charter schools. For students at average or above average district operated schools, there are few charter school alternatives where students on average show significantly higher educational growth. Eight district schools had growth percentiles that were significantly below average. Students at these schools might benefit from seeking to enroll in a charter school that had above average or average student growth scores.

Albany

The overall performance of district operated schools in Albany was significantly weaker than charter schools in the city and was the weakest of the three cities studied. Two of five charter schools had EM growth percentiles that were significantly above average, while three were average. Almost half (47%) of district operated schools had average growth percentiles that were significantly below average, while only 13% were significantly above average. There were exceptions, however. At the city’s Montessori Magnet School, the average EM growth percentile was 60%, while at William S. Hackett middle school, the EM growth profile was 55% — also above average.

Many Albany students in district operated schools might benefit by seeking to enroll in charter schools, particularly those attending the seven schools with below average growth percentages. For students at district operated schools with average EM growth percentiles, relatively little might be gained. For example, students at the Delaware Community school would be moving from a school with an average EM growth percentile of 51.8 to a charter school that would at best have an average growth percentile that is 3% higher.

Rochester

Rochester’s district operated schools overall performed slightly better than Albany’s, with 29% having EM growth percentages that were significantly below average, compared with 47% in Albany. But, as in Albany, only 13% of district schools were significantly above average. As a group, charter schools in the Rochester area performed the best among those in the three cities, with 55% having significantly above average EM growth percentiles.

Six of nine schools with significantly above average EM growth percentiles were charter schools, compared to only three district operated schools. Among schools that had EM growth percentiles that were significantly below average, seven of nine were district operated schools.

Most schools operated by the Rochester school district had EM growth percentiles that were statistically average. For many of these students, seeking to transfer to a school ranking higher might not offer a significant advantage, given the amount of variation in school performance that could be associated with statistical sample “noise.” For students in below average schools operated by the Rochester School district, moving to a better performing school could provide greater educational opportunity.

Conclusions

Although differences in student growth between city school districts were not large, within school districts there were relatively large differences between the best and worst performing schools. In two of three cities – Albany and Rochester – charter schools as a group outperformed district operated schools. But in those cities some charter schools performed poorly, and some district operated schools performed well.

For students, avoiding schools whose performance is significantly below average could be beneficial to student growth. But, geographic accessibility and realities of competition for limited seats in better performing schools can make it difficult to get into them.

For policy makers there are several challenges. Attempts to turn around poorly performing schools do not have a promising track record. Brian Backstrom of the Rockefeller Institute at the State University of New York writes, Over the past half-century, billions of dollars have been spent across the nation on efforts to transform persistently low-performing public schools — most of them urban, most of them low-income, and most of them disproportionately enrolled with students of color — into models of success. It hasn’t worked.”

School turnaround efforts are often frustrated by forces that contribute to organizational inertia, ranging from unions that fear that members will lose their jobs to uneven implementation because of differences in senior and middle level managers’ commitment and ability and insufficient long term financial commitments.

Backstrom points out that the most effective approach to improving schools that perform poorly has been to close them. But doing so faces practical impediments – most notably that better alternatives must be available to displaced students if efforts are to succeed. This can be difficult, because existing, better performing schools usually have enrollment constraints that limit the number of additional students that they can accommodate.

Where city school districts have seen improved results, charter schools often play an important role. The challenge here is two-fold. One issue is that scalability is a problem. Evidence suggests that some charter school operators with proven track records – KIPP and Uncommon Schools are two examples – are more likely to provide statistically superior results than independently operated charter schools, but the successful organizations are constrained by their organizational capacity to grow while maintaining quality. Charter schools take time to establish and, in many cases, need substantial private financial resources to support their efforts. Quality control is a concern as well. Student performance at some charter schools is significantly worse than at most district operated schools. These schools do not add meaningful choice to those seeking greater educational growth. Here, the State Education department should be vigilant in monitoring charter school performance.

————————————————

An abridged version of this post appears in the Rochester Beacon.

 




Rochester’s Broken School System

Kent Gardner argues forcefully in the Rochester Beacon that Rochester’s school system is broken and in need of radical change. Gardner’s post highlights the efforts of a local organization, ROC the Future, to bring about reform of the city’s school system. Gardner is correct – students in the city’s schools do poorly compared to others in the state, and the state designated Distinguished Educator Jaime Aquino’s recent report shows a series of failures of leadership, communication and implementation in the school district.   Here I take a close look at the dimensions of student performance in Rochester compared with other districts in Monroe County and the state outside New York City and look at what my findings suggest for possible reforms.

The Effect of Economic Disadvantage on Performance

Economic disadvantage is the strongest predictor of student performance among the socio-economic variables available for review in the data available from the state test database. The percentage of economically disadvantaged students in school districts is associated with 70% of the differences in performance on the 2018 Grade 8 New York State English Language Arts exam among school districts outside New York City.  

New York State defines economically disadvantaged students and family as those who take part in assistance programs “such as the free or reduced-price lunch programs, Social Security Insurance (SSI), Food Stamps, Foster Care, Refugee Assistance (cash or medical assistance), Earned Income Tax Credit (EITC), Home Energy Assistance Program (HEAP), Safety Net Assistance (SNA), Bureau of Indian Affairs (BIA), or Family Assistance: Temporary Assistance for Needy Families (TANF).

Grade 3 Results 

Third graders in the Rochester school system had the lowest percentage – 17% – of students passing the English Language Arts Exam of school districts in New York State outside New York City.  Among large school districts, Rochester had the highest percent of third grade students were economically disadvantaged – 93%.  Students in Syracuse performed nearly as badly – 20% of Syracuse third grade students passed. Ninety percent of Syracuse third grade students were economically disadvantaged.

Grade 8 Results

Eighty-seven percent of eighth grade Rochester School District students were economically disadvantaged in 2018, among the highest percentage in the state, and only 11.4% of 8th graders received a passing grade on the test – the lowest among the school districts studied. For districts in Monroe County outside the City of Rochester, an average of 37% of students were economically disadvantaged, and 50% passed the eighth grade ELA exam, receiving a grade of 3 or 4.

Other upstate cities also had relatively high percentages of disadvantaged students and small percentages of students passing the state exam, but Rochester’s case was the most extreme.

Five of the nine upstate cities in the chart above had a lower 8th grade passing rate than would be expected based on the relationship between student performance and economic disadvantage.  But, even if the schools in these cities had performed as well as expected based on the model, student performance would still have been poor.  In the case of Rochester, 17% of eighth graders would have passed and not 11%.  For Syracuse, 19% would have passed, not 15%.

The strong association between economic disadvantage and poor student performance found in Rochester and other upstate cities reflects other studies that find that student achievement at the school district level is strongly related to family income. For example, a group of Stanford University researchers in a study reported on by the New York Times in “Money, Race and Success: How Your School District Compares” found that sixth grade students in the richest districts are four grade levels ahead of those in the poorest districts.

How the Performance of Rochester Schools Compares with Others in Monroe County

The performance of schools within school districts varied, but the overall relationship between economic disadvantage and student performance continued to be strong. In this case, test results from grades 3 and 4  were combined, as were results from grades seven and eight, because small numbers of students attend typical elementary schools. By combining grades, random sampling variation is decreased.

Third and Fourth Grade Results

Rochester School District third and fourth grade students were both less likely to pass the ELA exam than students in other Monroe County school districts, and more likely to be economically disadvantaged.  The data for grade three and four shows that overall, more than 70% of students at all Rochester City schools were disadvantaged. At three schools, all of the tested students were disadvantaged. The performance of third and fourth grade Rochester School District students varied, ranging from six percent passing at one elementary school to 37.5% at another.

Of 17 Rochester School district schools studied, only four had passing rates that equaled or exceeded the percentage that would be expected based on a linear model of the relationship between student performance and the percentage of disadvantaged students. Among those four schools, the amount by which performance exceeded expectations was not large. Four of the 17 were in the bottom 20% of schools based on performance when the percentage of disadvantaged students was considered.

As a group charter schools had a lower percentage of disadvantaged students (78%) in third and fourth grade than schools in the Rochester School District (91%).  Overall, performance of third and fourth grade students at charter schools was better than at schools in the Rochester School District when the concentration of poverty was considered.  Of the six charter schools examined, performance was near what would be expected based on the percentage of disadvantaged students at three schools. Performance at two charter schools relative to the percentage of disadvantaged students was in the top 3% of all schools.

Readers should not conclude from this finding that charter school students at some charter schools performed better than at most Rochester City schools solely because of charter school teaching methods. Other studies have shown that self-selection accounts for some of the better performance of some charter schools. Charter schools are likely to be seen as more challenging than public school alternatives and may attract parents and students seeking a more rigorous alternative. See this study for aa review the issue. The kind of analysis employed here cannot account for self-selection.

Seventh and Eighth Grade Results

Smaller percentages of eighth grade students in Rochester schools passed the 8th grade ELA exam than in other Monroe County school districts.  Between 0% and 24% of students passed the state’s ELA exam. At the same time, concentrations of economically disadvantaged students in Rochester were much higher than in other Monroe County school districts.  73% to 94% of eighth grade Rochester City School students were economically disadvantaged. Performance at five of the ten schools in the group was in the bottom 20% of schools based on the relationship between performance and economic disadvantage. Performance at the other five schools was near average.

As with third and fourth grades, a smaller percentage of seventh and eighth grade charter school students was economically disadvantaged (75%) than students in the Rochester School District (88%). Charter school performance was mixed, though overall it was significantly better than at Rochester School District schools.

Performance at one school was slightly below what was predicted.  At two charter schools, performance was average. Performance at two schools was in the top 20% of schools based on the model. Performance at the remaining charter schools was in the top two percent. Cautions again apply about the causes of the relatively good performance at two of the charter schools – self-selection may have influenced the outcomes.

Conclusions

Rochester’s school system is at a crossroads. Student performance in Rochester schools is poor, even accounting for effect of the high percentage of students who are disadvantaged. The recent report by distinguished educator Jaime Aquino documents shortcomings in leadership, communications and policy execution. These factors all call for substantial changes in current practice.

The data about charter schools is somewhat equivocal. While student performance at several charter schools was substantially better than at public schools with similar percentages of disadvantaged students, the performance of others was not above average. Because charter school students apply for admission, selection bias is a possible explanation of better performance.  It should also be noted that though most students at charter schools were economically disadvantaged, the concentration of disadvantaged students in Rochester School District schools was 13% higher than at charter schools as a group.

In the best performing schools in the region, more than 70% of students pass the state’s ELA exams, compared to 15% for third graders and 16% for eighth graders in Rochester City Schools. But, only 20% to 30% of students in the best performing districts were economically disadvantaged compared with about 90% in Rochester.

The pervasiveness of the relationship between economic insecurity and poor student performance points to the need reduce the concentration of poverty in the City of Rochester. As a city, Rochester is highly dependent on other levels of government – primarily the Federal government — for assistance in combatting poverty. The city/county anti-poverty initiative – RMAPI operates with involved entities on strategy development and coordination. It is important that RMAPI and its partners develop clear strategies implemented at a scale that is large enough to have meaningful impacts. Rochester’s Center for Governmental Research developed a series of options for policy initiatives for RMAPI in 2014, some of which the organization followed.

The most recent progress report posted on the RMAPI website describes its 2017 accomplishments. The most recent press release is dated January 2018. Although the organization made some programmatic initiatives in the 2015-2017 time-frame, its website lacks information on the current status of its efforts.

There are potential federal actions that could benefit low income families. Because poor families cannot invest in their children as much as wealthier parents, solutions like proposed refundable family tax credits could help the already existing Earned Income Tax Credit reduce income insecurity. Increasing food stamp benefits, particularly for families with children might be another approach. Because there is evidence that disadvantaged students perform better in schools that have lower concentrations of disadvantaged students, efforts to provide lower income families with access to housing in wealthier neighborhoods has been proposed as a remedy.

In part, the organizational failures in the Rochester City School District may reflect the difficult conditions within which they operate. But, students at city schools do not perform well, even considering the concentration of economically insecure students who attend them. Given the lasting consequences of student failure, community expectations demand better. The Distinguished Educator’s report demonstrates that the district must make a sharp course correction if it hopes to meet the needs of Rochester families.

At the same time, as long as the concentration of poverty in Rochester continues to be very high, efforts to substantially improve student performance at city schools are likely to be only marginally successful.  The performance of students in Rochester City school reflects the nexus of income inequality and organizational failures. Ameliorating the problem will require a multi-pronged approach that addresses both problems.




Can Charter Schools break the Poverty-Poor Student Performance Link?

In an earlier post, I argued that school based solutions to the problem of the poor performance of students in central city schools were not likely to succeed because they ignored the impact of the concentration of disadvantaged students on student achievement.  The data showed that 79% of the variation in performance in school performance in upstate New York metropolitan areas was related to the concentration of economically disadvantaged students within them.

Discussions about the benefits of charter schools tend to be heated – inflamed by ideological differences.  But whatever one’s feelings are about the virtues of preserving public education, or of competition in improving educational opportunity, before making judgements, we should examine the available data about their effectiveness.

At the outset, it should be noted that evaluating the true impact of charter schools is difficult.  Ideally, the performance of charter and public schools should be compared by selecting and assigning students at random and following their progress over a period of years.  But, in reality, students in charter schools are not selected at random, and matched samples of public school students are not available for comparison. Published analyses on the subject have pointed out the need to adjust performance comparisons of students at public and charter schools for selection bias, because charter school students are to a large degree self-selected.

Where competent analyses comparing charter and public schools have been done, the findings have been mixed. One review of the available studies concluded:

“Taken in the aggregate, the empirical evidence to date leads one to conclude that we do not have definitive knowledge about the impacts of public charter schools on students and schools. But in reviewing the existing evidence, one is also struck by the fact that the impacts of charter schools appear to be very contextual. Some public charter schools are better than others. Some are very successful in meeting student needs, and others are not very successful…. Consequently, the impacts of public charter schools should not be painted with one broad brush stroke. Each should be judged on its own evidence and performance.”

Other studies  have found significant advantages for charter schools in central cities. Atila Abdulkadiroglu, Joshua Angrist, Susan Dynarski, Thomas J. Kane and Parag Pathak, in “Accountability and Flexibility in Public Schools: Evidence from Boston’s Charters and Pilots” found:

“A consistent pattern has emerged from this research. In urban areas, where students are overwhelmingly low-achieving, poor and nonwhite, charter schools tend to do better than other public schools in improving student achievement. By contrast, outside of urban areas, where students tend to be white and middle class, charters do no better and sometimes do worse than public schools.”

My research is based on a reanalysis of state education data on the performance of students on the 2015 Statewide Student Assessment.  It cannot provide a controlled analysis of the performance of charter school students, compared with those in public schools.  For that reason, the data available to me cannot produce conclusive evidence about the effectiveness of charter schools.

Because publicly available data is cross-sectional, it provides information about the performance of students at a given point in time, but unlike longitudinal studies, it does not directly measure their gains over a year or years.  For that reason, when a  cross-sectional study finds out-performance, or under-performance, there is the danger of making an attribution error, because we don’t know whether the out-performance or under-performance was a characteristic of the student population that was unrelated to the effectiveness of the schools being evaluated.  For example, the students at out-performing schools might have characteristics related to their selection that would predispose them to perform better than other students.

With those limitations in mind, it is worth looking at the New York State Education Department data on student performance from the 2015 Statewide Student Assessment, controlling for the concentration of poverty in schools, to see whether students at charter schools do significantly better than those at public schools with similar concentrations of disadvantaged students.  The chart below shows the performance of students in public and charter schools in all counties in metropolitan areas, except for the City of New York:

Public Charter Outside NYC

Note that data was available for only 33 charter schools outside New York City, so conclusions from this group of schools must be regarded as tentative.  Still, a few things stand out.  First, the performance of charter schools was quite varied – several charter schools were among the worst performers compared to schools with similar concentrations of disadvantaged students, while a number of others, particularly those with high concentrations of disadvantaged students performed better.  Second, for charter schools, unlike public schools, student performance was not related to the concentration of poverty.

As a group, students at charter schools did slightly better than at public schools with the same concentrations of disadvantaged students. However, the fact that 24% (8 of 33) schools exceeded the percent of students predicted to pass by 20% or more, based on the concentration of poor students, is significant.  Only 1.9% of public schools outside New York City had student performance reaching that level.   And, as Abdulkadiroglu, et. al. found, the benefit from charter schools was most significant for students in schools with high concentrations of poor students.

The performance of the better charter schools in urban counties outside New York City was significantly better than average schools with high concentrations of disadvantaged students, but not as good as at schools with few poor students.  Most of the better performing charter schools had about 40% of students passing the Statewide Assessment, compared with as many as 60% in schools with few disadvantaged students.

School Performance in New York City

The concentration of disadvantaged students in New York City schools is associated with 52% of the variation in student performance between the schools.  Compared to public schools in urban counties outside New York City economic disadvantage is a less powerful predictor of student performance in City schools – 52% vs. 79%.  For charter schools, the relationship between the concentration of poverty and student performance was very weak – explaining only 8% of the difference in student performance.  As with other counties, the performance of charter schools was quite heterogeneous. Students at charter schools in New York City as a group did better than those at public schools with similar concentrations of disadvantaged students.  At the same time, a number of Charter schools performed less well than the average of public schools with the same concentration of poor students.

The weaker relationship between the concentration of poverty and student performance in New York City schools appears to be in part a consequence of the city’s policy of creating specialized schools with selective admission criteria.  For example, the Medgar Evers College Preparatory School includes questions about student performance on the Statewide assessment in its application form.  Another example is the TAG Young Scholars School, which describes its admission policy this way:  “Prospective students must be tested by The New York City Department of Education to determine whether they qualify for a seat in one of the City’s Gifted and Talented programs.” Note that while charter schools often use lotteries to select students, they are not permitted to use test performance as a selection criterion.

These selective public schools raise the issue of causal attribution, since unlike schools that do not choose students based on test scores, it is likely that student bodies enter the selective public schools at higher levels of performance than students at other public and charter schools, and that their better performance may primarily be a result of selection criteria, rather than teaching at the schools.

Public Charter NYC

Some charter schools and public schools in New York City did as well as schools with low percentages of disadvantaged students.  Some of the best performing public schools with high concentrations of disadvantaged students use test performance as one criterion for admission.  Since charter schools are not permitted to exclusively serve high performing populations, the performance of the best charter schools is more remarkable.  At 34 of 148 (23%) of charter schools, 20% or more students than were expected to pass based on the concentration of disadvantaged students passed the statewide assessment. Among public schools in New York City, including those that have selective admissions, 8.9% of schools exceeded their predicted performance level by 20% or more.

While this data cannot prove that the excellent performance of some charter schools was the result of the schools themselves, rather than some other factor, it is consistent with studies that have shown charter schools to be advantageous for disadvantaged students in central cities.

Implications

Much of the discussion about the performance of schools, and how to improve outcomes, has focused on the common core and its testing requirements.  The purpose of these requirements was to provide a universal set of assessment tools that would provide comparable data about student progress across systems.

The results of the testing have been disappointing to many, since, as the figures above show, large percentages of students did not achieve passing grades.  For example, Governor Cuomo’s 2015 The State of New York’s Failing Schools report stated, “It is incongruous that 99% of teachers were rated effective, while only 35.8 percent of our students are proficient in math and 31.4 percent in English language arts. How can so many of our teachers be succeeding when so many of our students are struggling?”

Governor Cuomo’s proposal to improve student performance included the creation of a teacher evaluation system that relied more heavily (50%) on the performance of students in standardized tests, a process to make it easier to remove substandard teachers, and a process to place under-performing schools in receivership.  Several of the proposals have problems.  Teacher evaluation systems that rely heavily on the progress of students on standardized tests suffer from statistical defects that result in low reliability of results – a subject for a future blog post.  The process for identifying under-performing schools does not effectively identify schools that are under-performing relative to the concentration of students in poverty within them.

Most significantly, by focusing almost exclusively on accountability for under performing teachers and schools, the proposal does not offer a strategy for overall improvement of New York’s schools.  Accountability focused methods focus on remedying or removing the worst five or ten percent of schools and teachers in the system, but do nothing to help the great majority achieve better results.

If New York’s education system is to make strides in improving student outcomes, it must encourage schools and teachers to adopt known classroom teaching strategies and effective curriculum choices that have the potential to improve overall outcomes.  Since a significant number of charter schools have achieved excellent student outcomes, it would be helpful if the strategies they use could be considered for adoption in schools that do not perform well.  The state should focus on finding ways to encourage the use of effective strategies, by disseminating information and incentivizing their adoption.

Considerable research has been done on the strategies employed by effective charter schools in improving student performance.  For example, “Getting Beneath the Veil of Effective Schools: Evidence from New York City,” by Will Dobbie and Roland G. Fryer of Harvard University found that:  “traditionally collected input measures – class size, per pupil expenditure, the fraction of teachers with no certification, and the fraction of teachers with an advanced degree – are not correlated with school effectiveness.  In stark contrast…an index of five policies…explains approximately 45% of the variation in school effectiveness.”  They are consistent with the approaches used by “no excuses” model charter schools that emphasize selective teacher hiring, extensive teacher feedback, increased instructional time, and a focus on discipline and academic achievement.

For most schools in cities with high concentrations of disadvantaged students in central cities, academic performance remains poor. In some of these schools less than 10% of students received passing grades on the statewide assessment, and the overwhelming majority of schools with concentrations of disadvantaged students of 90% or more had less than 20% of students passing.

But almost one quarter of charter schools and a few public schools have broken the link between poverty and poor school performance.  At these schools, more than 40% of students passed the statewide assessment, despite very high concentrations of poverty within them.

Accountability based approaches aimed at weeding out ineffective teachers, or taking control of schools from boards of education will benefit only a small minority of students statewide.  Instead, we should focus on making use of what works in improving student performance at the best charter schools, encouraging poor performing schools to adopt effective techniques.