KIPP's response to the Western Michigan University report on KIPP

I'm forwarding KIPP CEO Richard Barth's response (below) to the just-released Western Michigan University report on KIPP because: 1) It's important to rebut its incorrect data and conclusions and 2) It's a brilliant case study on how to respond to such reports: it's elegantly written, reinforces what KIPP stands for, isn't defensive, and doesn't come across as an attack -- yet it utterly destroys the WMU report.

 

From: Richard Barth [mailto:RBarth@kipp.org]
Sent: Wednesday, March 30, 2011 8:45 PM
Subject: FW: A Wednesday Evening Thought

 

Dear Friends of KIPP:

 

As you will see from the attached note that I sent to our entire network this evening, we received a copy of a report by Gary Miron and his colleagues at Western Michigan University today that will most likely garner media attention.  I hope you will take away from my thoughts below that we believe  the report seeks to answer legitimate questions, questions which are core to our understanding of what constitutes school health both at KIPP and for the broader sector.  We are committed to pursuing inquiry into these topics, both internally as well as through independent third party research. 

 

Unfortunately, the report submitted by Western Michigan has significant methodological errors, data limitations, and speculative assertions.  I have included the response that we put together quickly this morning addressing the facts.

 

Please feel free to contact me if you have any questions or concerns.  As always, we appreciate your commitment to the nearly 28,000 KIPPsters who are working hard every single day to prove what's possible. 

 

Sincerely,

Richard

 

 

From: Richard Barth
Sent: Wednesday, March 30, 2011 8:18 PM
To: teamandfamily
Subject: A Wednesday Evening Thought

 

Good evening KIPP Team and Family:

 

Today we received a copy of a report on KIPP by Gary Miron and colleagues at Western Michigan University about which reporters are beginning to make inquiries.  We expect articles will run in several outlets tomorrow.  In having very little time to examine the report – but knowing we had to answer fair questions from the media – we provided the statement below to reporters today.  We expect several stories in the next few days including Education Week, The New York Times and The Washington Post.  Should you or others on your team receive inquiries, please direct them to Steve Mancini (smancini@kipp.org) for additional information. 

 

What I want you to know, more than anything else, is that we reject the speculative assertion the report is making:  that our results are based upon selective admission, attrition and outspending district schools.

 

I also want to emphasize, however, that as an organization, we absolutely need to embrace all of the positive that can come from folks raising fundamental questions about our work and engaging in debate.  And, I want to emphasize that we operate with the belief that, whether we like it or not, our fiercest critics can raise legitimate questions about our approaches and our impact.   In the case of the report published today, I would argue that the researchers actually were focused on questions about which we care deeply.  In fact, I would say that this report is focused on 3 of the 6 Essential Questions we use to examine the health of our network.  As every KIPP school leader, executive director and teacher knows by now, when we think of a healthy KIPP school, we ask ourselves six essential questions:

 

 

1. Are we serving the children who need us?

2.   Are our students staying with us?

3.   Are our students progressing and achieving academically?

4.   Are our alumni climbing the mountain to and through college?

5.   Are we building a sustainable people model?

6.   Are we building a sustainable financial model?

 

The report by Western Michigan – as I read it – focuses on questions 1, 2, and 6.    As we dug into the research today, however, we found that the quality of research was so riddled with errors in methodology that it is not helpful to us in shedding new light on how we are performing.    The attached document is what we put together quickly this morning for those who are calling with questions, and we wanted you all to have it.  As a reminder, for all media inquiries, please reach out to Steve Mancini at smancini@kipp.org.

 

As always, I want to thank you for all you do every day for our KIPPsters.  And, I want to make sure that even as this report is not one yielding helpful insights, we should always stay open to the fact that quality research is going to push us as to how we can keep getting better even as we get bigger.

 

Onward,

 

Richard

 

Statement by KIPP regarding report: "What Makes KIPP Work?" by Dr. Gary Miron and colleagues at Western Michigan University
March 30, 2010

At KIPP, we welcome any rigorous and objective review of our schools.  We have participated on several occasions with outside, independent reviews, such as the 2010 study by Mathematica Policy Research. 

 

Today we received a copy of the report "What Makes KIPP Work? A Study of Student Characteristics, Attrition and School Finance," by Dr. Gary Miron, Jessica L. Urschel, and Nicholas Saxton at Western Michigan University (WMU).  In our quick read of the WMU report we observe significant shortcomings in the methodologies used, and must therefore reject the core conclusions made by Miron, et al. about KIPP.

 

While this report focuses on some very fundamental issues for KIPP—student enrollment, attrition, and finances—its findings do not hold up to scrutiny.  We have identified, based on our quick review of the report, multiple factual misrepresentations and errors of analysis, which we would like to address.

 

For example, the WMU report claims that KIPP received $5,760 per student in private funding for the 2008 fiscal year.  However, this result is based on an analysis of only half of our schools, and includes at least two instances where private revenue for regions was misclassified -- including not distinguishing between funds for operating versus capital expenses.   When we look at correctly classified data for these two KIPP regions, and include all KIPP schools that were in operation that year, the number drops to approximately $2,500 per student.

 

We have included a fact sheet with a closer examination of inaccuracies the WMU report's data and methodology.  Given the limited time we have had to review the document, we are not assuming that this is by any means a comprehensive review.

Fact Sheet: A Closer Look at the Data: "What Makes KIPP Work?

In the March 2011 report "What Makes KIPP Work? A Study of Student Characteristics, Attrition and School Finance," Dr. Gary Miron and his colleagues at Western Michigan University raise some important questions about KIPP schools.   These three questions are ones that we take seriously and have studied over time.   

·        Who are the students that come through KIPP's doors?   Are KIPP schools serving the neediest students?

 

·        Why are students leaving KIPP?  How does KIPP student attrition compare to neighboring public schools?   

·        Are KIPP schools sustainable?  How reliant are KIPP schools on philanthropic dollars?

Taken collectively, the WMU report suggests that the progress made by KIPP schools is fundamentally dependent on selective admission of students, high levels of attrition, and private funding that supports significantly higher rates of spending than in traditional district schools.  We reject these assertions.

What follows is a closer look at the fact base around these issues:

Fact 1 – KIPP schools on average have higher concentrations of low income and minority students, and lower concentrations of special education and LEP students.  We do not attract the most able students.  

KIPP schools are open-enrollment public schools, and do not use selective criteria for admission.  The findings regarding student demographics in the WMU study are generally consistent with those in the Mathematica report, which concludes that "KIPP schools have student bodies characterized by higher concentrations of poverty and racial minorities and lower concentrations of special education and LEP students." 

What the WMU study does not address, however, is the considerable evidence that KIPP is not promoting selective entry, or "creaming" students.  In fact, our students most often come to us with lower academic achievement levels than their peers.  The Mathematica report concluded that "KIPP middle schools enroll students whose entering fourth grade achievement is lower than the district-wide average" and comparable to the average for neighborhood feeder elementary schools.

 

Fact 2 – KIPP schools do not have systematically higher (or lower) levels of attrition as compared to schools within their districts.   KIPP student achievement gains cannot be explained by student attrition or selective entry. 

The WMU report claims that KIPP schools have significantly higher levels of student attrition than neighboring school districts.  The researchers arrived at this conclusion by comparing annual school-level enrollment data for KIPP schools to total school district-level enrollment over a three-year period.

The WMU report relies on an apples-to-oranges comparison, as the district attrition figures do not take into account students who transfer from school to school within a district.

 

In addition, the WMU report overstates KIPP's enrollment decline due to incorrect data. For example, WMU reports that KIPP Academy of Opportunity in Los Angeles served 2 eighth grade students in 2008-09, when it actually served 64 eighth graders (see Appendix A, page 4).

 

A more rigorous analysis of student attrition is provided by Mathematica's 2010 report.  It uses student-level data and compared student attrition at KIPP schools to individual schools in neighboring districts with similar demographics across thirteen communities. In addition, Mathematica's report found that KIPP's achievement gains cannot be explained by student attrition. Even when explicitly accounting for student attrition, they found that KIPP's "impacts on students' state assessment scores in mathematics and reading are positive, statistically significant, and educationally substantial."

 

Fact 3 – KIPP schools received far less than $5,760 in private revenue per pupil in fiscal year 2008 for operating expenses.  

 

WMU claims that KIPP has a "$6,500 advantage" per student over neighboring district schools in fiscal year 2008—due in large part to their asserting that KIPP receives $5,760 in additional private funding per pupil.  WMU's private funding number is inflated for several reasons: the report uses a selective sample of our schools, misunderstands how KIPP schools use private funding (i.e., does not distinguish between operating and capital expenses), and contains incorrect data points.

 

First, WMU creates a non-representative sample of schools that leads to the overstatement of KIPP's private revenue per pupil.

 

To get to the $5,760 per pupil in private funding at KIPP, WMU used a sample of 28 of the 58 KIPP schools that operated during 2007-08 (Table 1, page 21). This is roughly half of the schools in operation that school year and for which WMU collected data (see Appendix D).  This sample of 28 schools included 71 percent of the network's private revenue but only 45 percent of students served in 2007-08.   If they had used the full sample size of KIPP schools operating in 2007-08, WMU would have calculated per pupil private revenue of $3,715—which is $2,000 less per pupil than their estimate.

         

Second, the report fails to acknowledge that KIPP and others turn to private funding to compensate for the inequities in public funding on several fronts – capital expenses, start-up costs and general operating costs.

 

Whereas districts receive taxpayer bond money to fund facilities and other capital projects, KIPP schools that are denied access to public facilities must look to private funding to match the bond proceeds received by districts.  Furthermore, private funding supports start-up costs which are only incurred in a school's early years.  Of the 58 KIPP schools operating in 2007-08, 24 of the schools were in their first, second or third year of operation during the year examined and had not yet reached full enrollment.   KIPP schools grow one grade level at a time, requiring them to fund an administrative staff on the public funding associated with a partially enrolled school, which makes the private funding need higher in the earlier years; as a school grows to full enrollment, the need for private funding diminishes.  Lastly, in addition to capital expenses and start-up expenses, studies have demonstrated that charters receive less public funding for general operating expenses.

 

Finally, at least two of the private revenue figures used in the analysis are incorrectly classified (see page 21).

 

For example, of the $17.3 million in private revenue reported by KIPP Houston in 2007-08, more than $10 million was restricted for purchasing land and building or renovating schools, not general operating expenses.  In addition, the stated figure of $8.1 million private revenue for KIPP schools in Newark, NJ (TEAM Academy) is the result of a misclassification on the 990 form.  Team Academy's audited financials for that year report $1.5 million in private revenue; the remaining revenue was actually state and local public revenue and, therefore, was double counted in the WMU analysis.  When adjusting KIPP's overall private revenue per pupil for just these two material corrections with KIPP Houston and KIPP Newark schools, the average revenue for KIPP schools in fiscal year 2008 decreases to approximately $2,500.