Error Rates in Measuring Teacher & School Performance Based on Student Test Score Gains

Attached is an important and valuable study by Mathematica entitled "Error Rates in Measuring Teacher & School Performance Based on Student Test Score Gains".  http://ies.ed.gov/ncee/pubs/20104004/ 

Thanks to Gary Rubinstein for sending it to me.  Here were his comments:

 

I know you respect Mathematica as they have produced many reports about KIPP which have been quite favorable.

Mathematica is also very involved with Value-Added models and actually do the statistics for D.C. schools for determining the 40% of teacher evaluation based on 'performance.'

Now, as a hedge-fund guy, you live and die by whether a model is an accurate predictor of something.  Please look, then, at the two pages of Chapter 5 of this paper prepared by Mathematica for the Department of Education where they say, about as clearly as they can:  "Our results strongly support the notion that policymakers must carefully consider system error rates in designing and implementing teacher performance measurement systems that are based on value-added models.  Consideration of error rates is especially important when evaluating whether and how to use value-added estimates for making high-stakes decisions regarding teachers (such as tenure and firing decisions)"

And this is THE research that was commissioned for trying to find out if this is a good thing to do.  The researchers said 'no', but they did it anyway.  Come on, Whitney.  As one math guy to another, this is not right.

There are better ways to motivate teachers than to use unreliable data to fire them.  One thing that I've been reading about, and which also would explain some of KIPPs success, is the importance of building an environment of teamwork.  I'm sure you have that at your company, it's not just every man (or woman) for him/herself, right?

 

Here's was my reply:

 

Thanks so much for sending this – it's an important study.

 

What I'm puzzled by is why you think it in any way contradicts what every sensible reformer thinks – it's much more devastating to the absurd union position that no test can ever be used in any part of a teacher's evaluation unless it's proven to be 100% accurate (an obviously impossible bar, which is exactly what it's intended to be).

 

Let's be clear what this study does (and doesn't) say.  It says that value-added models correctly identify above average, average, and below average teachers with 74% accuracy, but that's obviously far from perfect so they shouldn't be relied on solely.  I think we can both agree that no teacher should be fired based solely on value-added tests –  NOBODY thinks this! 

 

Here's what's needed:

 

1) Comprehensive evaluation systems like Michelle Rhee's IMPACT in DC or what Chris Cerf is setting up in NJ (per my last email).

 

2) Better tests to reduce the 26% error rate.

 

3) More principal discretion.  I also trust you saw this line on page 36: "…principals' assessments of teacher effectiveness are reasonably accurate at identifying the best and worst teachers in a given school…"  If I had my way, every principal in America would have the ability to build their team, just as every KIPP principal does.  Yes, I know there are lots of horrible, vindictive principals, which is why, concurrent with giving principals this power, they'd also truly be held accountable for results, just as KIPP principals are (and it's not just based on test scores, but that's certainly an important part of it).  I was heartened to see in the study that the error rate at the SCHOOL-LEVEL was a mere 10%:

 

Corresponding error rates for teacher-level estimates will be lower if the focus is on overall false positive and negative error rates for the full populations of low- and high-performing teachers. For example, with three years of data, misclassification rates will be about 10 percent.

 

Thus, if a principal fires or fails to recruit good teachers, hires his/her relatives and cronies, etc., the school's results will obviously suffer and the principal will soon be unemployed – pretty strong incentive not to engage in bad behavior.

 

Obviously, the unions are powerful enough to block true principal discretion, so might we agree to compromise on what AJ Duffy suggests (per my last email)?  He:

 

wants to make it harder for teachers to earn tenure protections and wants to lengthen that process. He even wants to require teachers to demonstrate that they remain effective in the classroom if they want to keep their tenure protections.

 

And if a tenured teacher becomes ineffective, he wants to streamline dismissals. The process now in place can stretch out for several years, even with substantial evidence of gross misconduct. Some union leaders, notably Duffy, have defended this "due process" as a necessary protection against administrative abuses.

 

"I would make it 10 days if I could," Duffy now says of the length of the dismissal process.