Whitney Tilson's School Reform Blog: DCPS's Response to Test Security Issues under Rhee

Michelle Rhee was of course aware of the risk of increased cheating and took strong steps to make sure it didn't happen, including hiring an outside firm, Caveon. In addition, she's strongly supported that independent investigation of issues raised by the USA Today story. Finally, it's nonsense that she's ducked this issue. Here's Rhee's statement yesterday:

"We believe that public trust is a critical component in ensuring a reliable system in how we assess progress. That is why we brought in independent investigators to look at how tests were administered. These investigators found no evidence of widespread wrongdoing, and we took whatever follow up steps they called for to improve testing security. It is for these same reasons that we spent hours with, and provided unprecedented access to, USA Today and why I was personally available to journalists after the story broke. And it is why we continue to support further investigation even in light of subsequent and contemporaneous tests corroborating the progress of the Districts students."

And below is Rhee's extensive response – here's an excerpt:

DCPS's Forceful Response to Test Security Issues under Rhee

· DCPS has responded forcefully to test security issues, and has continuously worked to improve testing policies and security. This has included working with OSSE to develop new and enhanced security protocols for schools administering tests and reviewing and enhancing that response each year. The district provided every school with an independent observer trained in test security protocols, and enhanced monitoring at schools identified as having prior issues. And the district conducted unscheduled surprise inspections of testing schools to ensure security of test material storage and handling.

· DCPS hired Caveon LLC – a national leader in testing security – to conduct thorough, independent investigations of testing data and school protocols. Caveon's review of testing security measures has resulted in stricter, strengthened protocols. Where Caveon's independent investigations showed improprieties, DCPS took the appropriate action – including dismissing teachers, removing them from administering tests, and other disciplinary steps.

· It should also be noted that this year's steady city test scores under the enhanced testing regime confirm the improvements in student achievement demonstrated in previous test scores, and which are also corroborated by the unquestioned, contemporary NAEP test results.

In summary: Was there cheating in DC under Rhee? Of course. Was there more cheating than before Rhee came in? I'd be shocked if there wasn't, given there was no accountability at all before Rhee, so there was no incentive to cheat. Was the cheating so widespread that it created a picture of progress under Rhee that didn't really happen? Nobody knows for sure until the results of the investigation are known, but I highly doubt it. There's too much other evidence – like NAEP scores that rose more than any other city – of real progress. Finally, did Rhee condone cheating to try to inflate test scores, or try to cover it up? Hell no!

-------------------

DCPS's Forceful Response to Test Security Issues under Rhee

DCPS's Transparent and Cooperative Approach to Inquiries Surrounding Erasure Analysis

· DCPS, under Chancellor Rhee, and later StudentsFirst, were highly cooperative with USA Today's inquiries that spanned several months and consumed several man hours. DCPS was similarly cooperative when the Washington Post ran a series of stories covering the same substance of the USA Today stories two years earlier.

· In the case of USA Today, Ms. Bello and Mr. Gillum were provided unprecedented time and access to report out their story. Several senior members of DCPS under, including the Chief of Data and Accountability, and a number of senior staff at StudentsFirst after Ms. Rhee left DCPS, spent hours with Ms. Bello and Mr. Gillum answering their questions. They were provided the information and data they requested up to the point of being able to identify individual students and teachers - which we simply were not willing to do.

· In addition to the access to the personnel, data and vendors including Caveon that were provided to Ms. Bello and Gillum, Chancelloe Rhee personally provided extensive responses to the questions posed to her as is documented by the unusual step of posting the correspondence between the reporters and her office online. Those responses answered the substance of Ms. Bello and Mr. Gillum's questions directly, especially in light of the fact that many of the questions that USA Today raised were previously raised and answered by a series of articles two years earlier by the Washington Post.

· Chancellor Rhee also made herself personally available for no less than six interviews requests in the first week after the USA Today story ran alone.

· While Ms. Bello and Mr. Gillum may be disappointed that DCPS wasn't willing to compromise student and teacher confidentiality or that the District wouldn't allow reporters to roam around campuses to gather more information during school hours – a decision made by the district itself after Chancellor Rhee's tenure - the access they were provided was extensive and formed much of the substance of the story they eventually published.

THE FACTS

Ø DCPS Took A Series of Proactive Security Measures. According to a press release from DCPS, "To support our commitment to integrity and accountability, we established a number of additional testing precautions this year to help ensure that each of our testing sites adheres to our administration protocols:

• We brought in Caveon, LLC, an independent firm with leading market expertise in test security, to conduct independent reviews of school testing protocols and staff training.

• We incorporated stricter protocols for receiving, storing and returning test materials into the school test plans per an independent review conducted by Caveon, LLC, after the 2010 CAS administration.

• We provided every school with an independent observer from Central Office trained in test security protocols for the duration of the DC CAS testing window as well as additional monitoring for schools that have been identified as having prior test security issues.

• We conducted unscheduled visits to testing schools to observe maintenance of secure test material storage and handling.

• We conducted an independent review of every school test plan." [DCPS Press Release, 3/28/11]

Ø Caveon President Said He Was Given Enough Data To Conduct a Thorough Investigation. In an interview with WUSA, Caveon President John Fremer was asked if he "given enough data to do a thorough job?" and responded that he was. [WUSA Interview]

Ø Independent Analysis Found No Evidence of Cheating In 2009. According to a memo from independent investigation firm Caveon, "Caveon did not find evidence of cheating at any of the schools. For one school, we suggested further follow up by DCPS, but even in that case there was no definitive evidence of cheating." [Caveon Memo Clarification Statement, 3/28/11]

Ø Independent Investigators Found Plausible Explanations For High Erasure Rates Across the District. According to a memo from independent investigators, "The complete responses to each question for each educator interviewed were reported to DCPS along with a school summary and an overall review of all the interviews. In no instance did Caveon conclude that cheating had been revealed by the process. Instead, plausible explanations were provided as to the reason for the high erasure rates." [Caveon Memo Clarification Statement, 3/28/11]

Ø Memo From Independent Investigators Cited Many Possible Reasons For Erasures and Gains. According to a memo from independent investigators, there were many reasons explaining the erasure levels, including teaching test and review strategies, encouraging students to mark questions they are unsure of or mark off wrong answers, and students having to correct a whole series of answers after marking the wrong row earlier in the test. The memo also speculated that more targeted or intensive teaching practices (such as new programs, Saturday school, after school tutoring, etc.) would increase scores. [Caveon Memo Clarification Statement, 3/28/11]

Ø Independent Investigators: We Had Complete Freedom to Investigate, No Pressure to Sugar Coat Results. According to a memo from independent investigation firm Caveon, "Caveon had complete freedom to carry out the interviews and review data with strong encouragement to use our best professional judgment and experience to inform our results and conclusions. There was no encouragement to minimize problems or "sugar coat" our results." [Caveon Memo Clarification Statement, 3/28/11]

Rhee Interviews/Statements in First Week Subsequent to USA Today story

Ø In an interview with Tavis Smiley on 3/29/11, Rhee discussed the USA Today story and issues raised.

http://www.pbs.org/wnet/tavissmiley/interviews/former-dc-schools-chancellor-michelle-rhee/

Ø In an interview with the Kojo Nnamdi show on 4/4/11, Rhee discussed testing issues and the issues raised by the USA Today story.

http://thekojonnamdishow.org/shows/2011-04-04/former-dc-schools-chief-michelle-rhee

http://thekojonnamdishow.org/shows/2011-04-04/former-dc-schools-chief-michelle-rhee/transcript

Ø In a series of statements to Politico, Rhee and a spokesman responded to the USA Today story.

http://www.politico.com/blogs/bensmith/0311/Rhee_DC_defends_test_investigation.html

Ø In an interview with the Washington Post's Jay Matthews, Rhee discussed the issues raised by the story (and her initial response)

http://www.washingtonpost.com/blogs/class-struggle/post/rhee-says-her-remarks-on-test-erasures-were-stupid/2011/03/30/AFTaTl3B_blog.html

Ø Rhee discussed testing issue and USA Today story with WUSA on 4/4/11

http://www.wusa9.com/news/local/story.aspx?storyid=145050