Nearly a decade of test-based accountability systems, from “adequate yearly progress” to high-stakes graduation exams, has produced little or no positive effect on student learning, a new blue-ribbon panel report from the National Academies concludes. The report also found there were insufficient safeguards against gaming the system, and that incentives to raise scores undercut the validity of the assessments as a gauge of academic progress, notes Education Week in its May 26 online article.

“Too often it’s taken for granted that the test being used for the incentive is itself the marker of progress, and what we’re trying to say here is you need an independent assessment of progress,” said University of California, Berkeley sociology Prof. Michael Hout, who chaired the 17-member committee of national experts. The panel, convened in 2002, has spent the last 10 years tracking the implementation and effectiveness of school improvement programs mandated by the No Child Left Behind Act and more than a dozen other test-based incentive programs. Those programs include:

• Test-based teacher incentive-pay systems in Texas, Chicago, Nashville, Tenn., and elsewhere;
• High school exit exams adopted by about half of states;
• Pay-for-scores programs for students in New York City and Coshocton, Ohio; and
• Experiments in teacher incentive-pay in India and student and teacher test incentives in Israel and Kenya.

On the whole, the panel found the accountability programs often used assessments too narrow to accurately measure progress on program goals, and used rewards or sanctions not directly tied to the people whose behavior the programs sought to change. The programs often had insufficient safeguards and monitoring to prevent students or staff from simply gaming the system to produce high test scores rather than measuring the learning the tests were meant to inspire.

There are “some real messages” for schools and districts, said committee member Kevin Lang, an economics professor at Boston University who has served as a school board member in Brookline, Mass. Among them: school boards need a way to monitor the district’s progress beyond just test scores. “Incentives can be powerful,” he noted, “but not necessarily in the way you would like them to be powerful.”

One of the most common problems identified by the report: most test-based accountability systems use the same test to apply sanctions and rewards as to evaluate learning. As a result, educators and students tend to focus on test-taking strategies, drilling students who are close to meeting proficiency cut-scores, or other behavior that improves test results rather than improving the overall learning that the scores are expected to measure. This undercuts the validity of the test itself. In New York, requiring all high school seniors to pass the Regents exam before graduating led to more students passing the Regents tests, but, as the report notes, scores on the lower-stakes National Assessment of Educational Progress, which tested the same subjects, didn’t budge.

“It’s human nature: Give me a number, I’ll hit it,” said panel chair Hout. “Consequently, something that was a really good indicator before there were incentives on it, be it test scores or the stock price, becomes useless because people are messing with it.”

In fact, the report found that, rather than leading to higher academic achievement, high school exit exams have decreased graduation rates nationwide by an average of about 2 percentage points. Similarly, NCLB accountability systems have generated minimal improvement in academic learning. When evaluated using outside comparison tests, such as the NAEP, student achievement gains dwindle to about .08 of a standard deviation on average, mostly clustered in elementary-grade mathematics. An intervention considered to have a small effect size is usually about .1 standard deviations. Even that disappointing effect was bigger than any the researchers found on individual student performance.

Jon Baron, president of the Washington, D.C.-based Coalition for Evidence-Based Policy and chairman of the National Board for Education Sciences, which advises the Education Department’s research arm, called the report’s findings “an antidote to what has been the accepted wisdom in this country, the belief that performance-based accountability and incentive systems are the answer to improving education.”

More K-12 Education News

Filed under: K-12 Education News

Tags: Education Policy, Public Policy