kruger@neu.edu

High School seniors were first required to pass the MCAS exams in order to receive their diplomas during the 2002-2003 school year. Since that time the overall dropout annual dropout rate has remained relatively stable. For grades 9 to 12, the annual dropout rates were 3.3% and 3.4% in 2002-2003 and 2007-2008, respectively. Thus, from examining the overall dropout rate it would seem that the MCAS diploma requirement has had little or no effect on high school students’ decisions to drop out of high school. However, as is often the case, overall numbers can mask the impact that a policy has on subgroups. There are three sources of data that support a connection between the MCAS and the dropout rate for vulnerable subgroups: (1) the increase in the dropout rates for the subgroups who most often fail the MCAS, (2) the dramatic increase in the dropout rate for students who have failed the high school version of the MCAS one or more times, and (3) a rigorous research study, which provides evidence for a causal relationship between failing the highs school MCAS and the on-time graduation rate.

1. The table below provides data on the increases in the annual dropout rates between 2002-2003 and 2007-2008 for two subgroups: students with limited English proficiency and students with disabilities. These two groups tend to fail the high school MCAS exams at higher rates than other groups. For the class of 2007-2008, the pass rates for students with limited English proficiency and students with disabilities on the MCAS high school tests during the 10^th grade administration were respectively 50% and 34% lower than the other members in their cohort group. Thus, it is not surprising that these two groups have had the sharpest increases in dropout rates since 2002-2003.

2. The impact of the MCAS diploma requirement on struggling students’ decision to drop out seems to be accelerating. In 2003-2004, the dropout rate for 12^th grade students who had not passed all the MCAS exams was 16.3%. In 2007-2008, the dropout rate for the same group had more than doubled to 34.6%. As of the 2007-2008 school year, if a student had not passed all the MCAS tests by the 12^th grade, he / she was 13 times more likely to dropout than a student who passed all the exams.

3. Papay, Murnane, and Willett’s (2008) study provide evidence of a causal connection between failing the high school MCAS math test and the high school graduation rate. They examined two groups of students: one group who just barely failed the high school MCAS math test and a second group, who barely passed the same test. Because these two groups of students’ scores were statistically indistinguishable from each other and well within the test’s margin of error, it allowed the researchers to isolate the effect of failing or passing the MCAS on the students’ perseverance in school. The results indicated that whereas failing the MCAS math test did not decrease the graduation rate for suburban students, low-income urban students were eight percent less likely to graduate if they failed the MCAS.

In summary, the above three sources of data provide compelling evidence that Massachusetts high stakes testing program is increasing the dropout rates for some of the most vulnerable students. 

References

Massachusetts Department of Education (2009). High School Dropouts 2007-08: Massachusetts Public Schools. Retrieved November 30, 2009 from http://www.doe.mass.edu/infoservices/reports/dropout

Massachusetts Department of Education (2008). Progress Report on Students Attaining the Competency Determination Statewide and by School and District: Classes of 2008 and 2009. Retrieved November 30, 2009 from http://www.doe.mass.edu/mcas/2008/results/CD.pdf

Massachusetts Department of Education (2005). Dropouts in Massachusetts Public Schools: 2003-04. Retrieved November 30, 2009 from http://www.doe.mass.edu/infoservices/reports/dropout

 Massachusetts Department of Education (2004). Dropout Rates in Massachusetts Public Schools: 2002-03. Retrieved November 30, 2009 from http://www.doe.mass.edu/infoservices/reports/dropout

Papay, J. P., Murnane, R. J., & Willett, J. B. (2008). The consequences of high school exit examinations for struggling low-income urban students: Evidence from Massachusetts. Cambridge, MA: National Bureau of Economic Research. Retrieved March 21, 2009 from http://www.doe.mass.edu/research/reports/papay-murnane-willett.doc

* Increase in number of dropouts is relative to 2007-2008 enrollments. Although the overall trends are evident in the above table, the reader should treat the change in numbers as estimates because the Massachusetts Department of Education made two modifications to their method calculating the dropouts between 2005 and 2007.

When the 1993 Massachusetts Education Reform Act was signed into law, little was known about the effectiveness of high school graduation exams, such as the MCAS. Sixteen years later, we have good evidence these high stakes tests do not accomplish their intended aims.  The most recent and rigorous research studies on this subject, most notably the investigations conducted by John Warren of the University Minnesota and Eric Grodsky of the University of California at Davis, show that graduation exams have no relationship to high school students’ academic achievement, no impact on closing the racial achievement gap, and no effect on post-high school education, employment status, or wage earnings.

Skepticism about these findings is understandable. It was assumed that the potential denial of a high school diploma would be a powerful motivating force on administrators, teachers and students, and therefore they would redouble their efforts and achieve at increasingly higher levels. The national test scores during the last several years appeared to confirm this theory. Massachusetts’ students have attained the highest mathematics and reading scores in the country on the Nation’s Report Card. This is an impressive accomplishment, but it probably has nothing to do with MCAS as a graduation requirement. The fact that the MCAS graduation requirement immediately preceded the rise in test scores does not prove that the MCAS caused the test scores to increase. Consumption of ice cream is correlated with a rise in shark attacks, but no sane person would argue that eating ice cream increases one’s risk of being attacked by a shark. Research is conducted to differentiate so called ‘spurious correlations’ from actual relationships between events, When researchers controlled for preexisting differences between states, such as per pupil spending and prior achievement scores, which might produce spurious correlations, they found that states without graduation exams were just as likely to have high national test scores as states with these exams. Thus, based on the best available research, there is no justification for using high school graduation exams as a strategy for improving academic achievement.

In addition to evaluating the policy’s intended effects, any informed judgment of the MCAS graduation requirement must also consider its potential for harm. Since the advent of the graduation requirement in 2003, more than 20,000 high school seniors have not passed the MCAS. Denying a student a high school diploma imposes a huge cost on both that individual and society. According to the Northeastern University’s Center for Labor Market Studies, the average yearly income in 2006-2007 for an individual without a high school diploma was a paltry $11,031.  Individuals without diplomas are often barred from entering the military, trade unions, and post-secondary programs. Furthermore, the cost to society of someone without a high school diploma is estimated to be at least $250,000 during that person’s lifetime. These individuals pay less money in taxes and are more likely to need social and correctional services. If high school graduation exams were pharmaceuticals, the Food and Drug Administration would ban their use. They do not have the intended effects on academic achievement, and they have serious side effects on our youth and society.

House Bill No. 3660, An Act To Improve Assessment And Accountability To Ensure Students Acquire 21st Century Skills, would rectify the principal flaw in our current statewide assessment system: the denial of a high school diploma because of the results on a single graduation exam. The Bill proposes a comprehensive set of assessment measures to determine a student’s eligibility for a high school diploma. The use of multiple assessments is not a quixotic search for a perfect assessment system. It is a practical approach supported by the overwhelming majority of professional organizations representing assessment experts.  The opponents of multiple assessment measures fear that this approach will dilute our high standards and undo the achievement gains of the last several years. Research does not support their theory, and public policy should not be driven by fear.

Massachusetts with its illustrious history of public education should be in the vanguard of improving the use of statewide assessments. House Bill No. 3660 will advance that noble goal. Nothing less than the wellbeing of our most vulnerable children hangs in the balance.

Policy Analyst, FairTest

Regarding H. 4163 and H. 4164

Joint Committee on Education Hearing

September 17, 2009

I’m Lisa Guisbond, a policy analyst for FairTest, which works to ensure that school assessments are fair, open, valid and educationally beneficial. I am a Brookline public school parent and I had the honor to serve on Gov. Patrick’s Readiness Subcommittee on MCAS. As an engaged stakeholder in the Readiness process, I thank you for the opportunity to share my concerns about these two bills.

I agree with Governor Patrick that it is time for the next chapter of Education Reform. Unfortunately, these two bills do not tackle educational assessment issues that have enormous influence on our schools. Whatever the governance model, whether charter, Readiness, pilot or regular public school, we need an assessment overhaul to obtain real learning improvements.

My colleagues on the Readiness Project MCAS subcommittee and I reviewed research on assessment, MCAS, and high-stakes testing systems. Our subcommittee report noted an emerging consensus that the high-stakes MCAS has had the “unintended outcome of narrowing curriculum, modifying instructional approaches without consideration of what is developmentally appropriate, and has resulted in notable decreases in student engagement.” We said that such negative consequences are magnified in poorer districts, the very schools, districts, and students that education reform was designed to help. Moreover, “while overall exam scores are going up, the achievement gaps remain and drop out rates are increasing … among vulnerable populations.”

To create an effective blueprint for Education Reform’s next chapter, we must absorb the lessons of the last chapter. The emphasis on preparing students for MCAS testing has not helped overcome achievement gaps or prepare students for college work.  New science MCAS results, for example, reflect large disparities in access to excellent science instruction. Just 13% or fewer African American, Hispanic and low-income students scored Proficient or higher on the grade 8 test, compared to 47% of whites.

But the answer is not to focus on test preparation to boost science scores. As Boston science teacher Garret Virchick wrote to the Boston Globe*, “A strong grounding in science means having students do real science in school, not cover everything from anatomy to zoology in the hopes of making sure students ‘know’ enough to pass the test. Doing science is messy, takes time, and should be filled with lots of room for students to make mistakes … This is happening less and less as pressure to do well on these tests has become the mantra in public education.”

There are some positive and some negative aspects of the Readiness bills. However, unless we move our schools from a culture of test compliance to one of deep learning, strong student engagement, and steady improvement, we will fail in our long-term objectives. To do this, we need a true Massachusetts Comprehensive Assessment System, as proposed in H. 3660.

*Virchick, Boston Globe, Sept. 10, 2009

In the 2009 cohort group alone, almost 2,000 students with disabilities failed to pass at least one of the high school MCAS exams, and didn’t graduate on time. 72% of the students in the Class of 2009 who failed to pass one or more high school MCAS tests had a disability. High school students who fail the MCAS are 13 times more likely to drop out of school (Massachusetts Department of Education, 2009). Download PowerPoint slides of these MCAS Results

1. The withholding of high school diplomas because of the results on a single type of assessment instrument has resulted in the disproportionately high denial of diplomas to students with disabilities, English Language Learners, and African-American students.

2. Virtually all professional and scholarly organizations representing assessment experts agree that it is inappropriate to withhold a high school diploma because of the results on a single type of assessment instrument. All tests, no matter how well designed, are subject to error, and multiple test-taking opportunities do not rectify the flaws of a test.

3. Research indicates that high stakes tests, such as the MCAS, do not achieve their aims of (a) better preparing students for the work force or college, (b) closing the achievement gap between racial groups, and(c) boosting the academic achievement of high school students.

4. Research also shows that high stakes tests put students at greater risk for dropping out of school.

5. Test preparation and statewide testing has diminished instructional time during the school day. State-mandated testing is occurring on 15% of the days students are in high school. Important skills are being neglected because of the excessive attention being given to preparing for specific test items.

House Bill 3660 would:

1. Reduce the overreliance on single type of test by using a variety of assessment approaches, including state and local tests, and portfolios.

2. Help create a truly comprehensive statewide assessment system and reverse the tendency to teach a specific set of test items.

3. Encourage instruction of the broad array of skills that students need to be successful in the 21st century.

4. Create a more balanced approach to student assessment, including student products and demonstrations that closely parallel the types of tasks students must perform in post-secondary education or real-world work settings.

5. Encourage district-level accountability and assessments that are strongly linked to improvement efforts.

6. Produce an assessment system that maintains high standards, is cost neutral, and eliminates the negative effects of high stakes testing.

You can find the names and contact information of your state representative and senator at: http://www.wheredoivotema.com

October 16th: The film will be shown immediately before the Massachusetts School Psychologists Association Conference. Time and location: TBA

October 17th: The film will be shown immediately before and after Educating the Whole Child Conference at Bunker Hill Community College before and after Educating the Whole Child Conference at Bunker Hill Community College