Validating a Mathematics Interim Assessment with Cognitively Diagnostic Error Categories

The stressors of the No Child Left Behind Act have thrust educators into a data-driven accountability culture. As school divisions are racing to keep up with increasingly higher achievement demands, educators are scrambling to find testing and instructional methods for improving mathematics achievement prior to students sitting for end-of-grade (EOG) and end-of-course (EOC) tests. Over the past several years, interim assessments have emerged as a possible solution, although there is a paucity of empirical research to support interim assessments as vehicles for improving mathematics achievement. The purpose of this mixed methods study was to create and validate a 7th grade mathematics interim assessment which incorporated cognitively diagnostic error categories. The interim assessment followed an ordered multiple-choice test design where distractors represented students’ common errors. Inspiration for the development of the error categories came from the cognitive and school improvement literature. The error categories comprise: conceptual, procedural, and attention errors. Validity evidence was gathered from qualitative sources (i.e., student cognitive think-alouds, expert teacher reviews), and quantitative sources (i.e., classical test theory analysis, distractor analysis, differential item functioning, and a partial credit item response theory analysis). Results suggest that there is validity evidence to support the development of the cognitively diagnostic error categories and the overall test design. Of the three error categories, the attention error category was the most problematic and erratic. Validity evidence to support the ordering of the error categories was not consistent. More research needs to be done in the development of the attention error category and the ordering of all three error categories. Limitations to the study and opportunities for future research were discussed.