Developing Enhanced Assessment Tools for Capturing Students Procedural Skills and Conceptual Understanding in Math

Developing Enhanced Assessment Tools for Capturing Students’ Procedural Skills and Conceptual Understanding in Math IES Cognition and Student Learning in Special Education Measurement, Goal 5 Abstract The purpose of this project is to develop, test, and refine sophisticated measurement tools for assessing the conceptual understanding and procedural skills of students with disabilities in math (MD). The assessments will build on our previous work with Enhanced Anchored Instruction (EAI) by shifting focus from instruction alone to making more meaningful links between instruction and assessment. Our previous research with EAI has shown that providing interesting, complex problems along with appropriate scaffolds in technology-assisted and hands-on learning environments can lead to statistically and practically important improvement in the math skills of low-achieving adolescents in middle schools, high schools, and alternative settings. EAI units are focused on improving student performance in math areas that prove most problematic for many students, such as computation with fractions and problem solving (e.g., Measurement and Data, Proportional Relationships, Geometry). The array of concepts and skills contained in the units are embedded throughout the Common Core State Standards Initiative - Mathematics (CCSSI-M, 2010). Our findings from two recent IES-funded randomized studies showed significant improvement in the performance of 880 students with and without MD who were taught with the EAI units on researcher-constructed and standardized tests compared to students in the Business- As-Usual classrooms. Although pleased with the results overall, we think the assessments fell somewhat short of capturing what students actually learned. We have come to this conclusion for two main reasons. First, the pencil-and-paper format of the test items prevented many of the students with disabilities from demonstrating their knowledge. We base our conclusion on 625 classroom observations we conducted during the course of the two studies, student and teacher interviews, and careful analyses of item errors. Second, we developed improved statistical analyses that showed students with MD in latent classes in our samples changed their response characteristics as a result of the EAI interventions. Our subsequent development of a multilevel multidimensional mixture IRT model further indicated that students in latent classes nested within teachers changed the kinds of errors they made as a direct result of the EAI intervention. The objectives of this project are 1) to refine our problem-solving assessments in ways that more adequately tap the knowledge and performance of students with MD, 2) to refine our fractions computation measures so teachers of students with MD can more efficiently conduct their own error analysis of student performance on each type of fractions computation items, and 3) to develop and extend sophisticated statistical analysis methods to uncover subtle differences in student performance over the course of instruction. Our plan is to build more useful and practical measurement systems that apply to both the input (i.e., test design) and the output (i.e., analysis) of assessment tools. The proposed project will employ quasi-experimental and experimental group designs to test the extent to which the new assessment methods compare to our current assessment methods. Based on our previous work, we anticipate the results from this development project will help establish more meaningful links between assessment and instruction, thereby improving the learning outcomes of students with MD