logo

Computer-Based Assessment

 

Supplemental

DAA Definitions
See the list of misconception categories and definitions


Technical Report,
Fall 2008


Sample Lesson Plan:
Download Graphing Lesson 1 (PDF)
Download graphing lesson 1 PDF

Sample Pages

See a sample test
Download algebra test sample page



See sample teacher pages
Download sample teacher page



"Reading the Teacher's Reports" PDF
Download Reading the Teachers' Reports PDF document

Contact

Mike Russell at russelmh@bc.edu

Diagnostic Algebra Assessment

daa logoThrough funding from the Institute for Education Sciences, the Technology and Assessment Study Collaborative at Boston College (inTASC) conducted a three-year study to develop and validate diagnostic tests to diagnose three algebraic misconceptions: concept of variable, equality and graphing. This research resulted in the development of a comprehensive online Diagnostic Algebra Assessment (DAA) system, which consists of:

  1. Diagnostic tests for algebraic misconceptions
  2. Immediate production of performance reports for individual students and classes, which consist of ability reports and misconception reports
  3. Lesson plans developed to target specific misconceptions students may have.

The DAA System is an inexpensive, practical, easy to access and use online assessment and instruction tool. The tests, reports, and lesson plans are available and delivered online, and may be used with students during a typical class period in a computer laboratory or classroom with Internet access.

After teachers have their students take the online assessments, the DAA System automatically scores each student's response for ability and misconception and immediately provides teachers with two detailed performance reports for each student – one with items scored for ability (correct/incorrect) and another with items scored for misconception (misconception option selected or not selected). In addition, teachers receive a summary performance report that describes how the class in aggregate performed in terms of ability and misconceptions. The summary report also identifies individual students who have been diagnosed with a misconception. Students who select misconception options for more than 33% percent of the items are diagnosed as having the misconception and are recommended for the instructional intervention activities provided through the DAA System.

These instructional interventions take the form of lesson plans and practice activities developed to help teachers target specific algebraic misconceptions among their students. These lesson plans and activities were designed specifically to help students re-conceptualize the particular algebraic concept with which they are struggling. Each lesson plan lists the objectives for the lesson, which reflect skills or concepts associated with the misconception that are corrected through the instructional intervention and detailed descriptions of small group and individual activities are included. These are accompanied by practice exercises and formative assessment exercises.

DAA Development

Phase I: Item Development and Pilot Studies
The DAA system was developed in three separate phases. During phase I, which was conducted between the fall of 2004 and the spring of 2005, 30 to 40 item-tests were developed and items were piloted online using a sample of approximately 2,000 students from across the country. Items consisted of released items from large-scale assessments and statewide tests such as such NAEP, TIMSS, MCAS, the New York Regents, and TAKS, among other sources. From these items, appropriate items were written and adapted to fit the misconceptions. Answer choices were developed so that one choice was correct and one of the options would indicate that a student had the misconception.

Phase II: Validity Studies
During the second phase of the research, which started in the winter of 2005, measures from three misconception tests were validated using a sample of approximately 1,500 students from across the country. The validity studies were identical for the three misconceptions in question. Misconception tests were developed from the best performing items obtained from Phase I. Isomorphs were developed for each of those original items. These isomorphs were designed to determine the extent to which students generalized their reasoning from the original items to the isomorphs. Based on response patterns on the diagnostic test (12 original items) students were classified into one of three performance categories:

Phase III: Pilot Study
Lesson plans were developed during the summer of 2006. These lesson plans were designed to help students re-conceptualize a specific misconception. Each of the lesson plans contains detailed descriptions of small group or individual activities followed by practice exercises, independent practice exercises, and formative assessment exercises.

A small experimental design with random assignment of teachers to groups was conducted in the spring of 2007 to determine if the lesson plans were effective tools to address the misconceptions. Sixty teachers and approximately 1,200 students from across the country participated in the study. At the time of registration, teachers were randomly assigned to one of the groups:

Preliminary Findings from Phase II and III Studies

Evidence from the validity study indicates that the diagnostic tests were successful at diagnosing three common algebraic misconceptions: concept of a variable, equality, and graphing. The groups of test items assessing each misconception were found to be very reliable both when items were scored for misconception and when items were scored for ability. Moreover, the assessments consistently classified students into three distinct performance categories: knowers, mistakers, or misconceivers. These findings confirm that students who hold these misconceptions can be identified using the DAA System diagnostic tests; once identified, teachers can target their instruction to meet the needs of individual students using the lesson plans and activities that are provided as part of the instructional intervention component of the DAA System. Likewise, findings from DAA System pilot study were positive and suggest that when teachers use evidence from the performance reports from the DAA System to diagnose students and use the lesson plans to tailor their instruction, students' ability scores increased and their misconception scores decreased. Evidence from the pilot study suggests that students are able to successfully re-conceptualize the common algebraic misconceptions and increase their algebra achievement.