Model Specification for Cognitive Assessment of Proportional Reasoning

In modern psychometric analysis of cognitive assessment, there is a choice between psychometric vs. cognitive science paradigms for modeling the latent scale. The first involves as few as one continuous latent ability parameter, while the second focuses on a set of binary latent skills. When the expert cognitive model is qualitatively specified (eg. paragraphs describing general trends in observable behavior for a set of developmental stages), interpretation of responses and latent variables is flexible, and may even be ambiguous. Models incorporating aspects from both psychometric and cognitive science paradigms can help in exploring response patterns, and refining both the exam design and the cognitive model. Here we present two such analyses of a pilot study of proportional reasoning. The first is a Rasch model using binary response coding with a continuous latent trait, which can approximate a set of developmental stages through careful item design and milestones. The second is a Bayes net model using polytomous response coding linked compensatorily to a set of latent skills, which allows for a factor-analytic approach to skill interpretation. We explore each scheme’s usefulness in inferring a student’s current state of knowledge, directing program planners, and directing assessment developers toward refining either the qualitative cognitive model or exam items.