Two case studies in using cognitive walkthrough for interface evaluation

Abstract: "Previous HCI-studies have compared usability evaluation methods quantitatively without supplementing these data with detailed qualitative data about how analysts actually learn and use methods. In contrast, we present two diary-based case studies that describe the processes of two novice analysts who learned about and applied the Cognitive Walkthrough (CW; Lewis, et al., 1990) to the specification of a multimedia authoring system. Results show that the two analysts easily learned to use CW but also that they found the technique tedious to use. Moreover, CW was neither reliable when comparing the two analysts' processes and outcomes to each other, nor accurate when comparing the analysts' problem predictions to results from usability tests applied to a running system. We examine these data in detail, searching for possible causes of the observed reliability and accuracy. Based on these analyses, we suggest three changes to CW method to improve its accuracy and two changes to improve its reliability. Further, we recommend developing a tool to reduce the tedium and integrate our suggested improvements to CW."