Towards robust Bayesian adaptive design methods for the study of human behavior

  Bayesian adaptive design is a powerful experimental design method that selects stimuli in sequence in response to observations made during the course of an experiment. Bayesian adaptive design has been increasingly adopted in the behavioral  sciences, for application to two experimental goals: parameter estimation, the measurement of latent variables that correspond to individually-varying traits, and model selection, selection between two or more mechanistic accounts of human behavior.  The goals of this dissertation are to provide a basic overview of Bayesian adaptive design (Chapter 1), accessible tools to facilitate its implementation and use (Chapter  2), and an understanding of the factors that can affect Bayesian adaptive design’s effectiveness in practice and steps users can take to improve its effectiveness (Chapters  3–6).  

Bayesian adaptive design requires both a priori specification of the sources of  variation and uncertainty in the behavioral process under study, and accurate math ematical representation of those sources of variation. In other words, Bayesian adaptive design relies on users’ ability to make a structured guess about the very  aspects of the world the designed experiment is intended to elucidate — making misrepresentation of these aspects effectively inevitable in practice. The focus of the  work presented in Chapters 3–5 of this dissertation is the robustness of Bayesian  adaptive design to inaccurate representation of the sources of variation and/or the  goals of the experiment itself. Through a combination of simulation experiments,  theoretical analysis and novel synthesis of existing literature, I will show that the flip side of the power of Bayesian adaptive design is its fragility: In the presence of  inaccurate representation, it can fail in sometimes counterintuitive ways. I discuss  and investigate three threats to the robustness of Bayesian adaptive design: overly  myopic specification of the goals of the experiment (Chapter 3), misspecification of  the prior parameter distribution (Chapter 4), and misspecification of the assumed  model itself (Chapter 5). The work presented will expose somewhat of a dilemma:  On one hand, Bayesian adaptive design for parameter estimation is more robust than Bayesian adaptive design for model selection in multiple senses, with the difference  stemming from the fact that the experimenter interested in model selection must contend with dual sources of uncertainty: uncertainty about the parameter value (level  of the individually-varying trait) and uncertainty about the model (mechanistic process) itself. At the same time, not acknowledging one’s uncertainty about the model  can result in model misspecification and consequentially in biased inferences.

  Taken together, the results and discussion in this dissertation enumerate the challenges and pitfalls that can arise both in the face of model specification and experimental aims that are too precise and in the face of specifications and aims that are  not precise enough — and provide some concrete recommendations in the context  of common paradigms encountered in the behavioral sciences.