Advances in Interactive Hypothesis Testing

Interactive testing is recently developed to allow to human experts participate in the hypothesis testing algorithms. Most testing methods are predefined algorithms that do not allow modifications after observing the data. However, in practice, analysts tend to choose a promising algorithm after observing the data; unfortunately, this violates the validity of the conclusion. In contrast, the interactive methods allow the algorithm to be much more flexible, such that a human (or a

computer program) may adaptively design the algorithm in a data-dependent manner if they adhere to a particular protocol of “masking” and “unmasking”. Interactive testing was first proposed for multiple hypothesis testing to control the false discovery rate (FDR). This thesis develops interactive tests in various problem settings. Following the problem setting in multiple testing, Chapter 2 and Chapter 3 propose interactive tests with global type-I error control and familywise error rate (FWER) control, respectively. The

interactive procedures can take advantage of covariates and repeated user guidance to focus on possible non-nulls, achieving high power in numerical experiments where the non-nulls are sparse and structured. In addition, we explore alternative forms of masking, which could be more robust to

conservative nulls. Moving outside of multiple testing with p-values, Chapter 4 studies the problem of comparing

multiple samples. Classical nonparametric tests, such as the Wilcoxon test, are often based on the ranks of observations. We design an interactive rank test called i-Wilcoxon with type-I error control. The i-Wilcoxon test is first proposed for two-sample comparison with unpaired data, and then

extended to paired data, multi-sample comparison, and sequential settings, thus also extending the Kruskal-Wallis and Friedman tests. As alternatives, we also numerically investigate (non-interactive) covariance-adjusted variants of the Wilcoxon test, and provide practical recommendations based on the anticipated population properties of the treatment effects. Out of the participants in a randomized experiment with anticipated heterogeneous treatment

effects, is it possible to identify which ones have a positive treatment effect, even though each has only taken either treatment or control but not both? While subgroup analysis has received attention, claims about individual participants are more challenging. Chapter 5 frame the problem in terms of multiple hypothesis testing: we think of each individual as a null hypothesis (the potential outcomes are equal, for example) and aim to identify individuals for whom the null is false (the treatment potential outcome stochastically dominates the control, for example). We develop a novel interactive algorithm that identifies such a subset, with nonasymptotic control of the false discovery rate (FDR).

We also propose several extensions: (a) relaxing the null to nonpositive effects, (b) generalizing the setting to observational studies with heterogeneous and unknown propensity scores, (c) moving from unpaired to paired samples, and (d) subgroup identification.