Harnessing the Wisdom of Crowds

  Decision makers can improve judgment accuracy by aggregating information from mul tiple sources, taking advantage of the Wisdom of Crowds. A variety of judgment aggrega tion procedures have been proposed in previous literature, from simple approaches such  as taking the simple average to complicated methods such as computing the theoretically  optimal weighting method, yet no single aggregation rule can outperform others across  different data contexts. My work tends to address two questions: assessing whether the  available information about individuals’ past performance is sufficient for decision mak ers to rely on a weighted average; and guiding decision makers to select the most appro priate weighting method given the various proposed aggregation rules that perform well  in varied empirical data contexts. I develop a general algorithm to test whether there are  sufficiently many observed judgments for practitioners to reject using the simple average  and instead trust a weighted average as a reliably more accurate judgment aggregation  method. I demonstrate this test algorithm provides better guidance than cross validation  in simulated and real-world data. I also develop two weighting methods for pursuing  the empirically best performing aggregation rule by using the regularization technique  and the stacking learning algorithm respectively. I find the regularized weighting method  achieves a competitive performance with other previous wisdom-of-crowd models, albeit  with limited improvement over them when the data is sparse. I also find that stacking  multiple wisdom-of-crowds weighting models is the most promising way to improve the  wisdom of crowds given historical data.