Culling the herding: Using real world randomized experiments to measure social bias with known costly goods
eer-ratings have become increasingly important sources of product information, particularly in markets for "information goods." However, in spite of the increasing prevalence of this information, there are relatively few academic studies that analyze the impact of peer-ratings on consumers transacting in "real world" marketplaces. In this paper, we partner with a major cable company to analyze the impact of peer-ratings in a real-world Video-on-Demand market where consumer participation is organic and where movies are costly and well-known to consumers. After experimentally manipulating the initial conditions of product information displayed to consumers, we find that, consistent with the prior literature, peer-ratings influence consumer behavior independently from underlying product quality. However, we also find that, in contrast to the prior literature, at least in our setting there is little evidence of long-term bias due to herding effects. Specifically, when movies are artificially promoted or demoted in peer-rating lists, subsequent reviews cause them to return to their true quality position relatively quickly. One explanation for this difference is that consumers in our empirical setting likely had more outside information about the true quality of the products they were evaluating than did consumers in the studies reported in prior literature. While tentative, this explanation suggests that in real-world marketplaces where consumers have sufficient access to outside information about true product quality, peer-ratings may be more robust to herding effects and thus provide more reliable signals of true product quality, than previously thought.