Evaluating and Recontextualizing the Social Impacts of Moderating Online Discussions

 In response to recent surges in abusive content in online spaces, large social media companies, such as Facebook, Twitter, and Reddit, have been pressured both legally and socially to strengthen their stances against offensive content on their platforms. While the standard practice for addressing abusive content is for human moderators to review whether content is appropriate, the vast scale of online content and psychological toll of abusive material on moderators has led to growing interest in natural language processing in developing technologies to aid in the moderation of offensive language. However, while there has been steady progress on the development of models centered on classifying offensive texts, there is limited consensus over what abusive language is and how NLP models can address practical issues within online moderation. In the complex sociotechnical systems where content moderation takes place, the answers to the questions of “what is abusive language?” and “how should language technologies be used to address abusive language?” can have a major impact on the participation experiences of users in online platforms. Research in online moderation from other disciplines, such as human-computer interaction, platform design, and law, often addresses these social consequences by taking a more interaction-focused view of the problem of moderating abusive language. However, when evaluating moderation issues at scale, these studies of interaction often end up relying on simplified approaches for considering sociolinguistic issues in online communities. 

In this thesis, my goal is to bridge the gap between the language-focused view of content moderation from NLP and the interaction-based view from platform design in two directions. In the first direction, I develop and apply more sophisticated language technologies techniques for evaluating the sociolinguistic impacts of mod?eration strategies at scale. Under this evaluation paradigm, I demonstrate the use of NLP techniques in measuring the social impacts of moderation strategies through three case studies over different online communities at different levels of impact. In the first case study, I examine volunteer-based moderation in Big Issues Debate, a political debate community on Ravelry, by investigating how users can perceive reactive moderation as a form of censorship. I introduce a framework for evaluating moderation bias while controlling for user behaviors displayed during the judgment. Using a probabilistic graphical model that accounts for user preferences and transitions between utterances, I then identify intent-based speech acts associated with a high-risk of moderation to examine whether users with minority viewpoints were targeted for moderation decisions. In the second case study, I apply techniques in framing analysis to examine user responses to a platform-wide policy change announcement regarding the expansion of the quarantine feature on Reddit. Through this analysis, I highlighted the ideological nature in how users discuss the tension between content moderation and free speech and the prioritization of different user experience goals with regards to moderation practices on the left and the right. As a followup to this second case study looking at quarantine policy, in the third case study, I examine the impacts of the quarantines of two major political subreddits, r/the Donald and r/ChapoTrapHouse. In addition to measuring changes in activity and the use of toxic language within the quarantined and related subreddits, I explore the impact of quarantines on different signals of polarization, engagement, and value association unique to political discussion communities. Based on the findings from these three evaluation case studies, I discuss some of the major social implications of the different moderation strategies used and provide recommendations for additional considerations when designing and evaluating moderation interventions. 

The second direction I propose to bridge the gap between language and interaction in abusive language studies is using insights from social theories and the study of online communities to recontextualize how normative and abusive language is defined in language technologies. Under this contextualization paradigm, I introduce an examination of how to operationalize descriptive linguistic norm differences across political subcommunities on Reddit. I first present an annotation experiment investigating how experiential factors influence human perception of ideological differences between content from different subreddits. Based on findings from the annotation experiment that the use of specific political associations may distinguish different interpretation patterns by annotators, I then introduce a framework for characterizing common types of assertions and associations made with political entities. Using this framework, I analyze differences in fine-grained assertion usage tendencies between various political subreddits on the left and right. Finally, I evaluate a subreddit embedding model on its ability to capture these differences in tendencies in how different political subreddits use assertion types. Based on these analyses, I reflect on common assumptions within NLP regarding the relationship between labels and normative linguistic behavior and provide future recommendations for taking a more interactional view of language issues in online communities.