file.pdf (139.33 kB)
Vox Populi Annotation: Measuring Intensity of Ideological Perspectives by Aggregating Group Judgments
journal contribution
posted on 2004-03-01, 00:00 authored by Wei-Hao Lin, Alexander HauptmannPolarizing discussions about political and social issues are common in mass media. Annotations on the degree to which a sentence
expresses an ideological perspective can be valuable for evaluating computer programs that can automatically identify strongly biased
sentences, but such annotations remain scarce. We annotated the intensity of ideological perspectives expressed in 250 sentences by
aggregating judgments from 18 annotators. We proposed methods of determining the number of annotators and assessing reliability, and
showed the the sentence-level annotations on ideological perspectives were reliable across different annotator groups.