A Psychometric Evaluation of Emotion Detection Lexicons: Construct Validity and Measurement Differences

Emotion detection (ED) encompasses a wide variety of tools and techniques to automatically extract emotion content from text. ED has become increasingly popular in psychology, linguistics, the data sciences, and many other fields, however the construct validity of ED methods has received minimal attention. General purpose emotion lexicons are one common ED tool that contain predetermined word-emotion associations. Though ED lexicons measure psychological constructs, many are justified with scant psychological theory. Further, different methods of constructing lexicons may also lead to differences in their word-emotion associations. The measurement similarities of different lexicons is currently unclear, which presents issues for researchers who are concerned with construct validity or who wish to compare results across multiple studies. This dissertation used a novel application of item response models directly on emotion lexicons to understand their similarities and differences. A dual confirmatory and exploratory psychometric approach was taken to compare how lexicons are typically used and how their categories actually inter- and intra-relate. In the confirmatory approach, a strict hypothetical structure was imposed on the lexicons where same-named discrete emotion variables were forced onto the same factors. The final confirmatory model fit poorly. In the exploratory approach, lexicon variables were free to associate. The final exploratory model indicated that while emotion lexicons generally had similar word-emotion associations for the same discrete emotions, there were significant distinctions. Limitations and future directions are discussed.