Pragmatic Measurement for Education Science: A Method-Substance Synergy of Validation and Motivation

Education researchers often require quick and efficient assessments of various student characteristics (e.g., motivation) to use in classroom settings. Unfortunately, guidelines for addressing measurement obstacles, such as scale length, are ambiguous at best and non-existent at worst. Many measures lack sufficient evidence that the conclusions they produce are merited, and short measures have received particular criticism from measurement experts. The result is a tension between technical and pragmatic constraints when conducting measurement in field research. This three-paper dissertation is aimed at identifying and addressing these tensions in one area of motivation research. Paper 1 provides the substantive frame for the overall dissertation. The goal was to understand short-term student motivation change in a classroom setting. Paper 2 provides a typical approach to assessing a scale’s quality and viability for use in the field. The goal was to use traditional psychometric approaches to evaluate a brief measure of motivation. Finally, Paper 3 presents a pragmatic approach to determining validity evidence (i.e., pragmatic measurement) by considering the underlying uses and restrictions of collecting data. The goal was to evaluate the pragmatic approach as a framework for measure users to identify the relevant validity evidence needed based on the potential uses and interpretations of a measure. Together, these papers highlight the nature and benefit of advancing methodological goals by pursuing substantive goals. The current research is a methodological-substantive synergy (i.e., work that advances a substantive domain, such as motivation, while developing and utilizing state-of-the-art methodology) aimed alleviating technical and practical tensions.

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant #R305B090002 to the University of Virginia, and by the National Science Foundation, through grant DRL 1534835 to the Chris Hulleman. The opinions expressed are those of the authors and do not represent views of the Institute, the U.S. Department of Education, or the National Science Foundation.