Methodological Considerations in School Climate Research

Methodological issues in school climate research have received somewhat less attention in the literature. These issues include the impacts of failing to screen data for potentially invalid responses in self-reports of school characteristics, the measurement issue of school climate as a level-two organizational construct, the common informant-based effect across different rater types (e.g., students and teachers), and the boundary and mechanism of how and why school climate influences student and school outcomes. These issues are addressed in a series of studies that comprise this three-paper dissertation. All three studies drew upon data from the statewide assessment of school climate surveys and department of education records in Virginia public high schools.
The first paper focused on the effects of validity screening in self-report data by investigating associations between students’ bullying victimization experiences and a series of student adjustment outcomes. Two methods of identifying invalid responders were used: survey completion time and built-in validity check items. Results revealed that inclusion of the invalid responders in the total sample inflated the prevalence of all reported risk behaviors, and deflated student reports of GPA, school engagement, and depression. The contrasts between the valid and invalid respondent groups were statistically significant (p < .001), with meaningful effect sizes for all investigated outcomes, after controlling for student and school demographics. The invalid group was significantly comprised of more males, non-Whites, and younger students. The associations between bullying and student outcomes varied as a function of group membership (valid vs. invalid). Results supported the study hypotheses that validity screening significantly affected the summary statistics/prevalence rates of self-reported outcomes and also impacted the associations among variables. This study provided additional evidence for the need to use validity screening in self-report surveys of adolescents.
The second paper investigated when and how different components of school climate, reported by students and/or teachers, were associated with school-level dropout rates. We attempted to examine how different elements of an authoritative school climate (i.e., disciplinary structure, academic expectations, and student support) were related to high school dropout rates, whether the components of school climate interact with each other to predict school dropout rates, and whether these relations were influenced by differences between student and teacher informants. The study found that student reports of teachers’ academic expectations, teacher reports of their support for students, and the interaction between the two, were significantly associated with school dropout rates. Results further indicated that high academic expectations were most strongly related to school completion when teacher-student relationships were supportive, and that both high academic expectations and supportive student-teacher relationships were associated with lower dropout rates even in schools with a high proportion of low-income students. The work provided an example of using appropriate statistical procedures to reveal complicated relationships in social science studies.
In the third paper, we illustrated the use of a doubly latent multilevel structural equation modeling framework (MSEM) in applications of mediation analysis, by examining the mediating role of student engagement in associations between school climate ratings obtained by teachers and a variety of student outcomes, both self-reported outcomes and external records. The illustrations demonstrated how to apply the relatively new procedure--doubly latent MSEM approach into school climate research with real data in applications of multilevel mediation analysis, while controlling for both measurement error and sampling error and providing bias-corrected estimates of the indirect effects. Substantive results supported the hypothesis that student engagement mediates the relationship between school climate reported by teachers and student outcomes of Prevalence of Teasing and Bullying, GPA, and suspension rates. It provided a functional model to explain why and how school climate affects student outcomes. This can inform school improvement efforts and provide evidence for the development of effective interventions. Together, this three-paper dissertation provided a window into several important methodological considerations in studying school climate and call for more attention and research on appropriate use of methodology in social science in general.