Improving College Students' Complex Systems Thinking with an Ecological Simulation: Does Scaffolding Help?

Complex systems comprise interconnected elements that follow simple rules without central controls and give rise to unpredictable behavior (Mitchell, 2009); characteristics of complex systems are called components. If understood, complex systems components may serve as unifying principles that help students understand systems across domains such as biology, economics, or engineering. Nevertheless, complex systems are difficult to understand because of the variety of changing interactions of their elements as well as the non-linear effects of such interactions. Such non-linear effects are often removed from causes through both time and distance. The purpose of this study was to examine adult understanding of complex systems components, to investigate whether an agent-based participatory simulation, with one of two types of scaffolding, might improve this understanding, and finally to determine if either simulation or scaffolding would help students transfer their new understanding to another context.
The study took place at a mid-sized, public university in the Mid-Atlantic region of the United States. Participants included 96 undergraduate and graduate students enrolled in a class about complex systems in the School of Architecture. The study and intervention were informed by a pilot study, as well as previous research by the study author. A 2x2 pretest-posttest quasi-experimental design was employed to test whether participation in an intervention improved students’ complex systems understanding and whether participation in one of either two scaffolding treatment groups helped improve this understanding. As part of their class, participants attended one of two workshops which served as treatment conditions (Self-Monitoring or Ontological Scaffolding) and then participated in a gameplay of the UVA Bay Game, an agent-based participatory simulation. Students completed identical pretest and posttest assessments of complex system component understanding, as well as an open-ended essay-style posttest transfer prompt.
Student understanding of complex systems components was analyzed using descriptive statistics, non-parametric tests, as well as coding for emergent themes amongst student responses. Non-parametric quantitative analyses revealed that student understanding significantly improved for the components Agent Actions (r = .17, p = .02) and Processes based Causality (r = .13, p = .045) while Action Effects understanding decreased (r = -.19, p = .01); 3 other components showed no changes. Student understanding differed by scaffolding condition only for the component Order (r = .24, p = .02), with Self-Monitoring students showing non-significant reduced understanding while Ontological Scaffolding student understanding increased non-significantly. Finally, students showed no difference for transfer due to treatment condition, though all students demonstrated better understanding of Action Effects and Order over other components, which varied depending on the type of system students chose as the topic of their essay.
This study is the first to use a quasi-experimental design to investigate the effectiveness of agent-participatory simulations in teaching college students complex systems understanding. Although most effects were small, the study shows promise for how such a classroom-based intervention might help students learn such a difficult topic.