Translating evaluation concepts used by multimedia developers through their evaluation approaches for instructional CD-ROMs

The purpose of this study was to describe the evaluation concepts used by SK multimedia developers for instructional CD-ROMs. This was achieved through interviews with the SK developers using a naturalistic, case study methodology. Transcripts of the interviews were content analyzed and presented in individual case study format. The discussion section comprises a cross-case analysis of all SK conversations.

Two major categories emerged from the content analysis: types of evaluation performed and overall views and uses of evaluations. The first primary category includes two sub-categories: formative and summative evaluations, each of which detailed various evaluation methods used by the developers during and after the development process. The second primary category described the developers' views of evaluation and how they believed that evaluations were helpful for improving their products.

The interviews revealed that developers are performing evaluations of CD-ROMs, but the evaluation language that they use is frequently different than the language used by program evaluators. The responses from the developers communicated their view of evaluation as an integral component of the development process and their belief that evaluation is critical for product improvement.

The evaluation approaches used by the SK developers offer practical methods by which to evaluate instructional media. Moreover, many of these approaches are especially useful for increasing the utility of evaluations and also are effective strategies to improve the utilization of scarce resources of money and time that are constraints to development and evaluation.

This study concluded that CD-ROMs can be properly developed and evaluated through the establishment of new partnerships between evaluators and developers in which they agree to share their evaluation methods and have them published as well. Currently, the disparity in the evaluation language used by evaluators and developers inhibits these partnerships. The researcher sought to translate the methods and terms used by developers to provide the impetus by which evaluators and developers might better communicate and establish mutually beneficial working partnerships. This study will provide those professionals who undertake evaluations of CD-ROMs and other multimedia with a valuable source to consider in their endeavors.