Can Level 1 evaluation be used to predict training transfer?

Patricia Windle Todd, Purdue University

Abstract

The purpose of this study was to determine whether trainees' responses on the end-of-course evaluation provide accurate information to make predictions about training transfer. The hypotheses on which this study focused were as follows: (a) responses to questions about perceived training transfer on the end-of-course questionnaire would predict behavioral change on the job; and (b) responses to questions about perceived barriers on the end-of-course questionnaire would predict barriers preventing behavior change on the job. The analyses showed no support that responses to questions on the end-of-course questionnaire predict behavior change. There was partial support that responses to questions on the end-of-course questionnaire predict barriers preventing behavior change on the job. Some agreement between Level 1 self-reports and the observation data was expected, but that was not the case. Furthermore, I expected to see agreement between the chi-square tests of the Level 1 "barriers" and "transfer" questions when tested with the Level 3 data. As it ended, the Level 1 "barriers" question and Level 3 self-reported data were dependent. In contrast the Level 1 "transfer" question and Level 3 self-reported data were independent. The findings from this study illustrate that attempting to predict training transfer from Level 1 responses is complex and requires the use of methodical assessments. It may not be enough to rely on self-reports to provide insight into training transfer. Although end-of-course questionnaires may be the most efficient method of collecting data, training professionals need to know that for various reasons, whether intentional or unintentional, there are many factors affecting the accuracy of self-reports. Admitting that Level 1 responses did not predict training transfer, still the most insightful discovery about training transfer was during the observations. The observations provided concrete data about some of the barriers prohibiting or preventing the use of the skills. Observations were also beneficial in calibrating the Level 1 data. Notwithstanding the results of this study, if self-reports are necessary due to practical considerations, training professionals should consult the literature regarding general reasons for response errors.

Degree

Ph.D.

Advisors

Russell, Purdue University.

Subject Area

Educational evaluation

COinS