The development of a content analysis model for assessing students' cognitive learning in asynchronous online discussions

Dazhi Yang, Purdue University

Abstract

The purpose of this study was to develop, validate, and test a content analysis model (framework) for assessing students' cognitive learning in asynchronous online discussions (AODs). The study adopted a fully mixed methods design, in which qualitative and quantitative methods were employed sequentially (a two-part process) at stages of data analysis and data interpretation (Hanson, Creswell, Plano-Clark, Petska & Creswell, 2005). Specifically, the design was a “sequential exploratory” (QUAL→ quan) mixed methods design (Hanson et al., 2005, p. 229) with priority given to the qualitative data and method, which was the first or qualitative process. Quantitative data supplemented the qualitative data. The data were students' online postings generated and collected previously in two online courses that used AODs as a main instructional method to carry out teaching and learning activities. The online postings were divided into two sets. The first set included 800 online postings, which were selected to develop a content analysis model (framework) and are referred to as the qualitative data. The second set included the remaining online postings (N=803), which were used to validate the newly constructed model (framework) and are referred to as the quantitative data. During the first or qualitative process, a grounded theory approach was adopted to find the components, in terms of knowledge acquisition and cognitive skills, for assessing students' cognitive learning in AODs, which was to develop a content analysis model (framework). The newly constructed framework has a two-dimensional structure (Anderson & Krathwohl, 2001) consisting of knowledge and cognitive skills, which is different from other existing content analysis models or frameworks for assessing AODs. The newly constructed framework has three categories of knowledge and five categories of cognitive skills and a total of 17 sub-categories. The three knowledge (K) categories (components) are (1) Factual Knowledge (FK), (2) Conceptual Knowledge (CK), and (3) Procedural Knowledge (PK). Each of the three knowledge categories has its respective sub-categories (indicators). In total, there are seven sub-categories of knowledge. The five cognitive skills (CS) categories (components) are (1) Sharing/Describing; Seeking information/solutions (CS-SDS), (2) Explaining; Comparing/Interpreting/Clarifying (CS-ECIC), (3) Analyzing; Concluding (CS-AC); (4) Applying (CS-A), and (5) Creating (CS-C). Each of the five cognitive skills categories has its respective sub-categories (indicators). In total, there are ten sub-categories of cognitive skills. During the second or quantitative process, chi-square tests were performed to check if all the 17 sub-categories in the newly constructed framework appeared in the second set or the quantitative data. All sub-categories except two were manifest according to the outcomes of chi-square tests. Subsequently, eight sub-categories of knowledge and cognitive skills were regrouped and one sub-category was excluded from the newly constructed framework according to the outcomes of the descriptive data analysis (proportions of occurrences) and chi-square tests. The regrouping and the resulting exclusion yielded an alternative measurement model with the same eight categories of knowledge and cognitive skills but with a total of 11 sub-categories. Finally, a confirmative factor analysis (CFA) was conducted to test if the alternative measurement model fit the quantitative data. The outcomes of the CFA indicated marginal fit of the alternative measurement model to the data. However, caution should be exercised in the adoption and use of the alternative measurement model (content analysis framework) in practice. Implications of the study include (1) suggestions for methods of; (a) assessing students' cognitive learning in AODs, and (b) designing discussion topics and questions; and (2) the research design's potential to help improve future studies in content analysis models or frameworks.

Degree

Ph.D.

Advisors

Richardson, Purdue University.

Subject Area

Educational psychology|Educational technology|Curriculum development

COinS