Beyond Aggregated Data: A Study of Group Differences in Conceptual Understanding and Resource Usage in an Undergraduate Dynamics Course

Nick Alan Stites, Purdue University

Abstract

As pedagogical innovations continue to be developed and adopted in engineering education, it is important to understand how these innovations affect the students’ experiences and achievements. A common data analysis practice when evaluating educational innovations is to aggregate the data from all of the students together. However, this data aggregation inherently biases the results toward the characteristics of the dominant student group, leaving the experiences of minority groups largely unexplored. In this dissertation, I investigate the students’ experiences and achievements in an undergraduate dynamics course, and I intentionally use analysis methods that disaggregate the data to better understand the behaviors and performance of smaller subgroups of students, not just the majority.This dissertation presents three studies that examine: 1) the validity, reliability, and fairness of a standardized set of conceptual questions on the final exam, with a focus on gender fairness, 2) how and why the students use the available resources, and 3) how the students’ holistic resource usage patterns relate to their academic achievement. My motivation for choosing these studies was that conceptual assessments and customized resources are two key components of the learning environment for the dynamics course. However, the quality of the conceptual exam questions used for the course had yet to be evaluated. Similarly, the learning environment for the course incorporates many customized resources, including a custom-written “lecturebook” (a hybrid of a textbook and a workbook) and an extensive online library of videos, but little was known about how the students used these resources, or how the students’ pattern of resource usage related to their performance in the course.The first study in this dissertation used multiple-group confirmatory factor analysis to investigate item-level gender bias in a 12-item Abbreviated Dynamics Concept Inventory (aDCI), which was a set of standardized conceptual questions included on the final exam. The results suggested that two items were slightly biased against women, with stereotypically-masculine contexts and content as possible sources of the bias. The bias in the aDCI items likely unfairly lowered some women’s final exam scores, highlighting the need for engineering educators to consider the fairness of their assessments.The second study used a cluster analysis of survey responses to identify nine archetypical patterns of resource usage, all of which differed from the average resource-usage pattern of the aggregated sample. An analysis of forty-four student interviews, organized by resource-usage cluster, determined that students exhibited their resource-usage behaviors largely because of how they perceived the resource’s availability, accessibility, and quality. The results illustrate that there is no “typical” way in which the students used the resources, so it is important for instructors to consider a wide array of usage behaviors when designing a course’s learning environment and resources.The third study utilized a multiple regression analysis to find that on average a student’s resource-usage pattern is not related to their achievement when controlling for many other demographic, cognitive, and non-cognitive factors that can affect resource usage and performance. However, two individual resource-usage patterns were significantly related to achievement. Students who primarily used their lecturebook and their peers for support performed better than their similar peers in other resource-usage clusters. Conversely, students who rarely used their lecturebook had lower course grades than their peers. Drawing from the results of the second study, general study-habit suggestions for the students in the course were extracted from the qualitative themes found in the interviews of the students in these two clusters.

Degree

Ph.D.

Advisors

Berger, Purdue University.

Subject Area

Demography|Sociology

COinS