Learning Analytics Approaches for Decision-Making in First-Year Engineering Courses

Laura Melissa Cruz Castro, Purdue University

Abstract

First-Year Engineering (FYE) programs are a critical part of engineering education, yet they are quite complex settings. Given the importance and complexity of FYE programs, research to better understand student learning and inform design and assessment in FYE programs is imperative. Therefore, this dissertation showcases various uses of data analytics and educational theory to support decision-making when designing and assessing FYE programs. Three case studies shape this dissertation work. Each study encompasses a variety of educational data sources, analytical methods, and decision-making tools to produce valuable findings for FYE classrooms. In addition, this dissertation also discusses the potential for incorporating data analytics into FYE programs. A more detailed description of the research methods, a summary of findings, and a list of resulting publications for each case study follows.The first case study investigated the relationship between two related Computational Thinking (CT) practices, data practices and computational problem-solving practices, in acquiring other CT competencies in a large FYE course setting. This study explored the following research questions: (1) What are the different student profiles that characterize their foundational CT practices at the beginning of the semester? and (2) Within these profiles, what are the progressions that students follow in the acquisition of advanced CT practices? To answer these questions, N-TARP Clustering, a novel machine learning algorithm, and sound statistical tools were used to analyze assessment data from the course at the learning objective level. Such a hybrid approach was needed due to the high-dimensionality and homogeneity characteristics of the assessment. It was found that early mastery of troubleshooting and debugging is linked to the successful acquisition of more complex CT competencies. This research was published in an article in the journal IEEEAccess.The second case study examined self-regulation components associated with students' successful acquisition of CT skills using students' reflections and assessment data. This research was grounded in three subprocesses of the Self-Regulated Learning (SRL) theory: strategic planning, access to feedback, and self-evaluation. This study responded to the following research question: What is the relationship between SRL subprocesses: access to feedback, self-evaluation, strategic planning, and the acquisition of CT skills in an FYE course? Results from a structural equation model, which reflects the complexity and multidimensionality of the analysis, provided evidence of the relevance of the three subprocesses in the acquisition of CT skills and highlighted the importance of self-assessment as key to success in the acquisition of programming skills. Furthermore, self-assessment was found to effectively represent the task strategy and access to feedback from the students. This analysis led to the understanding that even though the three SRL subprocesses are relevant for the student's success, self-evaluation serves as a catalyst between strategic planning and access to feedback. A resulting article from this case study will be submitted to theInternational Journal of Engineering Educationin the future.Lastly, the third study aimed to predict the students' learning outcomes using data from the Learning Management System (LMS) in an FYE course. The following research questions were explored in this case study: (1) What type of LMS objects contain information to explain students' grades in a FYE course? (2) Is the inclusion of a human operator during the data transformation process significant to the analysis of learning outcomes? Two different sections of a large FYE course were used, one serving as a training data set and the other one as a testing data set. Two logistic regression models were trained. The first model corresponded to a common approach for building a predictive model, using the data from the LMS directly. The second model considered the specifics of the course by transforming the data from aggregate user interaction to more granular categories related to the content of the class. A comparison was made between the predictive measures, e.g., precision, accuracy, recall, and F1 score for both models. The findings from the transformed data set indicate that students' engagement with the career exploration curriculum was the strongest predictor of students' final grades in the course. This is a fascinating finding because the amount of weight the career assignments contributed to the overall course grade was relatively low. This study will be presented at the 2022 American Society of Engineering Education (ASEE) national conference in Minneapolis, Minnesota.

Degree

Ph.D.

Advisors

Diefes-Dux, Purdue University.

Subject Area

Educational psychology|Educational technology|Management|Psychology

COinS