Write a 250 word summary of the ‘The Longitudinal Trajectories of Online Engagement Over a Full Program’ article I have attached. Include a discussion of the

Computers & Education 175 (2021) 104325

Available online 11 September 2021 0360-1315/© 2021 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

The longitudinal trajectories of online engagement over a full program

Mohammed Saqr a,b,*, Sonsoles López-Pernas c

a KTH Royal Institute of Technology, EECS – School of Electrical Engineering and Computer Science, Lindstedtsvägen 3, SE-100 44 Stockholm, Sweden b University of Eastern Finland, School of Computing, Joensuu, Yliopistokatu 2, Joensuu, fi-80100, Finland c Universidad Politécnica de Madrid, ETSI Telecomunicación, Departamento de Ingeniería de Sistemas Telemáticos, Avda. Complutense 30, 28040, Madrid, Spain

A R T I C L E I N F O

Keywords: Longitudinal engagement Trajectories of engagement Learning analytics Sequence mining Survival analysis

A B S T R A C T

Student engagement has a trajectory (a timeline) that unfolds over time and can be shaped by different factors including learners’ motivation, school conditions, and the nature of learning tasks. Such factors may result in either a stable, declining or fluctuating engagement trajectory. While research on online engagement is abundant, most authors have examined student engagement in a single course or two. Little research has been devoted to studying online lon- gitudinal engagement, i.e., the evolution of student engagement over a full educational program. This learning analytics study examines the engagement states (sequences, successions, stability, and transitions) of 106 students in 1396 course enrollments over a full program. All data of students enrolled in the academic year 2014–2015, and their subsequent data in 2015–2016, 2016–2017, and 2017–2018 (15 courses) were collected. The engagement states were clustered using Hidden Markov Models (HMM) to uncover the hidden engagement trajectories which resulted in a mostly-engaged (33% of students), an intermediate (39.6%), and a troubled (27.4%) trajectory. The mostly-engaged trajectory was stable with infrequent changes, scored the highest, and was less likely to drop out. The troubled trajectory showed early disengagement, frequent dropouts and scored the lowest grades. The results of our study show how to identify early program disengagement (activities within the third decile) and when students may drop out (first year and early second year).

1. Introduction

As online learning environments became increasingly common, along with all types of technology-enhanced learning (Palvia et al., 2018), a wealth of online data has driven interest in using modern analytical methods for understanding, profiling, and supporting student engagement (Barthakur et al., 2021; Gašević, Jovanović, Pardo, & Dawson, 2017; Henrie, Halverson, & Graham, 2015). Research in learning analytics has produced valuable insights regarding, e.g., engagement patterns, students’ profiles, and relationship of engagement with achievement (Barthakur et al., 2021; Henrie et al., 2015; Pardo et al., 2015). Such insights have relied on a diverse range of methods such as clustering, sequence mining, process mining, and predictive modelling (Bos, 2016; Cornwell, 2018; Gašević

* Corresponding author. University of Eastern Finland, School of Computing, Joensuu Campus, Yliopistokatu 2, fi-80100 Joensuu, Finland P.O. Box 111, fi-80101, Joensuu, Finland.

E-mail address: [email protected] (M. Saqr).

Contents lists available at ScienceDirect

Computers & Education

journal homepage: www.elsevier.com/locate/compedu

https://doi.org/10.1016/j.compedu.2021.104325 Received 7 April 2021; Received in revised form 28 August 2021; Accepted 4 September 2021

Computers & Education 175 (2021) 104325

2

et al., 2017; López-Pernas, Saqr, & Viberg, 2021; Matcha, Gašević, Uzir, Jovanović, & Pardo, 2019). A pervasive pattern of such research is that they have examined individual courses, or few courses with different students (Barthakur et al., 2021). While such studies have shed light on students’ learning strategies, approaches to learning, or engagement, little is known about the longitudinal online engagement in a full year of studies, a curriculum, or a program (Barthakur et al., 2021). The abundance of online data enables researchers a scalable method for collecting and analyzing data about learners. Additionally, the study of longitudinal engagement (or disengagement) enables researchers to understand learning strategies at the program scale and offer solutions to problems like disengagement, attrition, or poor academic performance in a program (Archambault & Dupéré, 2017; Barthakur et al., 2021; Gašević et al., 2017; Zhen et al., 2020).

Building on the existing body of literature, this article aims to study the longitudinal online engagement of students in a full ac- ademic program. Our method follows the latest research in the field of learning analytics and extends the existing methods with modern sequence typology methods. In particular, we use Latent Class Analysis (LCA) (Goodman, 1974; McCutcheon, 1987) to reveal the clusters of students’ learning strategies and engagement patterns in each course (Barthakur et al., 2021; Carroll & White, 2017; Park, Yu, & Jo, 2016). We further study the succession of engagement states using sequence pattern mining methods. We take advantage of Hidden Markov Models (HMM) to uncover the hidden engagement trajectories (sequence of similar engagement patterns across the program or typologies) (Helske et al., 2015, 2018). HMM has the advantage of being able to reveal hidden longitudinal patterns, as well as summarize complex information. For each trajectory, we study the patterns of engagement, stability, and de- viations (transitions from a trajectory to another). We conclude by studying the fate of engagement trajectories and their relation to performance and dropout. The contribution of this study is twofold: 1) it contributes to the body of the literature on online engagement trajectories, the criteria of the trajectories, stability thereof, and fate, and 2) it demonstrates the sequence typology methods and the wealth of insights they can bring. The research questions of our study are:

RQ1. What are the engagement states at the course level and what are their defining features?

RQ2. How stable are the learners’ engagement states at the course level (within and between students) and, if unstable, how do these engagement states change?

RQ3. What are the trajectories of students’ engagement across the full program? What are their defining features? How stable are these trajectories and, if unstable, how do they change?

RQ4. What is the temporal relationship between engagement trajectories and dropping out of the program or academic achievement (GPA)?

2. Background

2.1. Engagement as a construct

Many scholars posit that engagement is a “meta” concept that integrates –or overlaps with– other theoretical constructs such as motivation and self-regulation (Finn & Zimmer, 2012; Fredricks, Blumenfeld, & Paris, 2004). According to Fredricks et al. (2004), engagement is a multidimensional construct with integrated sub-components of behavior, emotion, and cognition. These components are dynamically intertwined and linked within the individual (Fredricks et al., 2004; Fredricks & McColskey, 2012). The behavioral component reflects following school norms, efforts to study, or participation in academic activities. Such behavior can be observed in regular attendance, participation, and commitment to homework and assignments. In online settings, behavioral engagement mani- fests in, e.g., students’ use of online resources, participation, regularity, and the patterns thereof (Azcona, Hsiao, & Smeaton, 2019; Rienties, Tempelaar, Nguyen, & Littlejohn, 2019). Emotional engagement reflects learners’ feelings about school, peers, and their learning experience, which includes emotional reactions such as interest, disinterest, happiness, sadness, and anxiety. Emotional engagement helps students engage behaviorally in school activities, feel belonging, and persist in education. Cognitive engagement reflects students’ efforts which can range from memorizing and trying to understand learning tasks to investing thoughtful energy to grasp advanced concepts or acquire complex skills. Cognitive engagement also includes the use of self-regulated learning strategies, preference for challenge, and positive coping in tough situations. As understanding of the concepts evolved, newer definitions emphasized the multidimensional nature of engagement (Finn & Zimmer, 2012; Fredricks et al., 2004; Fredricks & McColskey, 2012; Furrer & Skinner, 2003). Such dimensions are dynamically closely interrelated, e.g., positive feelings about school motivate behavioral engagement in school activities, and cognitive engagement (Azevedo, 2015; Finn & Zimmer, 2012; Fredricks et al., 2004). Researchers have demonstrated that all dimensions of engagement (behavioral, emotional, and cognitive) are significant catalysts of academic achievement. Such a relationship has been repeatedly confirmed in all levels of education (King, 2015; Lei, Cui, & Zhou, 2018; Wang & Degol, 2014).

Engagement has obvious qualities that can be observed, tracked, and easily understood by teachers (Fredricks & McColskey, 2012; Henrie et al., 2015). Since engagement is malleable, disengaged students are amenable to intervention and support (Finn & Zimmer, 2012; Fredricks et al., 2004). Furthermore, engagement may be seen as mediated by motivation. That is, students who have a positive belief about their capacity to succeed (academic self-concept) or see their learning tasks as valuable or enjoyable (subjective task value) are more likely to be engaged and succeed (Gottfried, Fleming, & Gottfried, 2001; Wang & Eccles, 2013). Therefore, an intervention that promotes students’ motivation, or improves school environment or learning tasks has the potential to enhance engagement and academic achievement (Gottfried et al., 2001).

M. Saqr and S. López-Pernas

Computers & Education 175 (2021) 104325

3

2.2. Engagement in online learning

Online learning has become an essential part of today’s education, courses and programs supported by technology or exclusively delivered online continue to grow in size, scale, and adoption (Escueta, Quan, Nickow, & Oreopoulos, 2017; Palvia et al., 2018). A remarkable upsurge in online education has been seen since the outbreak of the COVID-19 pandemic (Saqr & Wasson, 2020). The adoption of online learning has brought several opportunities, e.g., enabling the provision of learning resources that transcended the barriers of place and time, offering rich interactive multimedia, and facilitating collaborative learning (Escueta et al., 2017; López-Pernas, Gordillo, Barra, & Quemada, 2021; Schöbel et al., 2020). However, online learning requires certain qualities to succeed, e.g., self-regulation, agency, and engagement. Therefore, it has proven challenging for some students (Bol & Garner, 2011; Dabbagh & Kitsantas, 2004). As such, researchers and educators have searched for ways to better support students’ online learning, and therefore engagement has gained significant momentum (Bol & Garner, 2011; Dabbagh & Kitsantas, 2004).

Learning analytics affords researchers a seamless and unobtrusive “behind the scenes” method of collecting fine-grained data about learners’ behavioral engagement and holds the promise for optimizing learning and learning environments (Henrie et al., 2015). Such fine-grained data about students’ engagement can be collected in real time enabling the provision of timely feedback (Rienties et al., 2019). What is more, such feedback can be personalized, automated, and delivered at scale. Therefore, a significant corpus of research has examined students’ engagement in online learning. Such research has shed light on the measurement of engagement, early signs of disengagement, and the prediction of disengagement, to mention a few (Henrie et al., 2015; Lei et al., 2018; Saqr, Fors, & Tedre, 2017).

2.3. Person-centered methods

Variable-oriented methods capture the relationships and interactions among variables. Thus, variable-oriented methods are immensely helpful since they can produce valuable information about what is expected from a group of learners on average (Hick- endorff, Edelsbrunner, McMullen, Schneider, & Trezise, 2018; Rosato & Baer, 2012). However, “oftentimes, an ‘average’ learning pattern is not an adequate description for many learners because this ignores the unobserved heterogeneity between learners” (Hickendorff et al., 2018, p. 5). Therefore, variable-oriented methods have been criticized for being a misleading over-generalization: “that study findings represent the overall sample” (Rosato & Baer, 2012, p. 61). Person-centered methods, commonly implemented as Latent Class Analysis (LCA) or Latent Profile Analysis, aim at capturing distinct patterns of heterogeneity and variations within human behavior and experiences (Hagenaars and Jacques A, 2002; Hickendorff et al., 2018; Rienties et al., 2019; Rosato & Baer, 2012). In doing so, person-centered methods help researchers discover latent classes or “hidden subgroups” based on combinations of shared observable characteristics (Goodman, 1974; Hickendorff et al., 2018; McCutcheon, 1987; Rosato & Baer, 2012).

2.4. Engagement as a heterogeneous process

Using person-centered methods (e.g., LCA) to study engagement has the potential to advance our understanding of the hetero- geneity and diversity within students’ subpopulations and consequently offer the appropriate support (Wang & Degol, 2014). Evidence of sub-populations of students’ engagement has been reported by, e.g., Wang and Eccles (2013), who used latent profile analysis to find students’ engagement profiles. The authors reported five clusters of engagement profiles with varying patterns that influenced their educational and psychological functioning (Wang & Eccles, 2013). Rienties et al. (2019, p. 245) used person-centered methods to identify “distinct clusters of behavioral engagement in an online e-tutorial”. Further evidence of the heterogeneity of engagement among students’ sub-population was also reported in longitudinal studies (Archambault & Dupéré, 2017; Zhen et al., 2020). Zhen et al. (2020) reported four profiles of students’ engagement: persistent, climbing, descending, and struggling. Archambault and Dupéré (2017) reported five groups: stable high, stable moderate, transitory inclining, transitory declining, and constantly declining.

The heterogeneity within engagement profiles of online learners has also been a common finding among learning analytics re- searchers who reported distinct profiles of learners using other person-centered techniques such as clustering and sequence mining based on learners’ online activities, use of resources, and navigational behavior. Three levels of engagement (or subgroups) are commonly identified (Barthakur et al., 2021; Jovanović, Dawson, Joksimović, & Siemens, 2020, 2017; Kovanović et al., 2019; López-Pernas, Saqr, & Viberg, 2021; Matcha, Gašević, Uzir, Jovanović, & Pardo, 2019): 1) active, intense, or highly-engaged learners, who are intensive users of available resources and invest a considerable amount of energy in their approach to learning, 2) disengaged learners, who show low levels of participation and invest less time and effort doing learning activities, and 3) a third group of selective learners, who invest average time and efforts in their learning, directing most of their efforts to accomplish their tasks with reasonable efforts.

2.5. The trajectories of engagement

Engagement has a trajectory (a pathway or a timeline) that unfolds over time and can be shaped by different factors including learners’ motivation, school conditions, teachers, peers, social context, and the nature of learning tasks (Fredricks et al., 2004; Saqr, Nouri, & Jormanainen, 2019; Wang & Degol, 2014; You & Sharkey, 2009). Such factors may result in either a stable engaged tra- jectory, a declining trajectory, or a fluctuating trajectory between states over time (Finn & Zimmer, 2012; Skinner & Pitzer, 2012; Wang & Degol, 2014). So far, research in face-to-face settings is inconclusive regarding the longitudinal trajectory of engagement. Some scholars suggest that engagement shows marked inter-individual stability with low variation between the years, and high cross-time correlation (Gottfried et al., 2001; Janosz, Archambault, Morizot, & Pagani, 2008; Skinner & Belmont, 1993). That is, for an

M. Saqr and S. López-Pernas

Computers & Education 175 (2021) 104325

4

individual student, engagement during an academic year predicts engagement at the next year (Gottfried, Marcoulides, Gottfried, Oliver, & Guerin, 2007; Skinner & Pitzer, 2012). Others posit that engagement declines through the years (Wigfield, Eccles, Schiefele, Roeser, & Davis-Kean, 2007), while some suggest that engagement can grow under optimal conditions (You & Sharkey, 2009). Recently, some researchers have pointed to a heterogeneous pattern in which different groups of students exhibits different engagement profiles (Archambault & Dupéré, 2017; Li & Lerner, 2011; Pagani, Fitzpatrick, & Parent, 2012; Wigfield et al., 2007; Zhen et al., 2020), where some students maintain a stable state of engagement (usually the majority) and another group shows variations: ascending to higher engagement levels or declining to disengagement (Archambault & Dupéré, 2017; Li & Lerner, 2011; Zhen et al., 2020). The aforementioned studies have been conducted in face-to-face settings, while online longitudinal engagement over a full program remains an area in need of further research.

2.6. Modelling trajectories of engagement

Accounting for the sequential relationship between states/events is particularly important when modelling the timeline of certain observations or behavior. Therefore, sequence mining, a data analysis method that emphasizes the sequential order, was conceptu- alized (Agrawal & Srikant, 1995). Typology is a method aimed at identifying the patterns of the sequence of events or the trajectories. Such methods have been well-established in the study of longitudinal life events such as career pathways or sequences of marital life events (Helske, Helske, & Eerola, 2018; Malin & Wise, 2016; Ritschard & Studer, 2018). Since we are interested in revealing the trajectory of longitudinal engagement over a program, we used HMM, a method that has been used in modelling life events, as well as in several other fields, to model, e.g., temporal processes. HMMs have been frequently used to reveal students’ profiles (Jeong et al., 2008), and have proven particularly useful in mining longitudinal sequential patterns or trajectories (Helske et al., 2018; Jeong et al., 2008). In HMM, sequence data are observed states, which are considered probabilistic functions of hidden (latent) states (trajectories). Such “hidden states cannot be observed directly, but only through the sequence(s) of observations”. A trajectory is a typical succession of similar behaviors that are as homogeneous as possible as well as distinct from other trajectories: for instance, a trajectory of a career pathway. The hidden states are hypothesized to be generated by a similar mechanism, thus resulting in a certain trajectory. Furthermore, HMM can reveal the unobservable state in data and has a summarizing power that allows to compress information from different types of observations into meaningful insights (Helske & Helske, 2019). In this study, we use a combination of methods to identify students’ engagement states, describe the engagement states’ characteristics, group such states into homogeneous trajectories, and study such trajectories. The methods are described in detail in the next section.

2.7. Motivation for this research

The majority of existing works have focused on individual courses, with an obvious scarcity of research on the longitudinal engagement across a program. However, few examples exist. Lust, Elen, and Clarebout (2013) examined learners’ patterns of use of an online learning tool across two subsequent phases of a single blended learning course. They identified three distinct clusters of learners during the first phase whereas, in the second phase, they found an additional cluster, indicating changes in students’ patterns of engagement with the online tool across the two phases of the course. To address this dearth in existing literature, Barthakur et al. (2021) sought to cluster learners based on their program-level engagement profiles using their weekly activities across a full year online Massive Open Online Course (MOOC). They found three distinct clusters of learners according to their program-level strategies: 1) consistently engaged: students who engaged with the various learning activities throughout each course and across the program; 2) disorganized: students who were mostly disengaged all the way through and sometimes slightly more active towards the end of each course, and 3) get-it-done: students were mostly active at the start of each course across the program, focusing on the assessment components and disengaged for the rest of the course.

3. Methods

3.1. Context

The context of our study was Qassim University. While it was a blended context, the online component was an integral part of the teaching strategy. Since the program was based on Problem-Based Learning (PBL), students were required to read and contribute to the weekly online PBL discussions several times a week. The content and objectives of the online PBL were the subjects of the lectures, seminars, and practical sessions. While some of these were delivered face-to-face, students had to extract the learning objectives, lectures, and handouts from the online Learning Management System (LMS), as well as get course updates, follow teachers’ an- nouncements, and interact with colleagues. Therefore, engagement with the online component was an integral element of the pro- gram. Contributing to online PBL was “required” and students got a grade of (5%) for contributing and replying to colleagues. The online lectures were important (but not mandatory) as they were the main source of learning resources. The LMS also contained the learning objectives that students had to achieve as well as the schedules and deadlines of the various assessments of the courses. The courses were homogeneous and shared the same teaching strategy (PBL), organization of PBL sessions, and teaching and assessment methods including the breakdown of assessment methods. There were around four courses (always referred to as blocks) each year with durations that averaged six weeks. The courses were sequential, and students had to complete them in order. Some practical courses were longitudinal (full year) and were taught in parallel to the ongoing course (e.g., clinical skills). Such courses were excluded from our study as they did not share the same teaching strategy, had different online content, and had an evaluation method based on

M. Saqr and S. López-Pernas

Computers & Education 175 (2021) 104325

5

practical performance. Students’ performance was measured by the course grade, consisting of separate grades for 1) the final exam, 2) the level of

engagement in the online forums, and 3) students’ continuous assessment with the learning tasks and interaction during class. The final exam accounted for 80% of the overall grade, whereas the latter two components accounted for the remaining 20%, distributed as follows: 10% continuous assessment (e.g., practical assignments, seminar preparations, and engagement in lectures and course duties), 5% for participation in online forums (based on posting and responding to colleagues), and 5% for face-to-face PBL group sessions, and engagement in lectures and seminars. The GPA is the total grade for all courses over the full program. In the program under study, students who do not attend a course or fail get only 60% of the grade if they retake it. Withdrawn means excused to attend the course while keeping all grades. This could be a sick student, absent due to travelling, or having a personal issue. They are allowed to have or “re-take” a special exam. These exams and studies are done privately and so their data are not represented online. Including these students (although very few) would show up as disengaged (erroneously).

3.2. Data collection and operationalization

The data were collected from the institution’s LMS (Moodle) for all courses of all students enrolled in the academic year 2014–2015, and their subsequent data in 2015–2016, 2016–2017, and 2017–2018 (15 courses). Only learning-related events in Moodle logs were considered, in other words, teacher ‘events’ such as clicks on profiles pages, chats, and checking grades were not included in the analysis. In addition, very infrequent events (e.g., clicks on inactive wiki modules, workshops, and labels) were removed as they could not be used in the analysis. We operationalized online engagement –which is essentially behavioral engagement– following the dominant view of the literature on online learning (Azevedo, 2015; Henrie et al., 2015), as Henrie et al. (2015) sum- marized it in their literature synthesis on measuring engagement in online learning, “by computer-recorded indicators such as as- signments completed; frequency of logins to the website; number and frequency of postings, responses, and views; number of podcasts, screencasts, or other website resources accessed; time spent creating a post; and time spent online.” It is worth mentioning that such type of data can only capture behavioral engagement and less so cognitive engagement. To ensure we captured the full gamut of students’ activities in our PBL program, we collected three categories of indicators with eight variables for each student in each course:

Frequency of activities

These indicators reflect students’ investment in coursework, participation with colleagues, and contribution to the collaborative process (Redmond, Heffernan, Abawi, Brown, & Henderson, 2018; Saqr, Viberg, & Vartiainen, 2020).

• Frequency of Course Browsing: The number of times a student viewed the course main page, which acts as the gateway for all other resources, displays course announcements and updates (e.g., new announcements by the teacher, new lectures, posts from peers, assignments, etc.).

• Frequency of Forum Consumption: The number of times a student read other students’ forum posts. “Reading forums” is oper- ationalized as social engagement.

• Frequency of Forum Contributing: The number of times a student actively contributed to the forum content, i.e., created, updated, edited, or deleted posts. It is operationalized as student participation in the course collaborative activities, efforts to contribute with colleagues.

• Frequency of Lecture View: The number of times a student clicked on a learning resource, such as a recorded lecture or a “folder” with learning resources (group of learning materials).

2.4. Online time

These indicators reflect students’ on-task time and effort in online activities (Azevedo, 2015; Henrie et

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.