What impacts learning effectiveness of a mobile learning app focused on first-year students?

The use of mobile apps in higher education teaching—such as “learning management applications”, “vodcasts and podcasts”, “language learning applications”, “game-based learning applications” or “collaborative learning applications” (Goundar and Kumar 2022)—is discussed lively in the literature (e.g., Beatson et al. 2020; Gupta et al. 2021; Liu and Guo 2017; Ronzhina et al. 2021; Voshaar et al. 2023). In the following, we summarize related work on mobile learning apps in terms of potentials and challenges, technical and organizational issues, associated theories, as well as future developments. This should give readers a better overview of this mature research area.

In general, the potential of mobile student apps to support learning effectiveness is well-analyzed for various classroom and course examples (cf. Castek and Beach 2013). For instance, Larkin (2015) evaluates apps to foster the building of mathematical knowledge, while Diliberto-Macaluso and Hughes (2016) show that mobile apps may help psychology students achieve their learning objectives. Hence, apps can help to develop students’ self-regulation and deep thinking abilities or support them in labeling, summarizing, and discovering new knowledge amongst others (cf. Diliberto-Macaluso and Hughes 2016; Larkin 2015). In medical education, the benefits of mobile apps to offer an “enjoyable learning experience” are pointed out by Morris et al. (2016) for a neuroanatomy course. Mohapatra et al. (2015) present an overview of apps that are judged to be beneficial for medical education in general, with a particular focus on their ability to manage information from one or more sources to foster communication and support effective time management. Steel (2012) focuses on language students in particular and discusses the potential of mobile apps for this group, e.g., in terms of vocabulary acquisition. An overview of corresponding apps for language students is given by Gangaiamaran and Pasupathi (2017). In accounting and management, Beatson et al. (2020) and Voshaar et al. (2023) find out that students’ behavioral engagement with the help of mobile apps and gamification elements is positively associated with exam results. Seow and Wong (2016) introduce the so-called “Accounting Challenge (ACE)” app, which helps to keep up students’ motivation in studying accounting through gamification as well.

On the contrary, there are challenges of using mobile apps for student education. As Goundar and Kumar (2022) point out, the literature to date has a strong focus on “solution papers”, which introduce fully developed mobile applications that are supposed to improve learning performance. However, a discussion as to what degree singular app functionalities affect students’ cognitive knowledge processing or an explication of the implications for learning theories often come up short (e.g., Damyanov and Tsankov 2018). Along these lines, Mehdipour and Zerehkafi (2013) provide technical as well as social and educational challenges for mobile learning scenarios. These include content security and copyright issues, accessibility and cost barriers for end-users, or the lack of a learning theory for the mobile age in general, to mention just a few (cf. Mehdipour and Zerehkafi 2013). Furthermore, digital technologies may not adequately reproduce the emotional side of interactive learning, so attention should be given to the right balance between digital and human educational interactions (Montiel et al. 2020). A classification scheme for mobile learning challenges according to “management and institutional challenges”, “design challenges”, “technical challenges”, “evaluation challenges”, and “cultural/social challenges” is introduced by Damyanov and Tsankov (2018). In summary, education institutions need to establish a clear mobile learning policy, offer pedagogical support, consider the hardware capabilities of mobile devices, provide a suitable technical infrastructure, and deal with the cultural differences concerning perceptions and attitudes towards digital technologies (cf. Damyanov and Tsankov 2018).

From a technical perspective, the requirements on mobile learning environments, the core functionalities of apps to assure their practicability for educational purposes, and engineering processes for app realization are particularly important. In this context, Zhu et al. (2015) propose a design framework for mobile augmented reality education in healthcare. Further, Clayton and Murphy (2016) analyze mobile apps’ peer-learning and -teaching capabilities for conducting collaborative video design projects. The establishment of a content delivery infrastructure for educational material and suggestions on integrating mobile apps is done by Khaddage et al. (2011). Vázquez-Cano (2014) focuses on the mandatory capabilities of smartphones to support distance learning, while Pechenkina et al. (2017) identify the potential of gamification elements to increase student engagement, retention, and achievement. Finally, Papanikolaou and Mavromoustakos (2006) introduce critical success factors for learning app engineering processes, while Kumar and Mohite (2018) suggest approaches for testing their usability.

From an organizational perspective, the factors for successfully adopting digital technologies in higher education institutions are discussed (e.g., Chuchu and Ndoro 2019). It is accentuated that mobile learning initiatives are not limited to purchasing and deploying digital technologies but require a holistic consideration of diverse factors related to people, technology, or pedagogy (Krotov 2015). Thereby, principal factors that may impact user satisfaction, the intention to use, and the actual usage of mobile applications in higher education are examined (e.g., Almaiah and Alismaiel 2019; Chuchu and Ndoro 2019). As an example, Almaiah and Alismaiel (2019) focus on Jordanian universities and analyze two apps—one that provides student services (e.g., a timetable) and another one enabling “open virtual classes”—in light of the abovementioned factors. Thereby, so-called “quality factors” and “individual factors” that have been adapted from Delone and McLean (1992) and Davis (1989) seem to have a positive effect. Besides, also the variable “intention to use” was examined for this specific student group (cf. Almaiah and Al Mulhem 2019). Further, Chuchu and Ndoro (2019) present indicators that the “perceived usefulness” and “perceived ease-of-use” of a mobile learning app are central factors in creating a positive attitude among the target group and in ensuring its acceptance. An overview of critical success factors for mobile learning in organizations is provided by Krotov (2015). This study integrates the perspectives “organization” (e.g., executive involvement), “people” (e.g., personal innovativeness), “pedagogy” (e.g., quality of content provided), and “technology” (e.g., quality of mobile system) to arrive at a list of success factors from a socio-technical perspective (cf. Krotov 2015).

Considering the complex process of establishing mobile education technologies in organizations, a pedagogical and educational requirements model was proposed by Sarrab et al. (2018), which supports when searching for a suitable solution to deliver content for mobile learning. Besides, the role of mobile apps in facilitating the inclusion of students with handicaps into the university environment is a subject of investigation. For instance, Ok et al. (2016) introduce an evaluation scheme to purposefully select apps for students with learning disabilities. Moreover, people with developmental disabilities can benefit enormously from mobile apps, which hold true for educational, communication, and leisure purposes, helping them connect with their environment (Stephenson and Limbrick 2015). In addition, Bravou and Drigas (2019) reflect the suitability of mobile devices and apps for students with sensory, physical, and cognitive disabilities. In this respect, a comprehensive literature review on digital technologies for people with learning or cognitive disabilities was performed by Williams and Shekhar (2019).

Researchers are also engaged in theory building (cf. Hevner and Chatterjee 2010) to guide the purposeful usage of mobile apps in higher education. However, a widely accepted mobile learning theory has not yet been established (cf. Bernacki et al. 2020; Curum and Khedo 2021). Therefore, Park (2011) refers to the transactional distance theory (cf. Moore 1991), which defines “distance” as a pedagogical concept, and combines this theory with applications of digital technologies to arrive at a “pedagogical framework of mobile learning”. The framework distinguishes between four types of mobile learning depending on whether a (1) high or (2) low transactional distance is given and (3) an individualized or (4) socialized activity is to be solved. Thereby, the transactional distance is defined as the psychological gap between the learner and the instructor, whereas the activity type (i.e., individualized or socialized) assesses the importance of social aspects for a particular learning environment (Park 2011). Another mobile learning framework was introduced by Motiwalla (2007), who proposes to integrate the concepts “mobile connectivity” and “e-learning” for being able to delineate application requirements for mobile learning. Furthermore, a meta-framework to guide the establishment of mobile learning frameworks can be found in Liu et al. (2008). This meta-framework is, for instance, referenced by Nordin et al. (2010) as a theoretical base to create a lifelong, continuing learning framework.

Future developments of mobile learning apps will essentially emphasize the integration of Artificial Intelligence (AI) with learning environments (cf. Alzahrani et al. 2021; Chong 2019; Diaz et al. 2015; Kabudi et al. 2021). The purpose is to improve students’ learning performance via personalization of learning, facilitate the evaluation of student knowledge, or systematically assess learner requirements (Kabudi et al. 2021). Besides, the use of virtual reality (VR) and augmented reality (AR) to progress students’ learning experiences is intensively discussed (e.g., Fradika and Surjono 2018; Nicolaidou et al. 2021). For instance, Nicolaidou et al. (2021) show that a VR learning environment can positively affect vocabulary acquisition and learners’ experience when studying foreign languages. Further, the readiness of students to adapt VR technology to achieve learning goals is high (Ismail and Hashim 2020). Moreover, the use of chatbots and conversational agents is also rising (Hwang and Chang 2021; Liu et al. 2020; Smutny and Schreiberova 2020). Chatbots can serve as efficient information retrieval tools for specific domains to facilitate learning (cf. Liu et al. 2020). In this context, various platform-specific chatbots for learning (e.g., for the Facebook Messenger platform) at different maturity levels have been developed in recent years (cf. Smutny and Schreiberova 2020). Having said that, chatbots for education are primarily found for language courses as well as the disciplines of “engineering” and “computers”, while topics like “arts” or “mathematics” are less accentuated (Hwang and Chang 2021). Thus, chatbots may not be suitable for all types of courses alike, especially in case students’ hands-on competencies (i.e., arts) or computations and problem-solving skills (i.e., mathematics) are to be promoted. While the effectiveness of chatbots for learning purposes is usually measured by pre-/post-test questionnaires, profound insights on chatbots’ impact on behavioral aspects of the student learning process are still elusive (Hwang and Chang 2021).

To conclude this overview, our study aims to analyze factors contributing to the success of a mobile app, which was designed to meet the needs of first-year students in the “transition-in” phase of the student lifecycle. A particular interest is in the ability to positively impact their learning effectiveness, user satisfaction, and intention to reuse the app. To the best of our knowledge, a corresponding study concerning this stage of the student lifecycle has not been done yet. We provide insights that can help establish a mobile learning theory in the “transition-in” phase.

Throughout university life, students experience an evolution of their “student identity”, which goes along with a shift of priorities and agendas (Lizzio 2011). As mentioned above, our research focuses on “commencing” students who are just about to become acquainted with the university system and have an increased interest in opportunities for social interaction, active engagement, and early formative feedback (Matheson 2018). Generally, various propositions regarding the development stages of students exist (cf. Burnett 2007; Morgan 2013) that primarily differ in their conception of student transition (Gale and Parker 2014). A widely acknowledged proposition for an integrative framework was introduced by Lizzio (2011), which is depicted in Fig. 1 and differentiates between four major stages. Whereas future students (“transition-towards”) are engaged in finding an appropriate study program and university, commencing students (“transition-in”) work on the integration into the student world (Lizzio 2011). In the “transition-through” phase, continuing students work on developing graduate attributes and seek challenges by authentic curricula and assessments (Lizzio 2011; Matheson 2018; Msomi and Bansilal 2019). Finally, the “transition-up, out & back” stage addresses students that are graduating or returning for postgraduate studies to further strengthen their skills for employability (Lizzio 2011; Matheson 2018).

Against this background, most premature dropouts are observed in the “transition-in” phase (Chen 2012; Isleib et al. 2019; Neugebauer et al. 2019). An empirical study focused on German higher education institutions (60 universities and universities of applied sciences) identified a lack of social and academic integration as a major reason for premature dropouts (Isleib et al. 2019). More specifically, differences in the perception of study requirements were observed for dropout and non-dropout first-year students. These observations are generally also confirmed for other countries (cf. Chen 2012; Kehm et al. 2019; Xenos et al. 2002; Zvoch 2006). In terms of academic integration, many first-year students obviously struggle with self-organizing their studies and balancing the time slots for attending courses, obtaining credits, and preparing or post-processing lectures (Schulmeister 2007). In such cases, the danger of not meeting the academic standard and the probability of an early dropout increases significantly (Kehm et al. 2019). Consequently, the importance of promoting student retention and student achievements has been identified as a crucial responsibility of higher education institutions at that point (Matheson 2018; Sheader and Richardson 2006).

In the “Student Adjustment Model” of Menzies and Baron (2014), the stages experienced by first-year students when entering the university system are specified more in-depth, which helps to gain a deeper understanding of the overall “transition-in” phase. Hence, upon arrival, a sense of excitement can be observed among first-year students, which comes to a halt after some weeks when first negative experiences in the new environment have been made—a phase called the “party’s over stage” (Menzies and Baron 2014). Now students need to realistically assess their capabilities, identify gaps (e.g., self-organization skills) and carefully reflect on the institutional requirements (Matheson 2018). Here, universities can support by providing informative student feedback and curricula that offer opportunities for social networking and learning, or teaching methods that foster active learning and encourage student engagement (Matheson 2018; Whittaker and Brown 2012). Afterwards, students are able to enter the so-called “healthy adjustment stage” (Menzies and Baron 2014).

Furthermore, the base competencies of first-year students have been a subject of investigation (cf. Krumrei-Mancuso et al. 2013; Zehetmeier et al. 2014), which provides valuable insights into the academic skills of young people that contemporaneously enter the university system. Whereas some studies focus on special types of competencies—like digital (e.g., Reddy et al. 2020) or leadership competencies (e.g., Smart et al. 2002)—a more extensive investigation at a German higher education institute, comprising 18 competency types in total, was performed by Zehetmeier et al. (2014). Concerning first-year students, deficiencies in self-organization, accurateness, perseverance, intrinsic motivation, or self-criticism were described (Zehetmeier et al. 2014). Based on these findings, universities should develop solutions that can account for diverse student backgrounds, tackle insufficient experiences, and support individual learning and self-organizational strategies to achieve academic success.

Considering this, we argue that a mobile app may be a suitable solution to tackle the abovementioned challenges (e.g., insufficient student experiences, lack of self-organization, etc.) in the “transition-in” phase of the student lifecycle. To provide a general overview, Table 1 gives a brief selection of campus apps that come to use at German universities. Of course, campus apps can be found internationally (e.g., UC San Diego mobile app) (e.g., Almaiah and Alismaiel 2019; Holotescu et al. 2018).

According to the mobile learning app ontology of Notari and Hielscher (2016), the majority of mobile campus applications are designed as “learning and teaching support apps” with diverging functionalities and purposes. As a common denominator, almost all of them provide campus maps, an overview of cafeteria offerings, official timetables, or event directories, while some of them (e.g., apps of the University of Arts Bremen, University of Hohenheim) also enable access to learning content or the registration for exams. However, communication functionalities are rare since students usually evade to commercial apps like Facebook and Instagram to share information with their peers (cf. Statista 2020b).

Besides, a broad range of commercial apps supports users in organizing their daily lives, e.g., for scheduling daily routines and tasks (e.g., “24me”, “Todoist”), structuring brainstorming ideas (e.g., “MindNode”), or tracking fitness activities (e.g., “FitNotes”, “MyFitnessPal”). Such mobile apps are these days usually well-integrated into young peoples’ lives (Goodyear et al. 2019; Statista 2020a). However, this does not necessarily hold true for campus apps that run danger of not being further developed as soon as students lose interest or their commitment to using the app (Potgieter 2015). That may be especially true if a campus app provides (redundant) content already readily available elsewhere (e.g., the university’s homepage or LMS).

In light of the above explanations, we aimed to provide a mobile app that supports first-year students academically during the “transition-in” stage of the student lifecycle and is easily and practically integrable into everyday student life to ensure long-term student acceptance and commitment. The app was designed to combine functionalities of commercial apps to organize daily routines (e.g., “24me”, etc.) with university-related content, functionalities (e.g., timetables, define tasks and goals, etc.), and gamification elements. The design and development of the app are also described in a prior work in more detail (cf. Johannsen et al. 2021).

Our app’s target user group is business and economics students at the University of Bremen (Germany). We initially focus on this narrow group because requirements can be specified more precisely, and the authors are well acquainted with study-related challenges of this relatively homogeneous user group. Though, adapting the app to the needs of other departments and universities is generally possible. The app was developed in a DSR project (Baskerville et al. 2018; Peffers et al. 2007) with the goal of supporting students’ experience, learning strategies, and self-organization in the “transition-in” phase. Considering this, our artifact is based on three meta requirements (cf. Gregor and Hevner 2013) to address the student factors “student experience”, “learning strategies”, and “self-organization”, which are of utmost importance to successfully tackle the transition into the university environment (cf. Blüthmann et al. 2011; Heinze 2018; Neugebauer et al. 2019) (Fig. 2). In DSR, meta requirements define “what the system is for” (Gregor and Jones 2007, p. 325) and outline the purpose and scope of the type of artifact to be developed (Gregor and Jones 2007; Schmid et al. 2022) in our case a mobile solution for first-year students. Referring to the abovementioned student factors that have been derived from the literature (see Sect. 2), the following meta requirements were formulated:

Thereby, the term “student experience” subsumes “all experiences of an individual student” while being in the “identity as a ‘student’” including all “facets of the university” (e.g., administrative processes, IT support etc.), which “contribute” to the “personal development” as a learner (Baird and Gordon 2009, p. 194).

To specify the meta requirements and arrive at design requirements for the app, (I) user stories, (II) market research, (III) user requirements, and (IV) user journeys were used (Schilling 2016). In this context, also second- and third-year undergraduates (N = 54), who are still well familiar with the challenges experienced at the beginning of their studies were surveyed. Finally, we came up with eight major design requirements classified into the categories “course attendance/reminders” (Fig. 2—DR 1-2), “support of study phases” (DR 3-5), and “technical requirements” (DR 6-8) to support students’ experience, learning strategies, and self-organization (cf. Fig. 2, Johannsen et al. 2021).

The architecture of the app consists of a front- and a back-end. The frontend was developed with the help of the IONIC Framework (https://​ionicframework.​com/​), which works based on Angular (https://​angular.​io/​) (Green and Seshadri 2013). Further, the back-end was realized via the Spring Framework (https://​spring.​io/​) and the Spring Boot solution (cf. Walls 2016). Figure 3 shows exemplary screenshots. So, a new course is added to a student’s timetable (Screenshot 1), and sample functionalities for this course—derived from the design requirements—are shown, such as the conduction of quizzes (Screenshot 2) or the comparison with a peer group (Screenshot 3). The app can be classified as type 2 (i.e., high transactional distance and individualized mobile learning activity) in the “pedagogical framework of mobile learning” of Park (2011). Hence, this type allows a high degree of flexibility and portability, enabling students to integrate it flexibly into their mobile lifestyle (Park 2011).

With respect to the challenges in the “transition-in” phase (see Sect. 2.2) and the meta requirements, various functions that support students’ learning strategies, experience, and self-organization are offered by the app. For instance, the features of tracking learning time along with an overview of exam dates and events largely foster students’ self-organization abilities (cf. Zehetmeier et al. 2014). This is further supported by push notifications or newsfeeds about new learning content as well as a calendar function with reminders for lectures and important academic dates contributing to student experience (e.g., Staddon and Standish 2012; Trotter and Roberts 2006). Gamification is used to enrich students’ learning strategies (e.g., performance tests via quizzes), while they can work on exercises independent of time and place.

We developed a questionnaire based on the abovementioned established and validated scales from previous studies and modified them accordingly for the mobile learning context to test our hypotheses. As previously described, the constructs of “system quality”, “service quality”, and “information quality” were adapted from Aparicio et al. (2016) and Urbach et al. (2010) and are all measured by four underlying items. “Perceived enjoyment” consists of five items, three being adapted from Kim et al. (2007) and Wang et al. (2019b), and two used by Suki and Suki (2007). Three items were adapted from Wang et al. (2019b) and Wang (2008) and are complemented by one item each from Chiu et al. (2016) and Sun et al. (2008) to measure the construct “intention to reuse”. “Perceived user satisfaction” was measured by four underlying items used by Liaw (2008). Finally, we adopted three items from a previous study of e-learning effectiveness from Liaw (2008) and added one item each from Chiu et al. (2016) and Gable et al. (2008) in order to measure “learning effectiveness”.

Initially, we developed the survey in English, in accordance with prior research, and then translated it to German through a professional translation service in order to ensure a low-threshold participation opportunity and, thus, a high number of participants. Subsequently, a different professional translator translated it back into English to ensure conversion correspondence (Brislin 1970). As previously described, the constructs were unanimously measured with four or five items each. All items were assessed on a seven-point Likert scale (from 1 = “strongly disagree” to 7 = “strongly agree”). Table 4 in the Appendix presents the final survey consisting of the mentioned items used in our research model.

The mobile learning app was initially implemented in a mandatory introductory accounting course in the Winter semester 2020/21. Because of the COVID-19 pandemic, the social distancing requirements, and the sudden closures of university campuses, all lectures were held digitally in an asynchronous format via screencasts. Complementing the lectures, students could participate in synchronous, live tutorials and submit exercise sheets. Additionally, preparatory courses were also offered synchronously via Zoom. For our study, we invited all students who used the mobile learning app at some point in the Winter semester 2020/21 to participate in the online survey conducted during the final week of teaching (i.e., before the final exam) and administered on the university’s LMS. We did not offer any additional (e.g., monetary or extra course credit) incentives for participating, and the students were informed of the research purpose and their voluntary participation in the study. Even if they took part in the survey, they had the possibility to refuse to answer any question. Subsequently, one member of the research team, who was not involved with the empirical analysis, merged and pseudonymized the data from students’ questionnaires with data from several other sources, including students’ demographics being collected through another survey in the first week of the semester, students’ course attendance during the semester, and the academic performance data. More specifically, the students’ attendance at tutorials and workshops has been manually evaluated via Zoom participation protocols. Finally, the central examination office provided the student’s exam performance. In our analysis, we only use the final pseudonymized dataset, which does not allow identification of individual students.

The students were asked to answer the questionnaire according to their user experience throughout the semester. Thereby and due to the requirements of the IS success model, we were ex-ante limited to the population of 367 students who used the app during the semester and participated in the final exam to draw our sample. Our initial sample consists of 131 students who participated in our survey regarding their user experience. Out of the initial sample, we exclude 10 observations due to missing values in their survey responses and 1 without any variation in the responses. Hence, we received 120 usable responses, bringing our usable response rate to 91.60%. Further, we exclude 7 students because of missing values in their demographics, resulting in a final sample of 113 students who participated in the final exam¹ of the mandatory introductory accounting course and used the mobile learning app for learning purposes. Accordingly, our sample represents 30.79% of the underlying population that could be used for a study of this type.² The final sample comprises 63 female and 50 male students, with the overwhelming majority (94.69%) being 25 years old and younger. Table 2 presents the summarized descriptive statistics for the final sample of 113 students.³

Our research model was evaluated using PLS-SEM as the most favorable method to validate multistage models with complex relationships, interdependencies, constructs, and indicators (Hair et al. 2011; Sarstedt et al. 2016, 2021). We thereby followed recent recommendations as suggested by Hair et al. (2019) and Sarstedt et al. (2021). The minimum sample size was ascertained by multiplying the total number of constructs by ten (Hair et al. 2011; Marcoulides et al. 2009) and met with our 113 participants. The construct indicators in our model represent reflective measurements caused by latent variables (Churchill Jr 1979). We used SmartPLS (v. 3.3.2; Ringle et al. 2015) and applied a path weighting scheme with 300 iterations with \({10}^{-7}\) as the stop criterion. Bootstrapping was done via two-tailed bias-corrected and accelerated (BCa) confidence interval method with 4,999 subsamples followed by blindfolding with an omission distance of 7 (Henseler et al. 2016).

As mentioned, our app’s key user group are students who have just entered the university system. This focus on the “transition-in” phase of the student lifecycle differentiates our study from prior literature in this field (e.g., Almaiah and Al Mulhem 2019; Cidral et al. 2018; Wang et al. 2019a). Furthermore, we focus on students at a German university and, hence, in a typical Continental European higher education setting. The main differences between the Continental European and the Anglo-Saxon model of higher education arise primarily through universities’ funding and tuition fees. The Continental European model is characterized by state sponsorship of universities and free or very low tuition (e.g., Jongbloed 2004). As a result of this greatly reduced financial burden, students from disadvantaged backgrounds can also participate in higher education, and, therefore, the student population might be more (economically) diverse (Lenzen 2015). At the same time, limited state funding results in large lectures and a high student-lecturer ratio, which probably disadvantages students in need of greater guidance and, thus, increases the need for technological learning solutions.

At first, the research indicates a positive effect of system quality on perceived user satisfaction (H1b), a finding in line with prior results (e.g., Almaiah and Alismaiel 2019; Aparicio et al. 2017). As expected, factors like ease of use, easy navigation, structuredness, and the ability to efficiently retrieve relevant information help to increase user satisfaction. However, this effect could not be observed regarding the impact of system quality on the intention to reuse the app (H1a). A possible explanation for this finding, which is in line with prior literature (Aparicio et al. 2017; Chiu et al. 2016) is that students tend to use a system independently from its perceived quality, in case a university has committed to this particular system. Transferred to our context, the developed mobile app is the only solution of its kind. As such, first-year students obviously use it—regardless of their perception of system quality—as a means to (potentially) increase their learning performance. Nevertheless, further analysis of further factors seems promising to better understand the relationship between system quality and the intention to reuse the app (cf. Almaiah and Al Mulhem 2019).

Second, contrary to the proposed expectation in hypotheses H2a and H2b, service quality neither significantly impacts the intention to reuse nor perceived user satisfaction (see Table 3 or Fig. 5). Thus, service quality seems to be a minor issue in evaluating the mobile app’s benefits in terms of learning effectiveness. This is also reasonable as we noticed in the field that app users rarely contact the administrative or teaching staff for help with respect to mobile app usage. Likewise, previous research finds evidence that service quality is relatively less important in the context of knowledge-orientated information system success (e.g., Wang et al. 2019b; Wu and Wang 2006).

Third, information quality positively impacts the perceived user satisfaction in our study (H3b). Accordingly, the reliability, understandability, and relevance of the information provided by the app for students in the “transition-in” phase are greatly appreciated by our target group. However, no significant impact on the intention to reuse could be observed (H3a). A similar finding is presented by Chiu et al. (2016). As a potential explanation, students may believe that using the app is decisive for a successful start of studies, and therefore, they do not draw the presented information into question. This assumption aligns with the observation that students’ critical thinking abilities are only starting to take shape in their first year at university and significantly increase in the subsequent semesters (cf. Ralston and Bays 2015; Wallace and Jefferson 2015). Hence, analyzing students from more advanced semesters might lead to different results.

Fourth, the effects of perceived enjoyment on both the intention to reuse (0.407) and user satisfaction (0.346) are significantly positive and greater than the impact of system quality, service quality, and information quality. These findings support our hypotheses H4a and H4b. Therefore, since perceived enjoyment is the only construct that influences both intention to reuse as well as perceived user satisfaction, it can be stated that the enjoyment and joy-related aspects are of utmost importance to promote learning effectiveness in our research setting. This result is quite striking and provides further evidence on the value of gamification in higher education, a field of research which is still quite in its infancy.

Fifth, we find support that perceived user satisfaction mediates the effects of system quality, information quality, and perceived enjoyment on the intention to reuse (H5). This is also reasonable since it might be more likely that students intend to reuse the app when their level of satisfaction is high. Additionally, this mediating effect might be why the service quality (H2a) and the information quality (H3a) are not directly associated with the intention to reuse. We also find evidence that intention to reuse significantly positively affects learning effectiveness (H6). In other words, the greater the likelihood to reuse the app, the greater the app’s positive effect on students’ learning effectiveness.

Lastly, perceived user satisfaction plays the most important role in determining students’ learning effectiveness, supporting our hypothesis H7. Since user satisfaction influences learning effectiveness, both directly and indirectly as mediated by the intention to reuse, it has the most significant total impact on learning effectiveness (0.448 + (0.445 × 0.426) = 0.6376). Moreover, it also has the most substantial direct effect (0.448), while the intention to reuse has a slightly lower impact (0.426).

As described above, system quality, information quality, and perceived enjoyment positively affect perceived user satisfaction, while the latter also promote the intention to reuse. Furthermore, perceived user satisfaction and intention to reuse have a supportive influence on students’ learning effectiveness. In this light, we now compare and contrast the design requirements and their realization as app functionalities with the influencing factors established before (see Fig. 6). By that, we contribute to discussing how digital technologies can be purposefully designed to support student retention in higher education. Hence, our artifact and the design requirements may serve as propositions for developing similar student apps specifically for the “transition-in” phase. Furthermore, based on the findings and the instantiated artifact, we derive design principles (cf. Gregor and Jones 2007) for mobile app creation in this field.

First, considering information quality, the option to self-track one’s course attendance and analyze this data against the course offerings during the semester (DR 1), the functionality to allow pop-up messages as reminders for lectures and academic events (DR 2) as well as the availability of training and exam-oriented exercises to control the learning process (DR 3) have proven useful to provide understandable, interesting, and reliable information to first-year students. Thus, regarding DR 1, the user may navigate to a site called “course overview” after login, showing the portfolio of courses the user has registered for. There, the app provides the option to confirm attendance for lectures. On a more detailed level, the user can create a time tracker for each course on demand, which allows for tracking attendance and self-study periods. Concerning DR 2, students are notified via push messages if an important event (e.g., lecture, exam registration deadline) in a course for which they have registered is about to take place. As a further option, students can export the course or event dates to their personal smartphone calendars. Via the site “exercises”, exam-oriented exercises are offered to check one’s personal learning process (DR 3). At this point, the training content also comprises learning videos and course scripts, which can be accessed at any time, enabling students to adjust the pace of learning at their own discretion.

The information quality offered to students with the help of the abovementioned functionalities (see Fig. 6) supports them in planning their participation in academic events and, thus, integrating into the daily student life (student factor “student experience”). Furthermore, they can improve their self-organization and adjust their learning strategies in accordance with the results that were scored for the exam-oriented exercises for instance (student factors “self-organization” and “learning strategies”).

To increase users’ perceived enjoyment, gamification elements like quizzes (DR 4) and competitions with the peer group are available (DR 5). This is meaningful since a lack of motivation and engagement to participate in the learning process has been identified as a major challenge in contemporaneous education (Hassan et al. 2021; Kiryakova et al. 2014). In order to reach a higher level of commitment and motivation among students, we propose the functionalities DR 4 and DR 5. Therefore, the game design mechanisms “freedom to fail” and “rapid feedback” (Stott and Neustaedter 2013; Thakur et al. 2020) were purposefully transferred to our learning app. We consider both mechanisms as helpful for first-year students to reduce mental barriers to interact with fellow students and teachers. The principles can be ideally addressed by quizzes, which give students direct feedback, even beyond the classroom, and disassemble psychological barriers to “fail” or to give wrong answers (cf. Alberti et al. 2019; Gordon et al. 2021). Furthermore, rewarding students’ efforts immediately (e.g., by credits) is a recognized way to increase motivation (Kiryakova et al. 2014). Hence, during app usage, students may collect credits through various actions (e.g., quizzes, attending courses, correctly solving exam-oriented exercises, etc.), determining their ranking in a playful competition with their peer group. Further, the training exercises can also be offered in the form of an interactive survey during the lecture (i.e., a “clicker” functionality) to test students’ current knowledge and support their “progression” in becoming familiar with the course content (cf. Stott and Neustaedter 2013).

Against this background, we propose the described functionalities to increase students’ perceived enjoyment (when dealing with subject-related content) as means to address the student factors “learning strategies” and “self-organization” (see Fig. 6 and Sect. 2.3).

Additionally, we suggest the design of the solution as a hybrid app (DR 6), which differentiates between a front- and backend (DR 7) and transfers data with the help of HTTP and JSON (DR 8) as a way to effectively produce the required information and, hence, contribute to system quality in our setting (cf. Urbach et al. 2009). The design as a hybrid app allows us to offer the app for both common mobile phone platforms (i.e., iOS and Android), whereby the development efforts were less than for corresponding native apps (e.g., Schilling 2016). The app’s architecture enables easy maintenance, further development, as well as the addition of further services. Finally, the data (e.g., exam-oriented exercises, quiz questions, etc.) are stored in a database, which strongly facilitates content management and the provision of new content once users log in as “lecturers”. Figure 7 provides our proposition for the architecture.

Based on our findings, we propose the following four design principles to facilitate the design of related instances of the artifact (cf. Kruse et al. 2016; Sein et al. 2011), namely mobile solutions to support students in the “transition-in” phase. Generally, design principles build on the knowledge that is gained when developing and using a specific instance of an artifact and are formulated when reflecting upon the results from a more generic perspective (Kruse et al. 2016). Concretely, we propose the following design principles:

These design principles have been formulated based on a concrete instance of an artifact, considering the findings of this study. They can purposefully complement existing design principles, which are directed at the creation of innovative learning environments (cf. Herrington et al. 2009), the design of mobile course material (cf. Ally 2005), or making use of gamification elements (cf. Laine and Lindberg 2020). However, our design principles are different from existing propositions in the field of mobile learning (e.g., Palalas and Wark 2017) since we focus on support during the “transition-in” phase and the corresponding challenges. As such, the suggestions may be referenced and consolidated with further design principles to cover all stages of the student lifecycle in future research, contributing to an even better understanding of mobile learning in higher education.

As mobile phones are widely available among students and the field of higher education becomes aware of the potential to use them for learning purposes through the development and usage of mobile learning apps, it is crucial that researchers and practitioners develop a better understanding of what makes learning apps successful and—equally important—how to measure their success in the first place. Considering this, our study contributes to research in the following ways.

First, the mobile app to support students in the “transition-in” phase was developed with the help of a DSR procedure (cf. Peffers et al. 2007), as outlined in Sect. 2.3. The app represents the artifact (i.e., outcome) of the Design Science (DS) effort and, hence, a “human-made object” (Goldkuhl and Karlsson 2020, p. 1241) to solve a practical problem (March and Smith 1995). However, besides the artifact itself also its contribution to theory should be clearly highlighted in DSR (Baskerville et al. 2018). This contribution to scientific knowledge is addressed by Hevner’s “rigor cycle” (Hevner 2007) and a mandatory element from the perspective of the “design theory school of thought” (Baskerville et al. 2018, p. 359). Thereby, the IT-artifact (i.e., the mobile app) of our DS effort was built in previous work (see Sect. 2.3) and can be seen as an instance of a “type 2 app” according to the “pedagogical framework of mobile learning” (Park 2011) for students of an accounting course. In this paper, the app’s contribution to the scientific knowledge base is analyzed by identifying the factors that impact a student’s learning effectiveness, user satisfaction, and intention to reuse the app and linking these to design requirements (see Sects. 4 and 5). Furthermore, we present design principles that offer DS researchers “actionable knowledge useful in building new versions of similar artifacts” (Kruse et al. 2022, p. 1236). Accordingly, these insights may extend the “pedagogical framework of mobile learning” (Park 2011) by laying the foundation for success factors and design propositions that determine the acceptance of mobile apps among students in the “transition-in” phase.

Additionally, this research supports previous findings on the employment of system success models in the field of mobile learning. In this respect, we confirm the model of Wang (2008), which suggests that user satisfaction has a direct as well as indirect impact on other net benefits (e.g., learning effectiveness) through the mediation of intention to reuse. Another implication that can be drawn from this result is that the perceived user satisfaction and the intention to reuse are prerequisites for students’ learning effectiveness. Following Wang et al. (2019b), who implemented perceived enjoyment in the context of fee-based mobile learning, we now use the perceived enjoyment in our proposed success model for free mobile learning apps. Therefore, this study benefits future research by providing a validated IS success model for free mobile learning apps by combining perceived enjoyment and learning effectiveness with the established IS success model.

However, deviating from the traditional IS success model, we found service quality to have no significant impact on perceived user satisfaction and intention to reuse (see Fig. 5). Since this can be explained, as shown in Sect. 5.1, it can be stated that service quality is relatively less important in the context of knowledge-oriented IS success, which is in line with previous research (e.g., Wang et al. 2019b; Wu and Wang 2006) as well as rather good news for higher education organizations which mainly expend their resources on teaching staff instead of administrative personnel, which could provide user support for such apps.

In summary, the empirical results emphasize the importance of extending the traditional IS success model by other dimensions like perceived enjoyment when assessing mobile learning app success. Accordingly, future research can rely on this multidimensional approach, compare it to existing models, or examine the influence of the included constructs on mobile learning system success.

This research also provides several implications and benefits for practice. First, the results show that an app, which was collaboratively developed with the target groups (i.e., students and educators) and adapted to the particular needs of first-year students, can positively influence students’ learning effectiveness. This highlights the value of the design and development procedure of the app following a DSR approach (cf. Peffers et al. 2007) with several iterative steps.

Second, according to the employed and validated model, learning effectiveness is considered a more effective measure of mobile learning app success than the other six variables. In this regard, learning effectiveness should develop if the model components of system quality, service quality, information quality, perceived enjoyment, intention to reuse, and perceived user satisfaction are appropriately managed. Thus, to support learning effectiveness, an implication for developers of mobile learning apps is to focus on high system quality, information quality, and, foremost, an enjoyable learning experience. To this end, design requirements and principles have been proposed that allow the creation of similar artifacts. More detailed, our results show clear evidence that the impact of students’ perceived enjoyment on user satisfaction, intention to reuse, and learning effectiveness is substantially greater than the total effect of system quality, service quality, and information quality. This calls for strongly emphasizing gamification elements for mobile apps in corresponding DSR efforts.

Third, the four components of system quality, service quality, information quality, and perceived enjoyment have both direct and indirect effects. They all directly influence students’ perceived user satisfaction and the intention to reuse. The perceived user satisfaction, in turn, affects the intention to reuse as well as the learning effectiveness. Therefore, the intention to reuse and learning effectiveness are also influenced indirectly. The findings of this study suggest perceived user satisfaction has the most significant impact on intention to reuse and learning effectiveness. Moreover, perceived user satisfaction has the most substantial direct and total effect on students' learning effectiveness (see Fig. 4). Thus, the importance of student’s satisfaction with the learning app in improving their learning effectiveness is emphasized. However, the findings also suggest that developers as well as educators must track changes in both the perceived user satisfaction and intention to reuse, as user satisfaction does not totally mediate the impact of intention to reuse on students’ learning effectiveness.

To summarize, this study helps practitioners like developers and educators to identify the factors that make mobile learning applications more successful. The empirical findings encourage developers to consider the constructs of system quality, information quality, perceived enjoyment, perceived user satisfaction, intention to reuse, and learning effectiveness when designing their products. Moreover, the importance of students’ enjoyment and satisfaction while using the app for their learning outcomes is emphasized. Besides the implications for developers, this aspect is also a fundamental implication for educators, as it requests and motivates them to deliver learning content entertainingly to help their students succeed. Together, both aspects could help reduce early dropout rates among students and, thus, contribute to fighting the current shortage of skilled workforce and help meet the economy’s demand for qualified workers in the next decades.