Chatbot usage is evolving rapidly in various fields, including higher education. The present study’s purpose is to discuss the effect of a virtual teaching assistant (chatbot) that automatically responds to a student’s question. A pretest–posttest design was implemented, with the 68 participating undergraduate students being randomly allocated to scenarios representing a 2 × 2 design (experimental and control cohorts). Data was garnered utilizing an academic achievement test and focus groups, which allowed more in depth analysis of the students’ experience with the chatbot. The results of the study demonstrated that the students who interacted with the chatbot performed better academically comparing to those who interacted with the course instructor. Besides, the focus group data garnered from the experimental cohort illustrated that they were confident about the chatbot’s integration into the course. The present study essentially focused on the learning of the experimental cohort and their view regarding interaction with the chatbot. This study contributes the emerging artificial intelligence (AI) chatbot literature to improve student academic performance. To our knowledge, this is the first study in Ghana to integrate a chatbot to engage undergraduate students. This study provides critical information on the use and development of virtual teaching assistants using a zero-coding technique, which is the most suitable approach for organizations with limited financial and human resources.
Hinweise
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Introduction
There has been a rapid increase in chatbot usage in various fields in recent years. Notably, one essential field where chatbots and virtual assistants are increasingly employed is education (Clarizia et al., 2018). According to Agarwal et al. (2022) chatbots are software applications, which are able recognize patterns from inputs and produce outcomes as per the input. The chatbots are called virtual assistants, when they are designed to understand the needs of their users, through artificial intelligence (AI) methods, and reply back to them in natural language. In education, there has been a considerable growth in the representation of chatbots whose major goal is to develop knowledge for individual students, usually on a certain topic. The main goal of these chatbots is to develop new knowledge like a human teacher (Han & Lee, 2022; Pérez et al., 2020). Chatbots are now used as virtual assistants or agents to enhance learning and teaching. The increased use of chatbots is attributable to current advances in Natural Language Processing (Adamopoulou & Moussiades, 2020). More readily available computing power and communication technologies have facilitated the rapid development and deployment of chatbots (Maroengsit et al., 2019) in education. Besides, chatbots can help higher education institutions improve their current services, cut staff expenses, and develop innovative services (Hien et al., 2018). Pérez et al. (2020) identify two types of educational chatbots: Service‐oriented chatbots and Teacher‐oriented chatbots. Service‐oriented chatbots offer support for student queries during enrollment and admissions, and library services. Teacher-oriented chatbots act like classroom assistants (Chou et al., 2021) to generate knowledge, increase student engagement and provide smart feedback (Khan, 2019; Vázquez-Cano et al., 2021). Several chatbot development platforms produce education bots that engage students and contribute brief but relevant knowledge (Kumar, 2021). The development of some chatbots may require complex computational skills, while others may require zero coding techniques. Flow XO, DialogFlow, and Botsify are examples of chatbots that anyone, not necessarily tech-savvy, can consider when aiming to implement such solutions in their organizations (Satam et al., 2020). With the advancement of bot platform features, the user interface of the platform becomes very simple and intuitive to enable educational institutions, with limited software development and human resource capacity to create the bots by themselves.
Current research highlights that chatbots may address the challenge of insufficient student–instructor interaction (Clarizia et al., 2018). Especially in contexts like the Ghanaian higher education institutions, where the instructor–student ratio is high (Essel et al., 2019; Tsyhaniuk & Akenten, 2021), the use of a chatbot may provide automatic and instantaneous responses to students’ queries similar to chatting with a course instructor. This leads to reduced workload for the instructor and more engaging learning experience for the students. In this study we implemented a zero-coding chatbot, named KNUSTbot, in a multimedia programming course at Kwame Nkrumah University of Science and Technology (KNUST), which allows students to learn and reflect profoundly about multimedia programming via interaction. The effectiveness of student interaction with the KNUSTbot during the learning process was examined. The chatbot in the present study is a teaching assistant chatbot developed to accomplish a set of learning objects by determining intents and entities from a free text communication of a student leveraging Natural Language Processing without confining the student with a set of available options. This method allows for a more natural way of engaging (Yin et al., 2021).
Anzeige
The study’s Research Question (RQ) is the following:
How does the KNUSTbot, used as an intermediary between students and instructors, affect the student’s learning in a multimedia programming undergraduate course?
The following are the study’s sub-research questions:
1.
Is there a significant difference in academic results between students who interact with a chatbot and students interacting with a course instructor?
2.
What are the perceptions of the experimental cohort on the chatbot as a means to facilitate their learning?
Chatbots in teaching and learning
Chatbots combine artificial intelligence (AI) and Natural Language Processing to interact with a human interlocutor at a certain level of conversation via text or voice (Pérez et al., 2020; Smutny & Schreiberova, 2020). Clarizia et al. (2018) describe chatbots as virtual assistants capable of answering questions and providing appropriate responses. Other authors adopted a text-based chatbot, which typically responds to questions by following a built-in rule set, allowing them to respond to their users (Budiu, 2018; Salas-Pico & Yang, 2022; Topal et al., 2021). AI relates to systems or machines that mimic human intelligence and self-alter based on accumulated data (Angelov et al., 2021). Chatbots are an example of software applications (Salas-Pico & Yang, 2022; Topal et al., 2021) that understand questions faster and provide efficient answers (Angelov et al., 2021). Examples of chatbots include FAQ chatbot (Han & Lee, 2022; Ranoliya et al., 2017), ELIZA, an early Natural Language Processing computer program that simulated the communication between humans and machines (Natale, 2019), the colMOOC, an conversational virtual agent that promotes learners' interaction within MOOC platforms (Tegos et al., 2019), the StudBot, which is an academic information systems chatbot (Vijayakumar et al., 2019) and the artificially intelligent course teacher chatbots, like Sammy (Gupta et al., 2019), which is closer to what we are experimenting in our study. These chatbots have received significant recognition in the educational ecosystem in diverse learning contexts.
Anzeige
Okonkwo and Ade-Ibijola (2020) reported that most chatbots employed in higher education are teacher-oriented chatbots. Mendoza et al. (2020) highlighted positive impressions from a sample of students when they were engaged with a chatbot. Studies have also reported that students employ chatbots to ask questions, receive responses, and receive individualized support (Hiremath et al., 2018; Mikic-Fonte et al., 2018; Pham et al., 2018; Sinha et al., 2020).
According to Yin et al. (2021), no significant difference in the learning achievements of undergraduate students randomized into experimental and control groups (without or with the support of a chatbot) were found; however, the research reported higher levels of motivation for the learners interacting with the chatbot. Arruda et al. (2019) designed a chatbot for computer science students, employed for goal-oriented requirements modelling; the students found the chatbot functional and desired to use it in the future. A study performed by Kamita et al. (2019) using chatbots and web courses to improve students’ mental health reported a higher probability of efficacy as chatbots guided self-learning, enhanced motivation, and lessened stress. The University of Georgia designed a chatbot named ‘Jill Watson’ adopted in a computer science course. Participating students were more responsive and they stated that they wanted to use this chatbot in different lessons (Lipko, 2016).
Students’ ability to interact with instructors, by asking questions, is an essential process of learning that can contribute to enhanced academic performance (Harper et al., 2003; Sandu & Guide, 2019; Vlachopoulos & Makri, 2021). University students in Ghana have inadequate interaction with their course instructors during class sessions. This issue is due to the increase in the student–instructor ratio (Essel et al., 2019), reducing time instructors spend with their students. Furthermore, studies have shown that students are hesitant to ask questions because they are constantly scared of the teacher’s negative feedback (Oktaria & Soemantri, 2021; Verleger & Pembridge, 2018). In mitigating these issues, some instructors engage students after classroom didactics, with instant messengers (e.g. WhatsApp) and social media platforms (e.g. Facebook messenger) to provide personalized assistance to the students. However, the challenge is the instructor not having enough time to respond to questions and provide timely and individualized feedback to students. The unspontaneous student–instructor interaction leads to shortcomings in student’s knowledge. Late response to a student’s question is a significant concern as students continuously strive for precise and prompt responses (Farhan et al., 2012). In this context, chatbots become relevant in situations where course instructors cannot provide adequate response for students’ learning at any time of the day (Yang & Evans, 2019). A chatbot can simulate human-like dialogue-based interactive communications to assist students in revisiting learning resources (Göschlberger & Brandstetter, 2019; Jomah et al., 2016; Smith & Evans, 2018), promoting learning achievement and self-efficacy (Chang et al., 2021) and enhancing adaptive learning (Fadhil & Villafiorita, 2017).
Furthermore, chatbots can assist in overcoming this difficulty by initiating conversations based on the student’s context, making students seem individually addressed (Hien et al., 2018; Howlett, 2017). A chatbot can be an intermediary between a student and an instructor, which allows students to concurrently control their learning and improvement at their pace without constraining them (Wang et al., 2021). Also, chatbots tend to stimulate questions from students who may be restrained from engaging in a conventional learning space (Verleger & Pembridge, 2018).
Method
We employed a quasi-experimental pretest-posttest design in combination with a qualitative research method (focus groups), to collected more in-depth information on the students’ experience with the chatbot. The 68 participating undergraduate students were in their final year in a multimedia programming course offered by the Department of Publishing Studies at Kwame University of Science and Technology (KUNST). The experiment was conducted in the second semester of the 2021 academic year between April and August (16 weeks). Stratified sampling was chosen to create two groups with representative sample from the student population (Delice, 2010), which was divided into relatively similar subpopulations (strata) in terms of age, gender and academic performance. As part of the study, we garnered data on their socio demographics, including gender, age, cumulative weighted average grade, preferred instant messaging app, and previous experiences with AI chatbots.
The intervention procedure
We used the end-of-semester course achievement tests to measure the academic performance of the experimental and control groups. The achievement test had two sections. Section 1 contained different types of objective questions (multiple choice with single and double selection, matching, ranked order, and short answer). Section 2 of the test was a computer-based practical examination where students scripted a front-end website using HTML (Hypertext Markup Language) and CSS (Cascading Style Sheet). The lead investigator developed the test items. There were 40 objective-typed items in section A, and each item carried 1 point, and section B carried 60 points. The total maximum score for the test is 100 points. The reliability coefficient of the items was = 0.82, with the average difficulty level estimated at p = 0.45. According to Hasançebi et al. (2020), values ranging between 0.30 and 0.49 suggest average item difficulty. The number of correct responses provided by students was used to determine their achievement. The total time estimate for the achievement test was 60 min (20 min for section A and 40 min for section B). The investigators developed the table of specifications based on the university’s standards to measure the content and construct validity of the test.
In addition, the investigators designed group interview (focus group) guide to collect comprehensive data about the students’ experience with the chatbot. The guide was given to the experimental group to estimate their observations of the chatbot. The focus group discourse was done in a single sitting, with physical presence and one week after the experiment and every student in the experimental group partook. To give more opportunities for interaction to all students, three focus groups were conducted. The focus groups aimed at interpreting the students' positive and negative encounters with the chatbot (KNUSTbot) for course interaction and their predisposition to interact with the KNUSTbot in future academic endeavors.
The KNUSTbot was developed using WhatsApp instant messaging app and Flow program. FlowXO is a Natural Language Processing platform that can build human-like conversational AI with state-of-the-art virtual agents in multiple languages and platforms (Flow, 2020). Two experienced multimedia programming experts and an instructional designer verified the KNUSTbot. The chatbot’s intents and entities comprise a data repository of standard learning content queries (HTML and CSS) built based on chats database continuously accumulated in a Learning Management System over eight years by the lead investigator. Current studies (Snodgrass Rangel et al., 2017) highlight that student data obtained by teaching staff during teaching and learning process has the potential to improve education, as well as the way educational institutions work. We understand that some types of research using quantitative, qualitative or mixed methods, may require prior contact and dialogue with individuals or communities as a normal and integral component to establish the design of the research and understand its value for the society and the discipline in question. These activities don’t require ethics review and approval (Government of Canada.-Panel on Research Ethics, 2018). We considered that the accumulation of student queries prior to the beginning of the “approved” study fall in this category, since the instructor didn’t invite the students for this interaction and simply used the published queries within the Learning Management System (LMS) of the university. In addition, these queries are not linked to any personal information, such as name, email, group, year of studies, etc. Finally, all students gave their consent to participate in the LMS, where this information is published.
The term “intent” describes the question a student requested, to which the chatbot is supposed to answer. At the same time, the entity represents a trigger connected to the intent to render a distinct and individualized context for the intent. In all, 1000 intents and entities were used as a dataset to train the bot. We included additional 70 intents (short text expressions, e.g. ‘Hi super programmer’, ‘Good job’, ‘I’m always at your service’) to engage and motivate the students during the interaction. Besides, the bot was trained with internet sources (e.g. Websites and YouTube videos).
Whatsapp was selected because of its user-friendliness and popularity among undergraduate students (Afful & Akrong, 2019). The students interacted with the chatbot using natural language. The interface of the KNUSTbot and how a student can interact with it is illustrated in Fig. 1.
Fig. 1
The interface of the KNUSTbot
×
A digital literacy test was administered to the students to ascertain their prior abilities on basic computer skills, the internet and the web, productivity programs, computer security and privacy, and digital lifestyle. This activity occurred in week one of the course and constituted the pre-test score. All 68 students were exposed to multimedia programming with HTML and CSS through face-to-face instruction from the second week to the sixteenth week. The course instructor demonstrated how to develop a front-end application using the scripting languages in a practical session. During the sessions, the instructor created two WhatsApp platforms for the control and the experimental cohort. The students were instructed to direct content-related questions to the WhatsApp platform. The experimental cohort was assigned to the WhatsApp group via the platform integrated with the chatbot. They engaged the chatbot at any time after a topic was taught; however, the instructor did not intervene in the experimental cohort’s interactions. No additional communication channels or contact points were available for the two groups. The control cohort interacted on the platform without the chatbot. The two cohorts were assessed with a post-learning achievement test at the end of the sixteenth week.
After the experiment with the use of the chatbot, focus groups were conducted by the lead investigator, who is an experienced scholar in educational research. Students in each group were initially asked similar questions designed to explore their perceptions of the educational experience they had with the chatbot and to probe for the positive and adverse viewpoints they highlighted from their experience. Another question honed in on whether students felt they have derived similar benefits from their interaction with the chatbot compared to the interaction they were used to have with the instructor. The last question of the focus group was related to whether students consider the benefits of interaction with the chatbot significant, so it can be implemented in other courses of their program of studies. Each focus group lasted approximately 45 min. Figure 2 depicts the intervention procedure.
Fig. 2
The intervention procedure
×
Data analysis and ethical considerations
The Jamovi 2.0.0 package (Jamovi project, 2021) was used to analyze the data using descriptive and predictive statistics. The Shapiro–Wilk test (p > 0.05) indicated that the datasets were normally distributed. As a result, we obtained skewness and kurtosis values of ± 1.96 for the datasets, validating the normality of distribution (Essel et al., 2021). In the present study, the significance level for statistical tests was estimated at a p-value of less than p < 0.05. Frequency, percentage, Mean, and standard deviation were calculated for the students' sociodemographics. A Split-Plot Analysis of Variance (SPANOVA) was performed to assess the influence of the chatbot and instructor on students' Achievement Test scores over two-time sessions (pre-test and post-test). SPANOVA and independent-samples T-test were used to examine RQ1. All assumptions, including Homogeneity of variance (Levene’s: ppretest = 0.08; Pposttest = 0.56), and normality of distribution (Shapiro–Wilk: ppretest = 0.06; Pposttest = 0.27) were met.
To investigate RQ2, the investigators used the data from the focus groups, which took place at the University premises. The students were interviewed in groups of 11 or 12 people to create the dynamic of a conversation and to make the student feel more at ease (Witsenboer et al., 2022). The data were digitally recorded, transcribed, and manually coded under themes using content analysis. To create a theme, the content was coded in two levels. First level of coding included labels assigned to specifics fragments of the focus group, which could help us answer the RQ2. Following Witsenboer et al. (2022) it was also checked whether existing labels could be assigned to overlapping content. In the second level of coding we narrowed the focus to relatively fewer codes, directly related to RQ2.
The present study was conducted under the Helsinki Declaration (1975) and comparable ethical standards, with approval by the Humanities and Social Sciences Research Ethics Committee (HuSSRECC) of KUNST with number 233/22-08/2021. A written and verbal informed consent was solicited from the students.
Results
Participants’ profile and traits
This study included 68 final year undergraduate students, 34 of whom were in the experimental group and the other 34 in the control group. All the students had the WhatsApp app installed on their phones. Regarding students' experience with chatbots, all 68 reported that they have encountered chatbots in their online activities outside their studies, with the 79% (54 students) encountering services-oriented chatbots. Table 1 illustrates the profile of the students and some of key traits for the interpretation of the findings. The median split categorized overall academic performance measured with Cumulative Weighted Average (CWA, Median = 65.71) and the years of experience (Median = 5.0) with the use of WhatsApp app.
Table 1
Profile and traits of the 68 participants
Treatment group, n (%)
Control group, n (%)
Total, n (%)
Gender
Male
15 (44.1)
15 (44.1)
30 (44.1)
Female
19 (55.9)
19 (55.9)
38 (55.9)
Age
19–22
22 (62.9)
21 (63.6)
43 (63.2)
23–26
13 (37.1)
12 (36.7)
25 (36.8)
Experience with WhatsApp [years]
1–5 years
12 (35.3)
9 (26.5)
21 (30.9)
6 years and above
22 (64.7)
25 (73.5)
47 (69.1)
Daily whatsapp messenger use
1–5 times a day
9 (26.5)
10 (29.4)
19 (29.7)
6 times or more
35 (73.5)
24 (70.6)
45 (70.3)
Previous Academic performance
Less than 60.0
11 (32.4)
13 (38.2)
24 (35.3)
60.0 and above
23 (67.6)
21 (61.8)
44 (64.7)
Engagement with the chatbot
Figure 3 illustrates the preferred time during the day students engage with the chatbot, as well as the amount of interactions (number of chats) per hour. It is obvious that the students interact with the KNUSTbot mostly between 20.00 and 22.00, which is when the instructors are not usually available and responsive.
Fig. 3
Students’ engagement (number of chats per hour) with the KNUSTbot
×
Figure 4 illustrates the amount of students, who interacted with the chatbot during the course. It is obvious that there were more students engaging at the beginning of the course, when they needed more information to understand how the course works and towards the end of the course, when they were preparing for their final assessment.
Fig. 4
Number of students using the chatbot during the course
×
Figure 5 illustrates the number of student queries using the chatbot during the course. Again, it is obvious that there was more need for interaction during the last month of the course, when students were preparing for their final assessment.
Fig. 5
Daily queries by students
×
Academic achievement scores
As shown in Table 2, students in the experimental cohort received an average score of 40.6 (4.95) for course achievement before and 81.1 (3.19) after the experiment. Similarly, students in the control group received mean scores of 43.4 (4.09) and 65.2 (3.73) respectively. The results demonstrate that course achievement improved between the experimental group of students, who interacted on a real-time basis with the chatbot, and students who interacted with the instructor in real-time. Interaction between time and method of interaction, F(1, 66) = 87.5, p < 0.05, squared partial eta (ηp2) = 0.57 was found significant.
Table 2
Pretest and Posttest scores of the experimental and control cohorts
Score
Cohort
N
Mean
95% CI
Median
SD
Minimum
Maximum
Lower
Upper
Pretest score
Experimental
34
40.6
38.9
42.2
41.1
4.95
33.1
50.7
Control
34
43.4
42.0
44.7
43.1
4.09
34.9
50.7
Posttest score
Experimental
34
81.1
80.1
82.2
81.5
3.19
74.7
86.7
Control
34
65.2
64.0
66.5
65.3
3.73
56.6
73.7
CI confidence interval
Using Cohen's guidelines using effect size: 0.01 = small, 0.06 = medium, 0.14 = big, these results suggested a very large effect size for interaction. In terms of real-time interactions with course instructors (p < 0.001) and real-time chatbot interactions, there were significant differences between the posttest and pretest scores (p < 0.001) and no significant difference in cohorts before intervention in two groups (p > 0.05), the difference between cohorts following intervention was significant (p < 0.001). The results are illustrated in Tables 3 and 4.
Table 3
Within-subject effect (I)
Sum of squares
df
Mean square
F
p
ηp2
Time
33,105
1
33,105.4
2087
< 0.001
0.969
Time*cohort
2975
1
2974.9
188
< 0.001
0.740
Residual
1047
66
15.9
Type 3 sums of squares
Table 4
Within-subject effect (II)
Sum of squares
df
Mean square
F
p
ηp2
Cohort
1469
1
1468.7
87.5
< 0.001
0.570
Residual
1108
66
16.8
Type 3 sums of squares
Sociodemographic variables for experimental cohort regarding course achievement
The investigators also wanted to observe after the experiment whether there was a significant difference in course achievement regarding sociodemographics. The results in Tables 5 and 6 illustrate that there was no statistically significant variation in gender (p > 0.05) and course achievement, as well as age (p > 0.05) and course achievement.
Table 5
Independent samples T-test (gender)
Statistic
df
p
pre_score
Student's t
1.537
32.0
0.134
post_score
Student's t
-0.905
32.0
0.372
Table 6
Independent samples T-test (age)
Statistic
df
p
pre_score
Student's t
− 1.34
32.0
0.191
post_score
Student's t
− 1.39
32.0
0.173
In addition, the results show no statistically significant difference between prior academic performance and the posttest score (p = 0.51), as well as WhatsApp user experience and the posttest score (p = 0.34).
Findings from the focus group
Focus group discussions (after posttest) were conducted with the experimental cohort to garner data on the perceptions regarding their interaction with the KNUSTbot. Students were asked to appraise the positive and adverse viewpoints of their interaction with the KNUSTbot during this discussion. Table 7 comprises the thematic codes based on the results from the discussions. Student appraisals of the discourse were utilized to generate the thematic codes.
Table 7
Thematic codes generated from 34 students' evaluation of their interaction with the KNUSTbot
Category
Code
f (%)
Positive viewpoints
The AI chatbot is easy to use
10 (29.4)
Searching for and evaluating information
5 (14.7)
More self-belief to learn more effectively
4 (11.7)
Giving immediate feedback
15 (44.1)
Negative viewpoints
Concerns about responses being outdated/not relevant
3 (8.8)
Inability to think in depth
11 (32.4)
There is a dearth of detailed interactions
20 (58.8)
Recommend the implementation of such experience with chatbot in more courses?
Yes
27 (79.4)
No
2 (8.9)
Preference towards interacting with the chatbot comparing to traditional interaction with the instructors
No response/not sure
Yes
No
No response/not sure
5 (11.7)
20 (58.8)
5 (14.7)
9 (26.5)
Most students were satisfied with their interactions with the KNUSTbot during the course, as exposed in Table 7. Below there are some representative examples of student statements confirming the positive viewpoints.
The AI chatbot is easy to use
“I never imagined that a chatbot could be so easy to use through my mobile phone”
“It was very easy to share the responses from the chatbot with other students through WhatsApp. This is the app we use to communicate as well, so it was useful”
“It allowed me to study the course with ease”
Searching for and evaluating information
“I also got links to websites that provided access to video and text tutorials in relation to the course content”
“I didn’t only receive an answer to my question, but also sources to consult to better understand the chatbot’s answer”
More self-belief to learn more effectively
“I realized that I can find the answers on my own”
“At the beginning I was insecure with the information I received by soon I was able to confirm the answer on my own”
Giving immediate feedback
“I realized that the response to my questions was very swift”
“I was able to get 24/7 quick answers to my questions when learning HTML and CSS which I have never encountered in any other course”
Overall, the students were satisfied with the instantaneous and immediate responses they received to inquiries during chats. They were generally welcoming the usage of the chatbot as a learning tool. They didn’t encounter delayed responses to their questions asked within the chat platform compared to their experiences with the instructor, where their questions may have received a delayed response or no answer at all. Despite this positive feedback, student highlighted some negative viewpoints on the integration of the chatbot in their learning, which can be summarized in the statements below:
Concerns about responses being outdated/not relevant
“My experience with the chatbot was not pleasant. I realized I was getting similar answers to the variety of questions I asked during the chat sessions”
“Sometimes it was frustrating asking for one thing and getting an answer about a different topic”
“The majority of the information I had seemed outdated to me. Besides, links/URL provided were broken, and this situation worried me at times”
Inability to think in depth
“I missed more in-depth answers from the chatbot and not simply definitions and links to find additional resources. I feel only the instructor can do this”
“Often I needed more explanation or even justification of the answer I received from the chatbot. I wasn’t able to understand what to do with the given information”
“I wished the course instructor supported the AI chatbot at a point in time”
There is a dearth of detailed interactions
“Few times I just received a link or a short answer to my question”
“I wish there were more follow-up answers since not always I understood the information I got from the chatbot”
“Some of my questions couldn’t be answered because they were about how to apply the knowledge from the course. I needed more instructions”
After synthesizing the negative feedback received, we can say that the students complained about their interaction with the chatbot because it didn’t carry out in-depth learning and the human element was missing. They also acknowledged that the chatbot gave the same responses to different questions. The need for more instructions on how to apply the knowledge acquired from the course was also highlighted.
Notwithstanding, students formed good views after interacting with the chatbot and fully appreciated the interaction approach. The vast majority recommended the integration of chatbots in other courses of their studies and more than half of the participants preferred the chatbot comparing to the interaction with the instructor. Some representative responses are presented below:
“I felt really elated interacting with the chatbot in this course. I had responses to all the questions I asked. At one point, I felt like I was chatting with the instructor. It is simple interacting with the chatbot”
“Interacting with the chatbot is a great way to learn more about HTML and CSS. I expect that chatbot learning will be used in more tailored educational systems in the future”
“The integration of the chatbot into the teaching and learning of multimedia was engaging, motivating and exciting for me, as it was a new experience and I felt more sense of belonging in the course”
“I was surprised by the kind of feedback I received during my conversation with the chatbot. It supplied me with a lot of reference links”
After analyzing the results from the focus groups, it is reassuring that, given the right conditions, students might appreciate the integration of chatbots as part of a course since it simulates and assists them in learning abstract concepts in-depth. This positive response from the students suggests the possibility of increasing the limited interaction sessions between academic staff and students, ensuring self-regulated learning and experiencing novel learning cultures, where academic staff is assisted by integrated emerging technologies into their courses. As a whole, engaging with the chatbot can support students in connecting what they are learning with real-world challenges or precedents, encouraging them to reason in-depth regarding what they are studying.
Discussion
The study indicated that students in the experimental cohort who engaged with the chatbot performed better than students in the control cohort who interacted with the course instructor. The use of chatbots can be a significant progression and innovation for heightening challenging subject learning (Clarizia et al., 2018; Okonkwo & Ade-Ibijola, 2020) such as multimedia programming.
The findings indicate no significant difference related to gender, age, experience with Whatsapp, academic performance, and the post-test scores of the experimental cohort. This confirms Sandu and Gide (2019), who also reported no significant difference between gender and age and the adoption of a chatbot. Regarding WhatsApp, our findings demonstrate that years of experience in using it didn’t affect the students’ post-test score. This is probably because it is a very popular and intuitive app widely used in Ghanaian higher education (Boateng & Tindi, 2022). There was a significant difference between daily Whatsapp use and post-test scores of the experimental cohort. The possible reason for this finding can be attributed to the immediacy of the feedback provided by the chatbot reflecting the improvement of learning, while the instructor usually delayed more to answer due to the timing of the questions were sent (since 2 most questions were sent outside office hours) and the big student-instructor ratio (Essel et al., 2019), which doesn’t allow instructor to spend enough time with their students and provide them with timely responses.
The quantitative analysis demonstrates that engaging students with a teaching assistant chatbots positively impacts academic performance. The qualitative analysis provided evidence of students’ satisfaction with the use of the chatbot, which can be attributed to the instantaneous feedback they received from the chatbot, as well as the enormous contribution to the learning process via having more engagement with chatbot at different times, and without encountering any delays in the interaction process. One of the purposes of using AI-powered teaching assistant chatbots, according to Chang et al. (2021) and Sandu and Gide (2019), is to deliver timely knowledge to specific students to surmount difficulties that arise during the learning process. Besides, the comments from the experimental cohort suggest that the student gained understanding and confidence to complete the course which translated in their improved academic performance. They also found learning to be interesting and interactive as their engagement with the chatbot enhanced the organization and re-examination of knowledge acquired. This outcome is consistent with Chang et al. (2021) finding that students' awareness arose due to the possibility to grasp and perform in-depth thinking by studying pertinent information.
Conclusions, limitations and implications of the study
The present study's main findings and arguments support the value of using chatbots in higher education, since the students who interacted with the chatbot performed better than students in the control cohort who interacted with the course instructor. This is especially relevant for countries like Ghana, where student–teacher ratio is high and the provision of timely response and feedback to students is a challenge. At the same time, students were very satisfied with the use of the chatbot, mainly because it provided them with instantaneous feedback at different times, without encountering any delays in the interaction process.
However, specific difficulties associated with the AI-powered teaching assistant must be overcome to use this approach effectively. Academic staff must have access and knowledge to customize and integrate chatbots to assist students learning. Since students may have encountered other chatbots such as the service-oriented chatbots, it is also advisable to make the transition to the use of teaching assistant chatbot simpler. Furthermore, it is encouraged that different motivators should be used to urge students to engage the chatbot. For example, the integration of gamification to enhance students’ engagement and interaction with chatbot can be a reinforcer, in line with the recommendations of Fadhil and Villafiorita (2017). Moreover, a micro-learning approach is a viable strategy for integrating teaching assistant chatbots in the educational setting (Yin et al., 2021). Other instant messaging platforms (Telegram or WeChat) can be employed to encourage students' interactions with teaching assistant chatbots due to their familiarity with these platforms (Boateng & Tindi, 2022).
Though this study engaged students with a chatbot developed with zero coding and in one course, the results are encouraging for the use of a teaching assistant chatbot in similar contexts. Specifically, within an institution/country with very limited resources (human and technological), the fact that we were able to execute such innovation successfully, with positive impact on academic performance and student satisfaction, make us confident that it can enact positive change in the teaching and learning process.
In this context, it is important to mention that the present study has also some limitations. It consists entirely of fourth-year students in a single university department, so future studies may observe students’ experience in other years as well as other faculties. This would allow generalize the results on the integration of chatbots in higher education. For example, it is unclear whether students in this study engaged more with the chatbot because they come from a more “technical” discipline and they are already familiar with the use of technology. The outcome may have been different in a humanities course. Also, it is not clear how first year students, who are not yet familiar with the teaching and learning process of the university and the available communication channels, would react to the chatbot service, taking into consideration that first year students are more instructor-dependent (Hagenauer & Volet, 2014). Furthermore, it is important to mention that every instructor reacts differently when it comes to provide timely feedback and creating strong group dynamics with their students, even if they share the same workload with their peers. So the attitude of the instructor can also impact the use of the chatbot. Moreover, while the chatbot can be very useful for facts to be learned, which require clear right answers, it is not clear how the chatbot could support learning that is centered around students' developing ideas about a topic or a body of theory. For all the above-mentioned reasons, we are not aiming to make general claims about the use of the chatbot in higher education, but to explore the efficiency of this cost-effective additional student support service and to expand the research in contexts with high student–instructor ratio.
The results of our research confirmed the existing literature that the use of chatbots enhances self-efficacy and learner achievement. Universities should establish Educational Technology Centers managed by subject matter experts to assist instructors in integrating and engaging students with teaching assistant chatbots. To obtain reliable results, instructors with low levels of digital literacy should receive proper training and coaching. Moreover, hands-on workshops should be organized to urge instructors to embrace the teaching assistant chatbot interaction in supporting learning and teaching. Besides, any course instructor who engages a virtual teaching assistant chatbot with students must ensure that each student is conversant with the instant messaging (WhatsApp, Telegram, or WeChat). The instructor must also ensure that the students have consistent Internet access and examine whether specific students cannot use the instant messaging application. Students may have difficulty adapting to the teaching assistant chatbot when using it for the first time. An initial session should be organized in which the engagement procedure is detailed, and students are informed about its assets to alleviate the affirmed challenges.
Certain directions for additional investigations are made based on the findings and discussion of the present study. It remains unclear whether chatbot can support learning by responding to technical questions mainly (e.g. explaining what is HTML5), or it can help learners to understand the conceptual content better. Future research could look into the impact of chatbots in other areas of knowledge. Also, the academic performance of students who interacted with the chatbot the most and those who interacted with it less could be compared. In addition, the chatbot’s longitudinal influence on student engagement and motivation should be investigated. Another identified area of future research is the influence of chatbot use on postgraduate supervision activities. Lastly, future studies could investigate the control cohort’s perceptions and test the acceptance of the experimental cohort using the Unified Theory of Acceptance and Use of Technology models, focusing on the intentions of the students when they use the chatbot, as well as on the subsequent usage behavior.
Acknowledgements
Not applicable.
Declarations
Competing interests
The authors declare that they have no competing interests.
Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
