Predicting Academic Outcomes: A Survey from 2007 Till 2018

Since higher education plays an essential role in the development of a society (Pinheiro et al. 2015), increasing student success is a long-term goal for academic institutions. In order to increase students’ success rate, it is vital to understand and define academic success. The definition of academic success is rather complex and wide-ranging; therefore, it is frequently misused within educational research. However, the study of York et al. (2015) suggests a theoretically grounded definition of academic success that is made up of six components: (1) academic achievement, which is nearly entirely measured with course grades and grade point average (GPA), (2) satisfaction, which is often captured either by course evaluation or institutional surveys, (3) persistence, which is measured by retention between particular years of college and degree attainment rates, (4) acquisition of skills and competencies, which can be measured by assignments and course evaluations, (5) attainment of learning objectives, which can also be measured by assignments and course evaluations, and finally (6) career success, which can be determined by job attainment rates, promotion histories, career satisfaction and professional goal attainment.

A second crucial requirement for maximizing students’ success is the identification of the factors that have an effect over academic performance. Awareness of students’ success factors could assist in achieving the highest level of quality education (Yassein et al. 2017). It can potentially help in providing a clear and strong description of the types of knowledge and behaviour that are associated with adequate performance. Such awareness can be gained by using methods of data mining (DM) over educational records. The practice of DM methods applied to educational data is known as Educational Data Mining (EDM) (Baker and Yacef 2009). It is drawn from a variety of domains, including DM and machine learning, psychometrics and other areas of statistics, information visualization, and computational modelling (Romero and Ventura 2007). Generally, EDM refers to techniques and tools designed for automatically extracting useful information and patterns from huge data repositories related to people learning activities in an educational environment (Nithya et al. 2016). Those tools employ machine learning algorithms, database systems, statistical analysis, and artificial intelligence. The DM techniques include regression, clustering, classification, association, and prediction.

Figure 1 by Romero and Ventura (2007) shows the application of DM in educational systems, and we use it in order to position the approaches analyzed in this survey within the EDM landscape. The use of DM in educational systems is represented as an iterative cycle of hypothesis formation, testing, and refinement where the systems can be directed to support the specific needs of every participant in the educational process. EDM carries an array of DM techniques, to (1) support relationship analysis, classification, and clustering, (2) elaborate educational hypotheses, and (3) provide learning assistant (Baker and Yacef 2009).

The scope of this paper lies within the DM phase in the EDM cycle. There is an extensive variety of methods within DM. However, as this survey covers predictions in EDM, we address only supervised DM methods (also known as predictive or directive). Supervised techniques such as classification and regression predict the value of the output variables based on the inputs. To do this, a model is developed from training data where the values of input and output are previously labelled. The model generalizes the relationship between the inputs and outputs and uses it to predict other datasets where only inputs are known (Witten et al. 2017).

After having introduced the EDM concept and the supervised techniques, the rest of the paper is organized as follows: We start by giving an overview of existing EDM surveys, along with the research question and research methododogy. Afterwards, we provide an outline of the task of prediction in EDM. This task involves essential decision making regarding the DM tool to be used in performing the predictions, the prediction methods and techniques, and the selection of features (or predictors). We then continue with a rich literature review regarding predicting students’ academic achievement in the past 12 years. Lastly, we present a summary of the main results and the reached conclusions, as well as future lines of research.

The EDM literature is rapidly growing and needs to be brought up-to-date frequently to take new studies into account. Romero and Ventura (2007) conducted a literature survey on EDM covering published work between 1995 and 2005. They reviewed different types of educational systems (traditional classrooms and distance learning) and how EDM can be applied on them. They also described the DM techniques that have been applied in educational systems. Another EDM survey was conducted by Peña-Ayala (2014) and analyzes EDM studies published between 2010 and 2013. It provides an analysis of the EDM weakness, strengths, threats, and opportunities. Dutt et al. (2017) performed a systematic literature review covering research between 1983 and 2016 on clustering algorithms only, which are unsupervised techniques. Unsupervised techniques uncover hidden patterns in unlabeled data with the aim of finding patterns in a dataset. In unsupervised DM, there are no output variables to predict. In their study, they viewed the applicability and usability of clustering algorithms in the context of EDM. They concluded that the key benefit of the clustering algorithms is that it provides a fairly explicit schema of students’ learning style when using a number of attributes (e.g., time spent to complete tasks, students’ behavior in class, and students’ motivation towards learning) to cluster. Kumar et al. (2017) conducted a literature survey covering publications between 2007 and 2016 on students’ performance prediction. They report the used DM methods and their accuracies. However, they do not report the used DM tools. They found that GPA and students’ internal marks are important attributes for predicting academic achievement. Our literature survey covers different and more recently published papers (2007–2018) and focuses on prediction tasks in EDM using supervised methods. In this section of the paper, we outline our research questions and the research methodology used to collect relevant literature.

Being able to predict a student’s academic success serves as an essential research topic in many academic disciplines due to its benefits to both teaching and learning activities (Holland and Nichols 1964). Performing predictions in educational environments generally includes three steps: (1) selecting the right features (predictors), (2) selecting the right method or technique, and (3) selecting the proper data mining software. As is the case with most supervised machine learning tasks, these choices are essential for the reliability of the findings, for reducing the likelihood of errors, and ultimately for the overall performance of the solution. In the following, we give a short account of the features, methods and tools that we have frequently encountered in our survey, in the DM phase of the EDM process.

In this section, we survey the different types of predictions that have been performed in higher education institutions. We summarize 22 recent studies regarding the prediction of academic achievement in higher education. These studies have been conducted in different countries around the world between the year 2007 and 2018. We also demonstrate the most significant findings out of the literature study by discussing the outcomes of the previous works.

While this survey covers academic prediction studies performed between 2007 and 2018, there are also more recent studies that have been published in 2019 and 2020. Many of these approaches (Adekitan and Salau 2019; Berens et al. 2019; Bhutto et al. 2020) still rely on traditional machine learning methods, such as SVM, decision trees, logistic regression, and Naïve Bayes. However, there are also some new data mining methods that have been explored, such as Structural Equation Modeling (Nabizadeh et al. 2019) and probabilistic neural networks (Adekitan and Salau 2019). Although deep neural nets have seen a growing popularity in the machine learning community, particularly with applications to natural language processing, they are still not adopted in the EDM literature. This is probably due to their need of very large training data, whose sourcing is problematic in educational contexts.

With respect to the type of features used to perform the predictions, demographical features (Berens et al. 2019; Bhutto et al. 2020; Nabizadeh et al. 2019), previous GPA (Adekitan and Salau 2019; Berens et al. 2019; Bhutto et al. 2020; Nabizadeh et al. 2019), as well as students’ satisfaction and interaction with system (Bhutto et al. 2020) are still commonly used. However, some new features are also being investigated such as cognitive and metacognitive learning strategies (Nabizadeh et al. 2019). These strategies include students’ effort in rehearsal, elaboration, organization, critical thinking, and metacognitive self-regulation. Intrestengly, using such features shows promising results. This comes to confirm that prediction of students’ performance is still a very actively researched problem, whose current solutions can still be improved, and that the factors that mostly influence academic outcomes and hence can be used to predict future performances are still not widely understood.

With respect to opportunities for further research in the domain of EDM, we identify two critical gaps in the previous literature. First relates to the investigation of the relation between personality and academic achievement. In light of the recent personality measures such as IPIP-NEU (Goldberg et al. 2006), we identify the opportunity of using such measures for academic performance achievement. However, the complexity of collecting such data is an important challenge to overcome. The second gap we identify relates to student self-assessment. None of the studies reports on analyzing the relationship between self-assessment and actual performance. We expect that the inclusion of such measures has the potential to further improve the accuracy of academic performance prediction.

Educational data mining is an area that holds exciting opportunities for researchers and practitioners all around the world. This field assists in improving institutional effectiveness by supporting decision making and enhancing student learning to reach visible and measurable targets. This paper provides a rich literature review on the prediction of academic achievement in higher education for the past 12 years with the final aim of providing researchers and educational planners with information to assist them when attempting to carry out an EDM solution.

This paper revealed that a considerable amount of work has been performed in analyzing and predicting academic performance. It showed that classification and regression algorithms can be used successfully to predict students’ academic achievement in both course and degree level. It can be seen that most of the reviewed EDM research in the past decade has been completed using the open source Weka machine learning software. We found that the most used methods for predicting academic achievement are decision tree algorithms, with C4.5 being most popular among them, most likely because such white box classification algorithms obtain models that can explain predictions by IF–THEN rules. These rules can be simply interpreted by non-expert DM users, such as teachers, and can be directly applied in decision making. On the other hand, neural networks, support vector machines and K-nearest neighbour were not frequently used as compared to the rest of the DM methods. Such methods are not suitable for EDM purpose due to their black-box mechanisms.

We also found that the used features broadly differ based on the specific settings of the institute, culture, and country. However, gender, age, previous GPA and the proficiency level of the academic language are the features that most researchers agreed on when predicting students’ academic achievement in higher education regardless of their environment, i.e., where they come from and what they believe in. Nevertheless, an essential limitation of the surveyed literature is the fact that only a few studies investigate and report the significance of each predictor. Rather, the vast majority of studies report only the final results, making it challenging to judge the value of each feature, even for the very widely used ones. We therefore conclude that more research is needed first to deepen our understanding of the contribution of each traditionally used feature, and second, to extend the set of features and methodologies for further improving the current prediction accuracies.