A multimethod synthesis of Covid-19 education research: the tightrope between covidization and meaningfulness

The last months of 2019 have witnessed an unprecedented situation that humanity has not seen in a hundred years. The initial reactions of disbelief, hesitance and denial have wasted precious opportunities to prepare or at least to take much needed calculated steps [1‐3]. Perhaps, the way the situation developed at a staggering speed has made planning practically impossible. Universities around the world, in response to the global pandemic, were forced to cancel their face-to-face classes and shift to online education. Such a decision was taken overnight leading students, teachers, and families to a reality they had to accommodate with the wherewithal at hand [1‐4].

Online learning has become the crucial tool for the online transition, lectures were delivered though real-time video conferences, e.g., Zoom, Hangouts and Teams [5]. Several other forms were also adopted, e.g., video recordings, asynchronous forum discussions, or messaging through emails [3]. Such rapid changes, in the way learning was delivered, has influenced student satisfaction, mental well-being and a willingness to accept the “new normal” [5]. Teachers had to develop learning materials in new digital forms leading to a large increase in workload and possibly time trying to learn necessary digital skills or use new software [2]. Furthermore, teachers had to develop initiatives that help mitigate the unfolding situation, overcome the limitations of virtual teaching and possibly improve interactions with students [6, 7]. Families had to be involved in the teaching process, facilitate home schooling and help their children with the stressful situation [8]. Universities created—or we better say improvised– guidelines that detail how to respond to emergency in various shapes or forms, e.g., “Emergency Management Plan (EMP)”, “Crisis Management Plan,” or “Business Continuity Plan (BCP)” containing essentially the four phases of emergency management: preparedness, response, recovery and mitigation [9].

The accelerating situation has led to new realities where the educational community needed novel insights about different aspects, e.g., students, teachers, pedagogy, tools, and implementations. Therefore, researchers have been racing to offer their insights regarding their experience, students’ perceptions, tools, and ways to optimize learning and teaching, to mention a few [1, 4, 10]. Funding agencies have also tried to help researchers with fast-track grants targeting education during the pandemic, for instance some Erasmus + calls were launched in 2020. To that end, a large volume of research has been produced across vast and diverse areas that requires a synthesis. In this paper, we take a mixed methods approach combining (1) in-depth qualitative analysis of the top 54 cited papers, (2) bibliometric analysis of the publication meta-data, and (3) Structural Topic Models (STM) to make sense of the large number of publications and compile the published research into “topics” which we analyze and offer a concise analysis of the articles content.

Bibliometric analysis offers an overarching quantitative view of scientific research through the analysis of meta-data [11, 12]. Bibliometrics have been used widely across several fields to map scientific productivity, assess impact, dissemination, collaborative patterns, and research trends [13]. This approach relies on several analytical techniques, e.g., visualization, network analysis and statistical methods. However, bibliometrics is commonly criticized for the lack of qualitative and nuanced analysis [14]. Therefore, we augment our approach with qualitative analysis of the top 54 cited papers as well as STM for the analysis of research themes [15].

Despite the recency of STM as a technique, STM has gained an increasing role as a valuable tool for studying textual data across social sciences [16, 17]. Using STM, researchers are able to “mine” latent (often referred to as hidden) topics automatically from the large corpora of text using “unsupervised methods” [18]. That is, topics are inferred from the text without a priori assignment or manual coding of the data into predefined categories (“supervised methods”) [19]. The inferred topics represent themes within the dataset that have semantic associations. Two types of models exist, single membership models where each document belongs to a single topic and mixed membership where a document represents a mixture of topics which is used in our study. The use of STM could augment bibliometric analysis through discovery of the research themes and the “hidden topics” [16, 17]. In doing so, STM has an advantage over traditional keyword analysis which are usually dominated by most frequent keywords undermining several important themes within the corpus under study. STM has been used across several studies to reveal predominant research themes, e.g., [20‐22].

Few bibliometrics studies have tried to cover research about the pandemic, e.g., [21, 22]. Yet, such studies have focused mostly on online education, used a limited dataset or lacked a nuanced qualitative analysis that synthesizes the results beyond the metrics and indicators, e.g., [23]. Our study aims to bridge such gaps. The research questions of this study are:

The rest of the paper is structured as follows: the following section presents the methods employed in the study, followed by a section devoted to detailed description of the obtained results regarding each research question with extensive discussion. Finally, conclusions and remarks are presented in the last section.

The search was performed on Scopus database since it has a robust well-curated collection of articles that included almost all of Web of Science with a broader coverage for social science topics relevant to our study [24]. The search keywords were chosen to capture all variations of the Pandemic keyword as well as the education and teaching to reflect the context and therefore we choose the following keywords:

The search for the pandemic keywords involved only titles, keywords, and Scopus categorized keywords. Several iterations of search with different keywords were assessed, in which a sample of articles were assessed for relevance and accuracy. The final search was decided with consensus among researchers that the keywords bring most relevant results and avoids adding “noise.” A decision was made to exclude abstracts from the search for the pandemic keywords since initial searches with abstracts included a large number of irrelevant articles, and thereupon we decided to include articles which authors explicitly stated COVID-19 (or variations of the keyword) relevance through expressing it in the title or the keywords. On the other hand, the education and teaching keywords were searched in article abstracts, keywords, and titles. The keyword learning was also excluded since it brought lots of irrelevant articles, such as articles related to machine learning. The search was performed on 15^th of February 2022 and the meta-data was retrieved, processed, and prepared for analysis.

To answer RQ1: Bibliometric analysis was performed using Bibliometrix package [25], which is an open source R package that provides a toolset for analysis of bibliographic meta-data. Frequencies of citations, article statistics and top articles were computed and plotted using R statistical language with the help of Bibliometrix.

To answer RQ2: We used structural topic modeling (STM). STM has gained an increasing role as a valuable tool for studying textual data across social sciences [16, 17]. Using STM, researchers are able to “mine” latent (often referred to as hidden) topics automatically from the large corpora of text using “unsupervised methods” [18]. That is, topics are inferred from the text without a priori assignment or manual coding of the data into predefined categories (“supervised methods”) [15, 18]. The inferred topics represent themes within the dataset that have semantic associations. Two types of models exist, single membership models where each document belongs to a single topic and mixed membership where a document represents a mixture of topics which is used in our study. The use of STM could augment bibliometric analysis through discovery of the research themes and the “hidden topics” [16, 17]. In doing so, STM has an advantage over traditional keyword analysis which are usually dominated by most frequent keywords undermining several important themes within the corpus under study. STM has been used across several studies to reveal predominant research themes, e.g., for the analysis of education technology topics [18].

To identify the main themes of research through structural topic modeling we used R package stm which provides methods for probabilistic topic models, STM in our case. A topic is defined as a mixture of words where each word belongs to a topic with a certain probability. A document could have a mixture of topics, i.e., several topics could describe a single document with a certain probability. The stm package implements Latent Dirichlet Allocation (LDA) and uses a variational Expectation–Maximization algorithm to estimate the models and their parameters. The topics were modeled using the article's meta-data (title, abstract, keywords) as input [19]. The abstract and title were cleaned from stop words. Since different keywords may represent the same meaning and could result in erroneous results, we performed an exhaustive cleaning process where we combined similar keywords together using Google Openrefine [12, 26]. For instance, Learning Management system, LMS and learning management systems were combined together. The cleaning also removed keywords that are used to indicate COVID-19 (e.g., covid, covid19, covid-19 pandemic, Corona Virus) since they were among our search keywords. The estimation of the topic modeling was performed after the cleaning step.

An essential step of topic modeling is in choosing the number of topics. However, there is no optimum way to identify such numbers [27, 28]. Several methods exist to assist in this process, the most recommended of which are semantic coherence, exclusivity, and human judgment, which we applied in our study [15]. Semantic coherence is a criterion that is maximized when the most probable words co-occur together and correlates with human judgment. Nevertheless, as noticed by [19], semantic coherence is often dominated by frequent and common keywords, e.g., education and students in our case. Therefore, a measure for the specificity and uniqueness of the keyword was conceptualized to better separate different topics. Exclusivity, as the name suggests, reflects how exclusive the word is in a given topic [29]. Semantic coherence and exclusivity, while offering valuable guidance they “offer no particular statistical guarantees and should not be seen as estimating the “true” number of topics” [19], or as a substitute for careful examination, validation and extensive evaluation by human judgment [27]. Therefore, we followed the guidelines by augmenting the statistical parameters with consensus from experts about the most appropriate number of topics.

We estimated 40 models, the smallest of which had five topics and the largest had 45 topics. The semantic coherence and exclusivity were plotted and examined; ten topics had favorable yet close values. The topics were then examined by four experts who had to rank the best number of topics based on the following criteria [15]:

Each of the experts judged these criteria and the top three topics were examined, discussed and a consensus was reached among the experts that the number of topics that brings unified themes together, with least overlap and dissonance was sixteen topics.

To answer RQ3: The top 70 articles were retrieved according to the number of citations. While our intention was to report on all the 70 articles, we found that some of these articles were very short (less than a full page or just an extended abstract) and had no methods or results sections. Therefore, a quality assessment was performed so that very short articles (single page articles), articles without methods or results section, or articles with very small sample size (e.g., n = 3) were flagged. The quality criteria were agreed by the three researchers and applied to each of the analyzed papers, when a paper was flagged as a candidate to be excluded by one of the researchers the rest of the authors checked it also in order to make the proper decision and reach a census to exclude the paper. A total of 16 articles were rejected based on a consensus of the three authors and meeting the exclusion criteria. The remaining articles were qualitatively coded according to the themes representing the content of these articles by three researchers. The themes were developed using an inductive or grounded theory approach, i.e., developing the themes directly from the articles [28]. Three authors met and coded the articles and reached an agreement after several iterations on the following themes as representative of the main themes in the topics: challenges, guidance, impact, problem understanding, online migration, and tools and resources. In addition, during this classification the target group that the articles were dealing with was also considered, i.e., teachers, students.

We conducted this study with the aim of offering an overarching synthesis of COVID-19 research from the pandemic onslaught till now. A mixed methods approach was used, where we combined quantitative analysis of research productivity with pandemic statistics, structural topic modeling and qualitative synthesis of papers with most attention from the educational community. There are several key findings that warrant reflections.

The analysis has shown that the process of knowledge production about COVID-19 was less skewed compared to educational research in general [12, 93], with a large global participation of 137 different countries in research productivity. Whereas, research was concentrated in large and resourceful countries such as United States [35, 57, 67, 68], China [77], India [49], Germany [33], United Kingdom [51]; we also see several studies that addressed local and non-western contexts, e.g., Philippines [61], Rural South Africa [65], Jordan [63], Romania [56], Indonesia [66]. In fact, a global perspective [46, 64], with wide participation from different countries has helped in understanding the full breadth of impact of the pandemic [39, 41, 63]. In doing so, issues such as inequalities among different students’ subpopulations, as well as disparities in infrastructure and access to internet in, e.g., rural areas, received global attention and were prioritized [37, 42].

Several papers targeted teachers and teacher education [58, 61, 70], others have addressed students [49, 56, 63], yet, very few have researched the perspective of the families, despite that families were heavily involved in the process [50, 68]. Notable also that research was rather skewed toward some research fields, where medical [51, 63, 67, 68], engineering and mathematics education [45, 66, 92] received significant attention from researchers. A finding that could be explained by the idea that such disciplines may require practical face-to-face teaching which was an issue of concern during the pandemic [6, 30].

School closure, the consequences, and the alternative solutions occupied the public discourse as well as the research communities. Yet, schools have gone through several stages. Initially, many countries rushed to school closure which peaked around April 2020. About 1.3 billion students (81.8% of all enrolled) were instructed to stay home; a year later, where the pandemic was more rampant, school closure affected only 12.7% of students, reaching 2.7% as per the last recording in February 2022. Perhaps, the loss of learning time, the heavy toll on learners’ well-being as well as the remarkable burden vulnerable students had to endure [8, 48], has led to a policy where schools “were last to close and first to open” to avoid what the UNESCO called “a generational catastrophe.” Such a potential catastrophe would have resulted in stark inequalities of learning opportunities but also other aspects that school provides, e.g., school meals, physical activities and social interaction [37, 40‐41]. Of course, such decisions were aided by prioritizing teacher vaccination, health measures and infection tracing [39, 47].

If anything the pandemic is known for, it is the “impact,” an issue that has been studied from all points of views and perspectives. Therefore, researchers intensively studied the impact of pandemic on workload [34], academic work and personal lives [5, 56], student satisfaction [4, 76], confidence [51], quality of teaching and learning [43, 51], and on vulnerable groups [5, 62]. The impact on mental health and well-being has been a central theme in the pandemic research [8, 47, 49]. Along with the impact, came a long list of articles of recommendations and guidance regarding how to mitigate the impact, or address the challenges. For instance, we saw discussion about technical infrastructure [33, 40], online learning initiatives [4, 80], and sustainable online learning [36].

The rush to move online was accompanied by an accelerating stream of articles about the pandemic [94]. Thoughtful, well-planned, and meaningful research was hard to conceptualize or implement, and a sense of urgency led to a deluge of research with thin contributions in a time of dire need to genuine insights [94‐96]. Perhaps, as it has been argued by [94, 97], some may have found an opportune time to jump on the bandwagon of COVID-19 and the possibilities for research funding to capitalize on the need for research about the pandemic, a phenomenon that later became known as Covidization of research [94, 97].

We have used two methods for the analysis: STM and thematic analysis of the top cited papers. While STM is well-established for summarizing the general themes of a large textual dataset, such summarizing power should not be confused with retrieving the “true” content of the documents. As [27] points out, automated text analysis should not substitute careful and thoughtful text examination. Therefore, such methods are “best thought of as amplifying and augmenting careful reading and thoughtful analysis” [27]. Thereupon, a qualitative thematic analysis was performed, which revealed related but also varying themes. Of those themes, some may be hard to pick with a summarizing automatic text analysis, e.g., problem understanding, implications of the change and challenges. As such, we suggest that a careful qualitative analysis may be helpful to draw the full picture of text analysis.

Our article is not without limitations. Our search using keywords—which is the standard in all systematic reviews and bibliometrics—may have missed some articles that did not explicitly mention the pandemic keywords. Our results should not be viewed as encompassing all literature, but a large collection of articles based on systematic search. Using citations as measures of article impact is not ideal, yet it remains to be the most followed practice in the literature. To compensate for such shortcoming, we used structural topic modeling to gather all relevant topics and insights from the literature. One should not expect that synthesizing a few thousands of papers in a single article can be exhaustive, comprehensive, or complete. Nonetheless, our results should be viewed as a summary of the “important” take-home messages from these articles. Bibliometrics methods have known deficiencies such as over-reliance on metrics and skewed quantification of research which we tried to avoid in our article by combining several methods. The recency of the pandemic does not allow an accurate estimation of the impact of research or a temporal timeline and therefore, our estimation of such aspects remains to be verified in future research. Last, relying on a single database may have missed some articles that are not indexed in Scopus. Nevertheless, we had to choose one database to avoid erroneous mixing of citation counts between databases, and we selected Scopus since it has a wide coverage. Another limitation for our study is reliance on a database with poor selection of articles from the global south, a problem that all databases suffer from.

This work provides synthesis of COVID-19 research published by the educational community. A combination of quantitative analysis of research productivity with pandemic statistics, structural topic modeling and qualitative synthesis of papers with most attention from the educational community was used. A large volume of knowledge has been produced in education over the past couple of years that addressed various aspects of the pandemic, the majority of which had been published in open access journals, and few were in well-established publication outlets. From all papers that were taken into account, three main groups of topics were identified: (i) topics related to education in general, (ii) topics dealing with migration to online education, and (iii) diverse topics, e.g., perceptions, inclusion, medical education, engagement and motivation, well-being, and equality. A deeper analysis of the most cited papers revealed that problem understanding was the dominating theme of papers, followed by challenges, impact, guidance, online migration and tools and resources. While the conducted analysis may not be viewed as all encompassing, as some papers may have been missed by using one database, it does give an important synthesis of the findings in a large volume of knowledge as the insights were drawn from multiple perspectives and using different methods.